Class MicrosoftBPETokenizerService

Implements an ITokenizerService using the Microsoft BPE tokenizer (https://github.com/microsoft/Tokenizer). This class should always be instantiated as a singleton when used in dependency injection scenarios.

public class MicrosoftBPETokenizerService : ITokenizerService

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

Initializes a new instance of the MicrosoftBPETokenizerService class.

public MicrosoftBPETokenizerService(ILogger<MicrosoftBPETokenizerService> logger)

logger ILogger<MicrosoftBPETokenizerService>: The logger used for logging.

Decode an array of integer token ids.

public string Decode(int[] tokens, string encoderName)

Encode a string with a set of allowed special tokens that are not broken apart.

public List<int> Encode(string text, string encoderName)