Class MicrosoftMLTokenizer

Represents a tokenizer service that uses the Microsoft ML Tokenizer.

public class MicrosoftMLTokenizer : ITokenizerService

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Properties

Gets the underlying tokenizer instance.

public Tokenizer Tokenizer { get; }

Count the number of tokens in a given text.

public long CountTokens(string text, string? encoderName = null)

Decode an array of integer token ids.

public string Decode(int[] tokens, string? encoderName)

Encode a string with a set of allowed special tokens that are not broken apart.

public List<int> Encode(string text, string? encoderName)