Class TokenizerServiceNames

Namespace

FoundationaLLM.Common.Services.Tokenizers

Assembly

FoundationaLLM.Common.dll

Names of tokenizer services supported by the platform.

public static class TokenizerServiceNames

Inheritance

object

TokenizerServiceNames

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Fields

MICROSOFT_BPE_TOKENIZER

Tokenizer service implemented by Microsoft. For details, see https://github.com/microsoft/Tokenizer.

public const string MICROSOFT_BPE_TOKENIZER = "MicrosoftBPETokenizer"

Field Value

string