Class TikTokenizerEncoders

Namespace

FoundationaLLM.Common.Services.Tokenizers

Assembly

FoundationaLLM.Common.dll

Well-known TikTokenizer encoder names.

public static class TikTokenizerEncoders

Inheritance

object

TikTokenizerEncoders

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Fields

CL100K_BASE

Encoder used by OpenAI models like GPT-4, GPT-3.5-Turbo, and Text-Embedding-ADA-002.

public const string CL100K_BASE = "cl100k_base"

Field Value

string