Namespace FoundationaLLM.Common.Services.Tokenizers

Classes

LeasedTikTokenizer

Extends the TikTokenizer class to support leasing to multiple clients.

MicrosoftBPETokenizerService

Implements an ITokenizerService using the Microsoft BPE tokenizer (https://github.com/microsoft/Tokenizer). This class should always be instantiated as a singleton when used in dependency injection scenarios.

MicrosoftMLTokenizer

Represents a tokenizer service that uses the Microsoft ML Tokenizer.

TikTokenizerConfig

Provides the configuration values required to create a new TikTokenizer instance.

TikTokenizerEncoders

Well-known TikTokenizer encoder names.

TikTokenizerTokens

Well-known special TikTokenizer token names.

TokenizerServiceNames

Names of tokenizer services supported by the platform.

TryAGITokenizer

A tokenizer service that uses the Tiktoken library to encode and decode text into tokens.