Class TikTokenizerConfig

Provides the configuration values required to create a new TikTokenizer instance.

public record TikTokenizerConfig : IEquatable<TikTokenizerConfig>

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

Provides the configuration values required to create a new TikTokenizer instance.

public TikTokenizerConfig(string RegexPattern, string MergeableRanksFileUrl, Dictionary<string, int> SpecialTokens)

The raw content of the BPE rank file.

public byte[]? MergeableRanksFileContent { get; set; }

The URL used to download the BPE rank file.

public string MergeableRanksFileUrl { get; init; }

Regex pattern to break a long string.

public string RegexPattern { get; init; }

Special tokens mapping.

public Dictionary<string, int> SpecialTokens { get; init; }