Class TokenTextSplitterServiceSettings

Namespace

FoundationaLLM.Common.Models.Configuration.Text

Assembly

FoundationaLLM.Common.dll

Provides configuration settings that control token-based text splitting.

public record TokenTextSplitterServiceSettings : IEquatable<TokenTextSplitterServiceSettings>

Inheritance

object

TokenTextSplitterServiceSettings

Implements

IEquatable<TokenTextSplitterServiceSettings>

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

ObjectExtensions.GetPropertyValues(object, List<string>)

Constructors

TokenTextSplitterServiceSettings(string, string, int, int)

Provides configuration settings that control token-based text splitting.

public TokenTextSplitterServiceSettings(string Tokenizer, string TokenizerEncoder, int ChunkSizeTokens, int OverlapSizeTokens)

Parameters

Tokenizer string

The tokenizer used to split the test into tokens.

TokenizerEncoder string

The name of the encoder used for tokenization.

ChunkSizeTokens int

The target size in tokens for the resulting text chunks.

OverlapSizeTokens int

Teh target size in tokens for the overlapping parts of the adjacent text chunks.

Properties

ChunkSizeTokens

The target size in tokens for the resulting text chunks.

public int ChunkSizeTokens { get; init; }

Property Value

int

OverlapSizeTokens

Teh target size in tokens for the overlapping parts of the adjacent text chunks.

public int OverlapSizeTokens { get; init; }

Property Value

int

Tokenizer

The tokenizer used to split the test into tokens.

public string Tokenizer { get; init; }

Property Value

string

TokenizerEncoder

The name of the encoder used for tokenization.

public string TokenizerEncoder { get; init; }

Property Value

string

Methods

FromDictionary(Dictionary<string, string>)

Creates and instance of the class based on a dictionary.

public static TokenTextSplitterServiceSettings FromDictionary(Dictionary<string, string> settings)

Parameters

settings Dictionary<string, string>

The dictionary containing the settings.

Returns

TokenTextSplitterServiceSettings

A TokenTextSplitterServiceSettings instance initialized with the values from the dictionary.