Class TokenTextSplitterService

Namespace

FoundationaLLM.Common.Services.TextSplitters

Assembly

FoundationaLLM.Common.dll

Splits text based on number of tokens.

public class TokenTextSplitterService : ITextSplitterService

Inheritance

object

TokenTextSplitterService

Implements

ITextSplitterService

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

ObjectExtensions.GetPropertyValues(object, List<string>)

Constructors

TokenTextSplitterService(ITokenizerService, IOptions<TokenTextSplitterServiceSettings>, ILogger<TokenTextSplitterService>)

Splits text based on number of tokens.

public TokenTextSplitterService(ITokenizerService tokenizerService, IOptions<TokenTextSplitterServiceSettings> options, ILogger<TokenTextSplitterService> logger)

Parameters

tokenizerService ITokenizerService

The ITokenizerService used to tokenize the input text.

options IOptions<TokenTextSplitterServiceSettings>

The IOptions<TOptions> providing the settings for the service.

logger ILogger<TokenTextSplitterService>

The logger used for logging.

Methods

SplitPlainText(string)

Splits plain text into multiple chunks.

public List<TextChunk> SplitPlainText(string text)

Parameters

text string

The plain text to split.

Returns

List<TextChunk>

A list of TextChunk items containing the text chunks and their sizes in tokens.