Class DataPipelineStagePluginBase

Namespace

FoundationaLLM.Common.Services.Plugins

Assembly

FoundationaLLM.Common.dll

Implements a basic Data Pipeline Stage Plugin.

public class DataPipelineStagePluginBase : PluginBase, IDataPipelineStagePlugin

Inheritance

object

PluginBase

DataPipelineStagePluginBase

Implements

IDataPipelineStagePlugin

Derived

AzureAISearchIndexingDataPipelineStagePlugin

AzureAISearchRemovalDataPipelineStagePlugin

GatewayTextEmbeddingDataPipelineStagePlugin

KnowledgeExtractionDataPipelineStagePlugin

KnowledgeGraphDataPipelineStagePluginBase

TextExtractionDataPipelineStagePlugin

TextPartitioningDataPipelineStagePlugin

Inherited Members

PluginBase.Name

PluginBase._pluginParameters

PluginBase._packageManager

PluginBase._packageManagerResolver

PluginBase._pluginMetadata

PluginBase._serviceProvider

PluginBase._logger

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

ObjectExtensions.GetPropertyValues(object, List<string>)

Constructors

DataPipelineStagePluginBase(Dictionary<string, object>, IPluginPackageManager, IPluginPackageManagerResolver, IServiceProvider)

Initializes a new instance of the base data pipeline stage plugin.

public DataPipelineStagePluginBase(Dictionary<string, object> pluginParameters, IPluginPackageManager packageManager, IPluginPackageManagerResolver packageManagerResolver, IServiceProvider serviceProvider)

Parameters

pluginParameters Dictionary<string, object>

packageManager IPluginPackageManager

packageManagerResolver IPluginPackageManagerResolver

The package manager resolver for the plugin.

serviceProvider IServiceProvider

Fields

_dataPipelineStateService

The data pipeline state service.

protected readonly IDataPipelineStateService _dataPipelineStateService

Field Value

IDataPipelineStateService

_hasher

The hasher used to identify identical text content.

protected readonly XxHash128 _hasher

Field Value

XxHash128

Methods

ComputeHash(string)

Computes a hexadecimal hash string for the specified input using the current hashing algorithm.

protected string ComputeHash(string input)

Parameters

input string

The input string to compute the hash for. Cannot be null or empty.

Returns

string

A lowercase hexadecimal string representing the computed hash of the input.

GetStageWorkItems(DataPipelineDefinition, DataPipelineRun, List<string>, string, string)

Gets the list of data pipeline work items based on the provided content items canonical identifiers.

public virtual Task<List<DataPipelineRunWorkItem>> GetStageWorkItems(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<string> contentItemsCanonicalIds, string dataPipelineStageName, string previousDataPipelineStageName)

Parameters

dataPipelineDefinition DataPipelineDefinition

The definition of the data pipeline.

dataPipelineRun DataPipelineRun

The data pipeline run.

contentItemsCanonicalIds List<string>

The list of icontent items canonical identifiers.

dataPipelineStageName string

The name of the data pipeline stage.

previousDataPipelineStageName string

The name of the previous data pipeline stage.

Returns

Task<List<DataPipelineRunWorkItem>>

A list of data pipeline work items.

GetStartingStageWorkItems(DataPipelineDefinition, DataPipelineRun, List<DataPipelineContentItem>, string)

Gets the list of data pipeline work items based on the provided content items.

public virtual Task<List<DataPipelineRunWorkItem>> GetStartingStageWorkItems(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<DataPipelineContentItem> contentItems, string dataPipelineStageName)

Parameters

dataPipelineDefinition DataPipelineDefinition

The definition of the data pipeline.

dataPipelineRun DataPipelineRun

The data pipeline run.

contentItems List<DataPipelineContentItem>

The list of content items.

dataPipelineStageName string

The name of the data pipeline stage.

Returns

Task<List<DataPipelineRunWorkItem>>

A list of data pipeline work items.

ProcessWorkItem(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem)

Processes a specified data pipeline run work item.

public virtual Task<PluginResult> ProcessWorkItem(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem)

Parameters

dataPipelineDefinition DataPipelineDefinition

The definition of the data pipeline associated with the data pipeline run work item.

dataPipelineRun DataPipelineRun

The data pipeline run associated with the data pipeline run work item.

dataPipelineRunWorkItem DataPipelineRunWorkItem

The data pipeline run work item to process.

Returns

Task<PluginResult>

An object that contains the processing result.