Table of Contents

Interface IDataPipelineStateService

Namespace
FoundationaLLM.Common.Interfaces
Assembly
FoundationaLLM.Common.dll

Defines the interface for the Data Pipeline State Service.

public interface IDataPipelineStateService
Extension Methods

Methods

GetActiveDataPipelineRuns()

Gets a list of active data pipeline runs.

Task<List<DataPipelineRun>> GetActiveDataPipelineRuns()

Returns

Task<List<DataPipelineRun>>

The list of active data pipeline runs.

GetDataPipelineContentItem(DataPipelineRunWorkItem)

Gets a data pipeline content item associated with a data pipeline run work item.

Task<DataPipelineContentItem> GetDataPipelineContentItem(DataPipelineRunWorkItem dataPipelineRunWorkItem)

Parameters

dataPipelineRunWorkItem DataPipelineRunWorkItem

The data pipeline work item that references the content item.

Returns

Task<DataPipelineContentItem>

The content item referenced by the work item.

GetDataPipelineRun(string)

Gets a data pipeline run by its identifier.

Task<DataPipelineRun?> GetDataPipelineRun(string runId)

Parameters

runId string

The data pipeline run identifier.

Returns

Task<DataPipelineRun>

The requested data pipeline run object.

GetDataPipelineRunArtifactsPath(DataPipelineDefinition, DataPipelineRun)

Gets the path to the artifacts directory for a data pipeline run.

string GetDataPipelineRunArtifactsPath(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun)

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the run.

dataPipelineRun DataPipelineRun

The data pipeline run.

Returns

string

The path of the artifacts directory.

GetDataPipelineRunStageWorkItems(string, string)

Gets the list of data pipeline run work items associated with a specified stage of a run.

Task<List<DataPipelineRunWorkItem>> GetDataPipelineRunStageWorkItems(string runId, string stage)

Parameters

runId string

The data pipeline run identifier.

stage string

The stage of the data pipeline run.

Returns

Task<List<DataPipelineRunWorkItem>>

The list of data pipeline run work items associated with the specified stage of the run.

GetDataPipelineRunWorkItem(string, string)

Gets a data pipeline run work item by its identifier.

Task<DataPipelineRunWorkItem?> GetDataPipelineRunWorkItem(string workItemId, string runId)

Parameters

workItemId string

The data pipeline run work item identifier.

runId string

The data pipeline run identifier.

Returns

Task<DataPipelineRunWorkItem>

The requests data pipeline run work item object.

GetDataPipelineRuns(DataPipelineRunFilter)

Gets a list of data pipeline runs filtered by the provided filter criteria.

Task<List<DataPipelineRun>> GetDataPipelineRuns(DataPipelineRunFilter dataPipelineRunFilter)

Parameters

dataPipelineRunFilter DataPipelineRunFilter

The filter criteria used to filter data pipeline runs.

Returns

Task<List<DataPipelineRun>>

The list of requests data pipeline runs.

InitializeDataPipelineRunState(DataPipelineDefinition, DataPipelineRun, List<DataPipelineContentItem>)

Initializes the state of a data pipeline run.

Task<bool> InitializeDataPipelineRunState(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<DataPipelineContentItem> contentItems)

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition used to initialize the run state.

dataPipelineRun DataPipelineRun

The details of the data pipeline run.

contentItems List<DataPipelineContentItem>

The list of content items to be processed by the data pipeline run.

Returns

Task<bool>

true if the initialization is successful.

LoadDataPipelineRunParts<T>(DataPipelineDefinition, DataPipelineRun, string)

Loads the parts associated with a data pipeline run.

Task<IEnumerable<T>> LoadDataPipelineRunParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string filePath) where T : class, new()

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the run.

dataPipelineRun DataPipelineRun

The data pipeline run .

filePath string

The name of the file that contains the data pipeline run parts.

Returns

Task<IEnumerable<T>>

A list with the parts associated with the data pipeline run.

Type Parameters

T

The type of the data pipeline run part to be loaded.

Remarks

The filePath parameter must contain a path that is relative to the path of the data pipeline run artifacts directory.

LoadDataPipelineRunWorkItemArtifacts(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, string)

Loads the artifacts associated with a data pipeline run work item.

Task<List<DataPipelineStateArtifact>> LoadDataPipelineRunWorkItemArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, string artifactsNameFilter)

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the work item.

dataPipelineRun DataPipelineRun

The data pipeline run item associated with the work item.

dataPipelineRunWorkItem DataPipelineRunWorkItem

The data pipeline run work item.

artifactsNameFilter string

The name pattern used to identify a subset of the artifacts.

Returns

Task<List<DataPipelineStateArtifact>>

A list with the binary contents of the artifacts.

LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, string)

Loads the content item parts associated with a data pipeline run work item.

Task<IEnumerable<T>> LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, string fileName) where T : class, new()

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the work item.

dataPipelineRun DataPipelineRun

The data pipeline run item associated with the work item.

dataPipelineRunWorkItem DataPipelineRunWorkItem

The data pipeline run work item.

fileName string

The name of the file that contains the content item parts.

Returns

Task<IEnumerable<T>>

A list with the content item parts associated with the data pipeline run work item.

Type Parameters

T

The type of the content item parts to be loaded.

LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, string, string)

Loads the content item parts associated with a data pipeline run work item.

Task<IEnumerable<T>> LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string contentItemCanonicalId, string fileName) where T : class, new()

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the work item.

dataPipelineRun DataPipelineRun

The data pipeline run item associated with the work item.

contentItemCanonicalId string

The content item canonical identifier.

fileName string

The name of the file that contains the content item parts.

Returns

Task<IEnumerable<T>>

A list with the content item parts associated with the data pipeline run work item.

Type Parameters

T

The type of the content item parts to be loaded.

PersistDataPipelineRunWorkItems(List<DataPipelineRunWorkItem>)

Persists a list of data pipeline run work items.

Task<bool> PersistDataPipelineRunWorkItems(List<DataPipelineRunWorkItem> workItems)

Parameters

workItems List<DataPipelineRunWorkItem>

The list of data pipeline work items to be persisted.

Returns

Task<bool>

true if the items are successfully persisted.

SaveDataPipelineRunArtifacts(DataPipelineDefinition, DataPipelineRun, List<DataPipelineStateArtifact>)

Saves the artifacts associated with a data pipeline run.

Task SaveDataPipelineRunArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<DataPipelineStateArtifact> artifacts)

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the run.

dataPipelineRun DataPipelineRun

The data pipeline run.

artifacts List<DataPipelineStateArtifact>

The list of artifacts to be saved.

Returns

Task

SaveDataPipelineRunParts<T>(DataPipelineDefinition, DataPipelineRun, IEnumerable<T>, string)

Saves the parts associated with a data pipeline run.

Task SaveDataPipelineRunParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, IEnumerable<T> dataPipelineRunParts, string filePath) where T : class, new()

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the run.

dataPipelineRun DataPipelineRun

The data pipeline run.

dataPipelineRunParts IEnumerable<T>

The list of data pipeline run parts.

filePath string

The path of the file that contains the data pipeline run parts.

Returns

Task

Type Parameters

T

The type of the data pipeline part to be saved.

Remarks

The filePath parameter must contain a path that is relative to the path of the data pipeline run artifacts directory.

SaveDataPipelineRunWorkItemArtifacts(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, List<DataPipelineStateArtifact>)

Saves the artifacts associated with a data pipeline run work item.

Task SaveDataPipelineRunWorkItemArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, List<DataPipelineStateArtifact> artifacts)

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the work item.

dataPipelineRun DataPipelineRun

The data pipeline run item associated with the work item.

dataPipelineRunWorkItem DataPipelineRunWorkItem

The data pipeline run work item.

artifacts List<DataPipelineStateArtifact>

The list with the binary contents of the artifacts.

Returns

Task

SaveDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, IEnumerable<T>, string)

Saves the content item parts associated with a data pipeline run work item.

Task SaveDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, IEnumerable<T> contentItemParts, string fileName) where T : class, new()

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the work item.

dataPipelineRun DataPipelineRun

The data pipeline run item associated with the work item.

dataPipelineRunWorkItem DataPipelineRunWorkItem

The data pipeline run work item.

contentItemParts IEnumerable<T>

The list with the content item parts.

fileName string

The name of the file that contains the content item parts.

Returns

Task

Type Parameters

T

The type of the content item parts to be saved.

StartDataPipelineRunWorkItemProcessing(Func<DataPipelineRunWorkItem, Task>)

Starts processing data pipeline run work items.

Task<bool> StartDataPipelineRunWorkItemProcessing(Func<DataPipelineRunWorkItem, Task> processWorkItem)

Parameters

processWorkItem Func<DataPipelineRunWorkItem, Task>

The asynchronous delegate that is invoked for each data pipeline run work item.

Returns

Task<bool>

true if the processing is successfully started.

StopDataPipelineRunWorkItemProcessing()

Stops processing data pipeline run work items.

Task StopDataPipelineRunWorkItemProcessing()

Returns

Task

TryLoadDataPipelineRunArtifacts(DataPipelineDefinition, DataPipelineRun, string)

Tries to load the artifacts associated with a data pipeline run.

Task<(bool Success, List<DataPipelineStateArtifact> Artifacts)> TryLoadDataPipelineRunArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string artifactsNameFilter)

Parameters

dataPipelineDefinition DataPipelineDefinition

The data pipeline definition associated with the work item.

dataPipelineRun DataPipelineRun

The data pipeline run item associated with the work item.

artifactsNameFilter string

The name pattern used to identify the artifacts to load.

Returns

Task<(bool Success, List<DataPipelineStateArtifact> Artifacts)>

A boolean indicating whether the operation was successful or not and the list with the details of the artifacts if the operation is successful.

UpdateDataPipelineRunStatus(DataPipelineRun)

Updates the status of a data pipeline run.

Task<bool> UpdateDataPipelineRunStatus(DataPipelineRun dataPipelineRun)

Parameters

dataPipelineRun DataPipelineRun

The data pipeline run whose status is to be updated.

Returns

Task<bool>

true if the status update is successful.

UpdateDataPipelineRunWorkItem(DataPipelineRunWorkItem)

Updates a data pipeline run work item.

Task<bool> UpdateDataPipelineRunWorkItem(DataPipelineRunWorkItem workItem)

Parameters

workItem DataPipelineRunWorkItem

The data pipeline run work item to be updated.

Returns

Task<bool>

true if the data pipeline run work item is successfully updated.

UpdateDataPipelineRunWorkItemsStatus(List<DataPipelineRunWorkItem>)

Updates the status of data pipeline run work items.

Task<bool> UpdateDataPipelineRunWorkItemsStatus(List<DataPipelineRunWorkItem> workItems)

Parameters

workItems List<DataPipelineRunWorkItem>

The list of data pipeline work items whose status must be updated.

Returns

Task<bool>

true if the items statuses are successfully updated.