Interface IDataPipelineStateService
- Namespace
- FoundationaLLM.Common.Interfaces
- Assembly
- FoundationaLLM.Common.dll
Defines the interface for the Data Pipeline State Service.
public interface IDataPipelineStateService- Extension Methods
Methods
DataPipelineRunWorkItemArtifactChanged(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, string)
Indicates whether an artifact associated with a data pipeline run work item has changed.
Task<bool> DataPipelineRunWorkItemArtifactChanged(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, string artifactName)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRunDataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItemDataPipelineRunWorkItemThe data pipeline run work item.
artifactNamestringThe name of the artifact associated with the data pipeline run work item.
Returns
- Task<bool>
true if the artifact was changed by the data pipeline run indicated by
dataPipelineRun, false otherwise.
GetActiveDataPipelineRuns()
Gets a list of active data pipeline runs.
Task<List<DataPipelineRun>> GetActiveDataPipelineRuns()Returns
- Task<List<DataPipelineRun>>
The list of active data pipeline runs.
GetDataPipelineCanonicalRootPath(DataPipelineDefinition, DataPipelineRun)
Gets the canonical root path that corresponds to a data pipeline run.
string GetDataPipelineCanonicalRootPath(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRunDataPipelineRunThe data pipeline run.
Returns
- string
The path of the canonical root directory.
Remarks
The canonical root path is where we store artifacts shared by all individual runs that are trigerred for the same data pipeline definition and set of trigger parameters. This includes the content items that are processed by the data pipeline runs.
GetDataPipelineContentItem(DataPipelineRunWorkItem)
Gets a data pipeline content item associated with a data pipeline run work item.
Task<DataPipelineContentItem> GetDataPipelineContentItem(DataPipelineRunWorkItem dataPipelineRunWorkItem)Parameters
dataPipelineRunWorkItemDataPipelineRunWorkItemThe data pipeline work item that references the content item.
Returns
- Task<DataPipelineContentItem>
The content item referenced by the work item.
GetDataPipelineRun(string)
Gets a data pipeline run by its identifier.
Task<DataPipelineRun?> GetDataPipelineRun(string runId)Parameters
runIdstringThe data pipeline run identifier.
Returns
- Task<DataPipelineRun>
The requested data pipeline run object.
GetDataPipelineRunRootPath(DataPipelineDefinition, DataPipelineRun)
Gets the root path for a data pipeline run.
string GetDataPipelineRunRootPath(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRunDataPipelineRunThe data pipeline run.
Returns
- string
The path of the run root directory.
Remarks
The run root path is where the run-specific artifacts are stored.
GetDataPipelineRunStageWorkItems(string, string)
Gets the list of data pipeline run work items associated with a specified stage of a run.
Task<List<DataPipelineRunWorkItem>> GetDataPipelineRunStageWorkItems(string runId, string stage)Parameters
Returns
- Task<List<DataPipelineRunWorkItem>>
The list of data pipeline run work items associated with the specified stage of the run.
GetDataPipelineRunWorkItem(string, string)
Gets a data pipeline run work item by its identifier.
Task<DataPipelineRunWorkItem?> GetDataPipelineRunWorkItem(string workItemId, string runId)Parameters
workItemIdstringThe data pipeline run work item identifier.
runIdstringThe data pipeline run identifier.
Returns
- Task<DataPipelineRunWorkItem>
The requests data pipeline run work item object.
GetDataPipelineRuns(DataPipelineRunFilter)
Gets a list of data pipeline runs filtered by the provided filter criteria.
Task<List<DataPipelineRun>> GetDataPipelineRuns(DataPipelineRunFilter dataPipelineRunFilter)Parameters
dataPipelineRunFilterDataPipelineRunFilterThe filter criteria used to filter data pipeline runs.
Returns
- Task<List<DataPipelineRun>>
The list of requests data pipeline runs.
InitializeDataPipelineRunState(DataPipelineDefinition, DataPipelineRun, List<DataPipelineContentItem>)
Initializes the state of a data pipeline run.
Task<bool> InitializeDataPipelineRunState(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<DataPipelineContentItem> contentItems)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition used to initialize the run state.
dataPipelineRunDataPipelineRunThe details of the data pipeline run.
contentItemsList<DataPipelineContentItem>The list of content items to be processed by the data pipeline run.
Returns
LoadDataPipelineRunParts<T>(DataPipelineDefinition, DataPipelineRun, string)
Loads the parts associated with a data pipeline run.
Task<IEnumerable<T>> LoadDataPipelineRunParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string filePath) where T : class, new()Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRunDataPipelineRunThe data pipeline run .
filePathstringThe name of the file that contains the data pipeline run parts.
Returns
- Task<IEnumerable<T>>
A list with the parts associated with the data pipeline run.
Type Parameters
TThe type of the data pipeline run part to be loaded.
Remarks
The filePath parameter must contain a path that is relative to the path
of the data pipeline run artifacts directory.
LoadDataPipelineRunWorkItemArtifacts(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, string)
Loads the artifacts associated with a data pipeline run work item.
Task<List<DataPipelineStateArtifact>> LoadDataPipelineRunWorkItemArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, string artifactsNameFilter)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRunDataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItemDataPipelineRunWorkItemThe data pipeline run work item.
artifactsNameFilterstringThe name pattern used to identify a subset of the artifacts.
Returns
- Task<List<DataPipelineStateArtifact>>
A list with the binary contents of the artifacts.
LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, string, string)
Loads the content item parts associated with a data pipeline run work item.
Task<IEnumerable<T>> LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, string fileName, string contentSection = "content-items") where T : class, new()Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRunDataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItemDataPipelineRunWorkItemThe data pipeline run work item.
fileNamestringThe name of the file that contains the content item parts.
contentSectionstringThe section in the storage that contains the work item parts. Default is "content-items".
Returns
- Task<IEnumerable<T>>
A list with the content item parts associated with the data pipeline run work item.
Type Parameters
TThe type of the content item parts to be loaded.
LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, string, string, string)
Loads the content item parts associated with a data pipeline run work item.
Task<IEnumerable<T>> LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string contentItemCanonicalId, string fileName, string contentSection = "content-items") where T : class, new()Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRunDataPipelineRunThe data pipeline run item associated with the work item.
contentItemCanonicalIdstringThe content item canonical identifier.
fileNamestringThe name of the file that contains the content item parts.
contentSectionstringThe section in the storage that contains the work item parts. Default is "content-items".
Returns
- Task<IEnumerable<T>>
A list with the content item parts associated with the data pipeline run work item.
Type Parameters
TThe type of the content item parts to be loaded.
PersistDataPipelineRunWorkItems(List<DataPipelineRunWorkItem>)
Persists a list of data pipeline run work items.
Task<bool> PersistDataPipelineRunWorkItems(List<DataPipelineRunWorkItem> workItems)Parameters
workItemsList<DataPipelineRunWorkItem>The list of data pipeline work items to be persisted.
Returns
SaveDataPipelineRunArtifacts(DataPipelineDefinition, DataPipelineRun, List<DataPipelineStateArtifact>)
Saves the artifacts associated with a data pipeline run.
Task SaveDataPipelineRunArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<DataPipelineStateArtifact> artifacts)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRunDataPipelineRunThe data pipeline run.
artifactsList<DataPipelineStateArtifact>The list of artifacts to be saved.
Returns
SaveDataPipelineRunParts<T>(DataPipelineDefinition, DataPipelineRun, IEnumerable<T>, string)
Saves the parts associated with a data pipeline run.
Task SaveDataPipelineRunParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, IEnumerable<T> dataPipelineRunParts, string filePath) where T : class, new()Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRunDataPipelineRunThe data pipeline run.
dataPipelineRunPartsIEnumerable<T>The list of data pipeline run parts.
filePathstringThe path of the file that contains the data pipeline run parts.
Returns
Type Parameters
TThe type of the data pipeline part to be saved.
Remarks
The filePath parameter must contain a path that is relative to the path
of the data pipeline run artifacts directory.
SaveDataPipelineRunWorkItemArtifacts(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, List<DataPipelineStateArtifact>)
Saves the artifacts associated with a data pipeline run work item.
Task SaveDataPipelineRunWorkItemArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, List<DataPipelineStateArtifact> artifacts)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRunDataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItemDataPipelineRunWorkItemThe data pipeline run work item.
artifactsList<DataPipelineStateArtifact>The list with the binary contents of the artifacts.
Returns
SaveDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, IEnumerable<T>, string, string)
Saves the content item parts associated with a data pipeline run work item.
Task SaveDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, IEnumerable<T> contentItemParts, string fileName, string contentSection = "content-items") where T : class, new()Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRunDataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItemDataPipelineRunWorkItemThe data pipeline run work item.
contentItemPartsIEnumerable<T>The list with the content item parts.
fileNamestringThe name of the file that contains the content item parts.
contentSectionstringThe section in the storage that contains the work item parts. Default is "content-items".
Returns
Type Parameters
TThe type of the content item parts to be saved.
StartDataPipelineRunWorkItemProcessing(Func<DataPipelineRunWorkItem, Task>)
Starts processing data pipeline run work items.
Task<bool> StartDataPipelineRunWorkItemProcessing(Func<DataPipelineRunWorkItem, Task> processWorkItem)Parameters
processWorkItemFunc<DataPipelineRunWorkItem, Task>The asynchronous delegate that is invoked for each data pipeline run work item.
Returns
StopDataPipelineRunWorkItemProcessing()
Stops processing data pipeline run work items.
Task StopDataPipelineRunWorkItemProcessing()Returns
TryLoadDataPipelineRunArtifacts(DataPipelineDefinition, DataPipelineRun, string)
Tries to load the artifacts associated with a data pipeline run.
Task<(bool Success, List<DataPipelineStateArtifact> Artifacts)> TryLoadDataPipelineRunArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string artifactsNameFilter)Parameters
dataPipelineDefinitionDataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRunDataPipelineRunThe data pipeline run item associated with the work item.
artifactsNameFilterstringThe name pattern used to identify the artifacts to load.
Returns
- Task<(bool Success, List<DataPipelineStateArtifact> Artifacts)>
A boolean indicating whether the operation was successful or not and the list with the details of the artifacts if the operation is successful.
UpdateDataPipelineRunStatus(DataPipelineRun)
Updates the status of a data pipeline run.
Task<bool> UpdateDataPipelineRunStatus(DataPipelineRun dataPipelineRun)Parameters
dataPipelineRunDataPipelineRunThe data pipeline run whose status is to be updated.
Returns
UpdateDataPipelineRunWorkItem(DataPipelineRunWorkItem)
Updates a data pipeline run work item.
Task<bool> UpdateDataPipelineRunWorkItem(DataPipelineRunWorkItem workItem)Parameters
workItemDataPipelineRunWorkItemThe data pipeline run work item to be updated.
Returns
UpdateDataPipelineRunWorkItemsStatus(List<DataPipelineRunWorkItem>)
Updates the status of data pipeline run work items.
Task<bool> UpdateDataPipelineRunWorkItemsStatus(List<DataPipelineRunWorkItem> workItems)Parameters
workItemsList<DataPipelineRunWorkItem>The list of data pipeline work items whose status must be updated.