Interface IDataPipelineStateService
- Namespace
- FoundationaLLM.Common.Interfaces
- Assembly
- FoundationaLLM.Common.dll
Defines the interface for the Data Pipeline State Service.
public interface IDataPipelineStateService
- Extension Methods
Methods
GetActiveDataPipelineRuns()
Gets a list of active data pipeline runs.
Task<List<DataPipelineRun>> GetActiveDataPipelineRuns()
Returns
- Task<List<DataPipelineRun>>
The list of active data pipeline runs.
GetDataPipelineContentItem(DataPipelineRunWorkItem)
Gets a data pipeline content item associated with a data pipeline run work item.
Task<DataPipelineContentItem> GetDataPipelineContentItem(DataPipelineRunWorkItem dataPipelineRunWorkItem)
Parameters
dataPipelineRunWorkItem
DataPipelineRunWorkItemThe data pipeline work item that references the content item.
Returns
- Task<DataPipelineContentItem>
The content item referenced by the work item.
GetDataPipelineRun(string)
Gets a data pipeline run by its identifier.
Task<DataPipelineRun?> GetDataPipelineRun(string runId)
Parameters
runId
stringThe data pipeline run identifier.
Returns
- Task<DataPipelineRun>
The requested data pipeline run object.
GetDataPipelineRunArtifactsPath(DataPipelineDefinition, DataPipelineRun)
Gets the path to the artifacts directory for a data pipeline run.
string GetDataPipelineRunArtifactsPath(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun)
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRun
DataPipelineRunThe data pipeline run.
Returns
- string
The path of the artifacts directory.
GetDataPipelineRunStageWorkItems(string, string)
Gets the list of data pipeline run work items associated with a specified stage of a run.
Task<List<DataPipelineRunWorkItem>> GetDataPipelineRunStageWorkItems(string runId, string stage)
Parameters
Returns
- Task<List<DataPipelineRunWorkItem>>
The list of data pipeline run work items associated with the specified stage of the run.
GetDataPipelineRunWorkItem(string, string)
Gets a data pipeline run work item by its identifier.
Task<DataPipelineRunWorkItem?> GetDataPipelineRunWorkItem(string workItemId, string runId)
Parameters
workItemId
stringThe data pipeline run work item identifier.
runId
stringThe data pipeline run identifier.
Returns
- Task<DataPipelineRunWorkItem>
The requests data pipeline run work item object.
GetDataPipelineRuns(DataPipelineRunFilter)
Gets a list of data pipeline runs filtered by the provided filter criteria.
Task<List<DataPipelineRun>> GetDataPipelineRuns(DataPipelineRunFilter dataPipelineRunFilter)
Parameters
dataPipelineRunFilter
DataPipelineRunFilterThe filter criteria used to filter data pipeline runs.
Returns
- Task<List<DataPipelineRun>>
The list of requests data pipeline runs.
InitializeDataPipelineRunState(DataPipelineDefinition, DataPipelineRun, List<DataPipelineContentItem>)
Initializes the state of a data pipeline run.
Task<bool> InitializeDataPipelineRunState(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<DataPipelineContentItem> contentItems)
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition used to initialize the run state.
dataPipelineRun
DataPipelineRunThe details of the data pipeline run.
contentItems
List<DataPipelineContentItem>The list of content items to be processed by the data pipeline run.
Returns
LoadDataPipelineRunParts<T>(DataPipelineDefinition, DataPipelineRun, string)
Loads the parts associated with a data pipeline run.
Task<IEnumerable<T>> LoadDataPipelineRunParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string filePath) where T : class, new()
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRun
DataPipelineRunThe data pipeline run .
filePath
stringThe name of the file that contains the data pipeline run parts.
Returns
- Task<IEnumerable<T>>
A list with the parts associated with the data pipeline run.
Type Parameters
T
The type of the data pipeline run part to be loaded.
Remarks
The filePath
parameter must contain a path that is relative to the path
of the data pipeline run artifacts directory.
LoadDataPipelineRunWorkItemArtifacts(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, string)
Loads the artifacts associated with a data pipeline run work item.
Task<List<DataPipelineStateArtifact>> LoadDataPipelineRunWorkItemArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, string artifactsNameFilter)
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRun
DataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItem
DataPipelineRunWorkItemThe data pipeline run work item.
artifactsNameFilter
stringThe name pattern used to identify a subset of the artifacts.
Returns
- Task<List<DataPipelineStateArtifact>>
A list with the binary contents of the artifacts.
LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, string)
Loads the content item parts associated with a data pipeline run work item.
Task<IEnumerable<T>> LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, string fileName) where T : class, new()
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRun
DataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItem
DataPipelineRunWorkItemThe data pipeline run work item.
fileName
stringThe name of the file that contains the content item parts.
Returns
- Task<IEnumerable<T>>
A list with the content item parts associated with the data pipeline run work item.
Type Parameters
T
The type of the content item parts to be loaded.
LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, string, string)
Loads the content item parts associated with a data pipeline run work item.
Task<IEnumerable<T>> LoadDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string contentItemCanonicalId, string fileName) where T : class, new()
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRun
DataPipelineRunThe data pipeline run item associated with the work item.
contentItemCanonicalId
stringThe content item canonical identifier.
fileName
stringThe name of the file that contains the content item parts.
Returns
- Task<IEnumerable<T>>
A list with the content item parts associated with the data pipeline run work item.
Type Parameters
T
The type of the content item parts to be loaded.
PersistDataPipelineRunWorkItems(List<DataPipelineRunWorkItem>)
Persists a list of data pipeline run work items.
Task<bool> PersistDataPipelineRunWorkItems(List<DataPipelineRunWorkItem> workItems)
Parameters
workItems
List<DataPipelineRunWorkItem>The list of data pipeline work items to be persisted.
Returns
SaveDataPipelineRunArtifacts(DataPipelineDefinition, DataPipelineRun, List<DataPipelineStateArtifact>)
Saves the artifacts associated with a data pipeline run.
Task SaveDataPipelineRunArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, List<DataPipelineStateArtifact> artifacts)
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRun
DataPipelineRunThe data pipeline run.
artifacts
List<DataPipelineStateArtifact>The list of artifacts to be saved.
Returns
SaveDataPipelineRunParts<T>(DataPipelineDefinition, DataPipelineRun, IEnumerable<T>, string)
Saves the parts associated with a data pipeline run.
Task SaveDataPipelineRunParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, IEnumerable<T> dataPipelineRunParts, string filePath) where T : class, new()
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the run.
dataPipelineRun
DataPipelineRunThe data pipeline run.
dataPipelineRunParts
IEnumerable<T>The list of data pipeline run parts.
filePath
stringThe path of the file that contains the data pipeline run parts.
Returns
Type Parameters
T
The type of the data pipeline part to be saved.
Remarks
The filePath
parameter must contain a path that is relative to the path
of the data pipeline run artifacts directory.
SaveDataPipelineRunWorkItemArtifacts(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, List<DataPipelineStateArtifact>)
Saves the artifacts associated with a data pipeline run work item.
Task SaveDataPipelineRunWorkItemArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, List<DataPipelineStateArtifact> artifacts)
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRun
DataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItem
DataPipelineRunWorkItemThe data pipeline run work item.
artifacts
List<DataPipelineStateArtifact>The list with the binary contents of the artifacts.
Returns
SaveDataPipelineRunWorkItemParts<T>(DataPipelineDefinition, DataPipelineRun, DataPipelineRunWorkItem, IEnumerable<T>, string)
Saves the content item parts associated with a data pipeline run work item.
Task SaveDataPipelineRunWorkItemParts<T>(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, DataPipelineRunWorkItem dataPipelineRunWorkItem, IEnumerable<T> contentItemParts, string fileName) where T : class, new()
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRun
DataPipelineRunThe data pipeline run item associated with the work item.
dataPipelineRunWorkItem
DataPipelineRunWorkItemThe data pipeline run work item.
contentItemParts
IEnumerable<T>The list with the content item parts.
fileName
stringThe name of the file that contains the content item parts.
Returns
Type Parameters
T
The type of the content item parts to be saved.
StartDataPipelineRunWorkItemProcessing(Func<DataPipelineRunWorkItem, Task>)
Starts processing data pipeline run work items.
Task<bool> StartDataPipelineRunWorkItemProcessing(Func<DataPipelineRunWorkItem, Task> processWorkItem)
Parameters
processWorkItem
Func<DataPipelineRunWorkItem, Task>The asynchronous delegate that is invoked for each data pipeline run work item.
Returns
StopDataPipelineRunWorkItemProcessing()
Stops processing data pipeline run work items.
Task StopDataPipelineRunWorkItemProcessing()
Returns
TryLoadDataPipelineRunArtifacts(DataPipelineDefinition, DataPipelineRun, string)
Tries to load the artifacts associated with a data pipeline run.
Task<(bool Success, List<DataPipelineStateArtifact> Artifacts)> TryLoadDataPipelineRunArtifacts(DataPipelineDefinition dataPipelineDefinition, DataPipelineRun dataPipelineRun, string artifactsNameFilter)
Parameters
dataPipelineDefinition
DataPipelineDefinitionThe data pipeline definition associated with the work item.
dataPipelineRun
DataPipelineRunThe data pipeline run item associated with the work item.
artifactsNameFilter
stringThe name pattern used to identify the artifacts to load.
Returns
- Task<(bool Success, List<DataPipelineStateArtifact> Artifacts)>
A boolean indicating whether the operation was successful or not and the list with the details of the artifacts if the operation is successful.
UpdateDataPipelineRunStatus(DataPipelineRun)
Updates the status of a data pipeline run.
Task<bool> UpdateDataPipelineRunStatus(DataPipelineRun dataPipelineRun)
Parameters
dataPipelineRun
DataPipelineRunThe data pipeline run whose status is to be updated.
Returns
UpdateDataPipelineRunWorkItem(DataPipelineRunWorkItem)
Updates a data pipeline run work item.
Task<bool> UpdateDataPipelineRunWorkItem(DataPipelineRunWorkItem workItem)
Parameters
workItem
DataPipelineRunWorkItemThe data pipeline run work item to be updated.
Returns
UpdateDataPipelineRunWorkItemsStatus(List<DataPipelineRunWorkItem>)
Updates the status of data pipeline run work items.
Task<bool> UpdateDataPipelineRunWorkItemsStatus(List<DataPipelineRunWorkItem> workItems)
Parameters
workItems
List<DataPipelineRunWorkItem>The list of data pipeline work items whose status must be updated.