Class AzureCosmosDBDataPipelineService

Namespace

FoundationaLLM.DataPipelineEngine.Services.CosmosDB

Assembly

FoundationaLLM.DataPipelineEngine.dll

Provides the implementation for the Azure Cosmos DB data pipeline service.

public class AzureCosmosDBDataPipelineService : IAzureCosmosDBDataPipelineService

Inheritance

object

AzureCosmosDBDataPipelineService

Implements

IAzureCosmosDBDataPipelineService

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

ObjectExtensions.GetPropertyValues(object, List<string>)

Constructors

AzureCosmosDBDataPipelineService(IOptions<AzureCosmosDBSettings>, ILogger<AzureCosmosDBDataPipelineService>)

Initializes a new instance of the AzureCosmosDBDataPipelineService class.

public AzureCosmosDBDataPipelineService(IOptions<AzureCosmosDBSettings> options, ILogger<AzureCosmosDBDataPipelineService> logger)

Parameters

options IOptions<AzureCosmosDBSettings>

The IOptions providing the AzureCosmosDBSettings) with the Azure Cosmos DB settings.

logger ILogger<AzureCosmosDBDataPipelineService>

The ILogger used for logging.

Fields

_cosmosClient

protected readonly CosmosClient _cosmosClient

Field Value

CosmosClient

_dataPipelineContainer

protected readonly Container _dataPipelineContainer

Field Value

Container

_leasesContainer

protected readonly Container _leasesContainer

Field Value

Container

_logger

protected readonly ILogger<AzureCosmosDBDataPipelineService> _logger

Field Value

ILogger<AzureCosmosDBDataPipelineService>

_settings

protected readonly AzureCosmosDBSettings _settings

Field Value

AzureCosmosDBSettings

Methods

GetDataPipelineContentItem(string, string)

Get a data pipeline content item by its canonical identifier.

public Task<DataPipelineContentItem> GetDataPipelineContentItem(string dataPipelineRunId, string contentItemCanonicalId)

Parameters

dataPipelineRunId string

Thje data pipeine run identifier.

contentItemCanonicalId string

The canonical identifier of the content item run.

Returns

Task<DataPipelineContentItem>

GetDataPipelineRuns(DataPipelineRunFilter)

Gets a list of data pipeline runs filtered by the provided filter criteria.

public Task<List<DataPipelineRun>> GetDataPipelineRuns(DataPipelineRunFilter dataPipelineRunFilter)

Parameters

dataPipelineRunFilter DataPipelineRunFilter

The filter criteria used to filter data pipeline runs.

Returns

Task<List<DataPipelineRun>>

The list of requests data pipeline runs.

PatchDataPipelineRunWorkItemsStatusAsync(List<DataPipelineRunWorkItem>)

Patches the status of data pipeline run work items in Azure Cosmos DB.

public Task<bool> PatchDataPipelineRunWorkItemsStatusAsync(List<DataPipelineRunWorkItem> workItems)

Parameters

workItems List<DataPipelineRunWorkItem>

The list of data pipeline work items that need the status update.

Returns

Task<bool>

true if the batch is successfully processed.

PatchItemPropertiesAsync<T>(string, string, Dictionary<string, object?>, CancellationToken)

Patches the specified properties of an item in Azure Cosmos DB.

public Task<T> PatchItemPropertiesAsync<T>(string partitionKey, string id, Dictionary<string, object?> propertyValues, CancellationToken cancellationToken = default)

Parameters

partitionKey string

The partition key of the item.

id string

The identifier of the item.

propertyValues Dictionary<string, object>

Dictionary of the property names and values to patch.

cancellationToken CancellationToken

Cancellation token for async calls.

Returns

Task<T>

The patched object of type .

Type Parameters

T

The type of the item to patch.

RetrieveItem<T>(string, string)

Retrieves an item from Azure Cosmos DB.

public Task<T> RetrieveItem<T>(string id, string partitionKey)

Parameters

id string

The identifier of the item.

partitionKey string

The partition key of the item.

Returns

Task<T>

Type Parameters

T

The type of the item to retrieve.

RetrieveItemsAsync<T>(QueryDefinition)

Retrieves items from Azure Cosmos DB.

public Task<List<T>> RetrieveItemsAsync<T>(QueryDefinition query)

Parameters

query QueryDefinition

The query definition used to retrieve the items.

Returns

Task<List<T>>

The list of retrieved items.

Type Parameters

T

The type of the items to retrieve.

StartChangeFeedProcessorAsync(Func<DataPipelineRunWorkItem, Task>)

Starts the change feed processor for the data pipeline run work items.

public Task<bool> StartChangeFeedProcessorAsync(Func<DataPipelineRunWorkItem, Task> dataPipelineRunWorkItemProcessor)

Parameters

dataPipelineRunWorkItemProcessor Func<DataPipelineRunWorkItem, Task>

The asynchronous delegate that is invoked for each data pipeline run work item.

Returns

Task<bool>

true if the change feed processor is successfully started.

StopChangeFeedProcessorAsync()

Stops the change feed processor for the data pipeline run work items.

public Task StopChangeFeedProcessorAsync()

Returns

Task

UpsertDataPipelineRunBatchAsync(params dynamic[])

Batch create or update data pipeline run items.

public Task<bool> UpsertDataPipelineRunBatchAsync(params dynamic[] dataPipelineRunItems)

Parameters

dataPipelineRunItems dynamic[]

Data pipeline run items to create or replace.

Returns

Task<bool>

true if the batch is successfully processed.

UpsertItemAsync<T>(string, T, CancellationToken)

Upserts an item in Azure Cosmos DB.

public Task<T> UpsertItemAsync<T>(string partitionKey, T item, CancellationToken cancellationToken = default)

Parameters

partitionKey string

The partition of the item to upsert.

item T

The item to upsert.

cancellationToken CancellationToken

The cancellation token used to signal a cancellation request.

Returns

Task<T>

The upserted item.

Type Parameters

T

The type of the item to upsert.