Data Pipelines
Overview
A FoundationaLLM data pipeline has three main components:
- Data source: Where the pipeline reads content from. It references a data source resource and the data source plugin that reads it.
- Stages: The processing steps the content goes through. Stages form a forest: there are one or more starting stages, and each stage can have any number of next stages but only one parent.
- Triggers: How a run starts. A trigger is
Schedule(cron-based),Manual(started from the Management Portal or the API), orEvent. In the current release, onlyScheduletriggers start runs automatically, and only for active pipelines.
The data source and each stage are implemented by plugins. Each plugin defines the parameters it needs (name, type, and description). The pipeline stores default values for those parameters, and each trigger stores the values used when the trigger starts a run.
Note
Triggers of type Schedule (and Event) run without user interaction, so they must provide a complete set of parameter values. When you start a run manually from the Management Portal, Run Now uses the selected trigger's values, and Advanced run… lets you override them for that run. Values not provided by the run request are filled in from the trigger.
In the Management Portal, you build pipelines visually on the Stages and Triggers tabs of the pipeline editor. See Creating Data Pipelines.
Example
This pipeline reads files from an Azure Data Lake data source, extracts text, partitions it into token-based chunks, embeds the chunks, and indexes them in Azure AI Search through a knowledge unit:
{
"type": "data-pipeline",
"name": "DataPipeline01",
"display_name": "Data Pipeline 01",
"description": "Data Pipeline demo.",
"active": false,
"data_source": {
"name": "VGDataLake",
"description": "Victorious Ground data lake storage.",
"data_source_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.DataSource/dataSources/alchemy",
"plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-AzureDataLakeDataSource",
"plugin_parameters": [
{
"parameter_metadata": {
"name": "Folders",
"type": "array",
"description": "List of strings defining data lake folders (the first part identifies the container name)."
},
"default_value": null
}
],
"plugin_dependencies": []
},
"starting_stages": [
{
"name": "Extract",
"description": "Extract text from binary content items.",
"plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-TextExtractionDataPipelineStage",
"plugin_parameters": [],
"plugin_dependencies": [
{ "plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-PDFContentTextExtraction", "plugin_parameters": [] },
{ "plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-DOCXContentTextExtraction", "plugin_parameters": [] }
],
"next_stages": [
{
"name": "Partition",
"description": "Partition text into chunks.",
"plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-TextPartitioningDataPipelineStage",
"plugin_parameters": [
{
"parameter_metadata": {
"name": "PartitioningStrategy",
"type": "string",
"description": "Strategy used to partition text (can be Token or Semantic)."
},
"default_value": "Token"
}
],
"plugin_dependencies": [
{
"plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-TokenContentTextPartitioning",
"plugin_parameters": [
{
"parameter_metadata": {
"name": "PartitionSizeTokens",
"type": "int",
"description": "The size of the text partition in tokens."
},
"default_value": 400
},
{
"parameter_metadata": {
"name": "PartitionOverlapTokens",
"type": "int",
"description": "The size of the text partition overlaps in tokens."
},
"default_value": 100
}
]
}
],
"next_stages": [
{
"name": "Embed",
"description": "Embed chunks of text.",
"plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-GatewayTextEmbeddingDataPipelineStage",
"plugin_parameters": [
{
"parameter_metadata": {
"name": "KnowledgeUnitObjectId",
"type": "resource-object-id",
"description": "The FoundationaLLM resource identifier of the Knowledge Unit that provides the embedding configuration."
},
"default_value": "/instances/{{instanceId}}/providers/FoundationaLLM.Context/knowledgeUnits/dune-unit"
},
{
"parameter_metadata": {
"name": "EmbeddingRequestSizeTokens",
"type": "int",
"description": "The maximum size in tokens of each embedding request sent to the Gateway API."
},
"default_value": null
}
],
"plugin_dependencies": [],
"next_stages": [
{
"name": "Index",
"description": "Persist embeddings to a vector store.",
"plugin_object_id": "/instances/{{instanceId}}/providers/FoundationaLLM.Platform/plugins/Dotnet-FoundationaLLMDataPipelinePlugins-AzureAISearchIndexingDataPipelineStage",
"plugin_parameters": [
{
"parameter_metadata": {
"name": "KnowledgeUnitObjectId",
"type": "resource-object-id",
"description": "The FoundationaLLM resource identifier of the Knowledge Unit that provides the embedding configuration."
},
"default_value": "/instances/{{instanceId}}/providers/FoundationaLLM.Context/knowledgeUnits/dune-unit"
},
{
"parameter_metadata": {
"name": "VectorStoreId",
"type": "string",
"description": "The name of the Vector Store resource that identifies the logical partition of the Azure AI Search index."
},
"default_value": "Dune"
}
],
"plugin_dependencies": [],
"next_stages": []
}
]
}
]
}
]
}
],
"triggers": [
{
"name": "Default pipeline schedule",
"trigger_type": "Schedule",
"trigger_cron_schedule": "0 6 * * *",
"parameter_values": {
"DataSource.VGDataLake.Folders": [
"vectorization-input/Dune"
],
"Stage.Partition.PartitioningStrategy": "Token",
"Stage.Partition.Dependency.TokenContentTextPartitioning.PartitionSizeTokens": 400,
"Stage.Partition.Dependency.TokenContentTextPartitioning.PartitionOverlapTokens": 100,
"Stage.Embed.KnowledgeUnitObjectId": "/instances/{{instanceId}}/providers/FoundationaLLM.Context/knowledgeUnits/dune-unit",
"Stage.Embed.EmbeddingRequestSizeTokens": 8000,
"Stage.Index.KnowledgeUnitObjectId": "/instances/{{instanceId}}/providers/FoundationaLLM.Context/knowledgeUnits/dune-unit",
"Stage.Index.VectorStoreId": "Dune"
},
"canonical_id_parameters": [
"DataSource.VGDataLake.Folders"
]
}
]
}
Structure Rules
- The data source and every stage specify the plugin that implements them:
plugin_object_id: The resource identifier of the plugin, in the form/instances/{instanceId}/providers/FoundationaLLM.Platform/plugins/{pluginName}. Plugins come from the plugin packages installed on the Platform > Packages page.plugin_parameters: The parameters required by the plugin. Theparameter_metadatavalues come from the plugin definition. Parameter types arestring,int,float,bool,datetime,array, andresource-object-id.plugin_dependencies: Other plugins the plugin requires (for example, content text extraction plugins for the text extraction stage). The plugin definition says whether one dependency (Single) or several (Multiple) can be selected. Dependencies can have their own parameters.
- The data source specifies
data_source_object_id, the resource identifier of the data source it reads. The data source must be compatible with the data source plugin. - The data source name and the stage names must be unique within the pipeline and can contain only letters, numbers, underscores, and hyphens.
- Stages list the stages that run after them in
next_stages. activecontrols whetherScheduletriggers start runs for the pipeline. In the Management Portal, change it with the Activate column on the Data Pipelines page.- Triggers provide parameter values for the data source and stage parameters, keyed as described below.
canonical_id_parameters(optional) lists the trigger parameter keys used, together with the instance and pipeline name, to compute a run's canonical run ID. Runs with the same canonical run ID are treated as the same logical run when matching state with previous runs. When the list is empty, only the instance and pipeline name are used.
Trigger Parameter Naming Convention
| Origin | Key format | Example |
|---|---|---|
| Data source | DataSource.{DataSourceName}.{ParameterName} |
DataSource.VGDataLake.Folders |
| Stage | Stage.{StageName}.{ParameterName} |
Stage.Partition.PartitioningStrategy |
| Stage dependency | Stage.{StageName}.Dependency.{DependencyPluginName}.{ParameterName} |
Stage.Partition.Dependency.TokenContentTextPartitioning.PartitionSizeTokens |
{DependencyPluginName} is the last segment of the dependency plugin name, after the final hyphen (for example, TokenContentTextPartitioning for Dotnet-FoundationaLLMDataPipelinePlugins-TokenContentTextPartitioning).
In the Management Portal trigger editor, parameters are grouped by origin (Data Source, Stage: {name}, and Stage: {name} → {dependency}), and the full key is shown under each field.