Configuring Pipeline Stages
Learn how to configure the data source and processing stages of a data pipeline in the Management Portal.
Overview
A data pipeline starts with a data source that identifies the content items to process. Each content item then flows through one or more stages. Every stage is a plugin configuration: you pick a stage plugin, set its parameters, and (for some plugins) pick dependency plugins.
Stages form a tree:
- A stage that runs directly after the data source is a starting stage.
- Each stage has at most one parent stage. A stage can have several child stages, so a pipeline can branch.
- A child stage starts only after its parent stage has finished successfully for all of its work items. If any work item in a stage fails, the stage is marked as failed and its child stages do not run.
A pipeline can also consist of the data source only. In that case, the canvas shows "No stages — the pipeline will run the data source only."
Open the stage editor
- In the sidebar, select Data > Data Pipelines.
- Select Create Pipeline, or select the Edit (pencil) icon for an existing pipeline.
- Select the Stages tab.
The Stages tab shows a canvas with the data source node and one node per stage. Each node shows the stage name and the selected plugin (or "No plugin selected"). A warning icon appears on nodes that have validation issues.
| Canvas control | What it does |
|---|---|
| Add Stage | Adds a new stage named Stage<N> and opens its configuration dialog. |
| Show list view / Hide list view | Shows an accessible list of the data source and stages. For each stage, use Runs after: to choose its parent stage, or Data Source (root stage) to make it a starting stage. |
| Drag between node handles | Connects two nodes. The target stage runs after the source node. Because a stage has only one parent, a new connection replaces the previous one. |
| × on a connection | Removes the connection. The stage becomes a starting stage. |
| Trash icon on a node | Deletes the stage after you confirm in the Delete Stage dialog. |
Configure the data source
- On the canvas, select the data source node. The Data Source dialog opens.
- In Data source:, select a data source resource. The name and description are pre-filled from the selected resource.
- Review Data source name and Data source description. The name is used in trigger parameter keys (
DataSource.<name>.<parameter>), so keep it short and stable. - In Data source plugin, select the plugin that reads content from the data source.
- Set the plugin Parameters.
- Select Close.
The built-in data source plugins are:
| Plugin | Parameters |
|---|---|
| Azure Data Lake Data Source (FoundationaLLM) | Folders (array) — data lake folders; the first path segment is the container name. |
| SharePoint Online Data Source (FoundationaLLM) | DocumentLibraries (array) |
| Context File Data Source (FoundationaLLM) | ContextFileObjectId (resource object ID), ContentAction (add-or-update or remove) |
| Conversations Data Source (FoundationaLLM) | SelectionStrategy (Aged by agents or Inactive (no recent messages)), DaysCount (int), AgentNames (array) |
| Conversation Messages Data Source (FoundationaLLM) | ColumnAliases (string), CurrentDayOnly (bool) |
| Agent Tasks Data Source (FoundationaLLM) | None |
For details about the data source resources themselves, see Data Sources.
Configure a stage
- Select a stage node (or select Add Stage). The Stage: <name> dialog opens.
- Enter the Stage Name and Description. Stage names can contain only letters, digits, hyphens, and underscores, and must start with a letter. Each stage name must be unique in the pipeline.
- In Plugin, select a stage plugin. The Parameters section is filled with the plugin's parameters.
- If the plugin has dependencies, choose them under Dependencies. Some plugins accept a single dependency (drop-down), others accept several (multi-select). Parameters of each selected dependency appear under <dependency> Dependency Parameters:.
- Set the parameter values. Parameters of type resource object ID are shown as a drop-down of matching resources (for example, knowledge units, prompts, AI models, or data sources).
- Select Close, or select Delete Stage to remove the stage.
Use the Refresh parameters from plugin button (refresh icon next to the plugin drop-down) to reload the plugin's parameter list. Refreshing resets the parameters to the plugin defaults.
Important
The parameter values you enter on the data source and on a stage are defaults. When the pipeline runs, plugins receive the values stored on the trigger that started the run. New triggers are seeded from these defaults, and you can override them per trigger on the Triggers tab. See Setting Up Triggers.
Changing a stage's plugin or dependencies, refreshing its plugin, or changing the data source or its plugin clears the related parameter values from all triggers. Renaming a stage or the data source changes the trigger parameter keys, so review your triggers after a rename.
Built-in stage plugins
The FoundationaLLM Data Pipeline plugins package provides the following stage plugins. Additional plugins can be installed through plugin packages (see Plugins and Packages).
Text extraction and partitioning
| Plugin | Parameters | Notes |
|---|---|---|
| Text Extraction Data Pipeline Stage | MaxContentSizeCharacters (int) |
Must be a starting stage. Processes only content items that are added or updated. Text files are read directly. Binary documents are converted by the selected dependencies: PDF, DOCX, PPTX, XLSX, Image Content Text Extraction, and Image Metadata Text Extraction (multi-select). The image plugins take an AIModelObjectId parameter. A content type with no matching dependency fails the work item. When MaxContentSizeCharacters is not set, the maximum is 10,000,000 characters; larger content is discarded with a warning. |
| Text Partitioning Data Pipeline Stage | PartitioningStrategy (Token or Semantic) |
Requires one dependency that matches the strategy. Token Content Text Partitioning takes PartitionSizeTokens and PartitionOverlapTokens (both required). Semantic Content Text Partitioning is listed but not implemented yet; runs that use it fail. |
Shielding, embedding, and indexing
| Plugin | Parameters | Notes |
|---|---|---|
| Azure AI Content Safety Shielding Data Pipeline Stage | BlockEntireContentItem (bool, defaults to true) |
Runs after partitioning. Checks changed text parts for prompt-injection attacks. When BlockEntireContentItem is false, flagged parts are excluded from indexing and the work item completes with a warning. When it is true, the whole content item is blocked: its work item fails (which fails the stage and stops the stages after it), and the data source plugin is notified. In both cases the flagged parts can be downloaded from the run analysis. |
| Gateway Text Embedding Data Pipeline Stage | KnowledgeUnitObjectId, EmbeddingRequestSizeTokens (int, defaults to 100,000) |
Embeds only the text parts that changed in the current run. The embedding model and dimensions come from the vector database referenced by the knowledge unit. |
| Azure AI Search Indexing Data Pipeline Stage | KnowledgeUnitObjectId, VectorStoreId |
The knowledge unit's vector database must be an Azure AI Search database. The index is created if it doesn't exist. The knowledge unit's vector store ID is used when set; otherwise VectorStoreId is required. Existing entries for the content item are replaced. |
| Azure AI Search Removal Data Pipeline Stage | KnowledgeUnitObjectId, VectorStoreId |
Processes only content items marked for removal. Use it as a separate starting stage. |
For knowledge units and vector databases, see Creating Knowledge Units and Creating Vector Databases.
Knowledge graph
| Plugin | Parameters |
|---|---|
| Knowledge Extraction Data Pipeline Stage | PromptObjectId (prompts in the DataPipeline category), CompletionModel, EntityTypes, ModelTemperature, MaxOutputTokenCount |
| Knowledge Graph Consolidation Data Pipeline Stage | None |
| Knowledge Graph Summarization Data Pipeline Stage | PromptObjectId, CompletionModel, CompletionModelTemperature, CompletionMaxOutputTokenCount |
| Knowledge Graph Embedding Data Pipeline Stage | KnowledgeUnitObjectId |
| Knowledge Graph Indexing Data Pipeline Stage | KnowledgeUnitObjectId |
| Knowledge Graph Publishing Data Pipeline Stage | KnowledgeUnitObjectId |
Other stages
| Plugin | Parameters | Notes |
|---|---|---|
| Conversation Purge Data Pipeline Stage | BatchSize (int) |
Purges conversations identified by the Conversations data source. |
| Azure Data Lake Export Data Pipeline Stage | DataSourceObjectId, ConsolidatedExport (bool) |
Exports content to the location defined by an Azure Data Lake data source. |
| Agent Task Execution Data Pipeline Stage | None | Executes due agent tasks provided by the Agent Tasks data source. |
For a complete plugin reference, see Data Pipeline Plugins.
Example: document vectorization topology
A typical pipeline that indexes documents for agents uses these stages:
Data source
├── Extract (Text Extraction)
│ └── Partition (Text Partitioning, Token strategy)
│ └── Shield (Azure AI Content Safety Shielding, optional)
│ └── Embed (Gateway Text Embedding)
│ └── Index (Azure AI Search Indexing)
└── Removal (Azure AI Search Removal)
Use the same knowledge unit for the embedding, indexing, and removal stages so that embeddings match the index configuration.
Save and validate
Select Create Pipeline or Save Changes. Before saving, the portal checks the pipeline and lists every problem in a summary at the top of the page. Select an issue to go to the affected stage, trigger, or field. The checks include:
- A pipeline name is required and must be available.
- The data source must have a plugin selected.
- Every stage needs a non-empty, unique name and a plugin.
- Every dependency slot must have a plugin selected.
- At least one trigger is required.
Parameter values are not validated by the portal. A missing required parameter causes the affected work items to fail when the pipeline runs.