Table of Contents

Creating Data Pipelines

Learn how to create and edit data pipelines in the Management Portal.

Overview

A data pipeline reads content through a data source and processes it through a graph of stages, for example: extract text, partition it into chunks, embed the chunks, and index them in a vector database. Triggers define how the pipeline runs (on a schedule, on demand, or on events) and the parameter values each run uses.

Each part of a pipeline is implemented by a plugin. The data source uses a plugin from the Data Source category, and each stage uses a plugin from the Data Pipeline Stage category. The plugins you can choose depend on the plugin packages installed on the Platform > Packages page.

Prerequisites

The Data Pipelines List

In the sidebar, select Data > Data Pipelines.

Column Description
Name The pipeline name.
Description The pipeline description.
Schedule A plain-language summary of the first Schedule trigger, or Manual only when the pipeline has no schedule trigger.
Last Run The start time of the most recent run, colored by outcome (in progress, successful, or failed), or Never.
Activate A green check means the pipeline is active; a red x means it is inactive. Select the icon to activate or deactivate the pipeline.
Actions View JSON definition ({}), Edit (pencil), Run Now (play), and Delete (trash).

Use Search pipelines to filter by name, and the refresh button to reload the list. The list loads 20 pipelines at a time and shows Loading more pipelines... as you scroll.

Actions that change a pipeline are enabled only when you have write or delete permission on it. Running a pipeline is covered in Invoking Data Pipelines.

Activate or Deactivate a Pipeline

Select the icon in the Activate column. In the Activate Pipeline or Deactivate Pipeline dialog, select Activate or Deactivate to confirm. Schedule triggers start runs only for active pipelines. New pipelines are created inactive, and saving a pipeline in the editor doesn't change whether it is active.

Create a Pipeline

  1. On the Data Pipelines page, select Create Pipeline.
  2. Configure the three tabs: General, Stages, and Triggers.
  3. Select Create Pipeline at the top right.

A Data Pipelines link with a back arrow at the top of the editor, and the Cancel button, both ask you to confirm before discarding unsaved changes.

General Tab

Field Description
Display name A friendly name. The page title shows it as you type.
Name * The pipeline resource name. Letters, numbers, dashes, and underscores only. A check mark shows whether the name is available. The name can't be changed after creation; when editing, it is shown below Display name.
Description What the pipeline processes.

Stages Tab

The Stages tab shows a visual canvas. The Data Source node is always present. Each stage is a node, and arrows show which stage runs after which.

Toolbar and canvas controls

  • Add Stage creates a stage named Stage1, Stage2, and so on, and opens its dialog.
  • Show list view / Hide list view toggles an accessible list where each stage has a Runs after: dropdown. Select Data Source (root stage) to make a stage run first.
  • Use the zoom controls at the bottom left of the canvas to zoom and fit the view.
  • Nodes with a warning icon have no plugin selected or have validation issues.

Connect stages

  • Drag from the output handle on the right of a node to the input handle on the left of another stage. Dragging from the Data Source node makes the target a starting (root) stage.
  • Each stage has exactly one parent. Connecting a stage to a new parent replaces its previous connection.
  • Select the x on an arrow to remove the connection; the stage becomes a root stage again.
  • The canvas blocks connections that would create a cycle.
  • A pipeline with no stages runs only the data source.

Configure the data source

Select the Data Source node to open the Data Source dialog:

  1. In Data source, select a data source. The name and description are pre-filled from it.
  2. Optionally edit Data source name and Data source description. The name is used in trigger parameter keys.
  3. In Data source plugin, select a plugin, for example Azure Data Lake Data Source (FoundationaLLM) or SharePoint Online Data Source (FoundationaLLM).
  4. Set the plugin's Parameters, such as Folders or DocumentLibraries.
  5. Select Close.

Configure a stage

Select a stage node to open the Stage: {name} dialog:

  1. Edit Stage Name and the description. Stage names must be unique.
  2. In the plugin dropdown, select the stage plugin.
  3. Set the plugin's Parameters.
  4. If the plugin has dependencies, select them under Dependencies (a single dropdown or a multi-select, depending on the plugin), then set each dependency's parameters.
  5. Select Close, or select Delete Stage to remove the stage. You can also delete a stage with the trash icon on its node.

Select the refresh icon next to a plugin dropdown to reload the plugin's parameter definitions. This resets the parameters to the plugin defaults.

Parameter inputs

Parameter type Input
string, int, float, datetime Text box, or a dropdown when the plugin defines allowed values.
bool Toggle switch.
array Chips. Type values separated by commas.
resource-object-id Dropdown of matching resources (for example, knowledge units, prompts, AI models, or data sources).

Standard stage plugins

The standard FoundationaLLM data pipeline plugin package (.NET) includes these stage plugins, among others:

Stage plugin Key parameters and dependencies
Text Extraction Data Pipeline Stage (FoundationaLLM) MaxContentSizeCharacters. Dependencies (multiple): content text extraction plugins for PDF, DOCX, PPTX, and XLSX (image plugins are also listed but not yet functional; see Image Description). Must be a starting stage.
Text Partitioning Data Pipeline Stage (FoundationaLLM) PartitioningStrategy (Token or Semantic). Dependency (single): Token Content Text Partitioning (PartitionSizeTokens, PartitionOverlapTokens) or Semantic Content Text Partitioning.
Azure AI Content Safety Shielding Data Pipeline Stage (FoundationaLLM) BlockEntireContentItem.
Gateway Text Embedding Data Pipeline Stage (FoundationaLLM) KnowledgeUnitObjectId, EmbeddingRequestSizeTokens.
Azure AI Search Indexing Data Pipeline Stage (FoundationaLLM) KnowledgeUnitObjectId, VectorStoreId.
Azure AI Search Removal Data Pipeline Stage (FoundationaLLM) KnowledgeUnitObjectId, VectorStoreId.
Knowledge graph stages See Knowledge Graph Integration.

See Data Pipeline Plugins for the full reference.

Triggers Tab

Triggers are shown as cards with the trigger name, a type badge, a schedule summary (for example, the cron description, Runs on demand, or Runs on event), the number of parameter groups, and any issues.

  1. Select Add Trigger. A trigger named Trigger1, Trigger2, and so on is created with type Manual, and its dialog opens.
  2. In the Trigger: {name} dialog, set:
    • Trigger Name.
    • Trigger Type: Schedule, Event, or Manual.
    • Schedule (Schedule triggers only): build the cron schedule with the schedule editor. The default is daily at 06:00 (0 6 * * *).
  3. Under Trigger Parameters, expand each group to set values. Groups are labeled Data Source, Stage: {name}, and Stage: {name} → {dependency}. The full parameter key (for example, Stage.Partition.Dependency.TokenContentTextPartitioning.PartitionSizeTokens) is shown under each field.
  4. Optionally select Use in canonical identification of runs for parameters that identify the same logical run (for example, the folder being processed). The portal asks you to confirm, because changing these parameters affects how the next run is matched with the state of previous runs. Change them only on a newly created pipeline.
  5. Select Close, or Delete Trigger to remove it. You can also select Edit or the trash icon on the card.

Note: Only Schedule triggers start runs automatically. Manual triggers are started from the portal or the API. You can select Event, but the current trigger service does not start runs from events.

Schedule triggers run without user input, so give them a complete set of parameter values.

Validation

When you select Create Pipeline or Save Changes, the editor validates the pipeline. If there are problems, a red summary appears at the top ("n issues must be fixed before saving:"). Select an issue to jump to the affected tab, node, or trigger. Common issues:

  • Pipeline name is required, or not available.
  • Data source has no plugin selected.
  • A stage has no plugin selected, is missing a required dependency selection, has an empty name, or uses a name more than once.
  • Add at least one trigger so the pipeline can be run.
  • A trigger has an empty or duplicate name, or a Schedule trigger needs a valid cron schedule.

Parameter values are not validated in the editor.

Edit a Pipeline

  1. In the Data Pipelines list, select the Edit (pencil) icon.
  2. Change the settings on any tab.
  3. Select Save Changes. You return to the list after saving.

Delete a Pipeline

Select the Delete (trash) icon, then select Delete in the Delete Pipeline dialog.