Table of Contents

Optimizing Pipeline Performance

Learn which settings affect how fast data pipelines run and how much work they do.

Overview

Data pipeline runs split the content items found by the data source into work items, typically one per content item in each stage. Worker services process the work items in parallel. You can make runs faster and cheaper by:

  • Letting the pipeline skip content that hasn't changed.
  • Limiting the data source to the content you need.
  • Tuning the stage parameters that control content size and request size.
  • Choosing the right processor and worker capacity.

Use the run analysis to measure the effect of each change.

Measure before you tune

  1. In the sidebar, select Data > Data Pipeline Runs.
  2. Find the run and select the Analyze run (chart) icon.
  3. In the Data Pipeline Run Analysis dialog, review:
    • Execution Timeline: the duration of the run, of each stage, and of each stage's work items.
    • Stage Metrics Analysis: per stage, the number of work items, how many completed and succeeded, the number of warnings, and the start time and duration of each work item.

Compare these numbers between runs after each change. See Data Pipeline Runs for the full page description.

Process only what changed

Runs that share a canonical run ID share processing state. The built-in stages use that state to avoid repeating work:

  • Text Partitioning skips a content item when its extracted text hasn't changed.
  • Gateway Text Embedding and Azure AI Content Safety Shielding process only the text parts that changed in the current run.
  • Azure AI Search Indexing skips a content item when its text and metadata haven't changed and all of its parts are already indexed.

To benefit from this:

  • Keep the Use in canonical identification of runs settings on your triggers stable. Changing them starts a new state, and the next run reprocesses everything. See Setting Up Triggers.
  • When one pipeline processes several independent locations, select the location parameter (for example, Folders or DocumentLibraries) for canonical identification. Each location then keeps its own state, and runs for different locations can execute at the same time. Runs with the same canonical run ID cannot run in parallel.

Limit the data source scope

The data source plugin decides which content items a run processes. Point it only at the content you need, for example specific Folders for Azure Data Lake or specific DocumentLibraries for SharePoint Online. You can create several triggers with different parameter values instead of one trigger that covers everything.

Tune stage parameters

Plugin Parameter Effect
Text Extraction MaxContentSizeCharacters Content items whose extracted text exceeds this size are discarded with a warning. When not set, the limit is 10,000,000 characters. Lower it to keep very large documents out of the downstream stages.
Token Content Text Partitioning PartitionSizeTokens, PartitionOverlapTokens Larger partitions produce fewer parts, which means fewer texts to embed and fewer index entries. Smaller overlaps reduce duplicated tokens. Both values affect retrieval quality, so validate search results after changing them.
Gateway Text Embedding EmbeddingRequestSizeTokens Changed text parts are grouped into requests of up to this many tokens, and all requests for a content item are sent to the Gateway API in parallel. When not set, the size is 100,000 tokens.

Set these values on the stage (as defaults) and on each trigger that runs the pipeline, because runs use the trigger's values. See Configuring Pipeline Stages.

The embedding model and dimensions are defined by the vector database referenced by the knowledge unit, not by the pipeline. See Creating Vector Databases.

Choose a processor

Each run is processed by one of two worker services:

Processor Used by
DataPipelineBackendWorker Scheduled runs (always), Run Now, and the default selection in Advanced run….
DataPipelineFrontendWorker Selectable in Advanced run…. The platform also uses it for pipelines that process files uploaded by users.

Keep long-running batch runs on the backend worker so that they don't delay the processing of user uploads on the frontend worker.

Scale worker capacity

Each worker service processes up to a fixed number of work items at the same time. An administrator sets this number in Azure App Configuration:

Key Default
FoundationaLLM:APIEndpoints:DataPipelineFrontendWorker:Configuration:ParallelProcessorsCount 10
FoundationaLLM:APIEndpoints:DataPipelineBackendWorker:Configuration:ParallelProcessorsCount 10

The value is read when the worker service starts, so restart the worker after changing it. Higher values increase throughput only if the downstream services (for example, the Gateway API and Azure AI Search) have capacity for the additional requests.