Troubleshooting Data Pipelines
Learn how to diagnose common problems when creating, running, and monitoring data pipelines.
Overview
Most data pipeline problems fall into one of these areas:
- Permissions: the action is hidden, disabled, or rejected.
- Pipeline definition: the pipeline can't be saved, or a trigger is missing or misconfigured.
- Run start: the run is rejected before any content is processed.
- Run processing: one or more work items fail, which fails the stage and stops the stages after it.
Start with the run analysis, then use the sections below.
Investigate a run
- In the sidebar, select Data > Data Pipeline Runs.
- Adjust the filters so the run is listed. The Activation filter defaults to Manual; select Schedule or Any to see scheduled runs.
- Check the Status (Completed or In progress), Success (Success or Failure), Completed Stages, and Failed Stages columns.
- Select the Analyze run (chart) icon to open Data Pipeline Run Analysis:
- Errors (shown only when present) lists run-level errors, such as problems starting the next stages.
- Execution Timeline marks stages with failures and stages with warnings.
- Invocation Parameters shows the canonical run ID, processor, trigger, triggering user, and the parameter values the run used.
- Stage Metrics Analysis shows, per stage, how many work items completed, succeeded, and had warnings. Work items with warnings show a warning icon with the message, and a Download unsafe content parts icon that downloads the text parts flagged by content shielding.
The run analysis shows counts of failed work items but not their error messages. To find the error details, open Platform > Telemetry, select the Errors tab, and query the logs of the data pipeline worker services (for example, with the Most recent 50 AppExceptions template).
See Data Pipeline Runs for a full description of the page.
How failures propagate
- A work item that fails with a temporary error is retried. After 10 failed attempts, it is marked as failed with the error "Too many failed processing attempts."
- Some errors are permanent (for example, an unsupported content type or blocked content). Those work items are marked as failed without retries.
- A stage succeeds only if all of its work items succeed. If any work item fails, the stage is listed under Failed Stages, its child stages do not start, and the run completes with Failure.
- Warnings do not fail a run. A successful run that has warnings is shown with a distinct Success style.
To stop a run that is still in progress, select the Cancel run (ban) icon, then select Cancel Run in the Cancel Pipeline Run dialog. Completed runs can't be cancelled.
Permission problems
| Symptom | Cause and fix |
|---|---|
| The Create Pipeline button is missing. | The button is shown only to users with the Data Pipelines Contributor role. |
| Saving a pipeline fails with "Error saving pipeline. Please try again." | Every create and update requires the Data Pipelines Contributor role at a scope that covers the pipeline, even for Owners and Contributors. Updating an existing pipeline also requires write permission on it. |
| The Edit, Run Now, or Activate controls are disabled. | You don't have FoundationaLLM.DataPipeline/dataPipelines/write on the pipeline. |
| The Delete icon is disabled. | You don't have FoundationaLLM.DataPipeline/dataPipelines/delete on the pipeline. |
| The Data Pipeline Runs page shows no runs, or Last Run shows "Never" for pipelines that have run. | Viewing runs requires write permission on the pipeline (Owner, Contributor, or Data Pipelines Execution Manager). The Reader role is not enough. With Pipeline set to Any, the page queries all pipelines, which requires permission at the instance scope; if your access is limited to specific pipelines, select one in the Pipeline filter. |
See RBAC Roles and Permissions for Data Pipelines.
Pipeline definition problems
When you select Create Pipeline or Save Changes, the editor lists every problem it finds above the tabs. Select an issue to go to the affected stage or trigger. Common issues:
| Message | Fix |
|---|---|
| Pipeline name is required. / Pipeline name is not available. | Enter a unique pipeline name on the General tab. |
| Data source has no plugin selected. | Select the data source node on the Stages tab and choose a Data source plugin. |
| Stage "<name>" has no plugin selected. | Select a Plugin for the stage, or delete the stage. |
| Stage "<name>" is missing a required dependency selection. | Select a dependency under Dependencies. |
| Add at least one trigger so the pipeline can be run. | Add a trigger on the Triggers tab. |
| Trigger "<name>" needs a cron schedule. / has an invalid cron schedule. | Set a valid five-field schedule for the Schedule trigger. |
The editor does not validate parameter values. Missing or wrong values surface as work item failures when the pipeline runs.
Run start problems
| Symptom | Cause and fix |
|---|---|
| "Pipeline "<name>" has no configured trigger to run." | Add a trigger to the pipeline. |
| "Parameter <key> has no value." (in Advanced run…) | Enter a value for every listed parameter. |
| The run is rejected because it "is conflicting with an already existing run and cannot be started." | A run with the same canonical run ID is still in progress. Wait for it to finish or cancel it. To let runs for different parameter values execute in parallel, select those parameters for canonical identification on the trigger. See Setting Up Triggers. |
| The run is rejected because the trigger "does not exist in the data pipeline". | The trigger was renamed or removed. Runs always use the most recently saved pipeline definition. |
| The run fails with "Plugin metadata could not be retrieved for plugin object id …". | A plugin referenced by a stage is not registered. Check the plugin packages on the Platform > Packages page. |
Scheduled runs don't appear
Check the following in order:
- The pipeline is active. Only active pipelines are scheduled. New pipelines are inactive until you activate them in the Activate column on the Data Pipelines page.
- Enough time has passed. The scheduler reloads pipelines every five minutes, so new schedules and activation changes can take up to five minutes to take effect.
- The schedule is in UTC. Cron schedules are evaluated in UTC, not in your local time zone.
- The run had content to process. Scheduled runs that find no content items are not recorded.
- The runs page filter includes scheduled runs. Set Activation to Schedule or Any.
- No conflicting run is in progress. If a run with the same canonical run ID is still running, the scheduled occurrence is skipped.
Event triggers do not start runs automatically. Use a Schedule trigger for recurring processing.
Stage errors
The built-in stage plugins report the following common problems. Missing required parameters are reported as "The plugin <name> requires the <parameter> parameter." Check the parameter values on the trigger that started the run (shown under Invocation Parameters).
| Stage plugin | Problem | Fix |
|---|---|---|
| Text Extraction | The content type is not supported, or the stage "does not have a dependency content text extraction plugin to handle the <type> file type." | Add the matching content text extraction dependency (PDF, DOCX, PPTX, XLSX, or an image plugin), or remove unsupported files from the data source scope. |
| Text Extraction | Warning: the extracted text "exceeds the maximum allowed size … and has been discarded." | Increase MaxContentSizeCharacters or split the document. |
| Text Partitioning | No partitioning plugin for the selected strategy. | Select a dependency that matches PartitioningStrategy. Use the Token strategy; the Semantic partitioning plugin is not implemented yet. |
| Azure AI Content Safety Shielding | Warning: "Prompt injection detected in <n> content parts. The entire content item was blocked from further processing." The work item and the stage fail. | BlockEntireContentItem is true. Download the unsafe parts from the run analysis to review them. Set BlockEntireContentItem to false to exclude only the flagged parts and index the rest of the item. |
| Gateway Text Embedding | Failures in embedding chunk groups, or an operation that exceeds the maximum polling time (one hour). | Check the Gateway API and the embedding deployment used by the knowledge unit's vector database. Lower EmbeddingRequestSizeTokens if requests are too large. |
| Azure AI Search Indexing | The vector database type is not supported. | The knowledge unit must reference an Azure AI Search vector database. |
| Azure AI Search Indexing | The knowledge unit "does not specify a vector store id and the VectorStoreId is empty." | Set the vector store ID on the knowledge unit or the VectorStoreId parameter on the trigger. |
| Azure AI Search Indexing | "Failed to index <n> content item parts out of <m>." | Check the Azure AI Search service and the API endpoint configuration used by the vector database. |
A warning of "The content item has no content." means the item produced no text. It doesn't fail the run.