Table of Contents

Monitoring Data Pipelines

Learn how to track data pipeline runs, find failures, and review stage performance in the Management Portal.

Overview

The Management Portal shows pipeline activity in two places:

Page What it shows
Data > Data Pipelines Each pipeline's schedule, last run, and whether it is active.
Data > Data Pipeline Runs The run history, with filters, per-run status, and a detailed run analysis.

Check Pipeline Status

On the Data Pipelines page:

  • Schedule shows the schedule of the first Schedule trigger in plain language, or Manual only.
  • Last Run shows when the most recent run started. The pill is blue while the run is in progress, green when it succeeded, and red when it failed. Never means the pipeline has no runs.
  • Activate shows a green check for active pipelines and a red x for inactive ones. Schedule triggers start runs only for active pipelines.

Watch Active Runs

  1. In the sidebar, select Data > Data Pipeline Runs.
  2. Set Status to In progress. Set Activation to Any to include scheduled runs (the page opens filtered to Manual runs).
  3. Select the refresh icon at the top right of the page to reload the list, or the Refresh run icon on a row to update only that run.

The Completed Stages and Failed Stages columns update as stages finish.

Analyze a Run

Select the Analyze run (chart) icon on a run to open the Data Pipeline Run Analysis dialog. It shows:

  • Run ID, Owned By, and Triggered By, each with a copy icon.
  • Start Time, End Time (or In progress), and Total Duration.
  • Errors: Shown when the run recorded errors.
  • Execution Timeline: A bar for the whole pipeline run and one for each stage. Green stage bars mean all work items succeeded; red means some failed. A warning icon marks stages with warnings, and a Work Items (n) row shows when that stage's work items ran.
  • Invocation Parameters: Canonical Run ID, Processor, Trigger, Triggered By, and every trigger parameter value used by the run.
  • Stage Metrics Analysis: One section per stage with completed, successful, and warning counts and duration (mm:ss). Expand a stage to see its start and end time, work item counts, and a Work Items table (Work Item ID, Start Time, Duration) when telemetry is available. Work items with a warning show a warning icon (hover for the message) and a Download unsafe content parts icon that downloads a ZIP file.

Cancel a Run

Select the Cancel run (ban) icon on a run that is still in progress, then select Cancel Run in the Cancel Pipeline Run dialog (or Keep Running to close it). Processing stops and the run is marked as cancelled. The icon is disabled for completed runs.

Find Patterns Over Time

Use the filters on the Data Pipeline Runs page to review history:

  1. Select a pipeline in Pipeline.
  2. Set Period to a range such as Last 7 days or Last 30 days, or use Custom range under More filters.
  3. Set Success to Failure to list failed runs.
  4. Compare durations and the Failed Stages column across runs.
Pattern Possible cause
The same stage fails in every run Configuration error in that stage or its parameters.
Intermittent failures Transient errors, throttling, or network issues.
Durations increasing over time Growing data volume.
Success with warnings (amber Success pill) Some work items had warnings, for example from a shielding stage.