Table of Contents

Invoking Data Pipelines

Learn how to start a data pipeline run from the Management Portal.

Overview

A data pipeline run always uses one of the pipeline's triggers. Schedule triggers start runs automatically for active pipelines. To start a run yourself, use Run Now on the Data Pipelines page, either with the trigger's configured parameter values or with values you override for that run.

Use on-demand runs for initial data loads, re-processing specific folders or libraries, and testing a new configuration.

Prerequisites

  • The pipeline has at least one trigger. If it has none, the portal shows "Pipeline "{name}" has no configured trigger to run."
  • You have write permission on the pipeline. Otherwise the Run Now icon is disabled.

Run with the Configured Parameters

  1. In the sidebar, select Data > Data Pipelines.
  2. Select the Run Now (play) icon for the pipeline.
  3. In the Run Now dialog, select Run Now.

The run uses the pipeline's Manual trigger, or the first trigger if there is no Manual trigger, with that trigger's parameter values. It runs on the default processor, DataPipelineBackendWorker.

A notification confirms that the run started and shows the run name. The Last Run column updates to show the new run.

Run with Overridden Parameters (Advanced Run)

  1. Select the Run Now (play) icon for the pipeline.
  2. In the Run Now dialog, select Advanced run….
  3. In the Advanced run — {name} dialog:
    • Trigger: Shown only when the pipeline has more than one trigger. Selecting a trigger loads its parameter values.
    • Data Pipeline processor: DataPipelineBackendWorker (default) or DataPipelineFrontendWorker.
    • Parameter fields: One field per parameter, labeled with its full key (for example, DataSource.MyDataLake.Folders), pre-filled with the trigger value or the plugin default. Edit the values for this run only.
  4. Select Trigger.

Every parameter must have a value, or the portal shows "Parameter {key} has no value." Overridden values apply only to this run and are not saved to the trigger.

To change the values permanently, edit the trigger on the pipeline's Triggers tab. See Creating Data Pipelines.

Scheduled Runs

To run a pipeline on a schedule:

  1. Edit the pipeline and add a trigger with Trigger Type set to Schedule.
  2. Build the cron schedule and set all parameter values.
  3. Save the pipeline.
  4. In the Data Pipelines list, make sure the pipeline is active (green check in the Activate column).

The Schedule column on the list shows the schedule in plain language.

After Starting a Run

Track progress on the Data Pipeline Runs page (Data > Data Pipeline Runs). From there you can refresh a run, analyze it, or cancel it while it is in progress. See Data Pipeline Runs.

Troubleshooting

  • The Run Now icon is disabled: You don't have write permission on the pipeline.
  • "has no configured trigger to run": Add a trigger on the Triggers tab.
  • The run fails right away: Open the run's analysis on the Data Pipeline Runs page and check the Errors section. Common causes are missing parameter values, an unreachable data source, or an invalid plugin configuration.
  • A scheduled run doesn't start: Check that the pipeline is active and that the trigger type is Schedule.