Table of Contents

Setting Up Triggers

Learn how to configure the triggers that start data pipeline runs.

Overview

Triggers are part of the data pipeline definition. Each trigger has a name, a type, and a set of parameter values that are passed to the data source and stage plugins when a run starts. A pipeline must have at least one trigger before it can be saved.

Trigger type How runs start
Manual On demand, from the Run Now or Advanced run… actions on the Data Pipelines page, or through the Management API.
Schedule Automatically, on a cron schedule, while the pipeline is active.
Event Can be selected and saved, but the platform does not currently start runs automatically for Event triggers. An Event trigger runs only when it is used explicitly, for example by selecting it in Advanced run… or by naming it when triggering a run through the Management API.

Add or edit a trigger

  1. In the sidebar, select Data > Data Pipelines.

  2. Select Create Pipeline, or select the Edit (pencil) icon for an existing pipeline.

  3. Select the Triggers tab. Each trigger is shown as a card with its name, type, a schedule summary ("Runs on demand", "Runs on event", or a description of the cron schedule), the number of parameter groups, and the number of validation issues.

  4. Select Add Trigger to create a new trigger, or select a trigger card (or its Edit button) to change it. The Trigger: <name> dialog opens.

  5. Configure the trigger:

    Field Description
    Trigger Name The name of the trigger. Must be unique within the pipeline. New triggers are named Trigger<N>.
    Trigger Type Schedule, Event, or Manual. New triggers default to Manual.
    Schedule Shown only for Schedule triggers. A schedule builder that produces a standard five-field cron expression. The initial value is daily at 06:00 (0 6 * * *).
    Trigger Parameters The parameter values for this trigger, grouped by origin (see below).

  6. Select Close, then select Save Changes (or Create Pipeline) to save the pipeline.

To remove a trigger, select the trash icon on its card, or select Delete Trigger in the trigger dialog, and confirm.

Trigger parameters

When the pipeline runs, plugins receive their parameter values from the trigger, not from the stage definitions. The trigger dialog lists every parameter of the data source plugin, every stage plugin, and every selected stage dependency plugin. Parameters are grouped in collapsible sections:

Group Parameter key format
Data Source DataSource.<data source name>.<parameter>
Stage: <stage> Stage.<stage name>.<parameter>
Stage: <stage> → <dependency> Stage.<stage name>.Dependency.<dependency plugin>.<parameter>

The key is shown under each field. The dependency plugin segment is the last part of the plugin name, for example TokenContentTextPartitioning.

When a trigger is created, its values are seeded from the defaults entered on the Stages tab. After that, the trigger keeps its own values. Changing a stage's plugin or dependencies, refreshing a plugin, or changing the data source clears the related values from every trigger.

Canonical identification of runs

Each parameter has a Use in canonical identification of runs check box. The selected parameters, together with the instance and the pipeline name, determine the run's canonical run ID:

  • Runs with the same canonical run ID share processing state. Stages use that state to skip content that hasn't changed since the previous run.
  • Two runs with the same canonical run ID cannot run at the same time. A new run is rejected while a conflicting run is in progress.
  • When no parameters are selected, all runs of the pipeline share one canonical run ID.

For example, select the folder or document library parameter when the same pipeline processes different locations, so that each location keeps its own state and can run in parallel.

Changing this setting asks you to confirm, because the next run will no longer match the state of previous runs. Change it only on new pipelines.

Schedule triggers

Schedule triggers use standard five-field cron expressions (minute hour day-of-month month day-of-week), evaluated in UTC.

Expression Meaning
0 6 * * * Daily at 06:00 UTC
0 * * * * Every hour on the hour
0 0 * * 0 Weekly on Sunday at 00:00 UTC
0 2 1 * * Monthly on day 1 at 02:00 UTC

The Schedule column on the Data Pipelines page shows the first Schedule trigger of each pipeline, or "Manual only" when the pipeline has none.

How schedules are processed:

  • Only active pipelines are scheduled. New pipelines are created inactive; use the Activate column on the Data Pipelines page to activate a pipeline. Activation does not affect manual runs.
  • The scheduler reloads the list of pipelines and triggers every five minutes, so a new or changed schedule, or an activation change, can take up to five minutes to take effect.
  • Scheduled runs use the trigger's saved parameter values, run on the DataPipelineBackendWorker processor, and are attributed to the platform's service identity.
  • Scheduled runs that find no content items to process are not recorded, so they do not appear on the Data Pipeline Runs page.
  • If a scheduled run cannot start (for example, because a run with the same canonical run ID is still in progress, or a parameter is invalid), the occurrence is skipped and the scheduler waits for the next one.

Manual triggers

Manual triggers are used by the run actions on the Data Pipelines page:

  • Run Now uses the pipeline's first Manual trigger, or its first trigger when there is no Manual trigger, with the trigger's saved parameter values.
  • Advanced run… lets you choose the trigger (when the pipeline has more than one), the Data Pipeline processor, and override the parameter values for that run only.

See Invoking Data Pipelines for details.

Note

Pipelines that the platform invokes automatically for uploaded files (for example, context file and agent private storage pipelines) are started with a trigger named Default Manual Trigger. Keep that trigger name, and the stage names the platform expects, when you edit those pipelines.

Multiple triggers

A pipeline can have several triggers. Use this to combine a schedule with an on-demand trigger, or to run the same pipeline against different parameter sets (for example, one Schedule trigger per folder). Each trigger has its own parameter values and canonical identification settings.