Pipelines

Lifecycle

Start, pause, reload, archive, restore and delete a pipeline, and learn what each does to the source and the target.

A pipeline is always in one lifecycle state, and you move it with a small set of commands. The pipeline's agent applies a command within a few seconds and reports the new status. If the agent doesn't answer, the button says Still waiting for the agent to start this pipeline.

Start, pause, reload, archive, restore and delete are in the UI. Stop exists only in the CLI.

States and commands

Command	Where	Allowed when	Result
Start	Header, list row, Start selected, `sqlflow pipeline start`	The pipeline is paused, not archived, and its agent is online	Loading, then Healthy or Failing
Pause	Header, Pause selected, `sqlflow pipeline pause`	The pipeline is running	Paused
Stop	`sqlflow pipeline stop` only	The pipeline is running	Paused, with the agent's saved state erased
Reload from source	Actions menu, `sqlflow pipeline reload`	The agent is online	Loading
Archive	Actions menu, Archive selected, `sqlflow pipeline archive`	The pipeline isn't replicating on a live agent	Moves to the Archived list
Restore	Actions menu on an archived pipeline, `sqlflow pipeline restore`	The pipeline is archived	Paused
Delete	Actions menu on an archived pipeline, `sqlflow pipeline delete`	The pipeline is archived	Removed

Degraded, Failing and Restarting count as running, so you can pause them.

Start

Start brings a paused pipeline up on its assigned agent. A pipeline that runs on two agents needs both online.

First start, and the first start after a stop: the agent runs the initial load, then streams changes. Target tables that already hold rows are emptied and loaded again.
Start after a pause: the agent resumes from its saved position and loads nothing.
Start after an interrupted load or an edited table set: only the tables that still need it load. See When the load runs.

A refused start shows its reason beside the Start button. The common reasons are:

Message	Cause and fix
`Cannot start a pipeline that is <state>; it is already running.`	Only a paused pipeline starts.
`This pipeline is not assigned to an agent. Assign one first.`	Choose an agent with Actions > Assign agent….
`The assigned agent '<id>' is not online.`	Bring the agent online, or assign another.
`This pipeline is archived. Restore the pipeline first (it comes back paused), then start it.`	Restore it.
`Table <table> in <target> is also a target table of pipeline '<name>', which is <state>. … pause or archive pipeline '<name>' first, then start this one.`	Two active pipelines can't write the same target table. Pause or archive the other pipeline.
`Pipeline '<name>' was started on table <table> after this pipeline last wrote it, so the rows there are not this pipeline's. Start and Resume are refused: Reload this pipeline …`	Another pipeline wrote to the table while this one was paused. Reload this pipeline.
`Another DirectLog pipeline of this source database is captured by agent <names>. …`	All DirectLog pipelines of one SQL Server database must run on the same agent and standby. Assign this pipeline to them. To move the database, unassign its DirectLog pipelines, then assign them to the new agent.

Pause

Pause stops the pipeline on every agent that runs it. The agent keeps its saved position and its spool, so a later Start resumes without loading. Nothing is written to the target while the pipeline is paused, and rows already written stay.

What a pause leaves on the source depends on the engine:

Source and mode	While the pipeline is paused
SQL Server DirectLog	SQL Flow releases its hold on the log, so the log can be truncated at the next log backup or checkpoint. The tables stay marked. A start continues from the online log or your log backups.
Sybase ASE DirectLog and Reconstruction	SQL Flow releases its hold on the log, so `dump tran` or `trunc log on chkpt` can truncate it. DirectLog's table marking stays.
PostgreSQL	The replication slot stays and keeps WAL from the last confirmed position, up to the server's `max_slot_wal_keep_size`.
SQL Server and Azure SQL Native CDC	SQL Server's capture and cleanup jobs keep running on their own schedule.
Other sources and modes	SQL Flow holds nothing on the source.

There is no time limit on a pause. A paused pipeline can resume as long as the source still keeps the log from its saved position. If it doesn't, the start stops with a named message, loses no change silently, and needs a reload. For what each source must keep, see Source log retention.

Changes keep accumulating on the source while a pipeline is paused. A long pause can outlast what the source retains, and then the pipeline needs a reload.

Some settings change only while the pipeline is paused: the source and target connections, the table set, names and the atomic transaction limits.

A pause needs the pipeline's agents online, because an offline agent may still be running. If an agent is offline, the pause is refused with Agent '<id>' is not online, so it cannot acknowledge the stop. …. Bring the agent back. If it is lost for good, select Revoke on its card on the Agents page: a revoked agent counts as stopped, so its pipelines move to its standby if it has one, and otherwise pause. You can then archive the pipeline or assign it to another agent. See Agents and brokers.

Stop

Stop exists only in the CLI. It pauses the pipeline and also erases its saved state on the agents: the source position, the spool and the initial-load records. On a PostgreSQL source it also drops the replication slot and the publication, which releases the WAL the slot kept.

The next Start is then a first start: it loads the tables again and empties any target table that holds rows. stop asks for no confirmation, so use Pause unless you want that.

sqlflow pipeline stop <pipeline_name>

Reload

Reload from source re-seeds the pipeline. It empties each replicated table in the target, resets the source position, and copies the tables again from a fresh snapshot. The target is incomplete until the load finishes.

Open the pipeline and select Actions > Reload from source. The item is disabled unless the agent is online.
Read the Reload from source dialog. It states how many tables are emptied.
Select Reload.

The page shows Preparing to load, then Loading with a progress bar. See Initial load. On PostgreSQL, the reload also drops and recreates the replication slot and publication.

Reload is refused for Kafka, Google Pub/Sub and Fabric Open Mirroring targets, because a new snapshot can't remove the rows or events of the previous one. The error starts Pipeline <id>: Reload is not supported for source-keyed append/file target(s) <engine>. To re-seed such a pipeline, start from an empty target.

From the CLI, run sqlflow pipeline reload <pipeline_name> --yes. To re-seed a single table while the others keep streaming, use Resync on the Tables tab. See Reload one table.

Archive, restore and delete

Archive takes a pipeline out of service without destroying it.

Pause the pipeline. A pipeline replicating on an online agent can't be archived.
Select Actions > Archive, and confirm Archive pipeline?.

The archive is refused until every agent of the pipeline has confirmed it stopped. An archived pipeline appears on the Archived tab, doesn't run, and no longer blocks other pipelines from using its target tables. Nothing changes in the target.

Restore brings an archived pipeline back, Paused. Select Actions > Restore on its page, assign an agent if needed, and select Start.

Delete removes an archived pipeline for good.

Open the archived pipeline and select Actions > Delete.
Read Delete pipeline: the pipeline and its history are removed, and every agent that ran it erases its saved state and releases the capture objects it created on the source. It can't be undone.
Select Delete.

Rows already written to the target stay in place. Replication can't resume afterward, only be rebuilt with a fresh initial load. An agent that is offline erases its state when it reconnects. If the delete says the pipeline still holds source objects its agent is releasing, retry once they are released.

Force options

The UI has no force options. The CLI has these:

Command	Flag	Effect
`sqlflow pipeline pause <name>`, `sqlflow pipeline stop <name>`	`--force`	Changes the status without the agent, for a pipeline whose agent is unavailable. See Forcing a pause or stop.
`sqlflow pipeline delete <name>`	`--force`	Archives the pipeline and deletes it in one step. It never deletes a pipeline that is replicating on a live agent.
`sqlflow pipeline reload <name>`	`--yes`	Confirms the reload. Without it the command refuses.

sqlflow pipeline start has no force option. A start needs a live agent.

Forcing a pause or stop

Without --force, pause and stop go through the agent, and the pipeline shows Paused once every agent confirms. If the agent is offline, stale or no longer registered, the CLI stops and tells you to pass --force.

Agent online: run the command without --force.
Agent unavailable for now: use --force. A returning agent takes the new status and doesn't start the pipeline again by itself. For stop, the saved state on the agent is erased when the agent returns.
Agent lost for good: revoke it, from the Agents page or with sqlflow agent revoke. The pipeline then moves to the agent's standby or pauses, and can be archived or moved.

If this was useful,
our newsletter will be too.

Get monthly insights on data engineering, AI, and building critical infrastructure - direct from the Spectral Core team and CEO Damir Bulic.