Pipelines commands

Create, provision, assign, start, pause, reload, repair, compare, archive and delete pipelines with the sqlflow CLI.

These commands manage the pipelines of the account you are signed in to. They call the same operations as the web app. Set up the CLI first: see Overview and authentication.

Name a pipeline

Commands that take <name> accept a pipeline's name or its ID, ignoring capitalization. If two pipelines match, the command lists their IDs: Pipeline selector '<name>' is ambiguous; use one of these ids: …. restore and delete also look among archived pipelines.

Pipelines are created paused. Create one, make sure its target tables exist, then start it.

Create a pipeline

sqlflow pipeline create <name> --source <connection> --target <connection> --tables <table,table,...>
                        [--agent <name> | --agent-id <id>] [--recreate-target] [--idempotency-key <key>]
                        [--source-password-env <VAR> | --source-password <password> | --source-secret-ref <reference>]
                        [--target-password-env <VAR> | --target-password <password> | --target-secret-ref <reference>]
sqlflow pipeline create <name> --source <connection> --source-only --tables <table,table,...> [--agent <name>]

Give each connection one password source at most. SQL Flow refuses a password together with a reference: Source connection: supply either a secret reference or a password, not both. If a saved connection holds a reference and you give a password, the password wins, and the command says so. If you give neither, a saved reference is used. See Database credentials and secret references.

$secure = Read-Host -AsSecureString -Prompt 'Target password'
$env:WAREHOUSE_PG_PASSWORD = [System.Net.NetworkCredential]::new('', $secure).Password
sqlflow pipeline create orders --source orders-sql --target warehouse-pg --tables dbo.Orders,dbo.OrderLines --agent orders-agent --target-password-env WAREHOUSE_PG_PASSWORD

The command prints a line such as:

created pipeline 'orders' [orders]  sqlserver→postgres, 2 table(s), agent=orders-agent
  target verified on postgres: all selected table(s) in place.

When the pipeline has an agent and the target engine is one SQL Flow can create tables on (PostgreSQL, SQL Server, Azure SQL, SQL database in Fabric, MySQL, Oracle or Sybase ASE), the command then checks the target tables, for up to 2 minutes:

• If tables are missing or don't match, it deletes the pipeline it created and fails with 'orders': the postgres target is not ready: 2 table(s) missing/incompatible (dbo.Orders, dbo.OrderLines). Re-run with --recreate-target to create them (pipeline was not created).
• With --recreate-target, it creates the missing tables and keys instead, for up to 4 minutes. Without an agent or with another target engine, --recreate-target fails and removes the new pipeline.

If the control plane refuses the creation, the error shows only the status and code, because the request holds a password. The pipeline wizard in the web app shows the full reason.

With no agent, nothing is checked. Run pipeline provision once an agent is assigned.

create makes a pipeline with one source, one target and one agent. Relay agents, additional targets, in-flight processing rules and name mapping are set in the web app. See Pipeline wizard.

Check or create the target tables

sqlflow pipeline provision <name> [--recreate] [--idempotency-key <key>] [--target <connection>]
                                  [--target-password-env <VAR> | --target-password <password> | --target-secret-ref <reference>]

Without --recreate, provision checks that every selected table exists on the target with the right columns and key. It prints 'orders': target ready: all selected table(s) in place., or fails and lists up to five tables that are missing or incompatible. With --recreate, it creates what is missing and prints provisioned 'orders' → created missing target tables/keys on postgres.

The agent that runs the pipeline does the work, so the pipeline needs an online agent and selected tables. The check waits up to 2 minutes, and --recreate up to 4. --target uses a different saved connection as the target. If the target engine is one SQL Flow can't create tables on, the command prints '<engine>' has no target provisioner; create target tables for '<name>' manually, then start it. and exits with code 3. Run provision again after set-tables to cover new tables.

Show the mapped target tables

sqlflow pipeline schema-mapping <name> [--tables-only] [--verbose] [--target <connection>]
                                [--target-password-env <VAR> | --target-password <password> | --target-secret-ref <reference>]

Asks the agent which target tables each source table maps to, without touching the target. --tables-only prints the target tables as one comma-separated list, ready to pass as --tables when you chain a second pipeline that reads them. --verbose also prints the full column and key mapping as JSON. See Table and column names.

Assign and unassign an agent

sqlflow pipeline assign <name> --agent <name|id>
sqlflow pipeline unassign <name>

assign chooses the agent that runs the pipeline. If the pipeline is actively replicating on a live agent, both commands refuse: 'orders' is actively replicating on a live agent (healthy). Pause it before reassigning. If its agent is offline or gone, they allow it, which is how you recover a pipeline. If the target agent isn't online, assign prints a note that the pipeline won't run until it comes online. The control plane can refuse for other reasons, such as a standby agent. See High availability: Agents.

Change a paused pipeline

Both commands need the pipeline to be paused. Otherwise they stop with Pause 'orders' before editing its tables (currently healthy). Run `pipeline pause orders` first.

sqlflow pipeline set-tables <name> --tables <table,table,...>

Replaces the replicated tables. The change takes effect when you start the pipeline. Run pipeline provision first to create target tables for new tables. See Changing tables.

sqlflow pipeline set-connection <name> --which source|target
                                (--from <connection> | --host <host> --db <database> --user <user> [--port <port>]
                                                       [--mode <mode>] [--encryption require|prefer] [--opt <key=value> ...])
                                [--password-env <VAR> | --password <password> | --secret-ref <reference>]

Replaces the source or target connection, for example after a password rotation or a moved host. The engine of the pipeline doesn't change. Give the new details inline, or take them from a saved connection with --from. --encryption sets the connection's encryption setting, as for connection add. SQL Flow never returns a stored password, so give the password or reference again even if only the host changes. A source-only pipeline has no target to change. The command prints updated 'orders' target connection → <host>:<port>/<database>. Resume when ready.

sqlflow pipeline pause orders
sqlflow pipeline set-connection orders --which target --from warehouse-pg-2 --password-env WAREHOUSE_PG_PASSWORD
sqlflow pipeline start orders

Start, pause and stop

sqlflow pipeline start <name> [--idempotency-key <key>]
sqlflow pipeline pause <name> [--force] [--idempotency-key <key>]
sqlflow pipeline stop <name> [--force] [--idempotency-key <key>]
sqlflow pipeline start --all
sqlflow pipeline pause --all [--force]
sqlflow pipeline stop --all [--force]

The control plane sends the command to the pipeline's agents, which must be online. The commands print a line such as start → orders [orders] or pause → orders [orders].

• A pipeline on one agent. The CLI checks the agent first. If it is missing or offline, the command stops with Cannot start 'orders': its assigned agent 'orders-agent' is offline, not online. Assign one with `pipeline assign`, or bring the agent online.
• A pipeline on relay agents. Both the capture agent and the apply agent must be online. When either is unavailable, the command is refused, for example with [agent_offline].

start, pause and stop take --idempotency-key, which makes a retry safe and prints the operation record as a line of JSON. See Idempotency keys and CI/CD.

With --all, the command goes through every pipeline in the account that isn't archived, in name order. When it can't act on a pipeline, because the pipeline's agent is unavailable or the control plane refuses the command, for example start on a pipeline that is already running, it prints ! skipped orders: <reason> on standard error and carries on with the next pipeline. If it skipped any pipeline, it exits with code 2.

After stop, the pipeline shows as paused, but its agent no longer holds any position, spool or load progress for it. A later start runs the initial load again from a fresh snapshot, as for a new pipeline. See Initial load.

If a start is refused

The control plane answers with a reason and a code in brackets. The common ones are:

Pause or stop without an agent

When the pipeline's agent is unavailable, pause and stop refuse and suggest --force. With --force, they change the pipeline's status to paused directly in the control plane, without waiting for the agent, and print pause → orders [orders] (forced control-plane transition). An agent that comes back is brought into line with that status and never restarts the pipeline by itself.

If the agent is lost for good, revoke it with sqlflow agent revoke or Revoke on the Agents page. Its pipelines pause, or move to its standby agent if it has one, and you can then assign them to another agent or archive them.

Reload a pipeline

sqlflow pipeline reload <name> --yes

reload refuses to run without --yes. The pipeline's agent must be online. Kafka, Google Pub/Sub and Fabric targets refuse a reload. See Initial load. To reload a single table while the others keep streaming, use resync.

Repair a running pipeline

These commands repair a running pipeline without a full reload. Each one runs on the agent as an operation, which table-ops lists.

sqlflow pipeline resync <name> --table <table> --yes [--wait] [--timeout-seconds <n>]
sqlflow pipeline repair-keys <name> --table <table> (--keys <key,key,...> | --keys-file <path>) [--wait] [--timeout-seconds <n>]
sqlflow pipeline repair-drift <name> <compare-run-id> [--table <table>]
sqlflow pipeline table-ops <name>

resync and repair-keys print resync → orders table dbo.Orders queued as operation <id>. With --wait, the command follows the operation, printing a line each time its state changes, until it reaches completed (exit code 0) or failed, refused or expired (exit code 2). --timeout-seconds limits the wait, and defaults to 3600. If the operation hasn't finished by then, the command exits with 2.

--keys takes keys for a table with a one-column key, such as --keys 1,2,3. Whole numbers are sent as numbers and anything else as text. --keys-file takes the path of a JSON file that holds an array, or one JSON value per line. For a key with several columns, write each key as an object that names every key column:

[
  { "OrderId": 7, "LineNo": 2 },
  { "OrderId": 9, "LineNo": 1 }
]

table-ops prints one row for each operation, with its state and progress such as 3/8 slices, 1200000 rows for a resync, or 5 upserted, 1 absent of 6 keys for a key repair. See Repairing drift.

Compare source and target

sqlflow pipeline compare <name> [--target <id>] [--chunk-rows <n>] [--recheck-rounds <n>]
                         [--recheck-delay-seconds <n>] [--no-wait] [--diff-out <file>]
sqlflow pipeline compare-history <name>
sqlflow pipeline compare-diff <name> <run-id> [--out <file>]
sqlflow pipeline compare-schedule <name> --enabled true|false [--interval-minutes <n>]

--target takes the ID of the pipeline's target to compare when it has several. Here it names a target of the pipeline, not a saved connection. --chunk-rows, --recheck-rounds and --recheck-delay-seconds tune the compare. --diff-out also saves the diff document after the run.

compare waits for the run without a time limit. It prints verdict: clean drift=0 unresolved=0 refusedTables=0 sourceQueries=… sourceRows=…, a table of tables with their counts, and a sample of differing rows. The exit code is 0 for a clean result, 3 for drift, 4 when SQL Flow couldn't confirm the result or refused tables, and 1 if the run failed. A script can branch on it. See Compare and Automatic repair policy.

Archive, restore and delete

sqlflow pipeline archive <name>
sqlflow pipeline restore <name>
sqlflow pipeline delete <name> [--force]

archive and delete refuse a pipeline that is actively replicating on a live agent: Refusing to archive 'orders': it is actively replicating on a live agent (healthy). Pause it first. The control plane also refuses until every agent of the pipeline has confirmed that it stopped. If one is lost for good, revoke it: a revoked agent counts as stopped.

delete needs the pipeline to be archived first: 'orders' is not archived. Archive it first (`pipeline archive orders`), or pass --force to archive and delete in one step. --force does both steps, and still refuses a live pipeline. Deleting also runs the teardown that stop runs on each agent. A delete can answer that an agent is still releasing a source root; run it again once the agent has.

Check status

sqlflow pipeline status [<name>] [--archived]

PIPELINE                   STATUS    AGENT                    OBSERVED      APPLIED  LAG(ms)  ID
orders                     healthy   orders-agent (online)       18204        18204       42  orders
billing                    stale     billing-agent (offline)      9120         9120           billing
    ! assigned agent 'billing-agent' is offline, not online; replication is stalled until it returns.

Lines that start with ! explain why a started pipeline can't make progress, such as no agent or an offline agent, and show the last error an agent reported. With <name> the command lists one pipeline. --archived lists archived pipelines.

Check a processing rules file

These two commands check an in-flight processing rules file on your machine. They don't sign in or contact the control plane, so they need neither the URL nor the credentials.

sqlflow processing validate --file <path> --tables <table,table,...>
sqlflow processing describe --file <path> --tables <table,table,...>

--file is a JSON rules file and --tables lists the source tables the pipeline replicates, from 1 to 10,000. The check covers structure only. It doesn't look at the source schema or the target.

validate prints Processing structure valid; source schema and runtime eligibility have not been checked. describe prints a JSON summary of the rules. A rules file that breaks a rule fails with a code and a path, such as processing_table_untracked at processing.tables[0], and exit code 2. A missing option prints the usage and exits with 1. For the file format, the limits and every code, see Validation and limits.

See also

• Connections commands
• Agents commands
• Operations commands
• Lifecycle