Pipelines
Changing tables
Add or remove the tables a pipeline replicates, understand what the next start reloads, and resync a single table without stopping the rest.
You choose a pipeline's tables when you create it. You can add or remove tables later, but only while the pipeline is paused. The next start loads only the tables you added. To bring a single table back in line without pausing anything, use Resync.
Before you begin
- The pipeline is paused and not archived. Select Pause on its page and wait for the status to read Paused.
- New tables have a primary key or a unique index over non-null columns, as when you select tables in the wizard.
Add or remove tables
- Open the pipeline and select the Tables tab.
- Select Edit tables.
- Edit the list. It holds one table per line, schema-qualified as the source names them. Add a line to add a table. Delete a line to remove one.
- Select Apply.
- If you removed any table, confirm Stop replicating tables? with Apply changes.
The page confirms Tables updated. and checks the target for the new tables.
Check the target
SQL Flow checks the pipeline's first target that it can create tables on:
- Ready: the page says the target schema is ready.
- Tables missing: an amber box lists the tables that need creating. Select Show SQL to read exactly what will run, then Create tables & keys.
- Check failed: a red box says Tables updated, but the target schema could not be checked with the reason. The new list is saved. Fix the cause, then select Check again.
For a pipeline with several targets, create the tables in the other targets yourself. See Multiple targets.
Resume
Select Start.
What the next start does
After you save the list, the page tells you what the next start will load. The rule is the same for every source, capture mode and target:
| You | The next start |
|---|---|
| Add a table | Loads that table, and only that table. The other tables keep their rows and keep streaming from where they were. |
| Remove a table | Loads nothing and empties nothing. |
| Remove a table, then add it back | Loads it again in full, as a new table. |
A table you add is loaded like a table in a new pipeline. If its target table already holds rows that SQL Flow can't prove are its own, the start stops and doesn't empty them. See Rows already in the target.
Changing the source host, port or database is different: the next start loads every table again. See Initial load.
What happens to removed tables
A removed table is no longer captured, and SQL Flow stops writing to it. SQL Flow never drops, empties or alters the target table. The rows already there stay, so drop or empty them yourself if you no longer need them. Changes made to the source table while it is out of the set are not captured, which is why adding it back loads it in full.
When the change is refused
| Message | Cause and fix |
|---|---|
Pause the pipeline before editing its tables. | Pause it first. |
Restore the pipeline before editing its tables. | The pipeline is archived. Restore it. |
Dispatch Pause and wait for every stop acknowledgement before removing or moving target tables. | A pause hasn't reached every agent yet. Wait for Paused, then retry. |
Table <table> in <target> would be written by <n> targets of this pipeline (<ids>). … | Two targets of one pipeline would write the same table. Point one of them at another database, or remove one. |
Table <table> in <target> is also a target table of pipeline '<name>', which is <state>. … | Another active pipeline writes that target table. Pause or archive it before starting this pipeline. |
Use the CLI
sqlflow pipeline set-tables <pipeline_name> --tables <schema.table>,<schema.table>
sqlflow pipeline provision <pipeline_name>
sqlflow pipeline start <pipeline_name>set-tables replaces the whole list and, like the UI, needs the pipeline paused. provision creates the missing tables and keys on the target. See Pipelines commands.
Set names and rules for new tables
After you add tables, open Configure on the target in the Targets card on the Overview tab to give them names, filters and output columns. See Configure names and rules.
Reload one table
Resync brings one target table back in line with the source while every other table keeps streaming. Use it when one target table needs a clean copy. See Repairing drift.
- Make sure the pipeline is running (Healthy, Degraded or Failing) and its agent is online. Otherwise the button is disabled.
- On the Tables tab, select Resync on the table's row.
- Read the dialog: existing target rows can be updated or removed, and the table may be incomplete until the resync completes.
- Select Resync table.
The row shows resync running with progress, and the Table repairs list under the table keeps the history. A failure appears in a red Table repair: alert, for example Resync of <table> failed: <reason>. The pipeline's status doesn't change while a table resyncs.
On SQL Server Native CDC, Resync and Repair keys… run sys.sp_repltrans on the source, so the login needs permission to execute it (db_owner has it). Without it, the operation is refused and the message asks a DBA to authorize it.
From the CLI, sqlflow pipeline resync <pipeline_name> --table <schema.table> --yes starts the same operation, and --wait follows it to the end. To fix a few rows instead of a whole table, use Repair keys…. See Repairing drift.
