Targets
Source monitor
Run capture without a target to see source activity and spool state, then add a target when you are ready.
A source monitor is a pipeline that captures changes from the source but writes them nowhere. Use it to prove that capture works against a source, to watch how much change a database produces before you choose a target, or to stage a pipeline whose target isn't ready yet.
How it works
The agent runs the same capture it runs for any pipeline and writes the changes to its spool on disk, but no target reads them. A source monitor isn't a connection: you choose it on the Source step, and it has no settings.
Because capture runs in full, the source needs the same setup and carries the same costs as with a target. For example, a PostgreSQL source still gets its replication slot.
A source monitor runs no initial load. Its spool doesn't grow while no target reads it: it stays at most about 256 MiB, shown as Spool depth. Changes captured before you add a target aren't kept for that target.
Requirements
| Requirement | Details |
|---|---|
| Sources | Every source engine |
| Capture modes | Every mode except DirectLog. A DirectLog pipeline must have a target, so the wizard doesn't offer monitoring for it. |
| Tables | At least one. |
| Agent | An agent that can capture the source, as for any pipeline |
Create a source monitor
- Start a new pipeline and choose the source on the Source step.
- Under the capture mode, find Purpose: replicate to a target. and click monitor the source only. The line changes to Purpose: monitor the source only. To go back, click Replicate to a target instead.
- On the Target step, the wizard shows No target: this pipeline monitors the source only in place of the list of targets. Click Choose a target instead if you change your mind.
- On the Tables step, add the tables to monitor.
- The Prepare step has nothing to prepare, and the Review step says This will create a source monitor with no target. Add a target later when you are ready to write captured changes. Create the pipeline.
With the sqlflow CLI, create a source monitor with --source-only in place of --target. Replace <name>, <source_connection> and the table list.
sqlflow pipeline create <name> --source <source_connection> --source-only --tables <schema.table>,<schema.table>What it shows
On the pipeline's page:
- A notice: Source-only monitoring is active. SQL Flow can inspect source activity and load/spool state, but no target is receiving data., with an Add target button.
- A No target chip next to the pipeline's name.
- A Source monitor card, in place of Replication lag, with these facts:
| Fact | Meaning |
|---|---|
| Source head offset | How many records the agent has captured into the spool |
| Spool reader | How far a target is behind the head, or "no target is reading the spool yet" |
| Spool depth | The spool's size on disk |
| Active readers | How many targets read the spool |
| Configured tables | How many tables the pipeline captures |
Until the agent reports spool activity, the card shows Capture state and Configured tables only. The lag figure at the top of the page becomes Spool backlog, the records captured but not yet read by a target.
Add a target
Select Add target on the Targets card of the pipeline's Overview tab, and choose a target as you would in the wizard. You can add a target while the pipeline runs. The pipeline stays a source monitor with targets added to it; its list entry shows Source monitor + 1 target.
The new target gets its own initial load: SQL Flow copies every selected table to it, then applies the captured changes.
Limitations
- Not available with DirectLog.
- The pipeline's primary target stays Source monitor. You add targets to it, but you can't edit it into another target.
- Bidirectional replication isn't available for a source monitor.
