Overview
How SQL Flow works
Learn what the control plane, agents and log brokers do, how a change travels from source to target, and where each component runs.
SQL Flow has three components: a control plane that holds your configuration, agents that move the data, and log brokers that serve transaction-log files on database hosts. This page explains what each one does, how a change travels from the source to the target, and how the components connect.
Components
| Component | What it does | Where it runs |
|---|---|---|
Control plane (SqlFlow.App) | Serves the web UI and the endpoint the sqlflow CLI uses. Stores pipelines, connections, accounts and the registered agents and brokers in PostgreSQL. Sends commands to agents and brokers. It never reads or writes row data, and never connects to a database. | Hosted on the web in a hybrid deployment, or in a container on your own servers in an air-gapped deployment. |
Agent (SqlFlow.Agent) | Runs pipelines: the initial load, change capture, the change spool on local disk, and apply to each target. Every database connection starts at an agent, and connection tests in the wizard run on an agent. | A 64-bit Windows or Linux machine that reaches your sources and targets, as a service or in the foreground. See Agents. |
Log broker (SqlFlow.LogBroker) | Serves transaction-log files from the folders you allow, read-only, to the agents you allow, over TLS. It exists for security: it keeps all access to the database host in one small executable whose source code you can audit. Needed for SQL Server DirectLog, SQL Server Reconstruction, Oracle DirectLog, and both Sybase ASE modes (DirectLog and Reconstruction), even when the agent runs on the same host. | The database host, on Windows Server 2016 or later or on Linux. See Log brokers. |
A pipeline usually runs on one agent. With distributed acceleration, a capture agent near the source and an apply agent near the target share the work.
How the components connect
Arrows point from the side that opens the connection.
Row data travels only between the source, the log broker, the agent and the target: from the source, or the log broker, to the agent, and from the agent to the target. It never passes through the control plane.
The data path
Each change goes through four stages on the agent.
- Capture. The agent reads changes from the source in the pipeline's capture mode. Depending on the mode, it reads log bytes that a log broker serves, log records that the database returns over an ordinary or replication connection, or the change tables that the database's own CDC fills.
- Decode. The agent turns log records into row changes: inserts, updates and deletes with their column values. It keeps only committed changes, in commit order.
- Spool. The agent writes the changes to its spool, a durable, encrypted store on local disk. If a target is slow or unreachable, the spool grows and capture continues.
- Apply. The agent reads the spool and writes each target. Database targets get the changes in source transactions, with the pipeline's committed position recorded on the target, so a restart resumes from what the target holds. Event targets, such as Kafka, get each change as a message.
Before streaming starts, the initial load copies the existing rows of every selected table to each target. The pipeline then streams the changes captured since the load began. See Initial load.
A pipeline can write several targets. Each target reads the same spool at its own pace, with its own committed position. See Multiple targets.
With relay agents, the capture agent ships its changes, compressed and over mutual TLS, to an apply agent near the target, which writes the target. See How relay works.
The control path
Agents and log brokers open every connection to the control plane themselves, over outbound HTTPS. The control plane never connects to them, so nothing on the internet connects into your network in a hybrid deployment.
- Enrollment. An agent or broker registers with a join code from the control plane, then waits until a user approves it. See Agents.
- Configuration and commands. An approved agent pulls its configuration and receives commands, such as start, pause or reload, within one network round trip.
- Status. Every agent reports pipeline status, metrics, logs, alerts and schema events every few seconds.
If the control plane is unreachable, agents and brokers keep running the work they already accepted for up to 24 hours, and you can't change pipelines until it's back. See High availability: Control plane.
What each component keeps
| Component | Keeps |
|---|---|
| Control plane | Pipeline definitions, connection details, accounts, registrations, logs, alerts and metrics, in its PostgreSQL database. Database passwords only when you enter them, encrypted. |
| Agent | Its identity, its encryption keys and the spool, in its data directory. |
| Log broker | Its identity, TLS certificate and allowed folders, in its data directory. |
| Target | The replicated tables, and SQL Flow's bookkeeping for the pipeline's committed position. Each target page lists what SQL Flow creates there. |
Keep each agent's and broker's data directory across restarts and upgrades. See Upgrades and restarts.
