Overview
Core concepts
Learn the terms SQL Flow uses for its components, pipelines and the way data moves, with links to the details.
These are the ideas the rest of the docs build on. Each definition links to the page that covers it in full. For an alphabetical list of every term, see the Glossary.
Components
Control plane. The web application that holds your configuration and sends commands to agents and log brokers. You use it through the web UI or the sqlflow CLI. It never connects to a database, and row data never reaches it. See How SQL Flow works.
Agent. A service on one of your machines that runs pipelines: it connects to the source and the targets, captures changes, keeps them in its spool and applies them. See Agents.
Log broker. A small, auditable service on the database host that serves allow-listed transaction-log files, read-only, to the agents you allow, over TLS. It exists for security: the agent never gets access to the database host. DirectLog and Reconstruction need one. See Log brokers.
Join code and approval. A new agent or broker registers with a join code from Add agent or Add broker, valid for 15 minutes. It runs nothing until a user approves it on the Agents page. Revoke refuses its credential from then on; a revoked agent counts as stopped.
Organizing your work
Account. Agents, brokers, connections and pipelines belong to one account. A user can belong to several accounts. Deleting an account revokes its agents and log brokers and stops its pipelines. See Accounts and team.
Project. A named group of related pipelines. See Projects.
Connection. The details SQL Flow needs to reach one database or service: host, port, database, login and options. You can save a connection and reuse it in several pipelines. Its password is either stored encrypted by SQL Flow, or kept on the agent host as a secret reference. See Saved connections.
Encryption. A database connection's choice of transport: Require (default) encrypts the connection and validates the server certificate, and Prefer relaxes the check for a server you trust. Azure SQL and SQL database in Fabric connections always validate the certificate and have no such choice. See Encrypted data support.
Pipelines
Pipeline. One source, the tables selected from it, and one or more targets, run by an agent. You create it with the pipeline wizard. See Lifecycle.
Source. The database SQL Flow captures changes from. See Supported sources and targets.
Capture mode. How SQL Flow reads changes from a source, for example DirectLog, LogMiner or Native CDC. The mode decides the lag, the load on the source, and what you set up on the database. See Choosing a capture mode.
Target. Where a pipeline writes: a database, an event stream such as Kafka, or Fabric Open Mirroring. A pipeline's page lists its targets on the Targets card, and a pipeline can have several. See Multiple targets.
Source monitor. A pipeline that captures changes without a target, to watch a source's change volume or to stage a pipeline. See Source monitor.
How data moves
Initial load. The copy of the existing rows of each selected table, before streaming starts. See Initial load.
Streaming. The continuous capture and apply of changes after the initial load.
Spool. The agent's durable, encrypted store of captured changes on local disk. It lets capture continue while a target is slow or unreachable, up to a limit set by the free space on the agent's disk. See Lag, spool and log retention.
Committed position. The point in the source's change stream that a target has committed. SQL Flow records it on the target, and every restart resumes from it. See High availability: Agents.
Lag. How far a target is behind the source. Each pipeline is judged against a lag goal, which colors the figure. See Lag, spool and log retention.
Reload. Reload from source empties the pipeline's target tables and runs the initial load again, from a fresh source position. See Lifecycle.
Shaping the data
Mapping. The target name and column type of each source table and column. SQL Flow chooses them automatically, and you override names per target. See Mapping.
In-flight processing. Rules per target that filter rows, choose and compute columns, and mask values on the way. See In-flight processing.
Configure. The dialog where you set a target's names and rules, from the wizard's Tables step or a pipeline's Targets card. See Tables and configure.
Running in production
Relay agents. A capture agent near the source and an apply agent near the target, linked by a compressed, encrypted relay, for any source and capture mode. Also called distributed acceleration. See Distributed acceleration.
Standby. A second agent, or a second log broker, that takes over when the first one fails, or at once when you revoke a failed agent. See High availability.
Automatic restart. The agent restarts a faulted pipeline with growing delays, and parks it for an operator if the fault keeps coming back. See Automatic restart.
Compare and repair. A compare finds rows that differ between a target and its source. A repair rewrites those rows, or brings one whole table back in line with a resync, while the pipeline keeps streaming. See Compare and Repairing drift.
Diagnostics for support. A file you collect from a pipeline's Actions menu and send to SQL Flow support. Download diagnostics holds logs, status and configuration without row data. Save raw log bytes on agent holds row data, so it stays on the agent's own disk. See Diagnostics for support.
