Reference
FAQ
Short answers to the questions DBAs and architects ask most about SQL Flow's source impact, security, reliability and features, with links to the details.
Short answers to the questions DBAs and architects ask most often. Each answer links to the page with the details.
Your source database
Does SQL Flow write to my source database?
SQL Flow never writes to your source tables, and never changes a server option or a database setting: the setup pages give your DBA the statements to run. What it does on the source depends on the capture mode:
- SQL Server DirectLog: marks the selected tables for capture with
sys.sp_cdc_enable_table, when you select Mark now or your DBA runs the generated script, and drops the two CDC jobs that marking creates if the database had none. It advances the replication truncation point while it runs. Deleting the pipeline removes the marking it created. - SQL Server Reconstruction: nothing.
- Oracle DirectLog: a table resync, a key repair or a pipeline with processing rules can force a log switch with
ALTER SYSTEM ARCHIVE LOG CURRENT. - PostgreSQL: creates a replication slot and a publication, which a delete or
sqlflow pipeline stopdrops. - Sybase ASE DirectLog: marks the selected tables for replication with
sp_setreptable, when you select Mark now or your DBA runs the generated script, and sets the secondary truncation point while the pipeline runs. Deleting the pipeline removes the marking it set. - Sybase ASE Reconstruction: sets the secondary truncation point while the pipeline runs, and can run
checkpointin the database. - SQL Server Polling and Native CDC, Azure SQL, Oracle LogMiner and MySQL: nothing.
On SQL Server 2016 SP2 and 2017, Reconstruction, and DirectLog for tables with LOB columns or long rows, also need a signed helper procedure that a sysadmin installs once. It reads page headers only, never row data. See each mode's page under Sources.
What does SQL Flow install, and where?
An agent, a service on a Windows or Linux machine you choose, runs the pipelines. A log broker, a small read-only service, runs on the database host only for SQL Server DirectLog and Reconstruction, Oracle DirectLog and both Sybase ASE modes. SQL Flow adds no triggers or extensions to the database. The control plane is hosted for you, or runs in a container on your servers. Download the agent, the log broker and the sqlflow CLI from Downloads. See How SQL Flow works.
Why does SQL Flow use a log broker?
Security. The broker is a small, single-purpose executable that serves only the log files you allow-list, only to the agent addresses you allow in Agent addresses, and only read-only. It never connects to a database, holds no database credential, and records every file it opens in its audit log. Spectral Core provides its complete source code for your security audit on request. The agent needs no access to the database host. See Architecture: Log brokers.
How much load does SQL Flow put on the source?
It depends on the capture mode. DirectLog runs no query to find changes: the broker reads log files and the agent decodes them on its own machine. Polling queries the log on every read, Native CDC adds SQL Server's capture job and change tables, and LogMiner mines inside Oracle. The initial load and compares read whole tables. The Source queries/s card on each pipeline shows how many commands it sends to the source. See Performance considerations.
Does SQL Flow hold my transaction log?
Only in three cases:
- SQL Server DirectLog holds the log until the target commits. A pause releases the hold, and SQL Flow releases it before the log reaches 85 % of its capacity. When a pipeline holds the log but isn't capturing, SQL Flow warns 24, 6 and 1 hours before that point.
- A PostgreSQL replication slot keeps WAL, also while the pipeline is paused.
- Sybase ASE, in both capture modes, holds the log while the pipeline runs, and releases it when you pause.
Every other mode depends on your own log, archive or binlog retention. See Source log retention.
Which versions are supported?
SQL Server 2016 SP2 and later in every capture mode, Azure SQL Database including Hyperscale, PostgreSQL 14 and later, MySQL 8.0 and later including 8.4 and 9.x, MariaDB, and SAP ASE 16.0 SP02 PL02 (EBF 25320) for Linux x86-64 with a 16 KB page size. Oracle is tested on 12.2, 19.3, 21.3 and 23.26 (26ai), and other releases are expected to work. See Support matrix and known limitations.
Can I capture from a managed database service?
Yes for Azure SQL Database and Hyperscale, with Native CDC. Add an Azure SQL Managed Instance as a SQL Server source, with Native CDC or Polling. For SQL Server on a managed service that allows no log broker, such as Amazon RDS, use Native CDC. SQL Server on a virtual machine is an ordinary SQL Server source. See Choosing a capture mode.
A managed PostgreSQL or MySQL server is a source when it passes the same checks as any other server. For PostgreSQL, the login needs replication rights, for example through rds_replication on Amazon RDS and Aurora. See PostgreSQL: Managed services and MySQL: Managed services.
Security
Does my data pass through the control plane?
No. Row data moves only from the source, or its log broker, to the agent, and from the agent to the target, through an apply agent when you use relay. The control plane holds configuration and operational metadata, such as status, logs, alerts and the redacted diagnostics you request. Even a raw log export for support stays on the agent's own disk. See How SQL Flow works.
Is data encrypted in transit and at rest?
Yes. Agents and brokers reach the control plane over HTTPS and trust only its own certificate authority. Agents read from log brokers over TLS with the broker's certificate pinned, and relay agents use mutual TLS 1.3. Database connections use TLS as each connection's Encryption setting decides: Require, the default, validates the server certificate, and Prefer relaxes that for a server you trust. A Kafka target is unencrypted unless you choose TLS or SASL over TLS. Every agent's spool is encrypted with AES-256-GCM, and this can't be turned off. See Encrypted data support.
Where are database passwords stored?
Either the control plane stores the password encrypted and delivers it only to the agent that runs the pipeline, or you give a secret reference and the agent reads the password on its own host, so it never reaches the control plane. Sign-ins without a password, such as a Microsoft Entra managed identity or Windows authentication, use the agent host's identity. See Database credentials and secret references.
What network access does SQL Flow need?
Agents and log brokers open outbound HTTPS connections to the control plane, and nothing connects in from the internet. Agents connect to your databases, to log brokers on TCP 7003, and, for relay, to the apply agent on TCP 7420. For an environment with no internet access, run the control plane yourself in an air-gapped deployment. See Network and ports.
Does SQL Flow support TDE and Always Encrypted?
On SQL Server, Polling and Native CDC work on TDE databases; DirectLog and Reconstruction refuse them. Azure SQL supports TDE. Always Encrypted source columns are refused in every mode, except that DirectLog can copy the ciphertext unchanged into SQL Server or Azure SQL targets with the same keys. Targets can encrypt plaintext into Always Encrypted columns. On Oracle, DirectLog refuses TDE-encrypted tables by name, so capture those with LogMiner and keep the TDE keystore open. See Encrypted data support.
Reliability
What happens if an agent fails?
Its service manager restarts it, and every pipeline continues from the position its target last committed, without a reload. With a standby agent, the standby takes the pipelines over about 30 seconds after the failed agent's last heartbeat, or at once when you revoke the failed agent, and the old agent can't write again. Without a standby, revoke an agent that is lost for good on the Agents page or with sqlflow agent revoke; its pipelines pause so you can move them. See High availability: Agents.
What happens if the control plane is unavailable?
Pipelines keep running the work they already have for up to 24 hours, and agents reconnect on their own. You can't create, start, pause or edit pipelines until it is back. An agent with a standby partner is the exception: it stops its pipelines about 25 seconds after it loses the control plane, and they resume when the control plane returns. See High availability: Control plane.
What happens if the target or the network goes down?
Capture continues, changes wait in the agent's encrypted spool on disk, and lag grows. The agent keeps retrying and applies the backlog once the target accepts writes. Size the agent's disk for the longest outage you want to ride out. See Automatic restart and Lag, spool and log retention.
Can a change be lost or applied twice?
Capture moves its position only after a change is stored durably, and each target records the position it committed. After any restart, the pipeline resumes from that position, and a database target ends in the same state whether or not a change was replayed. Kafka and Google Pub/Sub deliver at least once, so consumers should upsert by key or drop repeats. See High availability overview.
How long can a pipeline stay paused?
As long as the source keeps the log the pipeline needs. SQL Server DirectLog, Reconstruction and Polling can resume from your log backups once the online log is gone. PostgreSQL keeps WAL up to max_slot_wal_keep_size, MySQL depends on binlog retention, Native CDC on the cleanup job's retention, Oracle on archived logs, and Sybase ASE on no log dump passing the saved position. When the log is gone, the start stops by name and the pipeline needs a reload. See Lifecycle: Pause.
Data and features
What happens to rows already in the target?
The first start empties every target table that holds rows and loads it again from the source. The Review step lists those tables, and Create and start asks you to confirm. A start after a pause loads nothing. On a SQL Server, Azure SQL or SQL database in Fabric target, a table that holds rows SQL Flow has no load record for is never emptied, which is always the case for a PostgreSQL, MySQL, Oracle or Sybase ASE source: the start is refused with target_initial_load_ownership_unproven until you empty it yourself. See Initial load.
Does SQL Flow create the target tables?
Only when you ask. On the wizard's Prepare step, Show SQL shows exactly what will run, and Prepare target schema creates the tables and keys. On an existing pipeline, Add target and Edit tables offer Create tables & keys for missing tables. SQL Flow never creates tables during a load, and never adds a missing column to an existing table. See Prepare.
Which column types does SQL Flow replicate?
The common built-in types of every source engine. When SQL Flow can't carry a type exactly, it refuses the table by name at the start, or stops the pipeline with a message that names the column, rather than truncating or rounding the value. Some types depend on the capture mode: for example, SQL Server sql_variant reaches SQL Server and Azure SQL targets only from DirectLog or Reconstruction, Oracle LogMiner refuses native BOOLEAN, JSON and VECTOR columns, and Sybase ASE DirectLog refuses tables with text, unitext or image columns. See Data type support by engine and Data type mapping.
Do my tables need a primary key?
SQL Flow applies changes by key. SQL Server DirectLog, Reconstruction and Polling accept a unique index over non-null columns in place of a primary key. Native CDC, Oracle, PostgreSQL logical replication, Sybase ASE, and the Kafka, Google Pub/Sub and Fabric Open Mirroring targets need a primary key, or for PostgreSQL a replica identity index. See Tables and configure.
Can I filter rows, rename tables or mask columns?
Yes, per target, in the Configure dialog. You can keep rows that match a condition, choose and compute output columns, mask values, and rename tables and columns. Saving changed rules reloads the pipeline, and names change only while it is paused. See In-flight processing and Table and column names.
How do I check that the target matches the source?
Run a compare on the pipeline's Compare tab; a new pipeline also compares daily. A compare reads every row while the pipeline streams and lists the keys that differ. Repair them with Repair keys… or Resync, or let an automatic repair policy do it. See Compare and Repairing drift.
Does SQL Flow replicate schema changes?
ADD COLUMN and DROP COLUMN reach every target at the same point in the stream as on the source. A change the target can't apply exactly holds that table for your review while the other tables keep streaming, and a change SQL Flow can't carry, such as a column type change or, on SQL Server, an added column of typed xml, stops the pipeline before it. See Schema changes.
Does SQL Flow replicate TRUNCATE TABLE?
Yes. Every source captures a TRUNCATE TABLE, and database targets empty the table in order with the changes around it. SQL Server itself refuses TRUNCATE TABLE on a table marked for DirectLog or enabled for Native CDC. Kafka, Google Pub/Sub and Fabric Open Mirroring stop the pipeline at a captured truncate of a keyed table. See Supported changes: TRUNCATE TABLE.
Can one pipeline write to several targets?
Yes. Each target has its own connection, names, rules, lag and committed position, and a slow or paused target doesn't hold back the others. See Multiple targets.
Operations
Can I automate SQL Flow?
Yes. The sqlflow CLI manages agents, connections and pipelines. See CLI overview and authentication.
How do I send SQL Flow support what they need?
Open the pipeline, select Actions > Download diagnostics, and send the file. It holds logs, status and configuration, with no row data, passwords or keys. If support asks for more, an Account admin can select Save raw log bytes on agent (contains row data)…: the file stays on the agent for up to 24 hours, and Spectral Core support gives you secure transfer instructions. See Diagnostics for support.
How is SQL Flow licensed?
Per deployment, as an annual subscription. A capacity plan (Core, Production, Scale or Enterprise) sets the CDC and initial load ceilings. Capability packs add distributed acceleration, high availability, bidirectional replication or an air-gapped deployment, and a support plan sets response times. There's no charge per CPU, pipeline or agent, and no limit on pipelines, sources, targets or agents. See Plans and licensing.
