Getting started

Go from an empty account to a first running pipeline, add an agent and a log broker, create the pipeline, and see a change arrive at the target.

This page takes you from an empty SQL Flow account to a pipeline that streams changes from a source database to a target. Each step is short and links to the page with the details. For a complete tutorial with copy-pasteable SQL for one pair of databases, follow one of the walkthroughs.

Before you begin

• A source database that SQL Flow supports, and the right to create a login on it and change its settings. See Supported sources and targets.
• A target: a database, a Kafka cluster, a Google Pub/Sub project or a Fabric open mirrored database.
• A 64-bit Windows or Linux machine for the agent. It must reach the source and the target, and the control plane over outbound HTTPS. See Agents.
• Administrator rights on the database host, if your capture mode reads the log through a log broker.

1. Sign in or install the control plane

• Hybrid: open the sign-in link from your welcome email, titled Your SQL Flow account. On Sign in to SQL Flow, enter the email and one-time password from that email, and choose your own password. Allow outbound HTTPS from your agent and broker hosts to the control plane. See Hybrid deployment.
• Air-gapped: run the control plane from its container image next to a PostgreSQL database, then sign in as the SuperAdmin. See Air-gapped deployment.

To choose between them, see Deployment models.

2. Choose a capture mode and prepare the source

Pick the capture mode with Choosing a capture mode. Then follow the Setup section of the mode's page. It lists the database settings and login privileges that SQL Flow checks before it captures anything.

3. Add an agent

1. Open Agents and select Add agent.
2. Choose Windows or Linux, select Download SqlFlow.Agent.exe or Download SqlFlow.Agent, and copy the file to the agent machine. You can also download it ahead of time from Downloads.
3. Select Copy command and run the command on the agent machine, in PowerShell as Administrator or with sudo. The join code in the command is valid for 15 minutes.
4. Back on Agents, check that the new agent's name matches the machine, and select Approve agent. Its chip reads Online within seconds.

See Agents for the service it installs and its options.

4. Add a log broker, if your capture mode needs one

SQL Server DirectLog, SQL Server Reconstruction, Oracle DirectLog and both Sybase ASE modes read the log through a log broker on the database host, even when the agent runs on that host. Every other capture mode skips this step.

1. On Agents, select Add broker.
2. Choose the database host's platform, download the broker, and move it to a permanent folder on the database host.
3. In Log folders, enter the folders that hold the database's log files. For SQL Server, add the folders your log backups are written to. For Oracle, add the online redo and archived log folders.
4. In Agent addresses, check that your agent's IP address is listed, so only it can connect. The panel fills in your known agents, and the command passes them as --allow-ips.
5. Select Copy command, and run the command on the database host as an administrator.
6. On Agents, under Brokers, compare the broker's Certificate SHA-256 with the fingerprint the install printed, then select Approve.

See Log brokers for service accounts and flags.

5. Create the pipeline

1. Open Pipelines and select New pipeline, or Create first pipeline on a new account.
2. Source: choose the engine and the Capture mode, check that Runs on names your agent, and enter the connection. Keep Encryption at Require (default), which validates the server certificate, unless the server has no certificate the agent can validate. Azure SQL has no such field, because it always validates. See Encryption. Select Test connection & list tables. For a mode that uses a broker, the Log broker panel reads Log broker active on and the host. See Source.
3. Target: choose the target tile, enter the connection, including Encryption for a database target, and select Test connection. See Target.
4. Tables: select the tables to replicate. Each needs a primary key or a unique index over non-null columns. See Tables and configure.
5. Prepare: if this step appears, fix what it lists. Mark now marks SQL Server tables for DirectLog capture, and Prepare target schema creates the target tables once you have read Show SQL. See Prepare.
6. Review: check the name and the summary, then select Create and start. If only Create paused is offered, a readiness check isn't green, and the notices at the bottom of the step say which. See Review and create.

6. Watch the initial load

The pipeline's page opens. It shows Loading and an Initial load card while the agent copies the existing rows, then Healthy once it streams changes. See Initial load.

7. Verify that changes arrive

1. Insert, update and delete a row in one of the selected source tables, and commit.
2. Query the same rows on the target. Names can change between engines: SQL Server dbo.OrderItems lands as public.order_items on PostgreSQL, for example. See Naming conventions per target engine.
3. On the pipeline's Overview tab, the Changes card counts the insert, the update and the delete under Observed and Applied. The Lag tile shows how far the target is behind the source.

If a change doesn't arrive, see Troubleshooting.

Next steps

• SQL Server → PostgreSQL walkthrough
• On-premises SQL Server → Azure SQL over a WAN with relay agents
• Compare a target with its source
• Alerts
• High availability