High availability
Overview
Learn what happens when the control plane, an agent, a log broker, a source or a target fails, and how each pipeline recovers.
SQL Flow keeps replicating through the failure of any single component. Each component recovers on its own, and every pipeline resumes from the position its target last committed, so a failure costs time but never data.
High availability covers three SQL Flow components: the control plane, the agents and the log brokers. It applies to every source, capture mode and target. It needs the High availability capability pack; see Plans and licensing.
What can fail
| Component | What happens while it is down | How it recovers |
|---|---|---|
| Control plane | Pipelines keep replicating. You can't create, start, pause or edit pipelines until it's back. | In an air-gapped deployment, run several instances against one PostgreSQL database and a standby instance takes over. In a hybrid deployment, it's hosted for you. See Control plane. |
| Agent | The pipeline roles it runs stop. The source keeps the log those pipelines still need. | The service manager restarts the agent, and the control plane resumes its pipelines. With a standby agent, the standby takes the roles over after about 30 seconds, or at once when you revoke the failed agent. See Agents. |
| Log broker | The agent can't read the source log, so the pipeline faults and restarts automatically. | The service manager restarts the broker. A standby broker that reads the same log files takes over on the next connection. See Log brokers. |
| Source or target | Capture or apply fails with a lost connection, a timeout, throttling or a failover. | The agent restarts the pipeline with increasing delays, and parks it for an operator if the fault keeps coming back. See Automatic restart. |
| Network link between relay agents | The capture agent keeps capturing into its local spool. | The capture agent reconnects and resends everything the apply agent hasn't stored yet. See How relay works. |
One restart path
Every restart resumes the same way, whatever caused it: an automatic restart after a fault, an agent service restart, a takeover by a standby agent, or an operator moving a pipeline to another agent. The agent reads the committed position from each target, reads the source again from there, and skips every change the target already holds.
Because the position lives on the target, a pipeline doesn't depend on the disk of the agent that ran it before. A new agent with an empty data directory resumes exactly where the old one stopped.
Leases and epochs
Each pipeline has a capture role, which reads the source, and one apply role per target. On a single-agent pipeline, one agent holds every role. With relay agents, the capture and apply roles run on different agents.
- Lease. The control plane grants each role to one agent under a 30-second lease, which the agent renews while it is healthy. A standby takes a role over only after its lease runs out.
- Epoch. Each grant carries an epoch, which increases every time the role changes owner. The target refuses a write from an older epoch, so an agent that was cut off from the network can't overwrite the work of the agent that replaced it.
What these settings don't cover
Standby agents and standby brokers protect SQL Flow's own components. They don't configure failover of your database hosts. A source or target that restarts or fails over within a few minutes is ridden out by automatic restart.
While a component is down, a log-based source may have to keep more transaction log than usual. See Source log retention.
