Agents

Pair agents so a standby takes over pipelines automatically, and learn how a pipeline resumes from its target's committed position on any agent.

An agent runs the capture and apply work of its pipelines. When an agent stops, its pipelines resume when it comes back. If you pair it with a standby agent, the standby takes its pipelines over automatically, without waiting for it.

After an agent restart

The agent's service manager restarts it within seconds after a crash: the Windows service, or the sqlflow-agent systemd service on Linux.

When the agent reconnects, the control plane resumes every pipeline that should be running on it, and each pipeline continues from its last committed position. A pipeline you paused stays paused. See Reconnect resume.

Roles, leases and epochs

Each pipeline role (capture, and one apply role per target) is held by one agent under a 30-second lease, and each change of owner raises the role's epoch. See Leases and epochs.

• An agent that has a standby partner stops a role when it can't renew its lease, before the lease runs out, so the old owner has stopped before the standby starts. An agent without a standby keeps running when it can't reach the control plane.
• The target refuses a write from an agent with an older epoch, with the error pipeline_owner_superseded, so an agent that was cut off can never overwrite the work of the agent that replaced it.

Resume from the target's committed position

Every target records the position the pipeline committed, in the same transaction as the data it writes. Any agent that starts a role resumes from that record and skips every change the target already holds, so it doesn't need the previous agent's disk.

• During an initial load, the tables a target records as fully loaded aren't loaded again. The new owner loads the rest.
• If the source no longer holds the log from the committed position, the pipeline continues from the database's log backups where the capture mode supports it. Otherwise it stops by name, and you reload the pipeline or the affected tables. See Source log retention.
• On a database target, the committed position is kept in the target's bookkeeping schema, sqlflow by default. Each target's page lists what SQL Flow creates there.

Set up a standby agent

A standby agent is linked to a primary agent as a pair. It takes over every pipeline role the primary holds, and the primary takes the roles back if the standby later fails while holding them.

Before you begin

• Install the standby on a different host from the primary. Two agents on one host protect against a crash of the agent process, but not against a host failure.
• Give the standby the same access as the primary: network routes and firewall rules to every source, target and log broker of the primary's pipelines, and the connector for every engine those pipelines use.
• Run both agents on a release inside the control plane's version window.
• Make sure the standby runs no pipeline work of its own. An agent that runs a source, relay or apply role can't become a standby.

Link an existing agent as the standby

1. Open Agents.
2. On the primary agent's card, click Configure standby.
3. In Standby agent for primary, choose an agent marked Eligible.
4. Click Use selected standby.

The two agents now appear together on one pair card, marked ↔ Linked.

If the primary isn't online when you link the standby, the standby doesn't take over yet, because the primary may still be running its pipelines without knowing about the pair. The standby takes over once the primary is back online, or at once if you revoke the primary.

You can also link from the standby's side: on its card, open Advanced: use this agent as a standby, click Make standby for, choose the Primary agent for standby, and click Link standby.

Install a new agent as the standby

1. On the primary agent's card, click Configure standby, then Install new standby.
2. Copy the install command from Install standby agent and run it on the standby host.
3. Approve the new agent on Agents. Approval links it to the primary as its standby.

Use the CLI

sqlflow agent standby <standby-name-or-id> --for <primary-name-or-id>
sqlflow agent ha-pairs
sqlflow agent unlink-standby <agent-name-or-id>

See Agents commands.

Verify the pair

The pair card on Agents shows:

• Active now: the agent that holds the protected pipeline roles.
• Standby: the agent ready to take them over.
• Protected pipelines: every pipeline either agent holds a role of. Each one lists its roles and the agent active for each.
• A readiness chip. Ready means "Both agents are online, inside the version window, and can run every role the other holds." Pair readiness unconfirmed comes with the reason.

Open Details on the pair card to see each role's Recorded owner, Epoch and Lease until.

On a pipeline's Overview tab, the Availability card lists the pair under Agent failover, with the agent Active now for each role and its Standby.

How a takeover happens

When the primary crashes, loses its host or loses its network, it stops renewing its leases and stops its roles. Once its leases in a pipeline run out, the standby takes all of that pipeline's roles over and resumes from the target's committed position. A paused pipeline moves too, and stays paused. The takeover starts about 30 seconds after the primary's last successful heartbeat.

You don't have to wait for the leases if you know the primary is lost. Click Revoke on its card on Agents: a revoked agent counts as stopped, so the standby takes its roles over at once. If the standby can't take over, for example because it's outside the version window, the primary's pipelines pause instead. The revoked agent's credential is refused from then on, so if it's still running somewhere, it stops on its next contact with the control plane.

What you see

• Agents shows a red alert that names the offline agents. For the pair, it says either that the protected roles are active on the standby and that you should restore the offline agent to regain standby protection, or that the takeover is pending or unconfirmed.
• The pipeline page shows Agent offline: with the same guidance, and the Availability card names the agent Active now.

When the old primary comes back

The returning agent finds that its roles have moved. It stops its old work without deleting anything, and becomes the standby for the roles its partner now holds. SQL Flow doesn't move the roles back automatically.

Move a pipeline without a standby

If an agent without a standby is down for good, move its pipelines to another agent:

1. Pause the pipeline, and wait for the agent to acknowledge that it stopped. Pause is refused while the agent is offline, because an agent that doesn't answer may still be running. If the agent is lost for good, click Revoke on its card on Agents instead: a revoked agent counts as stopped, and its pipelines pause.
2. Choose Actions > Assign agent… and pick the new agent.
3. Click Start. The new agent resumes from the target's committed position.

With the CLI, revoke a lost agent with sqlflow agent revoke <agent>, and use sqlflow pipeline assign <name> --agent <agent> after the pause.

Assign agent… isn't offered for a pipeline that uses relay agents. Protect both relay agents with standbys instead.

Every SQL Server DirectLog pipeline that reads the same database must have the same capture agent and the same standby. To move them, unassign every one of them first, then assign them all to the new agent.

Unlink a pair

Click Unlink from HA on the pair card. Both agents stay enrolled and approved. Each protected pipeline stays with the agent that runs it now, and is no longer taken over if that agent goes offline.

If a takeover is due at that moment, the control plane refuses and says which agent to wait for.

Troubleshooting

Agent is reserved as standby

This agent is reserved as standby. Remove its HA link before assigning or starting independent work.

A standby agent can't run pipelines of its own. Assign the pipeline to another agent, or unlink the pair first.

Agent with roles can't become a standby

This agent is running a source, relay, or apply role and cannot become a standby.

Pause or move the agent's pipelines, wait for them to stop, then link it again.

The pair shows Pair readiness unconfirmed

Read the reason under the chip. One of the agents:

• isn't registered, approved or online;
• is revoked: its partner takes over the roles it held. Unlink the pair, then remove the revoked agent;
• went offline before it learned of the pair, so it may still be running its roles. Its partner takes over once it's back online, or at once if you revoke it;
• runs a release outside the version window;
• runs a build that doesn't stop its roles at their deadline, which needs an agent upgrade;
• or lacks the connector for an engine of a pipeline the other agent holds.

Offline agent can't acknowledge the stop

Agent '…' is not online, so it cannot acknowledge the stop.

You paused a pipeline whose agent is offline. SQL Flow doesn't treat silence as a stop, because the agent may be cut off from the control plane and still running. Bring the agent back online and pause again. If it's lost for good, revoke it on Agents: its pipelines then pause, or move to its standby.

Agent still holds an unstopped pipeline

Agent … still holds a pipeline that has not acknowledged a stop, …

You clicked Remove on an agent that isn't revoked and still holds a pipeline that hasn't stopped. Bring the agent back online and pause its pipelines, or revoke it first, then remove it.

Pipeline can't be reassigned until it stops

To reassign this pipeline, dispatch Pause and wait for every runner to acknowledge it stopped.

The pipeline's current agent hasn't confirmed that it stopped. Wait for the pause to complete. If the agent is gone for good, revoke it, then assign the pipeline again.

Standby on the same host as the primary

Both agents are on the same host.

The pair protects against an agent process failure, but not a host failure. Install the standby on another host.

See also

• Overview
• Automatic restart
• Control plane
• Agents
• Monitoring agents and brokers