Control plane

Learn what keeps running while the control plane is down, and run several control plane instances in an air-gapped deployment.

The control plane stores configuration and sends commands to agents and log brokers, but no row data passes through it. While it's unavailable, pipelines keep replicating. What stops is your ability to see and change them.

Hybrid deployments

In a hybrid deployment, the control plane is hosted for you. You don't run its instances or configure its failover. The rest of this page describes what your agents and brokers do while it's unreachable, and how to run the control plane yourself in an air-gapped deployment.

While the control plane is unreachable

Agents and log brokers keep running the work they already accepted:

• Each agent and broker may keep running accepted work for up to 24 hours after its last successful contact with the control plane. After 24 hours without contact, it stops.
• Agents and brokers reconnect on their own.
• You can't create, start, pause or edit pipelines, approve agents or brokers, or see live status.
• An agent or broker that restarts while the control plane is unreachable doesn't start any work until it reaches the control plane again.
• An agent that belongs to a standby pair stops its pipeline roles about 25 seconds after its last successful heartbeat, because its partner could otherwise take them over. When the control plane is back, the pipelines resume on one of the two agents. An agent without a standby keeps running.
• A revoked or expired agent credential stops the agent at once.

Reconnect resume

When an agent reconnects, the control plane resumes every pipeline assigned to it that should be running but isn't. The pipeline continues from its last committed position without a reload. A pipeline you paused, or one with a start, pause or reload command already queued, is left alone.

Run several instances in an air-gapped deployment

In an air-gapped deployment you run the control plane yourself, and you can run two or more instances of it against one PostgreSQL database. One instance, the leader, serves every request. The others forward the requests they receive to the leader, and one of them takes over when the leader stops.

How leadership works

• Lease. The leader holds a lease stored in the PostgreSQL database and timed by the database's clock, so the instances' own clocks don't matter.
• Takeover. When the leader stops renewing its lease, a standby takes over about 15 seconds later. A leader that shuts down cleanly releases the lease, so a standby takes over within seconds. A leader that can't renew stops serving before any other instance can take over.
• Forwarding. A standby passes every request it receives to the leader, over HTTPS, and returns the leader's answer. Health endpoints are answered by the instance that receives them.
• No leader. While no instance leads, every instance answers 503 with the code control_plane_no_leader. Agents and brokers retry.

Every instance serving at once (active-active) isn't available in this release.

Before you begin

• A PostgreSQL database that every instance uses, set in ConnectionStrings:SqlFlow. The control plane's availability can't exceed its database's, so run the database with automatic failover.
• The same private certificate on every instance to protect the Data Protection key ring, which SQL Flow stores in the PostgreSQL database. Set DataProtection:CertificatePath and DataProtection:CertificatePassword, or DataProtection:CertificateThumbprint. Back the certificate up separately from the database: without it, stored secrets can't be read.
• A TLS setup that works for every instance. Either give every instance your own certificate (Kestrel:Certificates:Default) and set ControlPlane:CertificateAuthorityPath to the PEM file of the authority that issued it, or use the control plane's built-in authority and list every DNS name and IP address that clients use in ControlPlane:PublicHostNames.
• A load balancer in front of the instances that sends traffic to all of them, health-checks GET /healthz, keeps the original Host header, and connects to the instances over HTTPS. The control plane refuses to start without an HTTPS binding. It builds the control plane URL shown with each join code from the Host header it receives.
• Network access between instances: each instance reaches every other one at its advertised HTTPS address.

Configure the instances

Set these values identically on every instance. In environment variables, replace each : with __, for example ControlPlane__Ha__Mode.

Bring the cluster up

1. Run the migration step once, with the same configuration as the instances, including ControlPlane:Ha:ClusterName. It creates or upgrades the database schema and records the cluster name, then exits.

   SqlFlow.App migrate

2. Start every instance.

3. Add every instance to the load balancer, with GET /healthz as the health check.

Verify

Ask each instance for its role:

curl https://<instance-address>:<port>/healthz/role

Exactly one instance answers "role": "leader". The others answer "role": "standby", with the same epoch. An instance without ControlPlane:Ha:Mode answers "role": "single".

Health endpoints

All three are anonymous.

Troubleshooting

Cluster name not set

control_plane_cluster_name_required

ControlPlane:Ha:Mode is leader-serves, but ControlPlane:Ha:ClusterName isn't set. Set the same cluster name on every instance and on the migration step.

Database cluster name missing or different

control_plane_cluster_unset
control_plane_cluster_mismatch

The database has no cluster name yet, or it was migrated for another cluster. Run SqlFlow.App migrate with the cluster name, or point the instance at the right database.

In-memory store can't be shared

control_plane_ha_in_memory_store

ConnectionStrings:SqlFlow selects the in-memory store, which can't be shared between processes. Point every instance at the same PostgreSQL database.

Instances run in different modes

control_plane_ha_mode_mismatch

Live instances on the same database run in another mode. Give every instance the same ControlPlane:Ha:Mode, or stop the others first.

Key ring not shared between instances

control_plane_key_ring_not_shared
data_protection_certificate_required

The instance can't read the key ring the other instances share. Install the same Data Protection certificate on every instance.

Requests fail with 503 control_plane_no_leader

No instance holds the lease: the leader stopped and no standby has taken over yet, or no instance can reach the database. Check /healthz/role on each instance and the database's availability.

See also

• Air-gapped deployment
• Deployment models
• Agents
• Network and ports