High availability
Log brokers
Learn how pipelines recover when a log broker stops, and add standby brokers that serve the same source log files.
A log broker serves the source's log files to the agent. If the broker stops, capture stops with it, and the pipeline restarts automatically until a broker answers again. A standby broker on another host lets the pipeline continue while the preferred broker is down.
When a broker stops
When the agent can't read through the broker, the pipeline faults with broker-unavailable or broker-not-ready, both transient faults, and the agent restarts it on the automatic restart schedule. Meanwhile the broker's service manager (the Windows service, or systemd on Linux) restarts the broker within seconds. At the next restart, the agent connects to the first broker that answers and the pipeline continues from its last committed position.
If the broker is still unreachable after five automatic restarts within 30 minutes, the pipeline is parked as Failed: needs operator. Bring the broker back, then Pause and Start the pipeline.
A refusal that a restart can't fix, such as a failed authentication, a TLS error or a log path outside the broker's allowed folders, stops the pipeline at once instead. See Automatic restart.
While the broker is down, the source may keep more log than usual. See Source log retention.
Add standby brokers
Each pipeline that reads its source through a log broker can have up to two standby brokers besides its preferred broker. A standby must be able to read the same source log files:
- A second broker on the same host protects against a broker process failure.
- A broker on another host protects against a host failure, if that host can reach the same log files.
Before you begin
- Install the standby broker and approve it on Agents.
- Allow the same log folders on the standby as on the preferred broker, including the log backup folders. A standby whose allowed folders don't cover the source's log files isn't eligible.
- Open the standby's port to the pipeline's agent.
Configure the standby
- Open the pipeline and stay on the Overview tab.
- In the Availability card, under Broker failover, click Configure broker standby.
- Pick the standby:
- Choose existing broker lists the registered brokers, each with its eligibility.
- Install new standby broker shows an install command pre-filled with the source's log folders. Run it on the standby host, click Refresh after installation, and approve the new broker.
- Select up to two standby brokers, then click Save standby brokers.
A standby is eligible when it is approved and its allowed folders cover the source's log files. Network and file access are checked when the agent first connects to it.
How the agent picks a broker
Every time a pipeline starts or restarts, the agent tries the preferred broker first, then each standby in order, and reads the source log through the first broker that answers. It moves on to the next broker only when it can't connect to the current one. A TLS or authentication refusal ends the attempt, because another broker wouldn't change it.
Adding a standby doesn't move a running pipeline. The agent considers the standby the next time it connects.
Unlink a standby broker
In the Broker failover section, click Unlink from HA next to the standby. The broker stays approved and keeps serving its other pipelines. If the agent is reading through that standby right now, it switches back to the preferred broker or the other standby.
During a control plane outage
A broker keeps serving the work it already accepted for up to 24 hours without reaching the control plane. A broker that restarts while the control plane is unreachable waits for it before it serves. See Control plane.
