Distributed acceleration
Sizing and tuning
Size relay agents for CPU, memory, disk and bandwidth, and set the number of load workers when the default doesn't fit.
Relay tunes its own compression and timing from how far behind it is, so most pipelines need no tuning. This page covers what you do size: the two agents' hosts, the link and the number of load workers.
Size the agents
| Capture agent | Apply agent | |
|---|---|---|
| CPU | Reads and decodes the source, and compresses what it ships. Compression works harder while the relay is behind. | Decompresses and applies. During an initial load, runs one writer per load worker. |
| Disk | Its spool holds every change the apply agent hasn't acknowledged. During a link outage, it grows by the source's change volume for as long as the outage lasts. | Its spool holds what it received until the target commits it. |
To keep an agent from using more memory than you want on a shared machine, install it with --memory-limit, for example --memory-limit 8GiB. The limit is at least 512 MiB. A pipeline that the agent's memory can't hold is refused at start, and isn't restarted automatically. See Limit memory.
Size the capture agent's spool disk for the longest link outage you want to ride out without stopping capture, and plan the source's log retention for the same outage.
Bandwidth
The link must carry the change volume after compression. Measure it on a running pipeline:
- Open Details under the pipeline's metrics.
- Read serialized payload:, the uncompressed rate, and wire:, the rate on the link, with the compression ratio after it.
The link needs headroom above the steady wire: rate to catch up after an outage or during an initial load. While the relay is behind, it compresses harder, so catching up usually takes fewer bytes per change than steady streaming. See Monitoring relay.
Relay doesn't cap its bandwidth, so one pipeline catching up can use the whole link. When several pipelines share a link, size it for the ones that may catch up at the same time.
Load workers
SQLFLOW_LOAD_WORKERS sets how many workers an agent runs in parallel during an initial load: readers on the capture agent, and writers on the apply agent of a relayed load.
| Setting | Value |
|---|---|
| Default | The number of processors, at most 8 |
| Allowed values | 1 to 64 |
| Invalid value | The agent refuses to start: "SQLFLOW_LOAD_WORKERS='…' is not a whole number between 1 and 64; unset it to use the default (processor count, at most 8)." |
Raise it when the target's writers spend most of their time waiting on the service, for example an Azure SQL database whose log rate stays well below its tier's limit during a load. Lower it when a load takes too many connections from the source or the target.
Set the variable in the agent service's environment, then restart the agent. See Set an environment variable for the agent service.
