Agents

Install a SQL Flow agent as a Windows service or a systemd service, approve it, and remove it when you no longer need it.

An agent runs your pipelines: initial loads, change capture, the encrypted on-disk spool, and apply to targets. Install one on a machine that can reach your sources and targets. A pipeline can use one agent, or a capture agent and an apply agent with relay.

Before you begin

• A 64-bit (x64) Windows or Linux machine. On Linux, install needs systemd and useradd.
• An administrator PowerShell prompt on Windows, or sudo on Linux. Only install and uninstall need elevation.
• Outbound HTTPS from the machine to the control plane. The agent needs no inbound port, except as the apply agent of a relay pipeline. See Network and ports.
• Network access from the machine to your source and target databases, and to any log broker your pipelines read through. localhost in a connection means the agent's own machine.
• Local disk for the change spool, in the agent's data directory.

Install an agent

1. In SQL Flow, open Agents and select Add agent. The panel creates a join code, valid for 15 minutes.
2. Choose the platform of the agent machine, Windows or Linux, and select Download SqlFlow.Agent.exe or Download SqlFlow.Agent. Copy the file to the agent machine. To download ahead of time and verify the checksum, use Downloads, then select Set up agent there.
3. Keep Install as a service selected and select Copy command.
4. On the agent machine, open the folder that holds the download and run the command. The command differs only in the join code, and in --control-plane when your control plane isn't the hosted one:

.\SqlFlow.Agent.exe install --account <join_code>

Replace <join_code> with the code from the panel. A bad or expired code fails before anything is installed. The command installs and starts the service, and prints the agent's identity. 5. Approve the agent. See Approve an agent.

If the join code expires before you run the command, select Get new join code in the panel and copy the command again.

Approve an agent

A new agent appears on the Agents page as Pending. It runs nothing until you approve it.

1. Open Agents and find the card marked Approval required.
2. Check that the agent's name, which is its machine name, and its Agent id match the machine and the identity the install command printed.
3. Select Approve agent. To refuse it, select Reject instead.

The agent picks up its configuration within 30 seconds of approval. You can also approve with the CLI: sqlflow agent approve <name>.

What install creates

Install copies the executable into the program directory, so you can delete the download afterwards. Running install again on the same machine keeps the stored identity, so the agent needs no second approval.

The data directory

The data directory holds everything that makes an agent the same agent after a restart. Keep it, and back it up with the machine.

Run in the foreground

Select Run now in the Add agent panel for a session that runs until you close the shell, with no service and no elevation. Containers and process supervisors use the same run verb.

.\SqlFlow.Agent.exe run --account <join_code> --data-dir .\agent-data

Without --data-dir, run uses the service's default data directory, which on Linux needs root. Add --data-dir when your account can't create it.

run needs a join code only while the data directory holds no identity. In a container, set SQLFLOW_ACCOUNT, SQLFLOW_CONTROL_PLANE and SQLFLOW_DATA_DIR instead of flags, and keep the data directory on a persistent volume. Stop the agent with Ctrl+C or SIGTERM; it stops its pipelines cleanly and keeps its checkpoints.

Command-line reference

SqlFlow.Agent install --account <join code> [--control-plane <https url>] [--data-dir <path>] [--reset]
                      [--memory-limit <size>] [--service-account <account> [--service-password-file <path>]]
SqlFlow.Agent run     [--account <join code>] [--control-plane <https url>] [--data-dir <path>] [--reset]
                      [--memory-limit <size>]
SqlFlow.Agent uninstall [--data-dir <path>] [--purge]
SqlFlow.Agent --version

--version prints the agent's release version and exits.

A flag wins over its environment variable.

Choose the service account

The default virtual account needs no password and gets full control of the data directory. Choose another account with --service-account on Windows when your database authentication or file access needs it, for example a domain account for Windows authentication to SQL Server:

.\SqlFlow.Agent.exe install --account <join_code> --service-account 'CORP\svc-sqlflow' --service-password-file C:\secure\svc-sqlflow.txt

Install hands the password to Windows without storing it. Delete the password file afterwards. On Linux the agent always runs as sqlflow-agent, and --service-account is refused.

Limit memory

By default the agent sizes itself to the memory of its host or container. Set --memory-limit to cap the whole agent on a machine it shares:

.\SqlFlow.Agent.exe install --account <join_code> --memory-limit 8GiB

The value is a whole number of bytes, or a whole number with KiB, MiB, GiB or TiB and no space. The minimum is 512MiB. Decimal units such as GB are refused. The cap survives restarts, but running install again without --memory-limit clears it.

Set an environment variable for the agent service

An installed service doesn't inherit the environment of the shell you install it from. Set a variable the agent reads in the service's own environment, then restart the service. Two such variables are SQLFLOW_SECRET_<name>, for an env: secret reference, and SQLFLOW_LOAD_WORKERS, for the number of load workers. For the memory limit, use --memory-limit on install instead.

The examples set SQLFLOW_LOAD_WORKERS to 16. Replace the name and value with yours.

$name = 'SQLFLOW_LOAD_WORKERS'; $value = '16'
$key = 'HKLM:\SYSTEM\CurrentControlSet\Services\SqlFlow.Agent'
$current = @((Get-ItemProperty -Path $key -Name Environment -ErrorAction SilentlyContinue).Environment) | Where-Object { $_ }
$updated = @($current | Where-Object { $_ -notlike "$name=*" }) + "$name=$value"
Set-ItemProperty -Path $key -Name Environment -Value ([string[]]$updated) -Type MultiString
Restart-Service -Name SqlFlow.Agent

Install many agents

For scripted provisioning, mint one join code that admits several registrations, from 1 to 1000:

sqlflow agent join-code --uses 20

The code is still valid for 15 minutes, and every agent that registers with it still needs approval. The same works for log brokers.

Remove an agent

1. On the Agents page, take the agent out of service:
   • Revoke rejects its credential from now on, whether the agent is online or not. Use it for an agent that is lost for good. Its pipelines pause, or move to its standby agent if it has one. It stays listed with a Revoked chip.
   • Remove, on an offline or revoked agent, deletes its record. Pipelines assigned to it are unassigned. An agent that isn't revoked and still holds a pipeline it never confirmed stopping can't be removed until you revoke it.
2. On the agent machine, remove the service:

& 'C:\Program Files\SqlFlow\Agent\SqlFlow.Agent.exe' uninstall

Without --purge, uninstall keeps the data directory, so a later install on the same machine continues as the same agent. With --purge it also removes the data directory, the program directory, and the Linux user or the Windows event-log source. Run uninstall --purge from a copy of the executable outside the program directory, because it can't delete the folder it runs from.

A revoked installation can't register again under the same identity. To bring the machine back, run install --reset --account <join_code> with a new join code, then approve the new agent.

Troubleshooting

Install needs administrator rights

install must run elevated: it creates the service account, the service and the data directory.

Cause: the command ran without administrator rights. Fix: run it in PowerShell as Administrator, or with sudo.

Join code spent, expired or unknown

Registration refused: the join code is spent, expired or unknown.

Cause: the code is older than 15 minutes, was already used, or was minted by another control plane. Fix: select Get new join code and run the command again.

Agent installation was revoked

Registration refused: this installation has been revoked.

Cause: the agent on this machine was revoked. Fix: run install --reset --account <join_code> with a new code, and approve the new agent.

Control plane URL isn't HTTPS

--control-plane must be an https URL (got '<url>'); the control plane is only ever reached over TLS.

Cause: the control plane address starts with http://. Fix: use its https:// address. The Add agent panel prints the address the control plane reports for itself.

Control plane certificate authority missing

No control-plane certificate authority is stored at <path>.

Cause: run started with a data directory that holds no ca.pem, and without a join code. Fix: pass --account <join_code>, or point --data-dir at the directory the agent was installed with.

The agent log says identity <id> is awaiting approval

Cause: the agent registered but isn't approved yet. Fix: approve it on the Agents page.

Build outside the version window

Cause: the agent release is too old, or too new, for the control plane. Fix: install a supported release. See Upgrades and restarts.

See also

• Log brokers
• Network and ports
• Upgrades and restarts
• Monitoring agents and brokers
• High availability: Agents