Log brokers

Install a SQL Flow log broker on the database host, choose its allowed log folders and service account, and approve it.

A log broker is a small, single-purpose service on the database host that serves transaction-log files, and for Sybase ASE the database's device files, read-only, to SQL Flow agents over TLS. Install a broker for these capture modes, even when the agent runs on the same host:

• SQL Server DirectLog
• SQL Server Reconstruction
• Oracle DirectLog
• Sybase ASE DirectLog
• Sybase ASE Reconstruction

Every other capture mode reads changes over the ordinary database connection and needs no broker.

Why a log broker

The broker exists for security. It is the only SQL Flow component with access to the database host, and it is built so you can secure and verify it:

• Small and auditable. It serves file bytes and nothing else. Spectral Core provides its complete source code for your security audit on request.
• Only allow-listed files. It serves only files inside the folders you give it with --allow-paths.
• Only allow-listed agents. With --allow-ips, it accepts connections only from the agent addresses you list.
• Read-only. It never writes to the files it serves, never connects to a database and holds no database credential.
• Recorded. Its audit log records every connection it refuses, and every file it opens or refuses to open.

The agent, which runs the rest of SQL Flow, needs no administrator rights and no access to the database host. See Architecture: Log brokers.

Before you begin

• The database host runs Windows Server 2016 or later, or Linux with glibc 2.28 or later, systemd, useradd and setfacl (from the acl package). On older Windows the broker refuses to install. On Linux with an older glibc, it can't serve log files and reports the C library of this host exports no statx (glibc 2.28 or later is required).
• You can open PowerShell as Administrator, or use sudo, on the database host. Install creates the broker's service account and grants it read access to the log folders.
• You know the folders that hold the database's log files, or for Sybase ASE its device files. The source test in the pipeline wizard discovers them.
• Agents can reach the database host on TCP port 7003, and the host can reach the control plane over outbound HTTPS. See Network and ports.

Setup by database and host

Sybase ASE brokers run on Linux, because the supported ASE build runs there.

The broker serves files from the file system only. It can't serve Oracle ASM locations such as +DATA/…; use LogMiner when the logs exist only in ASM. It can't serve a Sybase ASE raw device either, so keep the database's devices in files. For a database in a container, the broker must see the log files at the same paths the database reports.

Install a broker

1. In SQL Flow, open Agents and select Add broker. The panel creates a join code, valid for 15 minutes. From the pipeline wizard, the same panel opens with the log folders the source test found.
2. Choose the database host's platform, Windows or Linux, and download the executable. To download ahead of time and verify the checksum, use Downloads, then select Set up broker there.
3. Move the executable to a permanent folder on the database host, such as C:\Program Files\SqlFlow\LogBroker or /opt/sqlflow/logbroker. The service runs the executable from where you install it, so leave it there.
4. Keep Install as a service selected. In Log folders, enter the absolute folders the broker may serve, separated by commas. Open Database guidance for the folders your database needs.
5. In Agent addresses, check the IP addresses or CIDR networks of the agents that will read through this broker, separated by commas. The panel fills in the addresses of your known agents. The command passes them as --allow-ips.
6. Select Copy command, then run the command in the permanent folder:

.\SqlFlow.LogBroker.exe install --account <join_code> --allow-paths 'D:\SQLLogs' --allow-ips <agent_addresses>

The panel fills in the join code, the folders and the agent addresses, such as 10.20.0.0/24,10.30.4.17, and adds --control-plane when your control plane isn't the hosted one. 7. Note the certificate fingerprint in the install output, on the line that starts Created broker identity. You compare it when you approve. 8. Approve the broker. See Approve a broker.

Install checks that its service account can read the log files before it uses the join code. If a check fails, install undoes its changes and the join code stays unspent, so you can fix the cause and run the same command again.

Approve a broker

A new broker appears under Brokers on the Agents page as Pending. Until you approve it, it serves nothing.

1. Open Agents and find the broker's card under Brokers.
2. Check the host name, and check that Serves lists the folders you allowed.
3. Open Details and compare Certificate SHA-256 with the fingerprint the install printed.
4. Select Approve. To refuse the registration, select Reject.

Agents reach the broker at its machine name on port 7003. If an agent can't resolve that name, set an address on the pipeline wizard's Broker step: next to Agents dial, select Change, then choose one of the addresses the broker reports or enter one.

What install creates

The data directory holds the broker's identity and certificate. Keep it. Running install again with the same data directory keeps the identity, so the broker stays approved.

Choose the service account

On Linux the broker always runs as sqlflow-logbroker, and --service-account is refused. On some Linux hosts an administrator must also grant that user traversal through restricted parent folders of the log folders.

On Windows, install chooses the account from the folders you allow, and prints the account it chose and why:

• LocalSystem when an allowed folder holds SQL Server database files (.mdf, .ndf or .ldf). SQL Server holds its live log open exclusively, and reading it needs the Debug programs privilege, which LocalSystem has.
• NT SERVICE\sqlflow-logbroker, the broker's own virtual account, otherwise. It reads what install grants it, which covers Oracle redo and archive folders.

To choose another account, pass --service-account:

Another service's virtual account, such as NT SERVICE\MSSQLSERVER, is refused, because Windows doesn't let one service log on as another's virtual account.

Change the allowed folders

The broker serves nothing outside its --allow-paths folders, and a pipeline can't use a log file outside them. To change the list, run install again on the host with the new --allow-paths and a new join code from Add broker. The broker keeps its identity and its approval, and the card's Serves row shows the new list.

Restrict which hosts can connect

Set --allow-ips on every broker, so only your agents can reach its data port. The Add broker panel adds it from Agent addresses. Without it, any address may open a connection to the data port, although each connection must still pass TLS and present a valid pipeline key. To set or change the list, run install with --allow-ips and a new join code from Add broker:

.\SqlFlow.LogBroker.exe install --account <join_code> --allow-paths 'D:\SQLLogs' --allow-ips 10.20.0.0/24,10.30.4.17

The broker refuses any other address and records the refusal in audit.jsonl.

Run in the foreground

Select Run now in the Add broker panel for a session without a service. The broker still runs elevated, because it opens the live log files, and keeps its state in broker-data beside the executable:

.\SqlFlow.LogBroker.exe run --account <join_code> --allow-paths 'D:\SQLLogs' --data-dir .\broker-data

Press Ctrl+C to end the session. In a container or under a supervisor, set SQLFLOW_ACCOUNT, SQLFLOW_CONTROL_PLANE and SQLFLOW_DATA_DIR instead of flags, and stop the broker with SIGTERM.

Command-line reference

SqlFlow.LogBroker install --account <join code> --allow-paths <dir[,dir...]> [--allow-ips <cidr[,...]>]
                          [--control-plane <https url>] [--data-dir <dir>]
                          [--service-account <account> [--service-password-file <path>]]
                          [--service-name <name> --listen-port <port>]
SqlFlow.LogBroker run     [--account <join code>] [--allow-paths <dir[,dir...]>] [--allow-ips <cidr[,...]>]
                          [--control-plane <https url>] [--data-dir <dir>]
SqlFlow.LogBroker uninstall [--data-dir <dir>] [--purge] [--service-name <name>]
SqlFlow.LogBroker --version

--version prints the broker's release version and exits.

Remove a broker

1. On the Agents page, find the broker under Brokers:
   • Revoke an approved broker. It stops serving at once, and pipelines reading through it stop. Its identity is refused until you cancel the revocation.
   • Delete a pending or revoked broker to forget its registration, its keys and its revoked identity.
   • Cancel revocation returns a revoked broker to Pending.
2. On the database host, remove the service:

& 'C:\Program Files\SqlFlow\LogBroker\SqlFlow.LogBroker.exe' uninstall --purge

Use the path where you installed the executable. Without --purge, uninstall keeps the data directory and the folder grants, so a later install brings back the same broker without a new approval.

Troubleshooting

Allowed folder doesn't exist

--allow-paths entry '<path>' does not exist or is not a directory

Cause: a folder is mistyped, or doesn't exist on this host. Fix: enter folders exactly as they exist on the database host.

Allowed folder contains a double quote

--allow-paths value '<value>' contains a double quote.

Cause: on Windows, a quoted folder that ends in a backslash, such as 'C:\dir\', swallows the rest of the command line. Fix: drop the trailing backslash.

Windows Server version too old

the SqlFlow log broker requires Windows Server 2016 or later on the SQL Server host

Cause: the host runs an older Windows version. Fix: move the database host to Windows Server 2016 or later, or use a capture mode that needs no broker, such as Polling.

ACL tools missing on a Linux host

'setfacl' is not available; install the 'acl' package

Cause: the Linux host lacks the POSIX ACL tools. Fix: install the acl package and run the command again.

Broker is waiting for approval

Waiting for the approval in the control plane...

Cause: the broker is registered but not approved. Fix: approve it on the Agents page.

Install says the broker's identity was revoked

Cause: the data directory holds an identity a user revoked. Fix: select Delete on the broker's card, or run uninstall --purge on the host, then install again with a new join code.

See also

• Agents
• Network and ports
• Pipeline wizard: Broker
• High availability: Log brokers
• Security considerations