Administration
Database credentials and secret references
Choose between a database password stored encrypted by SQL Flow and a secret reference resolved on the agent host, and set either one up.
A connection that signs in with a password takes exactly one password source. Either you enter the password and SQL Flow keeps it encrypted, or you give a secret reference and the agent reads the password on its own host. With a secret reference, the password never reaches the control plane.
Compare the two sources
| Password | Secret reference | |
|---|---|---|
| What you enter | The password | env:<name> or file:<name> |
| Where the value is stored | The control plane, encrypted | Only on the agent host: an environment variable of the agent service, or a file in the agent's data directory |
| Who receives it | Only the agent assigned to the pipeline, over its authenticated HTTPS connection | Nobody; the agent reads it locally |
| Saved connections | Not stored. You enter the password when a pipeline uses the saved connection. | Stored, because a reference is not a secret |
| Several agents (relay, standby) | Delivered to each agent that runs the pipeline | Must exist on each agent host that may run the pipeline |
Supplying both for one connection is refused with <connection>: supply either a secret reference or a password, not both., where <connection> names the connection, such as Source connection.
Some sign-in methods use no password at all, for example SQL Server Windows (agent service account), Microsoft Entra managed identity and Google Pub/Sub. They take credentials from the agent host. For a Microsoft Entra service principal, the password source holds the Client secret.
Enter a password
- In the connection form, set Password source to Password.
- Enter the password.
With the CLI, pass the password by the name of an environment variable in the CLI's own shell, so it doesn't appear in your command history or the process list:
export ORDERS_DB_PASSWORD='<password>'
sqlflow pipeline create orders --source orders-sql --target warehouse-pg --tables dbo.Orders --source-password-env ORDERS_DB_PASSWORDUse a secret reference
- In the connection form, set Password source to Secret reference.
- Under Secret location, choose Environment variable (SQLFLOW_SECRET_<name>) or File (<data dir>/secrets/<name>).
- Enter a Secret name, for example
orders-pw. - Provide the password on every agent host that may run the pipeline, as described in the next two sections.
- Select Test connection & list tables. The test runs on the agent, so it proves that the agent resolves the reference.
A name has 1 to 64 characters. It starts with a letter or a digit, and continues with letters, digits, _, - or ., but not ... A reference can reach only the agent's own secrets folder and its own SQLFLOW_SECRET_ variables. With the CLI, pass the reference as --secret-ref, --source-secret-ref or --target-secret-ref, for example --source-secret-ref file:orders-pw.
Provide a secret file
For file:<name>, create the file secrets/<name> in the agent's data directory, readable by the agent's service account and nobody else:
Run in PowerShell as Administrator:
$dir = Join-Path $env:ProgramData 'SqlFlow\Agent\secrets'
New-Item -ItemType Directory -Force -Path $dir | Out-Null
icacls $dir /inheritance:r /grant:r '*S-1-5-32-544:(OI)(CI)F' '*S-1-5-18:(OI)(CI)F' 'NT SERVICE\SqlFlow.Agent:(OI)(CI)R'
$secure = Read-Host -AsSecureString -Prompt 'Password'
[System.Net.NetworkCredential]::new('', $secure).Password | Set-Content -Path (Join-Path $dir 'orders-pw') -NoNewline -Encoding utf8The icacls line limits the folder to Administrators, SYSTEM and the agent's service account. If the agent runs under another account, name that account instead of NT SERVICE\SqlFlow.Agent.
The agent drops one trailing line break from the file and keeps every other character, including leading or trailing spaces.
Provide an environment variable
For env:<name>, set SQLFLOW_SECRET_<name> in the agent service's environment, not in your shell. For example, env:orders_pw reads SQLFLOW_SECRET_orders_pw. See Set an environment variable for the agent service for the steps, then restart the agent service so it sees the new value.
Prefer the file form when you can. A service's environment is visible to administrators in the Windows registry and to the service manager on Linux. On Linux, keep the value out of the unit with an EnvironmentFile= line that points to a file only root can read.
Rotate a password
- Password. Enter the new password on the pipeline's connection. The control plane encrypts it and delivers it to the agent.
- Secret file. Replace the file's contents, then pause and resume the pipeline. The agent reads the file when it starts or resumes a pipeline and when it runs a connection test.
- Environment variable. Change the variable, then restart the agent service.
Troubleshooting
Secret variable not set
The <engine> connection names the secret reference 'env:<name>', but the environment variable SQLFLOW_SECRET_<name> is not set for this agent.Cause: the variable isn't in the agent service's environment, or the service wasn't restarted after you set it. Fix: set it as described above and restart the service. A variable set only in your shell isn't visible to the service.
Secret file missing or unreadable
The <engine> connection names the secret reference 'file:<name>', but this agent has no readable secret file 'secrets/<name>' under its data directory.Cause: the file is missing, misnamed, in another agent's data directory, or not readable by the service account. Fix: create it in the data directory of the agent that runs the pipeline, and check its permissions.
Invalid secret reference
'<reference>' is not a valid secret reference.Cause: the reference doesn't start with env: or file:, or its name breaks the rules above. Fix: correct the name.
