Targets

Azure SQL

Replicate into Azure SQL Database or Hyperscale, set up the target identity, and write into Always Encrypted columns.

SQL Flow applies changes to Azure SQL Database, including Hyperscale, with the same transactional, exactly-once writes as the SQL Server target. This page covers what differs on Azure: authentication, transport security, serverless databases and resource limits.

Requirements

RequirementDetails
ServiceAzure SQL Database or Hyperscale. Azure Synapse pools and Fabric Warehouses are refused. For SQL database in Fabric, use the SQL database in Fabric target.
DatabaseThe target database already exists. SQL Flow never creates a database.
TablesEach target table has a primary key or a unique constraint that matches the source key. Prepare target schema creates missing tables and keys for you.
AuthenticationSQL login or Microsoft Entra ID. Windows authentication isn't available.
TransportAlways encrypted, with the server certificate validated. There is no Encryption setting.
NetworkThe apply agent reaches <server>.database.windows.net on port 1433, and ports 11000 to 11999 when the server uses the Redirect connection policy. Allow the agent's address in the server firewall.

Permissions

The target user needs the same permissions as on SQL Server. Membership in db_owner covers all of them. For a Microsoft Entra identity, create the user from the external provider. Replace <user_name>, and repeat the last statement for each schema that holds target tables:

CREATE USER [<user_name>] FROM EXTERNAL PROVIDER;
GRANT CREATE SCHEMA, CREATE TABLE, CREATE TYPE TO [<user_name>];
GRANT VIEW DEFINITION TO [<user_name>];
GRANT SELECT, INSERT, UPDATE, DELETE, ALTER ON SCHEMA::[dbo] TO [<user_name>];

To use adaptive resource governance, also grant VIEW DATABASE STATE.

What SQL Flow creates on the target

The objects are the same as on SQL Server: your tables, only when you select Prepare target schema or a create action, and the bookkeeping schema, sqlflow by default, created at the first start. These objects use storage and count against the database's size limit.

Options

The Target step offers the same fields as SQL Server, except Encryption:

Field on the Target stepOption keyDefault
AuthenticationauthenticationSQL login
Bookkeeping schemasqlServerMetadataSchemasqlflow
Collapse row versions (faster)applyMode (optimized or ordered)optimized
Maximum records per transactionmaxBufferedTransactionRecords1,000,000
Maximum transaction size (MiB)maxBufferedTransactionBytes512 MiB
Allow streamed large-object loads despite triggerslobInitialLoadAcceptsInsertThenStreamedUpdatesOff

The Authentication methods for Azure SQL are SQL login, Microsoft Entra managed identity, Microsoft Entra workload identity, Microsoft Entra service principal (with a client secret or a certificate), Microsoft Entra integrated and Microsoft Entra password. See Authentication methods. For unattended agents, use a managed identity or a service principal.

Serverless databases

A serverless database pauses when idle and resumes on the first connection, which can take longer than the default 15-second connect timeout. Set mssqlConnectResiliency=true with the CLI's --opt to connect with a 60-second timeout and 3 retries 10 seconds apart. During an Azure failover, a tier change or a resume, apply also retries its connection for up to 120 seconds. If that runs out, the pipeline faults without skipping anything, and a restart continues from its last committed position.

Adaptive resource governance

Set applyAdaptiveResourceGovernance=true with the CLI to have apply shrink its batches and parallelism when the database's log write rate nears its limit. It is off by default, and needs VIEW DATABASE STATE; without it, apply doesn't adapt but is otherwise unaffected.

Option keyDefaultMeaning
applyLogPressureHighPercent80avg_log_write_percent above which apply backs off.
applyLogPressureCriticalPercent95Level at which it backs off hardest.
applyLogPressureLowPercent60Level below which it may recover.

Always Encrypted targets

Azure SQL targets support Always Encrypted columns exactly as SQL Server targets do: alwaysEncrypted=encrypt to encrypt plaintext into the target's encrypted columns, or alwaysEncrypted=ciphertext to copy ciphertext from a DirectLog source.

On Azure, keep the column master key in Azure Key Vault and let the agent reach it with a managed identity or a workload identity that has the key permissions get, unwrapKey and verify:

sqlflow connection edit <connection_name> \
  --opt alwaysEncrypted=encrypt \
  --opt alwaysEncryptedKeyStore=azurekeyvault \
  --opt alwaysEncryptedKeyVaultCredential=managedidentity \
  --opt alwaysEncryptedKeyVaultClientId=<client_id>

Omit alwaysEncryptedKeyVaultClientId for a system-assigned identity. For a workload identity, set alwaysEncryptedKeyVaultCredential=workloadidentity with alwaysEncryptedKeyVaultTenantId, alwaysEncryptedKeyVaultClientId and alwaysEncryptedKeyVaultTokenFile. Untick Collapse row versions (faster): encrypted columns need ordered, row-by-row apply.

Limitations

  • Keys. A target table without a primary key or unique key that matches the source key is refused at the start. rowversion, timestamp, hierarchyid, sql_variant, xml, json, geography, geometry, vector and unbounded columns can't be part of a created key.
  • Types. CLR user-defined types are refused. sql_variant is supported outside the key from a DirectLog or Reconstruction source, and needs ordered apply (untick Collapse row versions (faster)); see Data type mapping.
  • Existing tables. Missing columns aren't added to an existing table.
  • Read-only routing. mssqlApplicationIntent=readonly is refused, because a readable secondary can't take writes.

See also

Previous
SQL Server

If this was useful,
our newsletter will be too.

Get monthly insights on data engineering, AI, and building critical infrastructure - direct from the Spectral Core team and CEO Damir Bulic.