Connections
Unity Catalog setup
Service principal
- In the workspace, go to Settings → Identity and access → Service principals and add one: a Databricks-managed principal, or on Azure an existing Entra ID application.
- Give it the Databricks SQL access entitlement.
- On the SQL warehouse, grant it CAN USE.
- For Service principal (Databricks OAuth secret), create an OAuth secret on the principal's Secrets tab. Copy the client ID and secret into Omni Loader.
Privileges
| Use | Privileges |
|---|---|
| Source (SQL Select) | USE CATALOG, USE SCHEMA, SELECT |
| Source (Parquet export) | the above, plus WRITE FILES on the external location over the export storage |
| Target | USE CATALOG, CREATE SCHEMA (when the run creates schemas), USE SCHEMA, CREATE TABLE, MODIFY |
| Target with cut-off | the above, plus CREATE EXTERNAL TABLE on the external location over the staging storage |
Unity Catalog gives the creator ownership of the schemas and tables a run creates. If the run connects as a service principal, other users need grants from that principal (or from a metastore admin) to read the new tables.
External location
Needed for a Parquet export and for a cut-off target. In Catalog → External data:
- Create a storage credential. On Azure, use the workspace's access connector, which needs Storage Blob Data Contributor on the storage account. On AWS, use an IAM role.
- Create an external location with that credential, at the container or folder Omni Loader uses, for example
abfss://<container>@<account>.dfs.core.windows.net/<folder>ors3://<bucket>/<folder>. - Grant the connecting principal the privileges from the table above.
