Connections
Reading from Databricks
Reading from Databricks
When Databricks is the source, the Data read option chooses how rows leave Databricks.
SQL Select
The default. Omni Loader reads each table through the ODBC driver, then writes it to the target like any other source. Works with every target.
Parquet export
Databricks writes each table as Parquet files directly to cloud storage (INSERT OVERWRITE DIRECTORY), and the target loads those files itself. Rows never pass through Omni Loader, which makes this the fastest path for large tables.
| Export storage | Targets that can load it |
|---|---|
| Azure Data Lake | Databricks, Fabric Lakehouse, Fabric Warehouse, Azure Synapse |
| Amazon S3 | Databricks |
Databricks export storage
- Storage: Azure Data Lake or Amazon S3. For Azure, enter the storage account, container and a SAS token. For S3, enter the bucket URL, access key ID and secret access key.
- Data format: Parquet.
- Compression: Snappy, Gzip or None.
- Data handling: what happens to the export folder between runs. Options are Timestamped, Clear before run, Folder per table and Clear after run.
Loading the export
The target uses the source's storage settings. Its connection shows Use export authentication: leave it on to load with the source's SAS token, or turn it off and enter a SAS token of the target's own. An S3 export brings its keys along, and the Databricks target turns them into temporary credentials for the load.
A Fabric Lakehouse target loads the export with a Spark notebook and needs Service principal authentication. A source filter (WHERE clause) applies to the export too.
