Omni Loader writes each table to staging files in cloud storage, then loads them with COPY INTO. Tables are created as Delta tables in the connection's catalog. Primary keys, unique constraints and foreign keys are created as Unity Catalog informational constraints. Indexes are not created, because Databricks has none.

Omni Loader connection to storage

Storage

• Azure Data Lake or Azure Blob: the storage endpoint, container, staging folder and a SAS token. COPY INTO passes the same SAS token to Databricks, so the staging storage needs no external location.
• Amazon S3: the bucket URL, access key ID and secret access key of an IAM user. For each load, Omni Loader exchanges the key for temporary credentials, because Databricks refuses a plain access key. Azure Databricks can load from S3 this way.

Data format

• Parquet (recommended) keeps types exact.
• CSV is text. Omni Loader casts every value to the column's declared Databricks type during the load. One difference from Parquet: an empty string arrives as NULL.

Compression

Snappy, Gzip or None for Parquet; Gzip or None for CSV.

Data handling

Timestamped, Clear before run, Folder per table or Clear after run.

Cut-off (warm tier)

With a cut-off column on a table (the table's Cut-off tab):

• rows below the warm value are skipped;
• rows from the hot value up are loaded into the Delta table;
• rows in between stay in the staging files, under <staging folder>/Warm/<schema>/<table>.

Omni Loader creates an external table <table>_external over those files. Querying the hot table and the external table together gives you all the rows.