Question 1

What does the connection need on each side?

Accepted Answer

Databricks takes a SQL warehouse hostname and a token, or an OAuth service principal. Azure takes a storage account and container plus one of an account key, a SAS token, service principal credentials, or OAuth. Which Azure credential you choose has no bearing on formats, paths or write behaviour, so pick whichever fits how your organisation handles cloud identity rather than what the sync needs.

Question 2

How do the write modes differ?

Accepted Answer

Replicate maintains one stable object replaced on every run, which suits a consumer pointed at a fixed path indefinitely. Snapshot writes a new timestamped object per run, building history in the container. An incremental option on replicate writes only changed records as new timestamped files, which suits a downstream process consuming deltas. Output subfolders and names accept datetime expressions, so files can organise themselves by date.

Question 3

Which format should I use?

Accepted Answer

Parquet where the files will be queried later, since it carries types and preserves nesting, which matters because Databricks exposes STRUCT, ARRAY and MAP. JSON Lines keeps nested values intact and suits row by row processing. CSV is the compatibility option and encodes nested values into a single cell, losing type information along the way. There is no compression setting to configure in any of them.

Question 4

What affects export runtime?

Accepted Answer

The SQL warehouse behind the connection, mainly. Its size, autoscaling and existing concurrency all shape how quickly a read completes, as does how much data the query scans. A cold warehouse adds start up latency before anything moves, and the connection is retried for around ten minutes while that happens. Serverless warehouses generally start faster than classic ones, which is worth knowing if exports run on a tight schedule.

Question 5

How do I connect Databricks to Azure Blob Storage?

Accepted Answer

Connect Databricks with a SQL warehouse hostname and a token, or an OAuth service principal, then connect Blob Storage with an account key, SAS token, service principal, or OAuth, naming the storage account and container. After that, choose the data to export, the output format, and whether each run replaces one object or writes a new timestamped one.

Connect Databricks with Azure Blob Storage

Connect Databricks and Azure Blob Storage with Polytomic

Several Azure credential types

Typed output with Parquet

Stable file or history

Trusted by businesses worldwide

Step-by-step integration guide

Connect Databricks in Polytomic

Connect Azure Blob Storage in Polytomic

Create a sync between Databricks and Azure Blob Storage

Map fields and enable the sync

Frequently asked questions

The sync and ETL platform that does it all

Powerful data sync

Advanced analytics

Enterprise security

Easy setup

Ready to connect Databricks with Azure Blob Storage?