Question 1

What is this pairing typically used for?

Accepted Answer

A number of things, and they tend to share a requirement: something needs the data as files rather than as a queryable table. That covers handing an extract to a partner or another team, feeding a job that expects objects, and keeping a copy under your own retention terms, among others. Because the export runs on a schedule, the bucket reflects current state rather than whenever an extract was last run by hand.

Question 2

Which format should I choose?

Accepted Answer

It depends on whether nesting matters to you. Databricks exposes STRUCT, ARRAY and MAP, and Parquet preserves typed nested data where the type is supported, so it is the safer choice for anything queried later. JSON and JSON Lines also keep nested values intact. CSV works but encodes nested values into a single cell, so a downstream reader has to parse them back out, and type information is lost along the way.

Question 3

What lands in the bucket on each run?

Accepted Answer

That depends on the mode. Replicate writes the complete dataset to one stable object and replaces it every run, which suits a consumer pointed at a fixed path. Snapshot writes a new timestamped object per run, building history. An incremental option on replicate writes only changed records into new timestamped files. Incremental cannot be combined with snapshot, since the two describe different intentions.

Question 4

What affects how long an export takes?

Accepted Answer

Mostly the SQL warehouse behind the connection. Reads run through it, so its size, autoscaling and how much concurrency it is already handling all shape the runtime, as does how much data the query scans. A cold warehouse adds start up latency before anything moves, and Polytomic will keep retrying the connection for around ten minutes while that happens. Serverless warehouses generally start faster than classic ones.

Question 5

How do I connect Databricks to Amazon S3?

Accepted Answer

Connect Databricks with a SQL warehouse hostname and a token, or an OAuth service principal, and connect S3 with access keys or an assumed IAM role pointing at the bucket and optional prefix. Then choose what to export, pick a format, and decide between one stable object, a timestamped snapshot per run, or changed records only. Output subfolders accept datetime expressions if you want files organised by date.

Connect Databricks with Amazon S3

Connect Databricks and Amazon S3 with Polytomic

Parquet keeps nesting

Replicate or snapshot

Expression driven paths

Trusted by businesses worldwide

Step-by-step integration guide

Connect Databricks in Polytomic

Connect Amazon S3 in Polytomic

Create a sync between Databricks and Amazon S3

Map fields and enable the sync

Frequently asked questions

The sync and ETL platform that does it all

Powerful data sync

Advanced analytics

Enterprise security

Easy setup

Ready to connect Databricks with Amazon S3?