Question 1

How do I replicate Postgres to Databricks?

Accepted Answer

Connect both, pick your tables, set a schedule. Polytomic loads everything once, then follows PostgreSQL logical replication. On the Postgres side you need a publication covering those tables, a replication slot, and wal_level set to logical, plus a user with the REPLICATION attribute or rds_replication on RDS. On the Databricks side you need a SQL warehouse endpoint and storage for staging, which is where this differs from a warehouse that handles loading internally.

Question 2

What storage does a Databricks destination need?

Accepted Answer

Cloud storage you provide, because load files are written there first and then pulled in with COPY INTO. On AWS that is an S3 bucket with either an access key or an IAM role. On Azure it is an ADLS Gen2 compatible storage account and container with a key or SAS, read over abfss. There is no GCP destination path today, so a Google Cloud workspace can be read from but not written to. With IAM role staging, Databricks itself also needs to read the bucket through a storage credential or Unity Catalog external location.

Question 3

Do deletes from Postgres reach Databricks?

Accepted Answer

Yes with logical replication. The delete arrives as an event and the merge marks the row with a deleted at timestamp, so it stays queryable and you can see when it went. The propagate hard deletes option physically removes those rows after the load, and it requires the Polytomic timestamp columns to stay enabled since the delete keys off them. The fallback read paths behave differently: a tracking column read and an xmin read both simply stop returning the row.

Question 4

Can I partition the Databricks table by a Postgres timestamp?

Accepted Answer

Yes. Choose a datetime source field as the partition key and the table is created with PARTITIONED BY on that column, which matters on large event tables. Two things are not automatic: Z-order and liquid clustering are left entirely to you, so if a table needs either, apply it yourself after creation. Polytomic will not overwrite that on an ordinary incremental run.

Question 5

Why replicate an application database into Databricks?

Accepted Answer

Because analytical work and serving your product pull in opposite directions, and the data you want to join to was never in Postgres. ETL into a data warehouse shifts heavy aggregation off the server your users depend on. Databricks suits this particularly well when the same tables also feed notebooks or model training, since raw application data lands beside everything else in Delta rather than in a separate copy someone has to maintain.

Sync PostgreSQL to Databricks

Sync PostgreSQL to Databricks with Polytomic

Deletes arrive as events

Your storage, S3 or ADLS

Partition on a timestamp

Or sync Databricks to PostgreSQL

Trusted by businesses worldwide

Step-by-step integration guide

Connect PostgreSQL in Polytomic

Connect Databricks in Polytomic

Create a sync between PostgreSQL and Databricks

Map fields and enable the sync

Frequently asked questions

The sync and ETL platform that does it all

Powerful data sync

Advanced analytics

Enterprise security

Easy setup

Ready to connect PostgreSQL with Databricks?