Question 1

Which file formats and compression are supported?

Accepted Answer

CSV, JSON documents, JSONL or newline delimited JSON, and Parquet. External compression covers gzip, bzip2 and Zstandard. Externally compressed Parquet is rejected, because the reader needs to seek within the file. There is no snappy wrapper support, though Parquet internal codecs are handled by the Parquet reader itself. Avro is not available as an S3 source format despite Avro appearing elsewhere in the codebase as a writer.

Question 2

How do files map to Redshift tables?

Accepted Answer

Three ways. A single file can become one table. All files matching a pattern can be concatenated into one configured table, which is the usual arrangement for partitioned exports. Directory and expression patterns can produce multiple tables at once, with matching files concatenated per table. Concatenated files need compatible schemas. Worth remembering that S3 has no real directories outside directory buckets, so patterns operate on key prefixes rather than folders.

Question 3

Is ingestion event driven or scheduled?

Accepted Answer

Both paths exist, and the distinction matters. The default is a scheduled listing and scan of the bucket. The optional incremental path uses S3 Event Notifications delivered to an SQS queue that Polytomic polls and drains. An event does not directly trigger a sync run. The integration is specifically with SQS rather than SNS or EventBridge, and notifications are unordered and at least once, so queue retention needs to exceed any outage you might have.

Question 4

Do we need a second S3 bucket for the Redshift side?

Accepted Answer

You need a staging bucket in the same region as your Redshift cluster, and it is worth keeping it separate from your source bucket to avoid the sync reading its own staged output. Every Redshift write stages files to S3 and then runs COPY, so there is no path that avoids staging. Polytomic needs read and write access on the staging bucket and cleans up staged files after a successful run, though a lifecycle rule is still sensible. That staging step happens inside every ETL run rather than being something you schedule separately.

Question 5

Why route files through an ETL pipeline rather than loading them directly?

Accepted Answer

Because a bucket full of exports is not a dataset until something gives it structure, tracks what has already been loaded, and keeps up when the shape changes. Hand written COPY jobs tend to work until a partition is late or a column appears. ETL into a data warehouse handles the discovery, typing and incremental bookkeeping, and turns path conventions into real columns so the partitioning you already have becomes queryable.

Sync Amazon S3 to Redshift

Sync Amazon S3 to Redshift with Polytomic

Four formats, three compressions

Path segments become columns

Scheduled scan or SQS events

Or sync Redshift to Amazon S3

Trusted by businesses worldwide

Step-by-step integration guide

Connect Amazon S3 in Polytomic

Connect Redshift in Polytomic

Create a sync between Amazon S3 and Redshift

Map fields and enable the sync

Frequently asked questions

The sync and ETL platform that does it all

Powerful data sync

Advanced analytics

Enterprise security

Easy setup

Ready to connect Amazon S3 with Redshift?