Question 1

How does change capture work?

Accepted Answer

The first ETL run reads the table. After that Polytomic consumes DynamoDB Streams shard by shard, checkpointing on sequence numbers. Streams have to be enabled with NEW_IMAGE or NEW_AND_OLD_IMAGES. KEYS_ONLY and OLD_IMAGE are not sufficient, and a sync pointed at a table configured either of those ways will not work.

Question 2

What happens if the sync falls behind?

Accepted Answer

This is the constraint to plan around. DynamoDB stream retention is twenty four hours. Checkpoints let Polytomic resume, but if the stream ages out past the point it needs, the gap is unrecoverable and a full resync is required. That window is far shorter than a PostgreSQL write ahead log or a MySQL binlog you control yourself, so a failure over a weekend is a real risk worth alerting on.

Question 3

Do I have to configure streams on every table?

Accepted Answer

Not if you let Polytomic manage them, in which case it enables and reconfigures streams through UpdateTable. Without that, you configure each selected table yourself. The trade is that letting Polytomic manage streams means granting dynamodb:UpdateTable, so the policy stops being strictly read only.

Question 4

Do deletes reach BigQuery?

Accepted Answer

Yes. REMOVE events become deletes, though they carry only the key attributes and are marked as partial records, because DynamoDB does not include the rest of the item on a delete. That is enough to remove or tombstone the row, and not enough to know what it contained.

Question 5

What does the IAM policy need?

Accepted Answer

For reads: BatchGetItem, Describe*, List*, GetItem, Query, Scan and PartiQLSelect. For streams: GetRecords, GetShardIterator, DescribeStream and ListStreams, scoped to the table's stream ARN. Add UpdateTable only if Polytomic manages stream settings. Role based authentication also needs an sts:AssumeRole trust policy, ideally pinned with the External ID Polytomic generates.

Sync DynamoDB to BigQuery

Sync DynamoDB to BigQuery with Polytomic

Streams after the first load

Twenty four hour window

Deletes carried

Trusted by businesses worldwide

Step-by-step integration guide

Connect DynamoDB in Polytomic

Connect BigQuery in Polytomic

Create a sync between DynamoDB and BigQuery

Map fields and enable the sync

Frequently asked questions

The sync and ETL platform that does it all

Powerful data sync

Advanced analytics

Enterprise security

Easy setup

Ready to connect DynamoDB with BigQuery?