Question 1

What does crossing clouds mean in practice?

Accepted Answer

Two things. First, credentials: a Google service account key, or workload identity federation, reads BigQuery, while separate AWS credentials or an IAM role write to the bucket, and neither side is passed to the other. Second, egress. Data leaves Google and lands in AWS, so cross cloud transfer costs apply on top of the bytes BigQuery scans. Worth costing out before scheduling a large table frequently.

Question 2

Which format should I use?

Accepted Answer

Parquet for anything that will be queried later, because it carries types and reads efficiently by column, so a reader on the AWS side does not have to reinterpret every value. That matters more here than in a same cloud export, since the consumer is usually a different stack entirely. JSON Lines keeps nested values intact for record by record processing. CSV is the compatibility choice and loses type information.

Question 3

How do I keep the BigQuery side cheap?

Accepted Answer

Narrow what each run reads, since on demand pricing bills for bytes scanned every time. Set a tracking field, which incremental reads need together with a unique identity, and partition and ideally cluster the table so the predicate prunes. Select only the columns you export and avoid SQL that defeats pruning. For large bulk reads there is an optional Extract mode that stages Avro shards through a Cloud Storage bucket and reads them in parallel, which helps throughput, though it does not reduce the bytes BigQuery scans and adds temporary storage of its own.

Question 4

What happens to deleted rows in an incremental export?

Accepted Answer

They can appear as tombstone records carrying the key and a deletion timestamp, rather than being edited out of files already written, which is not possible once a file is closed. A consumer reading the full set of incremental files has to apply those tombstones to arrive at current state. If that is unwelcome downstream, a scheduled replicate or snapshot run gives a complete picture instead.

Connect BigQuery with Amazon S3

Connect BigQuery and Amazon S3 with Polytomic

Two sets of credentials

Types survive in Parquet

Replicate or snapshot

Trusted by businesses worldwide

Step-by-step integration guide

Connect BigQuery in Polytomic

Connect Amazon S3 in Polytomic

Create a sync between BigQuery and Amazon S3

Map fields and enable the sync

Frequently asked questions

The sync and ETL platform that does it all

Powerful data sync

Advanced analytics

Enterprise security

Easy setup

Ready to connect BigQuery with Amazon S3?