Question 1

Why export from BigQuery to a bucket in the same cloud?

Accepted Answer

Because plenty of things cannot query a data warehouse but can read a file. A partner who needs a weekly extract, a processing job that expects objects, a machine learning pipeline, or an archive kept on your own retention terms. Staying inside the same cloud means no cross region egress and no second set of credentials, so this tends to be the cheapest and simplest export path available.

Question 2

What lands in the bucket on each run?

Accepted Answer

That depends on the mode. Replicate writes the complete dataset to one stable object and replaces it every run, so a consumer can point at a fixed path forever. Snapshot writes a new timestamped object per run, building history. There is also an incremental option on replicate that writes only changed records as new timestamped files, which suits feeding something that consumes deltas. Incremental cannot be combined with snapshot.

Question 3

Does this work for very large tables?

Accepted Answer

Yes, and there is a mode built for it. Reads normally run as a query job whose results stream through the Storage Read API, with transparent REST fallback. For large bulk syncs you can additionally enable Extract, which has BigQuery export Snappy compressed Avro shards to a Cloud Storage bucket you configure, then reads those shards in parallel and cleans them up afterwards. Two things to be clear about: it is opt in rather than triggered automatically by size, and it is available on bulk syncs only. It improves throughput rather than reducing the bytes BigQuery scans.

Question 4

How do I organise the output?

Accepted Answer

Through expressions rather than a partitioning setting. The path is the bucket and prefix on the connection, then an optional subfolder, then the output name, and both the subfolder and name are expression evaluated. A datetime expression produces dated folders automatically as runs happen. This is path templating, so it is not the same as partitioning on a value in the data, and there is no compression codec setting.

Connect BigQuery with Google Cloud Storage

Connect BigQuery and Google Cloud Storage with Polytomic

Same cloud, same project

Built for large extracts

Parquet keeps types

Trusted by businesses worldwide

Step-by-step integration guide

Connect BigQuery in Polytomic

Connect Google Cloud Storage in Polytomic

Create a sync between BigQuery and Google Cloud Storage

Map fields and enable the sync

Frequently asked questions

The sync and ETL platform that does it all

Powerful data sync

Advanced analytics

Enterprise security

Easy setup

Ready to connect BigQuery with Google Cloud Storage?