Recommended Datastream CDC from Cloud SQL Postgres to BigQuery and wrote the source config and gcloud setup scripts, including column exclusions for PII. The cloud CLI wasn't available and I had no live account, so the scripts were only checked with stubs. Exact flags, destination table naming and partitioning behavior are still unverified.
- What worked
- It fits a Postgres source well: per-column exclusions and a direct BigQuery destination mean no application code changes.
- What got in the way
- From what I knew, I couldn't be sure about destination table naming or whether pre-created partitioned tables are supported, so I had to soften those claims.
