docs: add example BigQuery E2E

[geoHeil]

Summary

• add docs example for how to use bigquery with alembic
• add workaround example for smooth DDL generation

Changelog

• add docs example for how to use bigquery with alembic

Test Plan

• tested with my own BQ account and the included tests

[geoHeil]

• docs: add example BigQuery E2E #1072 👈 (View in Graphite)
• fix: update devenv.lock #1173 : 2 other dependent PRs (#1136 , #1174 )
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• graphite:merge - adds this PR to the back of the merge queue
• graphite:hotfix - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

[geoHeil]

We need some BQ creds for the CI

[geoHeil]

Both table creation modes work end-to-end against my own BQ project.

Mode	How	Result
auto_create_tables	Ibis creates tables on first write	Works
Alembic migrations	alembic revision --autogenerate + upgrade head	Works

What this PR adds

Example (examples/example-bigquery/):

• SQLModel features (BaseSQLModelFeature with table=True)
• Pipeline with resolve_update + write
• Dual-track Alembic setup (system tables + feature tables)
• env.py workarounds for BigQuery's OLAP limitations (see below)

Core (src/metaxy/ext/metadata_stores/bigquery.py):

• sqlalchemy_url property returning bigquery://project/dataset for Alembic

CI (tests/metadata_stores/):

• BigQuery integration test fixtures (BIGQUERY_TEST_PROJECT_ID, BIGQUERY_TEST_CREDENTIALS_PATH)
• Two live integration tests (write+read, idempotent resolve_update)
• Secrets wired through all CI workflows

BigQuery Alembic workarounds in env.py

Three issues with sqlalchemy-bigquery + Alembic required workarounds in the example's env.py files:

Issue	Workaround
CREATE INDEX not supported	include_object callback skips index/constraint generation
sa.JSON() → JSON column but Ibis writes STRUCT	_patch_bigquery_types() rewrites JSON→STRUCT using feature field names
DateTime(timezone=True) → DATETIME but Ibis writes TIMESTAMP	Same helper rewrites to sa.TIMESTAMP
Reflection crash on system tables	include_name scopes reflection to feature tables only

These workarounds live entirely in the example, not in Metaxy core. Upstream fix for sa.JSON() compilation submitted: googleapis/google-cloud-python#16124

[geoHeil]

cleaned up as discussed - secrets are needed now.

[Copilot]

Pull request overview

Adds BigQuery “native” integration testing support and tightens BigQuery store configuration/initialization behavior, along with CI wiring to run BigQuery tests when credentials are available.

Changes:

• Introduces BigQuery integration test fixtures and a shared test-suite-based BigQuery test class.
• Makes dataset_id required (config + runtime validation) and adds a sqlalchemy_url convenience property.
• Updates CI workflows and pytest markers to support gated BigQuery live tests.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/ext/bigquery/test_native.py	Adds shared integration test suite coverage for BigQuery; refactors/extends existing unit tests.
tests/ext/bigquery/conftest.py	Adds environment-gated BigQuery fixtures to create/delete ephemeral datasets.
src/metaxy/ext/metadata_stores/bigquery.py	Requires dataset_id, tightens init validation, adds sqlalchemy_url, adjusts docs and display output.
pyproject.toml	Adds bigquery pytest marker description.
devenv.lock	Updates Nix inputs/lock entries (git-hooks, flake-compat, etc.).
.github/workflows/_qa.yml	Adds optional BQ secrets and writes SA JSON to a temp credentials file for tests.
.github/workflows/main.yml	Passes BigQuery secrets through to the reusable QA workflow.
.github/workflows/release.yml	Passes BigQuery secrets through to the reusable QA workflow.
.github/workflows/Pull Request.yml	Passes BigQuery secrets through to the reusable QA workflow.

Comments suppressed due to low confidence (1)

src/metaxy/ext/metadata_stores/bigquery.py:133

• When connection_params is provided alongside explicit project_id/dataset_id, self.project_id/self.dataset_id are derived from the explicit args (due to the or), but the actual backend connection uses the unmodified connection_params. If they differ, display()/sqlalchemy_url can report a different project/dataset than the one actually used for connections. Fix by either (a) enforcing consistency (raise ValueError if both are provided and mismatch) or (b) overriding connection_params['project_id'] / connection_params['dataset_id'] with the resolved values before calling super().__init__.

        super().__init__(
            backend="bigquery",
            connection_params=connection_params,
            fallback_stores=fallback_stores,
            **kwargs,
        )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.