feat(ol-dbt-cli): add SQL/YAML linting and column-level impact analysis CLI

[blarghmatey]

Summary

Adds a new ol-dbt CLI package at src/ol_dbt_cli/ with two commands for the engineering team to use with the src/ol_dbt dbt project.

Commands

ol-dbt validate

Structural consistency checks for dbt models:

Check	Level	Description
YAML/SQL sync	ERROR	Columns in YAML but absent from SQL output, or vice-versa
Missing upstream refs	ERROR	ref()/source() calls that don't resolve to known models
Upstream column existence	ERROR	Column referenced from upstream doesn't exist in that model
Duplicate aliases	ERROR	Repeated column alias within a single SELECT
SELECT *	INFO/WARN	Unresolved star expansion (use --warn-select-star to promote)

Options:

ol-dbt validate [--model MODEL]... [--errors-only] [--warn-select-star]
                [--compiled-dir PATH] [--output text|json]

• --model/-m — filter by model name(s), comma-separated list, or directory under models/
• --errors-only — suppress INFO/WARN, show only errors
• --warn-select-star — treat unresolved SELECT * as warnings
• --compiled-dir — use target/compiled/ SQL for Jinja-heavy models

ol-dbt impact

Column-level downstream impact analysis driven by git diffs:

ol-dbt impact [--base BRANCH] [--staged] [--compiled-dir PATH] [--output text|json]

• Reads git diff [--cached] to find changed/removed SQL column aliases
• Traverses upstream → downstream to find all models referencing each changed column
• Reports broken references, risky usages, and models needing review

Jinja pre-processing

All 588 dbt models in this project parse successfully via a regex-based Jinja stripper that handles:

• {{ ref() }} / {{ source() }} → stable SQL identifiers with reverse maps
• All {{ ... }} → unquoted placeholders (fixes doubled-quote bug: like '{{ var }}' → like '__jinja__' not like ''__jinja__'')
• Block-level macro calls alone on a line (e.g. {{ deduplicate_raw_table(...) }}) → SQL comment
• Macros that split a SQL column expression across the }} boundary (e.g. {{ array_join('partial SQL', "path") }}) → collapsed to __jinja__ as alias via post-cleanup regex pass

YAML registry improvements

• Parses both models: and sources: blocks for SELECT * column resolution
• Rescues model definitions accidentally nested inside another model's columns: list (authoring error in _stg_mitlearn_models.yml)

Tests

85 pytest tests covering all commands, library modules, and edge cases.

[Copilot]

Pull request overview

Introduces a new ol-dbt CLI workspace package (src/ol_dbt_cli/) intended to help engineers validate dbt model/YAML consistency and assess column-level downstream impact of in-progress changes in src/ol_dbt.

Changes:

• Adds ol-dbt validate and ol-dbt impact commands (cyclopts + rich output) plus supporting parsing/registry libraries.
• Implements YAML schema registry + sqlglot-based SQL parsing with Jinja stripping and partial SELECT * resolution.
• Adds a pytest suite covering registry/parsing/validation/impact helpers and wires the package into the uv workspace/dependencies.

Reviewed changes

Copilot reviewed 13 out of 18 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
uv.lock	Adds ol-dbt-cli workspace member and locks new deps (rich, sqlglot).
pyproject.toml	Adds src/ol_dbt_cli to uv workspace members and adds rich/sqlglot to root deps.
src/ol_dbt_cli/pyproject.toml	Declares the ol-dbt-cli package, deps, entrypoint, and ruff config.
src/ol_dbt_cli/ol_dbt_cli/init.py	Package marker.
src/ol_dbt_cli/ol_dbt_cli/cli.py	Defines ol-dbt CLI app and registers impact/validate commands.
src/ol_dbt_cli/ol_dbt_cli/commands/init.py	Commands package marker.
src/ol_dbt_cli/ol_dbt_cli/commands/validate.py	Implements structural validation checks and reporting/output.
src/ol_dbt_cli/ol_dbt_cli/commands/impact.py	Implements diff-based column change detection and downstream impact reporting.
src/ol_dbt_cli/ol_dbt_cli/lib/init.py	Lib package marker.
src/ol_dbt_cli/ol_dbt_cli/lib/yaml_registry.py	Parses dbt schema YAML into a registry of models/sources/columns.
src/ol_dbt_cli/ol_dbt_cli/lib/sql_parser.py	Strips Jinja and uses sqlglot to extract output columns (+ some star resolution).
src/ol_dbt_cli/ol_dbt_cli/lib/manifest.py	Parses dbt manifest.json into a registry with lineage helpers.
src/ol_dbt_cli/ol_dbt_cli/lib/git_utils.py	Git helpers for changed files/models and retrieving file content at refs.
src/ol_dbt_cli/tests/init.py	Tests package marker.
src/ol_dbt_cli/tests/test_yaml_registry.py	Tests YAML registry model/source parsing and nested-model rescue behavior.
src/ol_dbt_cli/tests/test_validate.py	Tests validate checks, formatting, and model target resolution helpers.
src/ol_dbt_cli/tests/test_sql_parser.py	Tests Jinja stripping and SQL parsing/star handling/compiled-dir logic.
src/ol_dbt_cli/tests/test_impact.py	Tests column diff helper used by impact analysis.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[rachellougee]

ol-dbt validate works well, as it detects the inconsistencies between sql and yml file.

ol-dbt impact has some limitations. It relies on the dbt compiled folder, so if downstream models have never been compiled locally, the result may be incomplete. So maybe we could add something like dbt compile --select model+ in the impact.py to ensure all downstream models are compiled before analysis?

Also, when I tested by renaming the user_email column in the stg__edxorg__program_entitlement, ol-dbt impact detected one downstream edxorg_to_mitxonline_users impact, but it missed the user_email reference in edxorg_to_mitxonline_program_entitlements. Not sure why it would miss that because both downstream models were compiled locally.

% ol-dbt impact
Using compiled SQL: /Users/rachellougee/repo/OL/ol-data-platform/src/ol_dbt/target/compiled/open_learning/models
Using manifest: /Users/rachellougee/repo/OL/ol-data-platform/src/ol_dbt/target/manifest.json

Analysing 1 model(s) for column-level impact

 BREAKING stg__edxorg__program_entitlement
   Column change(s) in 'stg__edxorg__program_entitlement' will break 1 downstream model(s).
   Column changes:
     🔄 user_email → user_emails
     →  user_emails
   Downstream impact:
       ↳ edxorg_to_mitxonline_users (uses: user_email)
       ↳ not_null_stg__edxorg__program_entitlement_program_title (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_program_type (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_organization_key (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_user_id (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_user_emails (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_purchase_date (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_expiration_date (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_number_of_entitlements (column metadata unavailable)
       ↳ not_null_stg__edxorg__program_entitlement_number_of_redeemed_entitlements (column metadata unavailable)
       ↳ dbt_expectations_expect_compound_columns_to_be_unique_stg__edxorg__program_entitlement_user_email__program_title (column metadata unavailable)
         ↳ not_null_edxorg_to_mitxonline_users_user_email (column metadata unavailable)
         ↳ unique_edxorg_to_mitxonline_users_user_email (column metadata unavailable)
         ↳ dbt_expectations_expect_compound_columns_to_be_unique_edxorg_to_mitxonline_program_entitlements_program_title__user_edxorg_email__order_id_is_null 
(column metadata unavailable)

Summary: 1 breaking, 0 warning(s) across 1 changed model(s)