DAT2-192/upgrade-dbt-version-1.11

[mattromano]

This command will auto install into all repos and run autofix tool. Only need to update requirements.txt and package tag locally.

 # ============================================
  # DBT 1.11 UPGRADE SCRIPT
  # ============================================
  # Pre-requisites (manual updates before running):
  # 1. Update requirements.txt with:
  #    dbt-core>=1.11.0,<1.12.0
  #    dbt-snowflake>=1.11.0,<1.12.0
  #    protobuf>=6.0,<7.0
  # 2. Update packages.yml fsc-evm revision to new tag (e.g., v1.40.0)
  # ============================================

  # Remove existing virtual environment if it exists
  rm -rf dbt-env && \

  # Create a new Python virtual environment
  python3 -m venv dbt-env && \

  # Activate the virtual environment
  source dbt-env/bin/activate && \

  # Upgrade pip to the latest version
  pip install --upgrade pip && \

  # Install all required Python packages from requirements.txt
  pip install -r requirements.txt && \

  # Deactivate and reactivate to ensure clean state
  deactivate && source dbt-env/bin/activate && \

  # Check dbt version to verify installation
  dbt --version && \

  # Clean install packages (critical: must delete package-lock.yml)
  rm -f package-lock.yml && dbt clean && dbt deps && \

  # Run dbt-autofix to automatically fix deprecations (arguments wrapper, etc.)
  uvx --from git+https://github.com/dbt-labs/dbt-autofix.git dbt-autofix deprecations && \

  # Fix dbt-autofix issues with custom test configs (partition_by should NOT be in config block)
  # This reverts the incorrect nesting that dbt-autofix does for custom macro arguments
  find models -name "*.yml" -type f -exec grep -l "fsc_evm.sequence_gaps\|fsc_evm.balances_sequence_gaps" {} \; | while read f; do
    sed -i '' 's/config:/partition_by:/g; s/partition_by:/partition_by:/g' "$f" 2>/dev/null || true
  done && \

  # Replace deprecated "-m " with "--select " in all workflow YAML files
  find .github/workflows -type f -name "*.yml" -exec sed -i '' 's/ -m / --select /g' {} + && \

  # Replace "-m " with "--select " in the Makefile (if exists)
  [ -f Makefile ] && sed -i '' 's/ -m / --select /g' Makefile || true && \

  # Parse to validate changes
  dbt parse && \

  echo "✅ Upgrade complete! Review changes before committing."

Changes:

1. All .ymls: argument configs (optional, but required in future) - the dbt-autofix that handles this has some issues with custom test configs so potentially may need to implement the fix from this commit here: 34d1f2f
2. Requirements.txt: Adds new DBT versions and other requirements. DBT install is now in two parts, one being core, the other being the snowflake specific adapter.
3. dbt_project.yml: Added behavior flags in +meta section: disallow-spaces-in-resource-names: true and source-freshness-run-project-hooks: true
4. dbt_project.yml: Added behavior flags in +meta section: disallow-spaces-in-resource-names: true and source-freshness-run-project-hooks: true
5. dbt_project.yml: Removed target-path (now default)
6. GHA Workflows: commands from -m to --select
7. Need to make sure custom_naming_macros.sql is updated to account for model naming changes in dbt 1.8. Otherwise models with multiple ephemeral model refs will resolve all ephemeral CTEs to __dbt__cte__ and overwrite each other. Commit: b87c064
8. Custom config variables, like score versioning, will need to be updated to the syntax in the commit here:652c838
9. incremental_predicates.sql macro will need to be update to this commit here:a32a98e (ignore get_merge_sql.sql changes)

Most changes can be automated via this tool here.

Additional Potential Changes:

1. We may want to also update DBT in livequery-models and fsc-utils repos - those are pretty old at v1.5 DBT. We do currently have udf tests in there that give deprecation warnings when pulled (Just warnings)
2. Update deprecated packages - expectations package has been deprecated for this one here. This update would need to happen in livequery-models as that's the furthest upstream install of the package.

Gnosis Testing Updates:

1. Run insert statment and kick history when updating to this PR here: gnosis/v4.41.0 gnosis-models#256