Skip to content

refactor: remove data_sources_expanded view (#845)#956

Open
maxachis wants to merge 1 commit intodevfrom
refactor/845-remove-data-sources-expanded
Open

refactor: remove data_sources_expanded view (#845)#956
maxachis wants to merge 1 commit intodevfrom
refactor/845-remove-data-sources-expanded

Conversation

@maxachis
Copy link
Copy Markdown
Contributor

@maxachis maxachis commented May 2, 2026

Summary

Closes #845.

Drops the data_sources_expanded SQL view (and its DataSourceExpanded SQLAlchemy shadow) so future data_sources schema changes no longer require dropping/recreating a view in every migration.

The view's only added value over the data_sources table was the joined record_types.name column. That is now exposed via a SQLAlchemy column_property on DataSource that issues a correlated subquery against record_types, producing the same record_type_name value with the same LEFT JOIN semantics (NULL when record_type_id is unset).

Replacements by call site

  • db/models/implementations/core/data_source/core.py — added agencies and data_requests relationships (moved off the deleted DataSourceExpanded) and the record_type_name column_property.
  • db/client/core.py, db/dynamic_query_constructor.py, endpoints/instantiations/data_sources_/get/{by_id,by_id/agencies,many}/query.py, endpoints/instantiations/data_sources_/get/convert.py — all select(DataSourceExpanded) / selectinload(DataSourceExpanded.*) switched to the DataSource model. No query semantics change; the column_property injects the same LEFT JOIN record_types projection.
  • db/subquery_logic.py, middleware/primary_resource_logic/{data_sources.py,data_requests_/related/related_source/get.py}Relations.DATA_SOURCES_EXPANDED references repointed to Relations.DATA_SOURCES.
  • db/models/implementations/core/{agency/core.py,data_request/expanded.py}data_sources relationships now point at DataSource instead of DataSourceExpanded.
  • db/models/table_reference.py, middleware/enums.py, middleware/column_permission/mapping.py — removed data_sources_expanded registrations; merged record_type_name into the data_sources permission entry (same role/permission settings as the deleted data_sources_expanded entry).
  • tests/utilities/test_result_formatter.py — instantiates DataSource directly.
  • New migration alembic/versions/2026_05_02_0839-a3fcd7a36a70_drop_data_sources_expanded_view.py drops the view in upgrade() and recreates it in downgrade() for reversibility.

Test plan

  • uv run ruff check . — clean
  • uv run basedpyright --level error — 0 errors
  • uv run alembic upgrade head then downgrade -1 then upgrade head against a local PG15 — view drops and re-creates as expected
  • uv run pytest tests/ (all sub-suites) — 257 passed locally; same tests_comprehensive failures (11 fail / 7 error) exist on dev baseline and are unrelated
  • CI run_pytest.yml against full DB

Risk areas for reviewer

  • The column_property runs a correlated scalar subquery for record_type_name whenever it is loaded. For the existing call sites this matches the LEFT JOIN the view used. If any caller did data_sources_expanded.record_type_name filtering in a WHERE clause it would still work (SQLAlchemy will inline the subquery), but no such filter exists in the current codebase — worth a quick second look.
  • The DataSourceExpandedDTO / DataSourceExpandedSchema names were left in place since they describe the API output schema, not the database view. Renaming is out of scope.

🤖 Generated with Claude Code

Drops the data_sources_expanded SQL view and its DataSourceExpanded ORM
shadow, replacing them with a column_property on DataSource that issues
an equivalent correlated subquery for record_type_name. The agencies and
data_requests relationships move onto DataSource directly. Removing the
view eliminates the migration friction it caused (every data_sources
schema change required dropping/recreating the view).

Closes #845

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant