perf: skip unnecessary metadata fetch calls for tags when not configured by tejassp-db · Pull Request #1387 · databricks/dbt-databricks

tejassp-db · 2026-04-10T12:45:36Z

Summary

Skips fetch_tags and fetch_column_tags queries to information_schema during incremental and view materializations when the model has no tags configured
Falls back to fetching metadata when model config is unavailable (safe default)
No behavior change for models with tags configured — diffs still computed correctly

What changed

Added requires_server_metadata_for_diff() to TagsConfig and ColumnTagsConfig to signal when server metadata is needed
IncrementalTableAPI._describe_relation and ViewAPI._describe_relation now check model config before making tag fetch calls
get_relation_config accepts the model config to pass through to _describe_relation
Fixed duplicate TagsProcessor in StreamingTableConfig.config_components

Test plan

Unit tests: empty tags (skip), non-empty tags (fetch), null config fallback (fetch), both tags + column tags present, hive_metastore (skip)
Functional tests: override fetch_tags/fetch_column_tags macros to raise errors, confirming calls are actually skipped when expected

PECOBLR-2497

Skip fetch_tags and fetch_column_tags information_schema queries during incremental and view materializations when the model has no tags configured. This avoids unnecessary server roundtrips on every run for models that don't use tags, while preserving full fetch behavior when tags are present or when the model config is unavailable. PECOBLR-2497

github-actions · 2026-04-10T13:00:03Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
impl.py					967, 975-984, 1070-1072, 1103-1106, 1146, 1164-1165
dbt/adapters/databricks/relation_configs
base.py					41
column_tags.py
tags.py
Project Total

_{This report was generated by python-coverage-comment-action}

Add unit and functional coverage for skipping tag metadata queries when model config does not require them. This protects the new fetch-planning logic across incremental, view, streaming table, and materialized view test paths without changing unrelated unstaged work.

partial_parse.msgpack does not persist compiled_code, causing get_config_from_model to raise "Cannot compile model ... with no SQL query" for materialized view and streaming table change tests. Co-authored-by: Isaac

Co-authored-by: Isaac

The previous `dict and len(...) > 0` short-circuited to a dict literal when empty, breaking the declared `-> bool` return type. Co-authored-by: Isaac

sd-db

Overall intentation of the change looks good, there are a few things specifically where we need changes

Function signature - make new param optional + naming + other related changes
Coverage - Can you confirm on coverage across all supported materialisation types (MV/ST missing; Views don't have support for column_tags)
Functional test changes

sd-db · 2026-05-07T06:40:18Z

+            else:
+                results["information_schema.tags"] = None
+
+            column_tag_config = (


Similar comments on naming + checks on desried_column_tag_config

Suffix _config and surrounding code makes it obvious that its coming from the user configuration.

Variable naming should still be intentional and descriptive, we should update it to model_column_tag_config

- Add functional tests for optional tag fetching in Materialized view. - Fix tests to check exceptions, and not log lines. - Make parameter optional to be backward compatible with jinja calls.

sd-db

Added some more comments, overall much better. I still feel we should add a comment in all the materialisation types where we are doing this optimisation that this only works because we don't support deleting/removing tags through dbt-databricks, as that will break the optimisations done and better to encode as a comment

sd-db · 2026-05-07T10:36:38Z

-        results["information_schema.tags"] = adapter.execute_macro("fetch_tags", kwargs=kwargs)
        results["show_tblproperties"] = adapter.execute_macro("fetch_tbl_properties", kwargs=kwargs)
+
+        table_tag_config = model_config.config.get(TagsProcessor.name) if model_config else None


rename to model_table_tag_config we should differentiate between desired(model) configs and existing (relation) configs better

sd-db

Reviewing tests 1/2

sd-db · 2026-05-08T09:04:17Z

+
+from tests.functional.adapter.helpers import FAIL_IF_TAG_FETCH_CALLED_MACROS
+
+VIEW_WITHOUT_TAGS_SQL = """


we should move this inside fixtures file

sd-db · 2026-05-08T09:12:04Z

+"""
+
+
+def get_model_config(project, relation: BaseRelation):


Do we need this ? Since model_config is optional in the tests we can just not pass it and that also tests for other behaviour overall. Look at the places where this gets called it is most a no-op. I think this is mostly an artifact of the config not being Optional earlier. Since now it is we can look to remove as well.

Currently this is adding extra complexity while mostly being no-op

sd-db · 2026-05-08T09:18:20Z

        adapter.execute_macro.return_value = MagicMock()

-        results = MaterializedViewAPI._describe_relation(adapter, MagicMock())
+        results = MaterializedViewAPI._describe_relation(adapter, MagicMock(), Mock())


nit: while the change is okay, it would be better if just did not pass model_config here, i.e no change as model_config being None, triggers the actual tag fetch macro which is what the test is looking for. This test was added when there no optimisation to not fetch and better to test that path instead of the else path.

sd-db

Reviewing Tests 2/2

sd-db · 2026-05-08T10:49:52Z

            )


+class TestDescribeRelationMetadataFetchPlanning:


nit: These tests look really good, but we missed MV materealisation here which would have this even more complete. Fine without as well.

tejassp-db self-assigned this Apr 10, 2026

tejassp-db requested review from benc-db, jprakash-db and sd-db as code owners April 10, 2026 12:45

tejassp-db had a problem deploying to azure-prod April 10, 2026 12:45 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 10, 2026 12:45 — with GitHub Actions Inactive

tejassp-db temporarily deployed to azure-prod April 13, 2026 10:22 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 13, 2026 10:22 — with GitHub Actions Failure

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

815ff55

tejassp-db temporarily deployed to azure-prod April 15, 2026 05:25 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 15, 2026 05:25 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 15, 2026 05:25 — with GitHub Actions Inactive

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

7839417

tejassp-db temporarily deployed to azure-prod April 17, 2026 04:10 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 17, 2026 04:10 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 17, 2026 04:10 — with GitHub Actions Inactive

tejassp-db and others added 9 commits April 27, 2026 10:28

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

91e0dea

fix: backfill compiled_code from raw_code in MV/ST test helper

a84e0f0

partial_parse.msgpack does not persist compiled_code, causing get_config_from_model to raise "Cannot compile model ... with no SQL query" for materialized view and streaming table change tests. Co-authored-by: Isaac

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

51a495e

chore: ruff format pass

0ed403b

Co-authored-by: Isaac

fix: return bool from requires_server_metadata_for_diff

f4399cd

The previous `dict and len(...) > 0` short-circuited to a dict literal when empty, breaking the declared `-> bool` return type. Co-authored-by: Isaac

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

e6adab7

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

ba26eb1

Fix failing tests.

f985dc8

Fix name shadowing

616239f

sd-db requested changes May 7, 2026

View reviewed changes

fix review comments.

12e008a

- Add functional tests for optional tag fetching in Materialized view. - Fix tests to check exceptions, and not log lines. - Make parameter optional to be backward compatible with jinja calls.

sd-db reviewed May 7, 2026

View reviewed changes

Comment thread dbt/adapters/databricks/impl.py

Rename variable

adee055

sd-db reviewed May 7, 2026

View reviewed changes

sd-db self-requested a review May 8, 2026 08:31

tejassp-db added 2 commits May 8, 2026 15:49

Add test for cascading table tags config

c727e76

Clarify backward compatible behavior with comments

8b57e75

sd-db reviewed May 8, 2026

View reviewed changes


		from tests.functional.adapter.helpers import FAIL_IF_TAG_FETCH_CALLED_MACROS

		VIEW_WITHOUT_TAGS_SQL = """

		)


		class TestDescribeRelationMetadataFetchPlanning:

Conversation

tejassp-db commented Apr 10, 2026

Uh oh!

github-actions Bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Apr 10, 2026 •

edited

Loading