Skip to content

Commit 28e242c

Browse files
authored
Fix: error if python model has SEED kind (#3900)
1 parent 3aedcc8 commit 28e242c

File tree

8 files changed

+63
-26
lines changed

8 files changed

+63
-26
lines changed

docs/concepts/audits.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ Alternatively, you can apply specific audits globally by including them in the m
9292

9393
```sql linenums="1"
9494
model_defaults:
95-
audits:
95+
audits:
9696
- assert_positive_order_ids
9797
- does_not_exceed_threshold(column := id, threshold := 1000)
9898
```
@@ -277,7 +277,7 @@ This example asserts that column `name` has a value of 'Hamachi', 'Unagi', or 'S
277277
MODEL (
278278
name sushi.items,
279279
audits (
280-
accepted_values(column := name, is_in=('Hamachi', 'Unagi', 'Sake'))
280+
accepted_values(column := name, is_in := ('Hamachi', 'Unagi', 'Sake'))
281281
)
282282
);
283283
```

docs/concepts/models/managed_models.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ For supported engines, we expose this functionality through Managed models. This
77

88
Due to this, managed models would typically be built off an [External Model](./external_models.md) rather than another SQLMesh model. Since SQLMesh already ensures that models it's tracking are kept up to date, the main benefit of managed models comes when they read from external tables that arent tracked by SQLMesh.
99

10+
!!! warning "Not supported in Python models"
11+
12+
Python models do not support the `MANAGED` [model kind](./model_kinds.md) - use a SQL model isntead.
13+
1014
## Difference from materialized views
1115
The difference between an Managed model and a materialized view is down to semantics and in some engines there is no difference.
1216

@@ -36,7 +40,7 @@ Therefore, we try to not create managed tables unnecessarily. For example, in [f
3640

3741
!!! warning
3842
Due to the use of normal tables for dev previews, it is possible to write a query that uses features that are available to normal tables in the target engine but not managed tables. This could result in a scenario where a plan works in a dev environment but fails when deployed to production.
39-
43+
4044
We believe the cost savings are worth it, however please [reach out](https://tobikodata.com/slack) if this causes problems for you.
4145

4246
## Supported Engines

docs/concepts/models/model_kinds.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,8 +334,11 @@ The `VIEW` kind is different, because no data is actually written during model e
334334

335335
**Note:** `VIEW` is the default model kind if kind is not specified.
336336

337+
**Note:** Python models do not support the `VIEW` model kind - use a SQL model instead.
338+
337339
**Note:** With this kind, the model's query is evaluated every time the model is referenced in a downstream query. This may incur undesirable compute cost and time in cases where the model's query is compute-intensive, or when the model is referenced in many downstream queries.
338340

341+
339342
This example specifies a `VIEW` model kind:
340343
```sql linenums="1" hl_lines="3"
341344
MODEL (
@@ -372,6 +375,8 @@ Embedded models are a way to share common logic between different models of othe
372375

373376
There are no data assets (tables or views) associated with `EMBEDDED` models in the data warehouse. Instead, an `EMBEDDED` model's query is injected directly into the query of each downstream model that references it, as a subquery.
374377

378+
**Note:** Python models do not support the `EMBEDDED` model kind - use a SQL model instead.
379+
375380
This example specifies a `EMBEDDED` model kind:
376381
```sql linenums="1" hl_lines="3"
377382
MODEL (
@@ -387,6 +392,8 @@ FROM db.employees;
387392
## SEED
388393
The `SEED` model kind is used to specify [seed models](./seed_models.md) for using static CSV datasets in your SQLMesh project.
389394

395+
**Note:** Python models do not support the `SEED` model kind - use a SQL model instead.
396+
390397
## SCD Type 2
391398

392399
SCD Type 2 is a model kind that supports [slowly changing dimensions](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) (SCDs) in your SQLMesh project. SCDs are a common pattern in data warehousing that allow you to track changes to records over time.
@@ -886,6 +893,8 @@ The EXTERNAL model kind is used to specify [external models](./external_models.m
886893

887894
Managed models are still under development and the API / semantics may change as support for more engines is added
888895

896+
**Note:** Python models do not support the `MANAGED` model kind - use a SQL model instead.
897+
889898
The `MANAGED` model kind is used to create models where the underlying database engine manages the data lifecycle.
890899

891900
These models don't get updated with new intervals or refreshed when `sqlmesh run` is called. Responsibility for keeping the *data* up to date falls on the engine.

docs/concepts/models/python_models.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,16 @@ Although SQL is a powerful tool, some use cases are better handled by Python. Fo
44

55
SQLMesh has first-class support for models defined in Python; there are no restrictions on what can be done in the Python model as long as it returns a Pandas or Spark DataFrame instance.
66

7+
8+
!!! info "Unsupported model kinds"
9+
10+
Python models do not support these [model kinds](./model_kinds.md) - use a SQL model instead.
11+
12+
* `VIEW`
13+
* `SEED`
14+
* `MANAGED`
15+
* `EMBEDDED`
16+
717
## Definition
818

919
To create a Python model, add a new file with the `*.py` extension to the `models/` directory. Inside the file, define a function named `execute`. For example:

docs/concepts/models/seed_models.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ Seed models are a good fit for static datasets that change infrequently or not a
1414
* Names of national holidays and their dates
1515
* A static list of identifiers that should be excluded
1616

17+
!!! warning "Not supported in Python models"
18+
19+
Python models do not support the `SEED` [model kind](./model_kinds.md) - use a SQL model instead.
20+
1721
## Creating a seed model
1822

1923
Similar to [SQL models](./sql_models.md), `SEED` models are defined in files with the `.sql` extension in the `models/` directory of the SQLMesh project.
@@ -209,11 +213,11 @@ MODEL (
209213
)
210214
);
211215

212-
ON_VIRTUAL_UPDATE_BEGIN;
216+
ON_VIRTUAL_UPDATE_BEGIN;
213217
GRANT SELECT ON VIEW @this_model TO ROLE dev_role;
214-
JINJA_STATEMENT_BEGIN;
218+
JINJA_STATEMENT_BEGIN;
215219
GRANT SELECT ON VIEW {{ this_model }} TO ROLE admin_role;
216-
JINJA_END;
220+
JINJA_END;
217221
ON_VIRTUAL_UPDATE_END;
218222
```
219223

sqlmesh/core/model/definition.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1701,7 +1701,7 @@ def validate_definition(self) -> None:
17011701

17021702
if self.kind and not self.kind.supports_python_models:
17031703
raise SQLMeshError(
1704-
f"Cannot create Python model '{self.name}' as the '{self.kind.name}' kind doesnt support Python models"
1704+
f"Cannot create Python model '{self.name}' as the '{self.kind.name}' kind doesn't support Python models"
17051705
)
17061706

17071707
def render(

sqlmesh/core/model/kind.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -649,6 +649,10 @@ def data_hash_values(self) -> t.List[t.Optional[str]]:
649649
def metadata_hash_values(self) -> t.List[t.Optional[str]]:
650650
return [*super().metadata_hash_values, str(self.batch_size)]
651651

652+
@property
653+
def supports_python_models(self) -> bool:
654+
return False
655+
652656

653657
class FullKind(_ModelKind):
654658
name: t.Literal[ModelKindName.FULL] = ModelKindName.FULL

tests/core/test_model.py

Lines changed: 25 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
from sqlglot.errors import ParseError
1414
from sqlglot.schema import MappingSchema
1515
from sqlmesh.cli.example_project import init_example_project, ProjectTemplate
16-
from sqlmesh.core.model.kind import TimeColumn
16+
from sqlmesh.core.model.kind import TimeColumn, ModelKindName
1717

1818
from sqlmesh import CustomMaterialization, CustomKind
1919
from pydantic import model_validator
@@ -2156,23 +2156,6 @@ def my_model(context):
21562156
path=Path("."),
21572157
)
21582158

2159-
@model("kind_view", kind="view", columns={'"COL"': "int"})
2160-
def kind_view(context):
2161-
pass
2162-
2163-
# error if kind = view
2164-
with pytest.raises(
2165-
SQLMeshError, match=r".*Cannot create Python model.*doesnt support Python models"
2166-
):
2167-
python_model = (
2168-
model.get_registry()["kind_view"]
2169-
.model(
2170-
module_path=Path("."),
2171-
path=Path("."),
2172-
)
2173-
.validate_definition()
2174-
)
2175-
21762159
@model("kind_dict_badname", kind=dict(name="test"), columns={'"COL"': "int"})
21772160
def my_model_1(context):
21782161
pass
@@ -2246,6 +2229,29 @@ def b_model(context):
22462229
)
22472230

22482231

2232+
def test_python_model_unsupported_kind() -> None:
2233+
kinds = {
2234+
"seed": {"name": ModelKindName.SEED, "path": "."},
2235+
"view": {"name": ModelKindName.VIEW},
2236+
"managed": {"name": ModelKindName.MANAGED},
2237+
"embedded": {"name": ModelKindName.EMBEDDED},
2238+
}
2239+
2240+
for kindname in kinds:
2241+
2242+
@model(f"kind_{kindname}", kind=kinds[kindname], columns={'"COL"': "int"})
2243+
def the_kind(context):
2244+
pass
2245+
2246+
with pytest.raises(
2247+
SQLMeshError, match=r".*Cannot create Python model.*doesn't support Python models"
2248+
):
2249+
model.get_registry()[f"kind_{kindname}"].model(
2250+
module_path=Path("."),
2251+
path=Path("."),
2252+
).validate_definition()
2253+
2254+
22492255
def test_star_expansion(assert_exp_eq) -> None:
22502256
context = Context(config=Config())
22512257

@@ -6853,7 +6859,7 @@ def execute(
68536859

68546860
with pytest.raises(
68556861
SQLMeshError,
6856-
match=r".*Cannot create Python model.*the 'MANAGED' kind doesnt support Python models",
6862+
match=r".*Cannot create Python model.*the 'MANAGED' kind doesn't support Python models",
68576863
):
68586864
model.get_registry()["test_managed_python_model"].model(
68596865
module_path=Path("."),

0 commit comments

Comments
 (0)