You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Docs: add info on specifying model kinds in python models (#1518)
* Elaborate on how to specify model kinds in python models
* Add model kind specification to python models concepts doc
* Streamline model configuration ref info
Copy file name to clipboardExpand all lines: docs/concepts/models/python_models.md
+38-7Lines changed: 38 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,6 +37,37 @@ The function takes an `ExecutionContext` that is able to run queries and to retr
37
37
38
38
If the function output is too large, it can also be returned in chunks using Python generators.
39
39
40
+
## `@model` specification
41
+
42
+
The arguments provided in the `@model` specification have the same names as those provided in a SQL model's `MODEL` DDL.
43
+
44
+
Most of the arguments are simply Python-formatted equivalents of the SQL version, but Python model `kind`s are specified with model kind objects. All model kind arguments are listed in the [models configuration reference page](../reference/model_configuration.md#model-kind-properties). A model's `kind` object must be imported at the beginning of the model definition file before use in the model specification.
This example demonstrates how to specify an incremental by time range model kind in Python:
58
+
59
+
```python linenums="1"
60
+
from sqlmesh import ExecutionContext, model
61
+
from sqlmesh.core.model import IncrementalByTimeRangeKind
62
+
63
+
@model(
64
+
"docs_example.incremental_model",
65
+
kind=IncrementalByTimeRangeKind(
66
+
time_column="ds"
67
+
)
68
+
)
69
+
```
70
+
40
71
## Execution context
41
72
Python models can do anything you want, but it is strongly recommended for all models to be [idempotent](../glossary.md#idempotency). Python models can fetch data from upstream models or even data outside of SQLMesh.
42
73
@@ -50,7 +81,7 @@ df = context.fetchdf("SELECT * FROM my_table")
50
81
In order to fetch data from an upstream model, you first get the table name using `context`'s `table` method. This returns the appropriate table name for the current runtime [environment](../environments.md):
Copy file name to clipboardExpand all lines: docs/guides/configuration.md
+57-55Lines changed: 57 additions & 55 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -775,6 +775,8 @@ Example configuration specifying a Postgres default connection, in-memory DuckDB
775
775
776
776
### Models
777
777
778
+
#### Model defaults
779
+
778
780
The `model_defaults` key is **required** and must contain a value for the `dialect` key. All SQL dialects [supported by the SQLGlot library](https://github.com/tobymao/sqlglot/blob/main/sqlglot/dialects/dialect.py) are allowed. Other values are set automatically unless explicitly overridden in the model definition.
779
781
780
782
All supported `model_defaults` keys are listed in the [models configuration reference page](../reference/model_configuration.md#model-defaults).
@@ -804,82 +806,82 @@ Example configuration:
804
806
)
805
807
```
806
808
807
-
#### Model Kind
808
-
The default model kind is 'view' unless overridden with the `kind` key. For more information, refer to [model kinds](../concepts/models/model_kinds.md).
809
+
The default model kind is `VIEW` unless overridden with the `kind` key. For more information on model kinds, refer to [model concepts page](../concepts/models/model_kinds.md).
809
810
810
-
Example:
811
+
#### Model Kinds
811
812
812
-
=== "YAML"
813
+
Model kinds are required in each model file's `MODEL` DDL statement. They may optionally be used to specify a default kind in the model defaults configuration key.
813
814
814
-
```yaml linenums="1"
815
-
model_defaults:
816
-
dialect: snowflake
817
-
kind: full
818
-
```
815
+
All model kind specification keys are listed in the [models configuration reference page](../reference/model_configuration.md#model-kind-properties).
819
816
820
-
=== "Python"
817
+
The `VIEW`, `FULL`, and `EMBEDDED` model kinds are specified by name only, while other models kinds require additional parameters and are provided with an array of parameters:
821
818
822
-
```python linenums="1"
823
-
from sqlmesh.core.config import Config, ModelDefaultsConfig
819
+
=== "YAML"
824
820
825
-
config = Config(
826
-
model_defaults=ModelDefaultsConfig(
827
-
dialect="snowflake",
828
-
kind="full",
829
-
),
830
-
)
831
-
```
821
+
`FULL` model only requires a name:
832
822
833
-
If a kind requires additional parameters it can be provided as an object:
823
+
```sql linenums="1"
824
+
MODEL(
825
+
name docs_example.full_model,
826
+
kind FULL
827
+
);
828
+
```
834
829
835
-
=== "YAML"
830
+
`INCREMENTAL_BY_TIME_RANGE` requires an array specifying the model's `time_column`:
836
831
837
-
```yaml linenums="1"
838
-
model_defaults:
839
-
dialect: snowflake,
840
-
kind:
841
-
name: incremental_by_time_range
842
-
time_column: ds
832
+
```sql linenums="1"
833
+
MODEL(
834
+
name docs_example.incremental_model,
835
+
kind INCREMENTAL_BY_TIME_RANGE (
836
+
time_column ds
837
+
)
838
+
);
843
839
```
844
840
845
-
=== "Python"
841
+
Python model kinds are specified with model kind objects. Python model kind objects have the same arguments as their SQL counterparts, listed in the [models configuration reference page](../reference/model_configuration.md#model-kind-properties).
846
842
847
-
The Python `model_defaults` `kind` argument takes a model kind object with a value of:
843
+
This example demonstrates how to specify an incremental by time range model kind in Python:
848
844
849
-
- EmbeddedKind
850
-
- ExternalKind
851
-
- FullKind
852
-
- IncrementalByTimeRangeKind
853
-
- IncrementalByUniqueKeyKind
854
-
- IncrementalUnmanagedKind
855
-
- SeedKind
856
-
- ViewKind
845
+
=== "Python"
857
846
858
847
```python linenums="1"
859
-
from sqlmesh.core.config import (
860
-
Config,
861
-
ModelDefaultsConfig,
862
-
IncrementalByTimeRangeKind
863
-
)
864
-
865
-
config = Config(
866
-
model_defaults=ModelDefaultsConfig(
867
-
dialect="snowflake",
868
-
kind=IncrementalByTimeRangeKind(
869
-
time_column="ds",
870
-
),
871
-
),
848
+
from sqlmesh import ExecutionContext, model
849
+
from sqlmesh.core.model import IncrementalByTimeRangeKind
850
+
851
+
@model(
852
+
"docs_example.incremental_model",
853
+
kind=IncrementalByTimeRangeKind(
854
+
time_column="ds"
855
+
)
872
856
)
873
857
```
874
858
859
+
Learn more about specifying Python models at the [Python models concepts page](../concepts/models/python_models.md#model-specification).
860
+
875
861
### Debug mode
876
862
877
-
To enable debug mode set the `SQLMESH_DEBUG` environment variable to one of the following values: `1`, `true`, `t`, `yes` or `y`.
863
+
To enable debug mode set the `SQLMESH_DEBUG` environment variable to one of the following values: "1", "true", "t", "yes" or "y".
878
864
879
-
Enabling this mode ensures that full backtraces are printed when using CLI. Additionally, the default log level is set to `DEBUG` when this mode is enabled.
865
+
Enabling this mode ensures that full backtraces are printed when using CLI. The default log level is set to `DEBUG` when this mode is enabled.
880
866
881
867
Example enabling debug mode for the CLI command `sqlmesh plan`:
Copy file name to clipboardExpand all lines: docs/reference/configuration.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -172,7 +172,9 @@ For example, you might have a specific connection where your tests should run re
172
172
173
173
## Debug mode
174
174
175
-
To enable debug mode set the `SQLMESH_DEBUG` environment variable to one of the following values: "1", "true", "t", "yes" or "y". Enabling this mode ensures that full backtraces are printed when using CLI. The default log level is set to `DEBUG` when this mode is enabled.
175
+
To enable debug mode set the `SQLMESH_DEBUG` environment variable to one of the following values: "1", "true", "t", "yes" or "y".
176
+
177
+
Enabling this mode ensures that full backtraces are printed when using CLI. The default log level is set to `DEBUG` when this mode is enabled.
176
178
177
179
Example enabling debug mode for the CLI command `sqlmesh plan`:
0 commit comments