You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/concepts/models/overview.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Overview
2
2
3
-
Models are comprised of metadata and queries that create tables and views, which can be used by other models or even outside of SQLMesh. They are defined in the `models/` directory of your SQLMesh project and live in `.sql` files.
3
+
Models are made up of metadata and queries that create tables and views, which can be used by other models or even outside of SQLMesh. They are defined in the `models/` directory of your SQLMesh project and live in `.sql` files.
4
4
5
5
SQLMesh will automatically determine the relationships among and lineage of your models by parsing SQL, so you don't have to worry about manually configuring dependencies.
Copy file name to clipboardExpand all lines: docs/concepts/models/seed_models.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -60,6 +60,10 @@ MODEL (
60
60
61
61
This is useful when you want to keep all seed CSV files in a top-level directory such as `seeds/` but don't want to keep track of or manage a bunch of relative paths.
62
62
63
+
### Encoding
64
+
65
+
SQLMesh expects seed files to be encoded according to the [UTF-8](https://en.wikipedia.org/wiki/UTF-8) standard. Using a different encoding may lead to unexpected behavior.
66
+
63
67
## Example
64
68
65
69
In this example, we use the model definition from the previous section saved in the `models/national_holidays.sql` file of the SQLMesh project.
Copy file name to clipboardExpand all lines: docs/concepts/models/sql_models.md
+12Lines changed: 12 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,18 +53,21 @@ UNCACHE TABLE countries;
53
53
```
54
54
55
55
### `MODEL` DDL
56
+
56
57
The `MODEL` DDL is used to specify metadata about the model such as its name, [kind](./model_kinds.md), owner, cron, and others. This should be the first statement in your SQL-based model's file.
57
58
58
59
Refer to `MODEL`[properties](./overview.md#properties) for the full list of allowed properties.
59
60
60
61
### Optional pre/post-statements
62
+
61
63
Optional pre/post-statements allow you to execute SQL commands before and after a model runs, respectively.
62
64
63
65
For example, post/post-statements might modify settings or create indexes. However, be careful not to run any statement that could conflict with the execution of another statement if the models run concurrently, such as creating a physical table.
64
66
65
67
Pre/post-statements are evaluated twice: when a model's table is created and when its query logic is evaluated. Since executing such statements more than once can have unintended side-effects, it is also possible to [conditionally execute](../macros/sqlmesh_macros.md#if) them depending on SQLMesh's [runtime stage](../macros/macro_variables.md#predefined-variables).
66
68
67
69
### The model query
70
+
68
71
The model must contain a standalone query, which can be a single `SELECT` expression, or multiple `SELECT` expressions combined with the `UNION`, `INTERSECT`, or `EXCEPT` operators. The result of this query will be used to populate the model's table or view.
69
72
70
73
## Python-based definition
@@ -113,6 +116,7 @@ The `@model` decorator is the Python equivalent of the `MODEL` DDL. In addition
113
116
**Note:** All of the [metadata](./overview.md#properties) field names are the same as those in the `MODEL` DDL.
114
117
115
118
## Automatic dependencies
119
+
116
120
SQLMesh parses your SQL, so it understands what the code does and how it relates to other models. There is no need for you to manually specify dependencies to other models with special tags or commands.
117
121
118
122
For example, consider a model with this query:
@@ -131,21 +135,29 @@ External dependencies not defined in SQLMesh are also supported. SQLMesh can eit
131
135
Although automatic dependency detection works most of the time, there may be specific cases for which you want to define dependencies manually. You can do so in the `MODEL` DDL with the [dependencies property](./overview.md#properties).
132
136
133
137
## Conventions
138
+
134
139
SQLMesh encourages explicitly specifying the data types of a model's columns through casting. This allows SQLMesh to understand the data types in your models, and it prevents incorrect type inference. SQLMesh supports the casting format `<column name>::<data type>` in models of any SQL dialect.
135
140
136
141
### Explicit SELECTs
142
+
137
143
Although `SELECT *` is convenient, it is dangerous because a model's results can change due to external factors (e.g., an upstream source adding or removing a column). In general, we encourage listing out every column you need or using [`create_external_models`](../../reference/cli.md#create_external_models) to capture the schema of an external data source.
138
144
139
145
If you select from an external source, `SELECT *` will prevent SQLMesh from performing some optimization steps and from determining upstream column-level lineage. Use an [`external` model kind](./model_kinds.md#external) to enable optimizations and upstream column-level lineage for external sources.
140
146
147
+
### Encoding
148
+
149
+
SQLMesh expects files containing SQL models to be encoded according to the [UTF-8](https://en.wikipedia.org/wiki/UTF-8) standard. Using a different encoding may lead to unexpected behavior.
150
+
141
151
## Transpilation
152
+
142
153
SQLMesh leverages [SQLGlot](https://github.com/tobymao/sqlglot) to parse and transpile SQL. Therefore, you can write your SQL in any supported dialect and transpile it into another supported dialect.
143
154
144
155
You can also use advanced syntax that may not be available in your engine of choice. For example, `x::int` is equivalent to `CAST(x as INT)`, but is only supported in some dialects. SQLGlot allows you to use this feature regardless of what engine you're using.
145
156
146
157
Additionally, you won't have to worry about minor formatting differences such as trailing commas, as SQLGlot will remove them at parse time.
147
158
148
159
## Macros
160
+
149
161
Although standard SQL is very powerful, complex data systems often require running SQL queries with dynamic components such as date filters. For example, you may want to change the date ranges in a `between` statement so that you can get the latest batch of data. SQLMesh provides these dates automatically through [macro variables](../macros/macro_variables.md).
150
162
151
163
Additionally, large queries can be difficult to read and maintain. In order to make queries more compact, SQLMesh supports a powerful [macro syntax](../macros/overview.md) as well as [Jinja](https://jinja.palletsprojects.com/en/3.1.x/), allowing you to write macros that make your SQL queries easier to manage.
0 commit comments