Merge branch 'main' into add_fabric_warehouse

Author: erindru

.gitignore

@@ -157,4 +157,4 @@ metastore_db/
 spark-warehouse/
 
 # claude
-.claude/
+.claude/

docs/concepts/models/overview.md

@@ -507,11 +507,15 @@ Some properties are only available in specific model kinds - see the [model conf
 :   Set this to true to indicate that all changes to this model should be [forward-only](../plans.md#forward-only-plans).
 
 ### on_destructive_change
-:   What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column).
+:   What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column or modifying column constraints in ways that could cause data loss).
 
     SQLMesh checks for destructive changes at plan time based on the model definition and run time based on the model's underlying physical tables.
 
-    Must be one of the following values: `allow`, `warn`, or `error` (default).
+    Must be one of the following values: `allow`, `warn`, `error` (default), or `ignore`.
+
+!!! warning "Ignore is Dangerous"
+
+    `ignore` is dangerous since it can result in error or data loss. It likely should never be used but could be useful as an "escape-hatch" or a way to workaround unexpected behavior.
 
 ### disable_restatement
 :   Set this to true to indicate that [data restatement](../plans.md#restatement-plans) is disabled for this model.

docs/guides/custom_materializations.md

@@ -64,6 +64,7 @@ class CustomFullMaterialization(CustomMaterialization):
         query_or_df: QueryOrDF,
         model: Model,
         is_first_insert: bool,
+        render_kwargs: t.Dict[str, t.Any],
         **kwargs: t.Any,
     ) -> None:
         self.adapter.replace_query(table_name, query_or_df)
@@ -78,6 +79,7 @@ Let's unpack this materialization:
     * `query_or_df` - a query (of SQLGlot expression type) or DataFrame (Pandas, PySpark, or Snowpark) instance to be inserted
     * `model` - the model definition object used to access model parameters and user-specified materialization arguments
     * `is_first_insert` - whether this is the first insert for the current version of the model (used with batched or multi-step inserts)
+    * `render_kwargs` - a dictionary of arguments used to render the model query
     * `kwargs` - additional and future arguments
 * The `self.adapter` instance is used to interact with the target engine. It comes with a set of useful high-level APIs like `replace_query`, `columns`, and `table_exists`, but also supports executing arbitrary SQL expressions with its `execute` method.
 
@@ -150,6 +152,7 @@ class CustomFullMaterialization(CustomMaterialization):
         query_or_df: QueryOrDF,
         model: Model,
         is_first_insert: bool,
+        render_kwargs: t.Dict[str, t.Any],
         **kwargs: t.Any,
     ) -> None:
         config_value = model.custom_materialization_properties["config_key"]
@@ -232,6 +235,7 @@ class CustomFullMaterialization(CustomMaterialization[MyCustomKind]):
         query_or_df: QueryOrDF,
         model: Model,
         is_first_insert: bool,
+        render_kwargs: t.Dict[str, t.Any],
         **kwargs: t.Any,
     ) -> None:
         assert isinstance(model.kind, MyCustomKind)

docs/guides/incremental_time.md

@@ -171,7 +171,12 @@ The check is performed at plan time based on the model definition. SQLMesh may n
 
 A model's `on_destructive_change` [configuration setting](../reference/model_configuration.md#incremental-models) determines what happens when SQLMesh detects a destructive change.
 
-By default, SQLMesh will error so no data is lost. You can set `on_destructive_change` to `warn` or `allow` in the model's `MODEL` block to allow destructive changes.
+By default, SQLMesh will error so no data is lost. You can set `on_destructive_change` to `warn` or `allow` in the model's `MODEL` block to allow destructive changes. 
+`ignore` can be used to not perform the schema change and allow the table's definition to diverge from the model definition.
+
+!!! warning "Ignore is Dangerous"
+
+    `ignore` is dangerous since it can result in error or data loss. It likely should never be used but could be useful as an "escape-hatch" or a way to workaround unexpected behavior.
 
 This example configures a model to silently `allow` destructive changes:
 

docs/integrations/dbt.md

@@ -44,16 +44,36 @@ Prepare an existing dbt project to be run by SQLMesh by executing the `sqlmesh i
 $ sqlmesh init -t dbt
 ```
 
-SQLMesh will use the data warehouse connection target in your dbt project `profiles.yml` file. The target can be changed at any time.
+This will create a file called `sqlmesh.yaml` containing the [default model start date](../reference/model_configuration.md#model-defaults). This configuration file is a minimum starting point for enabling SQLMesh to work with your DBT project.
+
+As you become more comfortable with running your project under SQLMesh, you may specify additional SQLMesh [configuration](../reference/configuration.md) as required to unlock more features.
+
+!!! note "profiles.yml"
+
+    SQLMesh will use the existing data warehouse connection target from your dbt project's `profiles.yml` file so the connection configuration does not need to be duplicated in `sqlmesh.yaml`. You may change the target at any time in the dbt config and SQLMesh will pick up the new target.
 
 ### Setting model backfill start dates
 
-Models **require** a start date for backfilling data through use of the `start` configuration parameter. `start` can be defined individually for each model in its `config` block or globally in the `dbt_project.yml` file as follows:
+Models **require** a start date for backfilling data through use of the `start` configuration parameter. `start` can be defined individually for each model in its `config` block or globally in the `sqlmesh.yaml` file as follows:
 
-```
-> models:
->   +start: Jan 1 2000
-```
+=== "sqlmesh.yaml"
+
+    ```yaml
+    model_defaults:
+      start: '2000-01-01'
+    ```
+
+=== "dbt Model"
+
+    ```jinja
+    {{
+      config(
+        materialized='incremental',
+        start='2000-01-01',
+        ...
+      )
+    }}
+    ```
 
 ### Configuration
 
@@ -63,47 +83,89 @@ SQLMesh derives a project's configuration from its dbt configuration files. This
 
 [Certain engines](https://sqlmesh.readthedocs.io/en/stable/guides/configuration/?h=unsupported#state-connection), like Trino, cannot be used to store SQLMesh's state.
 
-As a workaround, we recommend specifying a supported state engine using the `state_connection` argument instead.
+In addition, even if your warehouse is supported for state, you may find that you get better performance by using a [traditional database](../concepts/state.md) to store state as these are a better fit for the state workload than a warehouse optimized for analytics workloads.
 
-Learn more about how to configure state connections in Python [here](https://sqlmesh.readthedocs.io/en/stable/guides/configuration/#state-connection).
+In these cases, we recommend specifying a [supported production state engine](../concepts/state.md#state) using the `state_connection` configuration.
 
-#### Runtime vars
+This involves updating `sqlmesh.yaml` to add a gateway configuration for the state connection:
 
-dbt supports passing variable values at runtime with its [CLI `vars` option](https://docs.getdbt.com/docs/build/project-variables#defining-variables-on-the-command-line).
+```yaml
+gateways:
+  "": # "" (empty string) is the default gateway
+    state_connection:
+      type: postgres
+      ...
 
-In SQLMesh, these variables are passed via configurations. When you initialize a dbt project with `sqlmesh init`, a file `config.py` is created in your project directory.
+model_defaults:
+  start: '2000-01-01'
+```
 
-The file creates a SQLMesh `config` object pointing to the project directory:
+Or, for a specific dbt profile defined in `profiles.yml`, eg `dev`:
 
-```python
-config = sqlmesh_config(Path(__file__).parent)
+```yaml
+gateways:
+  dev: # must match the target dbt profile name
+    state_connection:
+      type: postgres
+      ...
+
+model_defaults:
+  start: '2000-01-01'
 ```
 
-Specify runtime variables by adding a Python dictionary to the `sqlmesh_config()` `variables` argument.
+Learn more about how to configure state connections [here](https://sqlmesh.readthedocs.io/en/stable/guides/configuration/#state-connection).
+
+#### Runtime vars
+
+dbt supports passing variable values at runtime with its [CLI `vars` option](https://docs.getdbt.com/docs/build/project-variables#defining-variables-on-the-command-line).
+
+In SQLMesh, these variables are passed via configurations. When you initialize a dbt project with `sqlmesh init`, a file `sqlmesh.yaml` is created in your project directory.
+
+You may define global variables in the same way as a native project by adding a `variables` section to the config.
 
 For example, we could specify the runtime variable `is_marketing` and its value `no` as:
 
-```python
-config = sqlmesh_config(
-    Path(__file__).parent,
-    variables={"is_marketing": "no"}
-    )
+```yaml
+variables:
+  is_marketing: no
+
+model_defaults:
+  start: '2000-01-01'
 ```
 
+Variables can also be set at the gateway/profile level which override variables set at the project level. See the [variables documentation](../concepts/macros/sqlmesh_macros.md#gateway-variables) to learn more about how to specify them at different levels.
+
+#### Combinations
+
 Some projects use combinations of runtime variables to control project behavior. Different combinations can be specified in different `sqlmesh_config` objects, with the relevant configuration passed to the SQLMesh CLI command.
 
+!!! info "Python config"
+
+    Switching between different config objects requires the use of [Python config](../guides/configuration.md#python) instead of the default YAML config.
+
+    You will need to create a file called `config.py` in the root of your project with the following contents:
+
+    ```py
+    from pathlib import Path
+    from sqlmesh.dbt.loader import sqlmesh_config
+
+    config = sqlmesh_config(Path(__file__).parent)
+    ```
+
+    Note that any config from `sqlmesh.yaml` will be overlayed on top of the active Python config so you dont need to remove the `sqlmesh.yaml` file
+
 For example, consider a project with a special configuration for the `marketing` department. We could create separate configurations to pass at runtime like this:
 
 ```python
 config = sqlmesh_config(
-    Path(__file__).parent,
-    variables={"is_marketing": "no", "include_pii": "no"}
-    )
+  Path(__file__).parent,
+  variables={"is_marketing": "no", "include_pii": "no"}
+)
 
 marketing_config = sqlmesh_config(
-    Path(__file__).parent,
-    variables={"is_marketing": "yes", "include_pii": "yes"}
-    )
+  Path(__file__).parent,
+  variables={"is_marketing": "yes", "include_pii": "yes"}
+)
 ```
 
 By default, SQLMesh will use the configuration object named `config`. Use a different configuration by passing the object name to SQLMesh CLI commands with the `--config` option. For example, we could run a `plan` with the marketing configuration like this:

docs/integrations/engines/bigquery.md

@@ -145,7 +145,7 @@ pip install "sqlmesh[bigquery]"
 | Option                          | Description                                                                                                                                                       |  Type  | Required |
 |---------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------:|:--------:|
 | `type`                          | Engine type name - must be `bigquery`                                                                                                                             | string |    Y     |
-| `method`                        | Connection methods - see [allowed values below](#connection-methods). Default: `oauth`.                                                                           | string |    N     |
+| `method`                        | Connection methods - see [allowed values below](#authentication-methods). Default: `oauth`.                                                                           | string |    N     |
 | `project`                       | The ID of the GCP project                                                                                                                                       | string |    N     |
 | `location`                      | The location of for the datasets (can be regional or multi-regional)                                                                                              | string |    N     |
 | `execution_project`             | The name of the GCP project to bill for the execution of the models. If not set, the project associated with the model will be used.                              | string |    N     |

examples/custom_materializations/custom_materializations/custom_kind.py

@@ -24,8 +24,9 @@ def insert(
         query_or_df: QueryOrDF,
         model: Model,
         is_first_insert: bool,
+        render_kwargs: t.Dict[str, t.Any],
         **kwargs: t.Any,
     ) -> None:
         assert type(model.kind).__name__ == "ExtendedCustomKind"
 
-        self._replace_query_for_model(model, table_name, query_or_df)
+        self._replace_query_for_model(model, table_name, query_or_df, render_kwargs)

examples/custom_materializations/custom_materializations/full.py

@@ -17,6 +17,7 @@ def insert(
         query_or_df: QueryOrDF,
         model: Model,
         is_first_insert: bool,
+        render_kwargs: t.Dict[str, t.Any],
         **kwargs: t.Any,
     ) -> None:
-        self._replace_query_for_model(model, table_name, query_or_df)
+        self._replace_query_for_model(model, table_name, query_or_df, render_kwargs)

examples/multi/.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "sqlmesh.projectPaths": ["./repo_1", "./repo_2"]
+}