Skip to content

Commit 5911fa3

Browse files
xinge-jixinge-ji
authored
Merge branch 'TobikoData:main' into doris-adapter
2 parents 5c69d76 + a5544e2 commit 5911fa3

File tree

21 files changed

+289
-61
lines changed

21 files changed

+289
-61
lines changed

‎.circleci/continue_config.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,10 @@ jobs:
258258
echo "export REDSHIFT_DATABASE='$TEST_DB_NAME'" >> "$BASH_ENV"
259259
echo "export GCP_POSTGRES_DATABASE='$TEST_DB_NAME'" >> "$BASH_ENV"
260260
echo "export FABRIC_DATABASE='$TEST_DB_NAME'" >> "$BASH_ENV"
261+
262+
# Make snowflake private key available
263+
echo $SNOWFLAKE_PRIVATE_KEY_RAW | base64 -d > /tmp/snowflake-keyfile.p8
264+
echo "export SNOWFLAKE_PRIVATE_KEY_FILE='/tmp/snowflake-keyfile.p8'" >> "$BASH_ENV"
261265
- run:
262266
name: Create test database
263267
command: ./.circleci/manage-test-db.sh << parameters.engine >> "$TEST_DB_NAME" up
@@ -310,7 +314,7 @@ workflows:
310314
matrix:
311315
parameters:
312316
engine:
313-
#- snowflake
317+
- snowflake
314318
- databricks
315319
- redshift
316320
- bigquery

‎.circleci/manage-test-db.sh‎

Lines changed: 1 addition & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ function_exists() {
2525
# Snowflake
2626
snowflake_init() {
2727
echo "Installing Snowflake CLI"
28-
pip install "snowflake-cli-labs<3.8.0"
28+
pip install "snowflake-cli"
2929
}
3030

3131
snowflake_up() {
@@ -40,20 +40,6 @@ snowflake_down() {
4040
databricks_init() {
4141
echo "Installing Databricks CLI"
4242
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh || true
43-
44-
echo "Writing out Databricks CLI config file"
45-
echo -e "[DEFAULT]\nhost = $DATABRICKS_SERVER_HOSTNAME\ntoken = $DATABRICKS_ACCESS_TOKEN" > ~/.databrickscfg
46-
47-
# this takes a path like 'sql/protocolv1/o/2934659247569/0723-005339-foobar' and extracts '0723-005339-foobar' from it
48-
CLUSTER_ID=${DATABRICKS_HTTP_PATH##*/}
49-
50-
echo "Extracted cluster id: $CLUSTER_ID from '$DATABRICKS_HTTP_PATH'"
51-
52-
# Note: the cluster doesnt need to be running to create / drop catalogs, but it does need to be running to run the integration tests
53-
echo "Ensuring cluster is running"
54-
# the || true is to prevent the following error from causing an abort:
55-
# > Error: is in unexpected state Running.
56-
databricks clusters start $CLUSTER_ID || true
5743
}
5844

5945
databricks_up() {

‎.github/workflows/pr.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ jobs:
3030
test-vscode-e2e:
3131
runs-on:
3232
labels: [ubuntu-2204-8]
33+
# As at 2026-01-12 this job flakes 100% of the time. It needs investigation
34+
if: false
3335
steps:
3436
- uses: actions/checkout@v5
3537
- uses: actions/setup-node@v6

‎Makefile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -215,14 +215,14 @@ doris-test: engine-doris-up
215215
# Cloud Engines #
216216
#################
217217

218-
snowflake-test: guard-SNOWFLAKE_ACCOUNT guard-SNOWFLAKE_WAREHOUSE guard-SNOWFLAKE_DATABASE guard-SNOWFLAKE_USER guard-SNOWFLAKE_PASSWORD engine-snowflake-install
218+
snowflake-test: guard-SNOWFLAKE_ACCOUNT guard-SNOWFLAKE_WAREHOUSE guard-SNOWFLAKE_DATABASE guard-SNOWFLAKE_USER engine-snowflake-install
219219
pytest -n auto -m "snowflake" --reruns 3 --junitxml=test-results/junit-snowflake.xml
220220

221221
bigquery-test: guard-BIGQUERY_KEYFILE engine-bigquery-install
222222
$(PIP) install -e ".[bigframes]"
223223
pytest -n auto -m "bigquery" --reruns 3 --junitxml=test-results/junit-bigquery.xml
224224

225-
databricks-test: guard-DATABRICKS_CATALOG guard-DATABRICKS_SERVER_HOSTNAME guard-DATABRICKS_HTTP_PATH guard-DATABRICKS_ACCESS_TOKEN guard-DATABRICKS_CONNECT_VERSION engine-databricks-install
225+
databricks-test: guard-DATABRICKS_CATALOG guard-DATABRICKS_SERVER_HOSTNAME guard-DATABRICKS_HTTP_PATH guard-DATABRICKS_CONNECT_VERSION engine-databricks-install
226226
$(PIP) install 'databricks-connect==${DATABRICKS_CONNECT_VERSION}'
227227
pytest -n auto -m "databricks" --reruns 3 --junitxml=test-results/junit-databricks.xml
228228

‎docs/integrations/engines/trino.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ hive.metastore.glue.default-warehouse-dir=s3://my-bucket/
9090
| `http_scheme` | The HTTP scheme to use when connecting to your cluster. By default, it's `https` and can only be `http` for no-auth or basic auth. | string | N |
9191
| `port` | The port to connect to your cluster. By default, it's `443` for `https` scheme and `80` for `http` | int | N |
9292
| `roles` | Mapping of catalog name to a role | dict | N |
93+
| `source` | Value to send as Trino's `source` field for query attribution / auditing. Default: `sqlmesh`. | string | N |
9394
| `http_headers` | Additional HTTP headers to send with each request. | dict | N |
9495
| `session_properties` | Trino session properties. Run `SHOW SESSION` to see all options. | dict | N |
9596
| `retries` | Number of retries to attempt when a request fails. Default: `3` | int | N |

‎docs/integrations/github.md‎

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -286,21 +286,22 @@ Below is an example of how to define the default config for the bot in either YA
286286

287287
### Configuration Properties
288288

289-
| Option | Description | Type | Required |
290-
|---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------:|:--------:|
291-
| `invalidate_environment_after_deploy` | Indicates if the PR environment created should be automatically invalidated after changes are deployed. Invalidated environments are cleaned up automatically by the Janitor. Default: `True` | bool | N |
292-
| `merge_method` | The merge method to use when automatically merging a PR after deploying to prod. Defaults to `None` meaning automatic merge is not done. Options: `merge`, `squash`, `rebase` | string | N |
293-
| `enable_deploy_command` | Indicates if the `/deploy` command should be enabled in order to allowed synchronized deploys to production. Default: `False` | bool | N |
294-
| `command_namespace` | The namespace to use for SQLMesh commands. For example if you provide `#SQLMesh` as a value then commands will be expected in the format of `#SQLMesh/<command>`. Default: `None` meaning no namespace is used. | string | N |
295-
| `auto_categorize_changes` | Auto categorization behavior to use for the bot. If not provided then the project-wide categorization behavior is used. See [Auto-categorize model changes](https://sqlmesh.readthedocs.io/en/stable/guides/configuration/#auto-categorize-model-changes) for details. | dict | N |
296-
| `default_pr_start` | Default start when creating PR environment plans. If running in a mode where the bot automatically backfills models (based on `auto_categorize_changes` behavior) then this can be used to limit the amount of data backfilled. Defaults to `None` meaning the start date is set to the earliest model's start or to 1 day ago if [data previews](../concepts/plans.md#data-preview) need to be computed.| str | N |
297-
| `pr_min_intervals` | Intended for use when `default_pr_start` is set to a relative time, eg `1 week ago`. This ensures that at least this many intervals across every model are included for backfill in the PR environment. Without this, models with an interval unit wider than `default_pr_start` (such as `@monthly` models if `default_pr_start` was set to `1 week ago`) will be excluded from backfill entirely. | int | N |
298-
| `skip_pr_backfill` | Indicates if the bot should skip backfilling models in the PR environment. Default: `True` | bool | N |
299-
| `pr_include_unmodified` | Indicates whether to include unmodified models in the PR environment. Default to the project's config value (which defaults to `False`) | bool | N |
300-
| `run_on_deploy_to_prod` | Indicates whether to run latest intervals when deploying to prod. If set to false, the deployment will backfill only the changed models up to the existing latest interval in production, ignoring any missing intervals beyond this point. Default: `False` | bool | N |
301-
| `pr_environment_name` | The name of the PR environment to create for which a PR number will be appended to. Defaults to the repo name if not provided. Note: The name will be normalized to alphanumeric + underscore and lowercase. | str | N |
302-
| `prod_branch_name` | The name of the git branch associated with production. Ex: `prod`. Default: `main` or `master` is considered prod | str | N |
303-
| `forward_only_branch_suffix` | If the git branch has this suffix, trigger a [forward-only](../concepts/plans.md#forward-only-plans) plan instead of a normal plan. Default: `-forward-only` | str | N |
289+
| Option | Description | Type | Required |
290+
|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------:|:--------:|
291+
| `invalidate_environment_after_deploy` | Indicates if the PR environment created should be automatically invalidated after changes are deployed. Invalidated environments are cleaned up automatically by the Janitor. Default: `True` | bool | N |
292+
| `merge_method` | The merge method to use when automatically merging a PR after deploying to prod. Defaults to `None` meaning automatic merge is not done. Options: `merge`, `squash`, `rebase` | string | N |
293+
| `enable_deploy_command` | Indicates if the `/deploy` command should be enabled in order to allowed synchronized deploys to production. Default: `False` | bool | N |
294+
| `command_namespace` | The namespace to use for SQLMesh commands. For example if you provide `#SQLMesh` as a value then commands will be expected in the format of `#SQLMesh/<command>`. Default: `None` meaning no namespace is used. | string | N |
295+
| `auto_categorize_changes` | Auto categorization behavior to use for the bot. If not provided then the project-wide categorization behavior is used. See [Auto-categorize model changes](https://sqlmesh.readthedocs.io/en/stable/guides/configuration/#auto-categorize-model-changes) for details. | dict | N |
296+
| `default_pr_start` | Default start when creating PR environment plans. If running in a mode where the bot automatically backfills models (based on `auto_categorize_changes` behavior) then this can be used to limit the amount of data backfilled. Defaults to `None` meaning the start date is set to the earliest model's start or to 1 day ago if [data previews](../concepts/plans.md#data-preview) need to be computed. | str | N |
297+
| `pr_min_intervals` | Intended for use when `default_pr_start` is set to a relative time, eg `1 week ago`. This ensures that at least this many intervals across every model are included for backfill in the PR environment. Without this, models with an interval unit wider than `default_pr_start` (such as `@monthly` models if `default_pr_start` was set to `1 week ago`) will be excluded from backfill entirely. | int | N |
298+
| `skip_pr_backfill` | Indicates if the bot should skip backfilling models in the PR environment. Default: `True` | bool | N |
299+
| `pr_include_unmodified` | Indicates whether to include unmodified models in the PR environment. Default to the project's config value (which defaults to `False`) | bool | N |
300+
| `run_on_deploy_to_prod` | Indicates whether to run latest intervals when deploying to prod. If set to false, the deployment will backfill only the changed models up to the existing latest interval in production, ignoring any missing intervals beyond this point. Default: `False` | bool | N |
301+
| `pr_environment_name` | The name of the PR environment to create for which a PR number will be appended to. Defaults to the repo name if not provided. Note: The name will be normalized to alphanumeric + underscore and lowercase. | str | N |
302+
| `prod_branch_name` | The name of the git branch associated with production. Ex: `prod`. Default: `main` or `master` is considered prod | str | N |
303+
| `forward_only_branch_suffix` | If the git branch has this suffix, trigger a [forward-only](../concepts/plans.md#forward-only-plans) plan instead of a normal plan. Default: `-forward-only` | str | N |
304+
| `check_if_blocked_on_deploy_to_prod` | The bot normally checks if a PR is blocked from merging before deploying to production. Setting this to `False` will skip that check. Default: `True` | bool | N |
304305

305306
Example with all properties defined:
306307

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ dependencies = [
1818
"ipywidgets",
1919
"jinja2",
2020
"packaging",
21-
"pandas",
21+
"pandas<3.0.0",
2222
"pydantic>=2.0.0",
2323
"python-dotenv",
2424
"requests",

0 commit comments

Comments
 (0)