Commit ba33d13
committed
Feat: Add StarRocks engine support
### What
- **Add StarRocks engine support to SQLMesh** via StarRocks’
MySQL-compatible protocol.
- Ship **engine adapter + docs + real integration tests** to ensure
generated SQL works on StarRocks.
### Why
- **User demand / adoption**: StarRocks is a common OLAP choice; SQLMesh
users want to run the same model lifecycle (build, incremental
maintenance, views/MVs) on StarRocks without bespoke SQL.
- **Engine-specific semantics**: StarRocks differs from vanilla MySQL in
DDL/DML constraints (e.g., key types, delete behavior, rename caveats).
An adapter is needed to produce correct and predictable SQL.
- **Confidence & maintainability**: Documenting config patterns +
codifying behavior with integration tests prevents regressions and makes
support “real” (not just “it parses”).
### Scope (what’s supported)
- **Connectivity**: Connect through MySQL protocol (e.g., `pymysql`).
- **Table creation / DDL**:
- Key table types via `physical_properties`: **DUPLICATE KEY
(default)**, **PRIMARY KEY (recommended for incremental)**, **UNIQUE
KEY**
- **Partitioning**: simple `partitioned_by` and advanced
`partition_by` (complex expression partitioning) + optional initial
`partitions`
- **Distribution**: `distributed_by` structured form or string
fallback (HASH / RANDOM; buckets required)
- **Ordering**: `order_by` / `clustered_by`
- **Generic PROPERTIES passthrough** (string key/value)
- **Views**:
- Regular views
- **Materialized views** via `kind VIEW(materialized true)` with
StarRocks-specific notes/constraints
- **DML / maintenance**:
- Insert/select/update basics
- Delete behavior handled with StarRocks compatibility constraints
(PRIMARY KEY tables recommended for robust deletes)
### Changes
- **Engine adapter**: `sqlmesh/core/engine_adapter/starrocks.py`
- **Docs**: `docs/integrations/engines/starrocks.md`
- **Integration tests**:
`tests/core/engine_adapter/integration/test_integration_starrocks.py`,
and `tests/core/engine_adapter/test_starrocks.py`
### Verification
- **Integration tests require a running StarRocks** instance.
- Ran:
- set `STARROCKS_HOST/PORT/USER/PASSWORD`
- `pytest -m "starrocks and docker"
tests/core/engine_adapter/integration/test_integration_starrocks.py`
### Known limitations / caveats
- **No sync MV support (currently)**
- **No tuple IN**: `(c1, c2) IN ((v1, v2), ...)`
- **No `SELECT ... FOR UPDATE`**
- **RENAME caveat**: rename target can’t be qualified with a database
name
### Notes on compatibility
- **Changes are StarRocks-scoped** (adapter/docs/tests) and should not
impact other engines.
Signed-off-by: jaogoy <jaogoy@gmail.com>1 parent 529ed00 commit ba33d13
File tree
22 files changed
+8566
-7
lines changed- .circleci
- docs
- guides
- integrations
- engines
- sqlmesh/core
- config
- engine_adapter
- snapshot
- tests/core
- engine_adapter
- integration
- docker
22 files changed
+8566
-7
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
304 | 304 | | |
305 | 305 | | |
306 | 306 | | |
| 307 | + | |
307 | 308 | | |
308 | 309 | | |
309 | 310 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
53 | 81 | | |
54 | 82 | | |
55 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
208 | 208 | | |
209 | 209 | | |
210 | 210 | | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
211 | 214 | | |
212 | 215 | | |
213 | 216 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
920 | 920 | | |
921 | 921 | | |
922 | 922 | | |
| 923 | + | |
923 | 924 | | |
924 | 925 | | |
925 | 926 | | |
| |||
952 | 953 | | |
953 | 954 | | |
954 | 955 | | |
| 956 | + | |
955 | 957 | | |
956 | 958 | | |
957 | 959 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
| 93 | + | |
93 | 94 | | |
0 commit comments