Skip to content

Commit e3e32bf

Browse files
eakmanrqtreysp
andauthored
feat: add helm chart and docker compose for hybrid executors (#4084)
Co-authored-by: Trey Spiller <1831878+treysp@users.noreply.github.com>
1 parent 0fa0534 commit e3e32bf

File tree

9 files changed

+748
-124
lines changed

9 files changed

+748
-124
lines changed
626 KB
Loading
Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Tobiko Cloud Hybrid Executors - Docker Compose Setup
2+
3+
This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose.
4+
5+
Hybrid executors allow you to run operations on your own infrastructure while leveraging Tobiko Cloud for orchestration.
6+
7+
## What this setup provides
8+
9+
This setup deploys two hybrid executors that pass work tasks from Tobiko Cloud to your data warehouse in a secure way:
10+
11+
- **Apply Executor**: Handles applying changes to the data warehouse
12+
- **Run Executor**: Handles scheduled model execution
13+
14+
Both executors must be properly configured with environment variables to connect to Tobiko Cloud and your data warehouse.
15+
16+
## Prerequisites
17+
18+
- Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery)
19+
- Docker and Docker Compose
20+
- A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials)
21+
22+
## Quick start guide
23+
24+
1. **Get docker-compose file**:
25+
26+
Download the [docker-compose.yml](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) files to a local directory.
27+
28+
2. **Create your environment file**:
29+
30+
Copy the downloaded example environment file into a new `.env` file:
31+
32+
```bash
33+
cp .env.example .env
34+
```
35+
36+
3. **Edit the .env file** with your project's configuration:
37+
38+
- Set your Tobiko Cloud organization, project, client ID, and client secret
39+
- Configure your gateway connection details
40+
- Adjust resource limits if needed
41+
42+
4. **Start the executors**:
43+
44+
```bash
45+
docker compose up -d
46+
```
47+
48+
5. **Check the logs**:
49+
50+
```bash
51+
docker compose logs -f
52+
```
53+
54+
## Configuration options
55+
56+
### Gateway configuration
57+
58+
The default configuration in the `docker-compose.yml` file uses Postgres, but you can use [any supported SQL engine](../../../integrations/overview.md#execution-engines) by adjusting the connection parameters in your `.env` file.
59+
60+
#### Multiple gateways
61+
62+
To configure multiple gateways, add additional environment variables for each gateway the `docker-compose.yml` file:
63+
64+
```yaml
65+
environment:
66+
# First gateway
67+
SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres}
68+
# ... other GATEWAY_A configuration ...
69+
70+
# Second gateway
71+
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake
72+
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT}
73+
# ... other GATEWAY_B configuration ...
74+
```
75+
76+
## Health checking
77+
78+
Verify the health of your executors by running these commands:
79+
80+
```bash
81+
docker compose exec apply-executor /app/pex executor apply --check
82+
docker compose exec run-executor /app/pex executor run --check
83+
```
84+
85+
Example successful output:
86+
87+
```bash
88+
> docker compose exec apply-executor /app/pex executor apply --check
89+
2025-04-09 21:24:49,873 - MainThread - httpx - INFO - HTTP Request: GET https://cloud.tobikodata.com/sqlmesh/<YOUR ORG>/<YOUR PROJECT>/api/state-sync/enterprise-version/upgrade "HTTP/1.1 200 OK" (_client.py:1025)
90+
2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180)
91+
```
92+
93+
In addition, ensure the executors are healthy by running `echo $?` to confirm the check command returned exit code 0.
94+
95+
## Stopping the executors
96+
97+
To stop the executors:
98+
99+
```bash
100+
docker compose down
101+
```
102+
103+
## Troubleshooting
104+
105+
If you encounter issues:
106+
107+
1. Check the logs: `docker compose logs -f`
108+
2. Verify your connection settings in the `.env` file
109+
3. Ensure your client ID and client secret are correct
110+
4. Check that your SQL engine is accessible from the Docker containers
111+
112+
## Security considerations
113+
114+
!!! warning "Never commit .env to version control"
115+
116+
The `.env` file contains sensitive information. Never commit it to version control.
117+
118+
- Consider using Docker secrets or a secrets management solution in production environments.
119+
- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options.

0 commit comments

Comments
 (0)