|
| 1 | +# Tobiko Cloud Hybrid Executors - Docker Compose Setup |
| 2 | + |
| 3 | +This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose. |
| 4 | + |
| 5 | +Hybrid executors allow you to run operations on your own infrastructure while leveraging Tobiko Cloud for orchestration. |
| 6 | + |
| 7 | +## What this setup provides |
| 8 | + |
| 9 | +This setup deploys two hybrid executors that pass work tasks from Tobiko Cloud to your data warehouse in a secure way: |
| 10 | + |
| 11 | +- **Apply Executor**: Handles applying changes to the data warehouse |
| 12 | +- **Run Executor**: Handles scheduled model execution |
| 13 | + |
| 14 | +Both executors must be properly configured with environment variables to connect to Tobiko Cloud and your data warehouse. |
| 15 | + |
| 16 | +## Prerequisites |
| 17 | + |
| 18 | +- Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery) |
| 19 | +- Docker and Docker Compose |
| 20 | +- A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials) |
| 21 | + |
| 22 | +## Quick start guide |
| 23 | + |
| 24 | +1. **Get docker-compose file**: |
| 25 | + |
| 26 | + Download the [docker-compose.yml](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) files to a local directory. |
| 27 | + |
| 28 | +2. **Create your environment file**: |
| 29 | + |
| 30 | + Copy the downloaded example environment file into a new `.env` file: |
| 31 | + |
| 32 | + ```bash |
| 33 | + cp .env.example .env |
| 34 | + ``` |
| 35 | + |
| 36 | +3. **Edit the .env file** with your project's configuration: |
| 37 | + |
| 38 | + - Set your Tobiko Cloud organization, project, client ID, and client secret |
| 39 | + - Configure your gateway connection details |
| 40 | + - Adjust resource limits if needed |
| 41 | + |
| 42 | +4. **Start the executors**: |
| 43 | + |
| 44 | + ```bash |
| 45 | + docker compose up -d |
| 46 | + ``` |
| 47 | + |
| 48 | +5. **Check the logs**: |
| 49 | + |
| 50 | + ```bash |
| 51 | + docker compose logs -f |
| 52 | + ``` |
| 53 | + |
| 54 | +## Configuration options |
| 55 | + |
| 56 | +### Gateway configuration |
| 57 | + |
| 58 | +The default configuration in the `docker-compose.yml` file uses Postgres, but you can use [any supported SQL engine](../../../integrations/overview.md#execution-engines) by adjusting the connection parameters in your `.env` file. |
| 59 | + |
| 60 | +#### Multiple gateways |
| 61 | + |
| 62 | +To configure multiple gateways, add additional environment variables for each gateway the `docker-compose.yml` file: |
| 63 | + |
| 64 | +```yaml |
| 65 | +environment: |
| 66 | + # First gateway |
| 67 | + SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres} |
| 68 | + # ... other GATEWAY_A configuration ... |
| 69 | + |
| 70 | + # Second gateway |
| 71 | + SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake |
| 72 | + SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT} |
| 73 | + # ... other GATEWAY_B configuration ... |
| 74 | +``` |
| 75 | + |
| 76 | +## Health checking |
| 77 | + |
| 78 | +Verify the health of your executors by running these commands: |
| 79 | + |
| 80 | +```bash |
| 81 | +docker compose exec apply-executor /app/pex executor apply --check |
| 82 | +docker compose exec run-executor /app/pex executor run --check |
| 83 | +``` |
| 84 | + |
| 85 | +Example successful output: |
| 86 | + |
| 87 | +```bash |
| 88 | +> docker compose exec apply-executor /app/pex executor apply --check |
| 89 | +2025-04-09 21:24:49,873 - MainThread - httpx - INFO - HTTP Request: GET https://cloud.tobikodata.com/sqlmesh/<YOUR ORG>/<YOUR PROJECT>/api/state-sync/enterprise-version/upgrade "HTTP/1.1 200 OK" (_client.py:1025) |
| 90 | +2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180) |
| 91 | +``` |
| 92 | + |
| 93 | +In addition, ensure the executors are healthy by running `echo $?` to confirm the check command returned exit code 0. |
| 94 | + |
| 95 | +## Stopping the executors |
| 96 | + |
| 97 | +To stop the executors: |
| 98 | + |
| 99 | +```bash |
| 100 | +docker compose down |
| 101 | +``` |
| 102 | + |
| 103 | +## Troubleshooting |
| 104 | + |
| 105 | +If you encounter issues: |
| 106 | + |
| 107 | +1. Check the logs: `docker compose logs -f` |
| 108 | +2. Verify your connection settings in the `.env` file |
| 109 | +3. Ensure your client ID and client secret are correct |
| 110 | +4. Check that your SQL engine is accessible from the Docker containers |
| 111 | + |
| 112 | +## Security considerations |
| 113 | + |
| 114 | +!!! warning "Never commit .env to version control" |
| 115 | + |
| 116 | + The `.env` file contains sensitive information. Never commit it to version control. |
| 117 | + |
| 118 | +- Consider using Docker secrets or a secrets management solution in production environments. |
| 119 | +- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options. |
0 commit comments