diff --git a/docs/cn/guides/20-self-hosted/01-quickstart/index.md b/docs/cn/guides/20-self-hosted/01-quickstart/index.md index 3257cbd554..b3de22a96d 100644 --- a/docs/cn/guides/20-self-hosted/01-quickstart/index.md +++ b/docs/cn/guides/20-self-hosted/01-quickstart/index.md @@ -2,13 +2,11 @@ title: 快速入门 --- -Databend 快速入门：5 分钟体验 Databend -本指南将帮助您快速设置 Databend，连接到它，并执行基本的数据导入。 +5 分钟内在本地启动 Databend。 -## 1. 使用 Docker 启动 Databend -运行以下命令在容器中启动 Databend： +## 1. 启动 Databend -``` +```shell docker run -d \ --name databend \ --network host \ @@ -19,124 +17,48 @@ docker run -d \ --restart unless-stopped \ datafuselabs/databend ``` -检查 Databend 是否成功运行： +:::tip 国内用户 +如果拉取镜像较慢，可替换为国内镜像： +```shell +registry.databend.cn/public/databend:latest ``` -docker logs -f databend -``` -等待直到您看到日志显示 Databend 和 MinIO 已准备就绪。 +::: -## 2. 连接到 Databend -安装 bendsql (Databend CLI)： +等待 Databend 就绪： -``` -curl -fsSL https://repo.databend.com/install/bendsql.sh | bash -echo "export PATH=$PATH:~/.bendsql/bin" >>~/.bash_profile -source ~/.bash_profile +```shell +docker logs -f databend ``` -连接到 Databend： -``` -bendsql -udatabend -pdatabend -``` +看到 `Databend Query started` 即表示启动成功。 -## 3. 执行基本数据导入 -### 步骤 1：创建外部存储桶 (myupload) -安装 mc (MinIO 客户端) 并创建存储桶： +## 2. 连接 -``` -wget https://dl.min.io/client/mc/release/linux-amd64/mc -sudo cp mc /usr/local/bin/ && sudo chmod +x /usr/local/bin/mc -mc alias set myminio http://localhost:9000 minioadmin minioadmin -mc mb myminio/myupload -mc ls myminio -``` -预期输出： -``` -[0001-01-01 08:05:43 LMT] 0B databend/ -[2025-04-12 08:43:59 CST] 0B myupload/ -``` +安装 BendSQL： -### 步骤 2：生成 CSV 文件并上传到 myupload -``` -echo -e "id,name,age,city\n1,John,32,New York\n2,Emma,28,London\n3,Liam,35,Paris\n4,Olivia,40,Berlin\n5,Noah,29,Tokyo" > data.csv -mc cp data.csv myminio/myupload/ -mc ls myminio/myupload/ -``` -### 步骤 3：创建外部 Stage 并检查数据 -在 bendsql 中运行： -``` -CREATE STAGE mystage 's3://myupload' -CONNECTION=( - endpoint_url='http://127.0.0.1:9000', - access_key_id='minioadmin', - secret_access_key='minioadmin', - region='us-east-1' -); -``` -显示外部 Stage @mystage 中的文件： -``` -LIST @mystage; -``` -| name | size | md5 | last_modified | creator | -|----------|--------|-------------------|-----------------------|-------------| -| String | UInt64 | Nullable(String) | String | Nullable(String) | -| data.csv | 104 | "a27fa15258911f534fb795a8c64e05d4" | 2025-04-12 00:51:11.015 +0000 | NULL | - -预览 CSV 数据： -``` -SELECT $1, $2, $3, $4 FROM @mystage/data.csv (FILE_FORMAT=>'CSV') LIMIT 10; -``` -| \$1 | \$2 | \$3 | \$4 | -|-------------------|-------------------|-------------------|-------------------| -| Nullable(String) | Nullable(String) | Nullable(String) | Nullable(String) | -| id | name | age | city | -| 1 | John | 32 | New York | -| 2 | Emma | 28 | London | -| 3 | Liam | 35 | Paris | -| 4 | Olivia | 40 | Berlin | -| 5 | Noah | 29 | Tokyo | - - -### 步骤 4：创建表并加载数据 +```shell +curl -fsSL https://repo.databend.com/install/bendsql.sh | bash +echo "export PATH=$PATH:~/.bendsql/bin" >> ~/.bash_profile +source ~/.bash_profile ``` -CREATE DATABASE wubx; -USE wubx; -CREATE TABLE t_person ( - id INT, - name VARCHAR, - age INT UNSIGNED, - city VARCHAR -); - -COPY INTO t_person FROM @mystage PATTERN='.*[.]csv' FILE_FORMAT=(TYPE=CSV, SKIP_HEADER=1); +连接到 Databend： +```shell +bendsql -u databend -p databend ``` -| File | Rows_loaded | Errors_seen | First_error | First_error_line | -|-----------|-------------|-------------|------------------|------------------| -| String | Int32 | Int32 | Nullable(String) | Nullable(Int32) | -| data.csv | 5 | 0 | NULL | NULL | +## 3. 验证 -### 步骤 5：查询数据 -``` -SELECT * FROM t_person; +```sql +CREATE TABLE t (id INT, name VARCHAR); +INSERT INTO t VALUES (1, 'Databend'); +SELECT * FROM t; ``` -| id | name | age | city | -|----------|----------|----------|----------| -| Nullable(Int32) | Nullable(String) | Nullable(UInt32) | Nullable(String) | -| 1 | John | 32 | New York | -| 2 | Emma | 28 | London | -| 3 | Liam | 35 | Paris | -| 4 | Olivia | 40 | Berlin | -| 5 | Noah | 29 | Tokyo | -🚀 现在您已经成功将数据导入到 Databend 中了！ +完成！如需了解分组件部署的方式，请参考[在 Docker 上部署](../02-deployment/01-non-production/00-deploying-local.md)。 ## 替代方案：Databend Cloud -如果设置本地环境比较麻烦，您可以尝试 [Databend Cloud](https://www.databend.com) 获得完全托管的体验。 -> 💬 **社区支持** -> 有疑问？联系我们的团队： -> 💬 [Slack 讨论](https://link.databend.com/join-slack) \ No newline at end of file +如果不想搭建本地环境，可以直接使用 [Databend Cloud](https://www.databend.com) 的全托管服务。 diff --git a/docs/cn/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md b/docs/cn/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md index a67cd200f1..2ae7988160 100644 --- a/docs/cn/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md +++ b/docs/cn/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md @@ -2,243 +2,144 @@ title: 在 Docker 上部署 --- - +本指南使用 Docker Compose 分别启动 **databend-meta**、**databend-query** 和 **MinIO** 三个容器，帮助你理解 Databend 各组件的关系。 -本指南将引导你使用 [Docker](https://www.docker.com/) 和 [MinIO](https://min.io/) 部署 Databend，以便在本地机器上进行完全容器化的设置。 - -:::note non-production use only -本指南中介绍的 MinIO 部署仅适用于开发和演示。由于单机环境中的资源有限，因此不建议将其用于生产环境或性能测试。 +:::note 仅用于非生产环境 +本方案仅适用于开发和本地测试，不适合生产环境或性能测试。 ::: ### 开始之前 -在开始之前，请确保你已具备以下先决条件： - -- 确保你的本地机器上已安装 [Docker](https://www.docker.com/)。 -- 确保你的机器上已安装 BendSQL。有关如何使用各种包管理器安装 BendSQL 的说明，请参阅 [安装 BendSQL](/guides/connect/sql-clients/bendsql/#installing-bendsql)。 +- 已安装 [Docker](https://www.docker.com/)（推荐 v26 及以上版本）。 +- 已安装 [BendSQL](https://docs.databend.com/guides/connect/sql-clients/bendsql/#installing-bendsql)。 -### 部署 MinIO +### 第一步：创建 docker-compose.yml -1. 使用以下命令拉取并运行 MinIO 镜像作为容器： +创建文件 `docker-compose.yml`，内容如下： -:::note - -- 我们在这里将控制台地址更改为 `9001`，以避免端口冲突。 -- 该命令还设置了 root 用户凭据 (`ROOTUSER`/`CHANGEME123`)，你需要在后续步骤中提供这些凭据进行身份验证。如果你此时更改 root 用户凭据，请确保在整个过程中保持一致。 - ::: +:::tip 国内用户 +如果拉取 Docker Hub 镜像较慢，可将镜像替换为国内镜像源： +- `minio/minio:latest` → 保持不变（MinIO 官方镜像） +- `datafuselabs/databend-meta:latest` → `registry.databend.cn/public/databend-meta:latest` +- `datafuselabs/databend-query:latest` → `registry.databend.cn/public/databend-query:latest` +::: -```shell -docker run -d --name minio \ - -e "MINIO_ACCESS_KEY=ROOTUSER" \ - -e "MINIO_SECRET_KEY=CHANGEME123" \ - -p 9000:9000 \ - -p 9001:9001 \ - minio/minio server /data \ - --address :9000 \ - --console-address :9001 +```yaml +services: + minio: + image: minio/minio:latest + network_mode: "host" + environment: + MINIO_ACCESS_KEY: minioadmin + MINIO_SECRET_KEY: minioadmin + volumes: + - minio-data:/data + entrypoint: | + sh -c " + minio server --address :9000 /data & + until mc alias set myminio http://localhost:9000 minioadmin minioadmin; do + echo 'Waiting for MinIO...'; + sleep 1; + done; + if ! mc ls myminio/databend > /dev/null 2>&1; then + mc mb myminio/databend; + fi; + wait; + " + + databend-meta: + image: datafuselabs/databend-meta:latest + network_mode: "host" + depends_on: + - minio + volumes: + - databend-meta-data:/var/lib/databend/meta + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:28101/v1/health"] + interval: 5s + timeout: 3s + retries: 10 + command: > + --log-file-level=warn + --log-file-dir=/var/log/databend + --admin-api-address=0.0.0.0:28101 + --grpc-api-address=0.0.0.0:9191 + --raft-listen-host=0.0.0.0 + --raft-api-port=28103 + --raft-dir=/var/lib/databend/meta + --id=1 + --single + + databend-query: + image: datafuselabs/databend-query:latest + network_mode: "host" + depends_on: + databend-meta: + condition: service_healthy + environment: + QUERY_DEFAULT_USER: databend + QUERY_DEFAULT_PASSWORD: databend + QUERY_STORAGE_TYPE: s3 + AWS_S3_ENDPOINT: http://127.0.0.1:9000 + AWS_S3_BUCKET: databend + AWS_ACCESS_KEY_ID: minioadmin + AWS_SECRET_ACCESS_KEY: minioadmin + META_ENDPOINTS: 0.0.0.0:9191 + volumes: + - query-logs:/var/log/databend + +volumes: + minio-data: + databend-meta-data: + query-logs: ``` -2. 运行命令 `docker logs minio` 以在日志消息中查找 MinIO API 和控制台 (WebUI) 地址： +### 第二步：启动 Databend ```shell -docker logs minio -``` - +docker compose up -d ``` -INFO: WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated. - Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD -INFO: Formatting 1st pool, 1 set(s), 1 drives per set. -INFO: WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable. -MinIO Object Storage Server -Copyright: 2015-2025 MinIO, Inc. -License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html -Version: RELEASE.2025-04-03T14-56-28Z (go1.24.2 linux/arm64) - -API: http://172.17.0.2:9000 http://127.0.0.1:9000 -WebUI: http://172.17.0.2:9001 http://127.0.0.1:9001 - -Docs: https://docs.min.io -INFO: - You are running an older version of MinIO released 5 days before the latest release - Update: Run `mc admin update ALIAS` -``` - -3. 在本地机器上打开 Web 浏览器，然后使用日志中显示的 WebUI 地址 (`http://127.0.0.1:9001`) 访问 MinIO 控制台。 - -![Alt text](/img/deploy/docker-minio.png) - -4. 使用凭据 `ROOTUSER`/`CHANGEME123` 登录到 MinIO 控制台，然后创建一个名为 `databend` 的 bucket。 - -![Alt text](/img/deploy/docker-bucket.png) - -### 部署 Databend -1. 使用以下命令拉取并运行 Databend 镜像作为容器： - -:::note - -- 启动 Databend Docker 容器时，可以使用环境变量 `QUERY_DEFAULT_USER` 和 `QUERY_DEFAULT_PASSWORD` 指定用户名和密码。如果未提供这些变量，将创建一个没有密码的默认 root 用户。 -- 以下命令还会创建一个 SQL 用户 (`databend`/`databend`)，你将需要使用该用户稍后连接到 Databend。如果你此时更改 SQL 用户，请确保在整个过程中保持一致。 - ::: +检查三个容器是否正常运行： ```shell -docker run -d \ - --name databend \ - -p 3307:3307 \ - -p 8000:8000 \ - -p 8124:8124 \ - -p 8900:8900 \ - -e QUERY_DEFAULT_USER=databend \ - -e QUERY_DEFAULT_PASSWORD=databend \ - -e QUERY_STORAGE_TYPE=s3 \ - -e AWS_S3_ENDPOINT=http://host.docker.internal:9000 \ - -e AWS_S3_BUCKET=databend \ - -e AWS_ACCESS_KEY_ID=ROOTUSER \ - -e AWS_SECRET_ACCESS_KEY=CHANGEME123 \ - datafuselabs/databend +docker compose ps ``` -2. 运行命令 `docker logs databend` 以检查 Databend 日志消息，并确保 Databend 容器已成功启动： +`minio`、`databend-meta`、`databend-query` 均应处于运行状态。查看日志： ```shell -docker logs databend +docker compose logs -f databend-query ``` -```shell -==> QUERY_CONFIG_FILE is not set, using default: /etc/databend/query.toml -==> /tmp/std-meta.log <== -Databend Metasrv - -Version: v1.2.697-d40f88cc51-simd(1.85.0-nightly-2025-02-14T11:57:01.874747910Z) -Working DataVersion: V004(2024-11-11: WAL based raft-log) - -Raft Feature set: - Server Provide: { append:v0, install_snapshot:v1, install_snapshot:v3, vote:v0 } - Client Require: { append:v0, install_snapshot:v3, vote:v0 } - -Disk Data: V002(2023-07-22: Store snapshot in a file); Upgrading: None - Dir: /var/lib/databend/meta - -Log File: enabled=true, level='Warn,databend_=Info,openraft=Info', dir=/var/log/databend, format=json, limit=48 - Stderr: enabled=false(To enable: LOG_STDERR_ON=true or RUST_LOG=info), level=WARN, format=text -Raft Id: 0; Cluster: foo_cluster - Dir: /var/lib/databend/meta - Status: single - -HTTP API listen at: 127.0.0.1:28002 -gRPC API listen at: 127.0.0.1:9191 advertise: - -Raft API listen at: 127.0.0.1:28004 advertise: 055b9e5d09a9:28004 - -Upgrade ondisk data if out of date: V002 - Find and clean previous unfinished upgrading -Upgrade on-disk data - From: V002(2023-07-22: Store snapshot in a file) - To: V003(2024-06-27: Store snapshot in rotbl) - No V002 snapshot, skip upgrade - Finished upgrading: V003 -Upgrade on-disk data - From: V003(2024-06-27: Store snapshot in rotbl) - To: V004(2024-11-11: WAL based raft-log) - Upgrade V003 raft log in sled db to V004 - Clean upgrading: V003 -> V004 (cleaning) - Remove V003 log from sled db - Removing sled tree: header - Removing sled tree: raft_log - Removing sled tree: raft_state - Done: Remove V003 log from sled db - Removing: /var/lib/databend/meta/heap - Removing: /var/lib/databend/meta/conf - Removing: /var/lib/databend/meta/db - Removing: /var/lib/databend/meta/DO_NOT_USE_THIS_DIRECTORY_FOR_ANYTHING - Finished upgrading: V004 -Upgrade ondisk data finished: V004 -Wait for 180s for active leader... -Leader Id: 0 - Metrics: id=0, Leader, term=1, last_log=Some(3), last_applied=Some(T1-N0.3), membership={log_id:Some(T1-N0.3), {voters:[{0:EmptyNode}], learners:[]}} - -Register this node: {id=0 raft=055b9e5d09a9:28004 grpc=} - - Register-node: Ok - -Databend Metasrv started - -==> /tmp/std-query.log <== -Databend Query - -Version: v1.2.697-d40f88cc51(rust-1.85.0-nightly-2025-02-14T11:30:59.842308760Z) - -Logging: - file: enabled=true, level='INFO', dir=/var/log/databend, format=json, limit=48 - stderr: enabled=false(To enable: LOG_STDERR_ON=true or RUST_LOG=info), level=WARN, format=text - -Meta: connected to endpoints [ - "0.0.0.0:9191", -] - -Memory: - limit: unlimited - allocator: jemalloc - config: percpu_arena:percpu,oversize_threshold:0,background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000 - -Cluster: standalone - -Storage: s3 | bucket=databend,root=,endpoint=http://host.docker.internal:9000 -Disk cache: - storage: none - path: DiskCacheConfig { max_bytes: 21474836480, path: "./.databend/_cache", sync_data: true } - reload policy: reset - -Builtin users: databend - -Builtin UDFs: - -Admin - listened at 0.0.0.0:8080 -MySQL - listened at 0.0.0.0:3307 - connect via: mysql -u${USER} -p${PASSWORD} -h0.0.0.0 -P3307 -Clickhouse(http) - listened at 0.0.0.0:8124 - usage: echo 'create table test(foo string)' | curl -u${USER} -p${PASSWORD}: '0.0.0.0:8124' --data-binary @- -echo '{"foo": "bar"}' | curl -u${USER} -p${PASSWORD}: '0.0.0.0:8124/?query=INSERT%20INTO%20test%20FORMAT%20JSONEachRow' --data-binary @- -Databend HTTP - listened at 0.0.0.0:8000 - usage: curl -u${USER} -p${PASSWORD}: --request POST '0.0.0.0:8000/v1/query/' --header 'Content-Type: application/json' --data-raw '{"sql": "SELECT avg(number) FROM numbers(100000000)"}' -``` +看到 `Databend Query started` 即表示启动成功。 -### 连接到 Databend - -在本地机器上启动一个终端，然后运行以下命令以连接到 Databend： +### 第三步：连接到 Databend ```shell bendsql -u databend -p databend ``` -```shell -Welcome to BendSQL 0.24.1-f1f7de0(2024-12-04T12:31:18.526234000Z). -Connecting to localhost:8000 as user databend. -Connected to Databend Query v1.2.697-d40f88cc51(rust-1.85.0-nightly-2025-02-14T11:30:59.842308760Z) -Loaded 1411 auto complete keywords from server. -Started web server at 127.0.0.1:8080 +执行一条简单查询验证： + +```sql +SELECT NOW(); ``` -一切就绪！现在，你可以执行一个简单的查询来验证部署： +### 停止 Databend -```sql -databend@localhost:8000/default> SELECT NOW(); +```shell +docker compose stop +``` -SELECT NOW() +彻底删除容器和数据卷： -┌────────────────────────────┐ -│ now() │ -│ Timestamp │ -├────────────────────────────┤ -│ 2025-04-10 03:14:06.778815 │ -└────────────────────────────┘ -1 row read in 0.003 sec. Processed 1 row, 1 B (333.33 rows/s, 333 B/s) +```shell +docker compose down -v ``` - \ No newline at end of file +### 下一步 + +- [使用 COPY INTO 加载数据](/guides/load-data/load/s3) +- [通过 MySQL 客户端或 JDBC 连接](/guides/connect/) +- 生产环境部署请参考 [在 Kubernetes 上部署](/guides/deploy/deploy/production/deploying-databend-on-kubernetes) diff --git a/docs/cn/guides/20-self-hosted/02-deployment/01-non-production/02-deploying-offline.md b/docs/cn/guides/20-self-hosted/02-deployment/01-non-production/02-deploying-offline.md new file mode 100644 index 0000000000..73aa756e3f --- /dev/null +++ b/docs/cn/guides/20-self-hosted/02-deployment/01-non-production/02-deploying-offline.md @@ -0,0 +1,79 @@ +--- +title: 离线环境 Docker 部署 +--- + +本指南适用于无法直接访问 Docker Hub 的离线或受限网络环境。 + +## 第一步：在有网络的机器上下载镜像 + +在可以联网的机器上拉取并导出所需镜像： + +```shell +docker pull datafuselabs/databend-meta:latest +docker pull datafuselabs/databend-query:latest +docker pull minio/minio:latest + +docker save datafuselabs/databend-meta:latest | gzip > databend-meta.tar.gz +docker save datafuselabs/databend-query:latest | gzip > databend-query.tar.gz +docker save minio/minio:latest | gzip > minio.tar.gz +``` + +:::tip 国内用户 +也可以从国内镜像源拉取，速度更快： +```shell +VERSION=v1.2.735-nightly +docker pull registry.databend.cn/public/databend-meta:${VERSION} +docker pull registry.databend.cn/public/databend-query:${VERSION} +``` +::: + +如需固定版本（生产环境推荐）： + +```shell +VERSION=v1.2.735-nightly +docker pull datafuselabs/databend-meta:${VERSION} +docker pull datafuselabs/databend-query:${VERSION} +docker save datafuselabs/databend-meta:${VERSION} | gzip > databend-meta-${VERSION}.tar.gz +docker save datafuselabs/databend-query:${VERSION} | gzip > databend-query-${VERSION}.tar.gz +``` + +## 第二步：传输并加载镜像 + +将 `.tar.gz` 文件上传到目标机器后，执行加载： + +```shell +docker load -i databend-meta.tar.gz +docker load -i databend-query.tar.gz +docker load -i minio.tar.gz +``` + +验证镜像已加载： + +```shell +docker images | grep -E "databend|minio" +``` + +## 第三步：启动 + +使用[在 Docker 上部署](./00-deploying-local.md)中的 `docker-compose.yml`，无需修改，镜像已在本地可用。 + +```shell +docker compose up -d +``` + +## 可选：推送到私有镜像仓库 + +如果公司内部有私有镜像仓库（例如 `registry.example.com`）： + +```shell +VERSION=v1.2.735-nightly +REGISTRY=registry.example.com + +docker tag datafuselabs/databend-meta:${VERSION} ${REGISTRY}/databend-meta:${VERSION} +docker tag datafuselabs/databend-query:${VERSION} ${REGISTRY}/databend-query:${VERSION} + +docker push ${REGISTRY}/databend-meta:${VERSION} +docker push ${REGISTRY}/databend-query:${VERSION} +``` + +然后将 `docker-compose.yml` 中的镜像地址替换为私有仓库地址即可。 diff --git a/docs/en/guides/20-self-hosted/01-quickstart/index.md b/docs/en/guides/20-self-hosted/01-quickstart/index.md index 1ac5514977..9686c264af 100644 --- a/docs/en/guides/20-self-hosted/01-quickstart/index.md +++ b/docs/en/guides/20-self-hosted/01-quickstart/index.md @@ -2,13 +2,11 @@ title: QuickStart --- -Databend Quick Start: Experience Databend in 5 Minutes -This guide will help you quickly set up Databend, connect to it, and perform a basic data import. +Get Databend running locally in under 5 minutes. -## 1. Start Databend with Docker -Run the following command to launch Databend in a container: +## 1. Start Databend -``` +```shell docker run -d \ --name databend \ --network host \ @@ -19,125 +17,41 @@ docker run -d \ --restart unless-stopped \ datafuselabs/databend ``` -Check if Databend is running successfully: -``` +Wait for Databend to be ready: + +```shell docker logs -f databend ``` -Wait until you see logs indicating that Databend and MinIO are ready. -## 2. Connect to Databend -Install bendsql (Databend CLI): +Look for `Databend Query started` in the output. -``` +## 2. Connect + +Install BendSQL: + +```shell curl -fsSL https://repo.databend.com/install/bendsql.sh | bash -echo "export PATH=$PATH:~/.bendsql/bin" >>~/.bash_profile +echo "export PATH=$PATH:~/.bendsql/bin" >> ~/.bash_profile source ~/.bash_profile ``` Connect to Databend: -``` -bendsql -udatabend -pdatabend -``` - -## 3. Perform a Basic Data Import -### Step 1: Create an External Bucket (myupload) -Install mc (MinIO client) and create a bucket: - -``` -wget https://dl.min.io/client/mc/release/linux-amd64/mc -sudo cp mc /usr/local/bin/ && sudo chmod +x /usr/local/bin/mc -mc alias set myminio http://localhost:9000 minioadmin minioadmin -mc mb myminio/myupload -mc ls myminio -``` -Expected output: -``` -[0001-01-01 08:05:43 LMT] 0B databend/ -[2025-04-12 08:43:59 CST] 0B myupload/ -``` -### Step 2: Generate a CSV and Upload to myupload +```shell +bendsql -u databend -p databend ``` -echo -e "id,name,age,city\n1,John,32,New York\n2,Emma,28,London\n3,Liam,35,Paris\n4,Olivia,40,Berlin\n5,Noah,29,Tokyo" > data.csv -mc cp data.csv myminio/myupload/ -mc ls myminio/myupload/ -``` -### Step 3: Create an External Stage and Check Data -Run in bendsql: -``` -CREATE STAGE mystage 's3://myupload' -CONNECTION=( - endpoint_url='http://127.0.0.1:9000', - access_key_id='minioadmin', - secret_access_key='minioadmin', - region='us-east-1' -); -``` -Show files in the external stage @mystage: -``` -LIST @mystage; -``` -| name | size | md5 | last_modified | creator | -|----------|--------|-------------------|-----------------------|-------------| -| String | UInt64 | Nullable(String) | String | Nullable(String) | -| data.csv | 104 | "a27fa15258911f534fb795a8c64e05d4" | 2025-04-12 00:51:11.015 +0000 | NULL | - -Preview the CSV data: -``` -SELECT $1, $2, $3, $4 FROM @mystage/data.csv (FILE_FORMAT=>'CSV') LIMIT 10; -``` -| \$1 | \$2 | \$3 | \$4 | -|-------------------|-------------------|-------------------|-------------------| -| Nullable(String) | Nullable(String) | Nullable(String) | Nullable(String) | -| id | name | age | city | -| 1 | John | 32 | New York | -| 2 | Emma | 28 | London | -| 3 | Liam | 35 | Paris | -| 4 | Olivia | 40 | Berlin | -| 5 | Noah | 29 | Tokyo | - - -### Step 4: Create a Table and Load Data -``` -CREATE DATABASE wubx; -USE wubx; - -CREATE TABLE t_person ( - id INT, - name VARCHAR, - age INT UNSIGNED, - city VARCHAR -); -COPY INTO t_person FROM @mystage PATTERN='.*[.]csv' FILE_FORMAT=(TYPE=CSV, SKIP_HEADER=1); +## 3. Verify +```sql +CREATE TABLE t (id INT, name VARCHAR); +INSERT INTO t VALUES (1, 'Databend'); +SELECT * FROM t; ``` -| File | Rows_loaded | Errors_seen | First_error | First_error_line | -|-----------|-------------|-------------|------------------|------------------| -| String | Int32 | Int32 | Nullable(String) | Nullable(Int32) | -| data.csv | 5 | 0 | NULL | NULL | - -### Step 5: Query the Data -``` -SELECT * FROM t_person; -``` -| id | name | age | city | -|----------|----------|----------|----------| -| Nullable(Int32) | Nullable(String) | Nullable(UInt32) | Nullable(String) | -| 1 | John | 32 | New York | -| 2 | Emma | 28 | London | -| 3 | Liam | 35 | Paris | -| 4 | Olivia | 40 | Berlin | -| 5 | Noah | 29 | Tokyo | - -🚀 Now you’ve successfully imported data into Databend! +You're all set. For a more realistic setup with separate components, see [Deploying on Docker](../02-deployment/01-non-production/00-deploying-local.md). ## Alternative: Databend Cloud -If setting up a local environment is troublesome, you can try [Databend Cloud](https://www.databend.com) for a fully managed experience. - -> 💬 **Community Support** -> Have questions? Connect with our team: -> 💬 [Slack Discussion](https://link.databend.com/join-slack) +Skip the local setup and try [Databend Cloud](https://www.databend.com) for a fully managed experience. diff --git a/docs/en/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md b/docs/en/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md index acacb53345..0ba1057c31 100644 --- a/docs/en/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md +++ b/docs/en/guides/20-self-hosted/02-deployment/01-non-production/00-deploying-local.md @@ -2,117 +2,137 @@ title: Deploying on Docker --- - +This guide walks you through deploying Databend using Docker Compose with separate **databend-meta**, **databend-query**, and **MinIO** containers — giving you a clear picture of how the components fit together. -This guide walks you through deploying Databend with [MinIO](https://min.io/) using [Docker](https://www.docker.com/) for a fully containerized setup on your local machine. - -:::note non-production use only -The MinIO deployment covered in this guide is only suitable for development and demonstration. Due to the limited resources in a single-machine environment, it is not recommended for production environments or performance testing. +:::note Non-production use only +This setup is intended for development and local testing only. It is not suitable for production environments or performance benchmarking. ::: ### Before You Start -Before you start, ensure you have the following prerequisites in place: - -- Ensure that [Docker](https://www.docker.com/) is installed on your local machine. -- Ensure that BendSQL is installed on your machine. See [Installing BendSQL](/guides/connect/sql-clients/bendsql/#installing-bendsql) for instructions on how to install BendSQL using various package managers. - -### Deploy Databend - -1. Pull and run the Databend image as a container with the following command: +- [Docker](https://www.docker.com/) (v26 or later recommended) installed on your machine. +- [BendSQL](https://docs.databend.com/guides/connect/sql-clients/bendsql/#installing-bendsql) installed on your machine. -:::note +### Step 1: Create docker-compose.yml -- When starting the Databend Docker container, you can specify the username and password using the environment variables `QUERY_DEFAULT_USER` and `QUERY_DEFAULT_PASSWORD`. If these variables are not provided, a default root user will be created without a password. -- The command below also creates a SQL user (`databend`/`databend`) which you will need to use to connect to Databend later. If you make changes to the SQL user at this point, ensure that you maintain consistency throughout the entire process. - ::: - -```shell -vim docker-compose.yml +Create a file named `docker-compose.yml` with the following content: +```yaml services: minio: - image: docker.io/minio/minio - command: server /data - ports: - - "9000:9000" + image: minio/minio:latest + network_mode: "host" environment: - - MINIO_ACCESS_KEY=MINIO_ADMIN - - MINIO_SECRET_KEY=MINIO_SECRET + MINIO_ACCESS_KEY: minioadmin + MINIO_SECRET_KEY: minioadmin + volumes: + - minio-data:/data + entrypoint: | + sh -c " + minio server --address :9000 /data & + until mc alias set myminio http://localhost:9000 minioadmin minioadmin; do + echo 'Waiting for MinIO...'; + sleep 1; + done; + if ! mc ls myminio/databend > /dev/null 2>&1; then + mc mb myminio/databend; + fi; + wait; + " + + databend-meta: + image: datafuselabs/databend-meta:latest + network_mode: "host" + depends_on: + - minio volumes: - - ./data:/data + - databend-meta-data:/var/lib/databend/meta healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] - interval: 30s - timeout: 5s - retries: 3 - databend: - image: datafuselabs/databend - environment: - - QUERY_DEFAULT_USER=databend - - QUERY_DEFAULT_PASSWORD=databend - - QUERY_STORAGE_TYPE=s3 - - AWS_S3_ENDPOINT=http://minio:9000 - - AWS_S3_BUCKET=databend - - AWS_ACCESS_KEY_ID=MINIO_ADMIN - - AWS_SECRET_ACCESS_KEY=MINIO_SECRET - ports: - - "3307:3307" - - "8000:8000" - - "8080:8080" + test: ["CMD", "curl", "-f", "http://localhost:28101/v1/health"] + interval: 5s + timeout: 3s + retries: 10 + command: > + --log-file-level=warn + --log-file-dir=/var/log/databend + --admin-api-address=0.0.0.0:28101 + --grpc-api-address=0.0.0.0:9191 + --raft-listen-host=0.0.0.0 + --raft-api-port=28103 + --raft-dir=/var/lib/databend/meta + --id=1 + --single + + databend-query: + image: datafuselabs/databend-query:latest + network_mode: "host" depends_on: - minio: - condition: service_started + databend-meta: + condition: service_healthy + environment: + QUERY_DEFAULT_USER: databend + QUERY_DEFAULT_PASSWORD: databend + QUERY_STORAGE_TYPE: s3 + AWS_S3_ENDPOINT: http://127.0.0.1:9000 + AWS_S3_BUCKET: databend + AWS_ACCESS_KEY_ID: minioadmin + AWS_SECRET_ACCESS_KEY: minioadmin + META_ENDPOINTS: 0.0.0.0:9191 + volumes: + - query-logs:/var/log/databend + +volumes: + minio-data: + databend-meta-data: + query-logs: +``` + +### Step 2: Start Databend + +```shell +docker compose up -d ``` -2. Start Databend +Check that all three containers are running: ```shell -docker compose up +docker compose ps ``` -### Connect to Databend +You should see `minio`, `databend-meta`, and `databend-query` all in a running state. To follow the logs: -Launch a terminal on your local machine, then run the following command to connect to Databend: +```shell +docker compose logs -f databend-query +``` + +Wait until you see a line like `Databend Query started`. + +### Step 3: Connect to Databend ```shell bendsql -u databend -p databend ``` +Run a quick check: + +```sql +SELECT NOW(); +``` + +### Stop Databend + ```shell -Welcome to BendSQL 0.24.1-f1f7de0(2024-12-04T12:31:18.526234000Z). -Connecting to localhost:8000 as user databend. -Connected to Databend Query v1.2.697-d40f88cc51(rust-1.85.0-nightly-2025-02-14T11:30:59.842308760Z) -Loaded 1411 auto complete keywords from server. -Started web server at 127.0.0.1:8080 +docker compose stop ``` -You're all set! Now, you can execute a simple query to verify the deployment: +To remove containers and volumes entirely: -```sql -🐳 databend@default:) CREATE OR REPLACE TABLE students (uid Int16, name String, age Int16); -🐳 databend@default:) INSERT INTO students VALUES (8888, 'Alice', 50); - -╭─────────────────────────╮ -│ number of rows inserted │ -│ UInt64 │ -├─────────────────────────┤ -│ 1 │ -╰─────────────────────────╯ -1 row written in 0.059 sec. Processed 1 row, 19 B (16.95 rows/s, 322 B/s) - -🐳 databend@default:) SELECT * FROM students; - -╭──────────────────────────────────────────────────────╮ -│ uid │ name │ age │ -│ Nullable(Int16) │ Nullable(String) │ Nullable(Int16) │ -├─────────────────┼──────────────────┼─────────────────┤ -│ 8888 │ Alice │ 50 │ -╰──────────────────────────────────────────────────────╯ -1 row read in 0.008 sec. Processed 1 row, 28 B (125 rows/s, 3.42 KiB/s) +```shell +docker compose down -v ``` - \ No newline at end of file +### Next Steps + +- [Load data with COPY INTO](/guides/load-data/load/s3) +- [Connect with MySQL client or JDBC](/guides/connect/) +- For production deployment, see [Deploying on Kubernetes](/guides/deploy/deploy/production/deploying-databend-on-kubernetes) diff --git a/docs/en/guides/20-self-hosted/02-deployment/01-non-production/02-deploying-offline.md b/docs/en/guides/20-self-hosted/02-deployment/01-non-production/02-deploying-offline.md new file mode 100644 index 0000000000..a311fb4dcd --- /dev/null +++ b/docs/en/guides/20-self-hosted/02-deployment/01-non-production/02-deploying-offline.md @@ -0,0 +1,70 @@ +--- +title: Offline Deployment with Docker +--- + +This guide covers deploying Databend in air-gapped or restricted network environments where direct access to Docker Hub is unavailable. + +## Step 1: Download Images on a Connected Machine + +On a machine with internet access, pull and export the required images: + +```shell +docker pull datafuselabs/databend-meta:latest +docker pull datafuselabs/databend-query:latest +docker pull minio/minio:latest + +docker save datafuselabs/databend-meta:latest | gzip > databend-meta.tar.gz +docker save datafuselabs/databend-query:latest | gzip > databend-query.tar.gz +docker save minio/minio:latest | gzip > minio.tar.gz +``` + +To pin a specific version (recommended for production): + +```shell +VERSION=v1.2.735-nightly +docker pull datafuselabs/databend-meta:${VERSION} +docker pull datafuselabs/databend-query:${VERSION} +docker save datafuselabs/databend-meta:${VERSION} | gzip > databend-meta-${VERSION}.tar.gz +docker save datafuselabs/databend-query:${VERSION} | gzip > databend-query-${VERSION}.tar.gz +``` + +## Step 2: Transfer and Load Images + +Copy the `.tar.gz` files to the target machine, then load them: + +```shell +docker load -i databend-meta.tar.gz +docker load -i databend-query.tar.gz +docker load -i minio.tar.gz +``` + +Verify the images are available: + +```shell +docker images | grep -E "databend|minio" +``` + +## Step 3: Deploy + +Use the same `docker-compose.yml` from [Deploying on Docker](./00-deploying-local.md) — no changes needed since the images are now available locally. + +```shell +docker compose up -d +``` + +## Optional: Push to a Private Registry + +If your organization uses a private Docker registry (e.g., `registry.example.com`): + +```shell +VERSION=v1.2.735-nightly +REGISTRY=registry.example.com + +docker tag datafuselabs/databend-meta:${VERSION} ${REGISTRY}/databend-meta:${VERSION} +docker tag datafuselabs/databend-query:${VERSION} ${REGISTRY}/databend-query:${VERSION} + +docker push ${REGISTRY}/databend-meta:${VERSION} +docker push ${REGISTRY}/databend-query:${VERSION} +``` + +Then update the image references in your `docker-compose.yml` accordingly.