Skip to content

Commit b4b88bf

Browse files
authored
Merge pull request #16 from zwk4zhendeC/alpha
sink describe
2 parents eb67971 + 2f2023d commit b4b88bf

2 files changed

Lines changed: 107 additions & 1 deletion

File tree

docs-en/00-release/0.20.md

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,26 +4,77 @@
44

55
### Added Elasticsearch Sink.
66
Supports batch writing parsed records to the Elasticsearch `/_bulk` API over HTTP/HTTPS, with timeout, retry, and password authentication support.
7+
Example:
8+
```toml
9+
[connectors.params]
10+
protocol = "http" # Transport protocol; switch to https if needed
11+
host = "localhost" # Elasticsearch server host
12+
port = "9200" # Elasticsearch service port
13+
index = "wp_nginx" # Target index name
14+
username = "elastic" # Username
15+
password = "" # Password
16+
batch_size = 1_0000 # Number of records per batch
17+
```
718
- Usage doc: [Elasticsearch Sink](../10-user/05-connectors/02-sinks/20-elasticsearch_sink.md)
819
- Example index: [wp-examples/extensions/elasticsearch/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/elasticsearch/README.md)
920

1021
### Added PostgreSQL Sink.
11-
Supports writing Record data to PostgreSQL tables in batches based on the configured `columns`, which fits relational storage and idempotent retry scenarios.
1222
- Usage doc: [PostgreSQL Sink](../10-user/05-connectors/02-sinks/23-postgresql_sink.md)
1323
- Example index: [wp-examples/extensions/tcp_postgres/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/tcp_postgres/README.md)
1424

1525
### Added Doris Sink.
1626
Supports batch loading data into Apache Doris through the Stream Load API, with custom Stream Load headers, timeout, retry, and password authentication support.
27+
Example:
28+
```toml
29+
[connectors.params]
30+
endpoint = "http://localhost:8040" # Use the BE HTTP port
31+
user = "root" # Username
32+
password = "" # Password
33+
database = "" # Database name
34+
table = "" # Table name
35+
batch_size = 1_0000 # Request batch size
36+
37+
# Optional: custom Stream Load parameters
38+
[connectors.params.headers]
39+
strip_outer_array = "false"
40+
max_filter_ratio = "0.1"
41+
strict_mode = "false"
42+
```
1743
- Usage doc: [Doris Sink](../10-user/05-connectors/02-sinks/17-doris.md)
1844
- Example index: [wp-examples/extensions/doris/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/doris/README.md)
1945

2046
### Added ClickHouse Sink.
2147
Supports writing to ClickHouse with the RowBinary format over HTTP, with timeout, retry, and password authentication support.
48+
Example:
49+
```toml
50+
[connectors.params]
51+
endpoint = "http://localhost:8123"
52+
username = "" # Username
53+
password = "" # Password
54+
database = "" # Database name
55+
table = "" # Table name
56+
batch_size = 1_0000 # Request batch size
57+
```
2258
- Usage doc: [ClickHouse Sink](../10-user/05-connectors/02-sinks/21-clickhouse.md)
2359
- Example index: [wp-examples/extensions/clickhouse](https://github.com/wp-labs/wp-examples/tree/main/extensions/clickhouse)
2460

2561
### Added HTTP Sink.
2662
Supports sending parsed records to HTTP/HTTPS endpoints for webhooks, API integrations, and custom pipelines, with `json`, `ndjson`, `csv`, `kv`, `raw`, and `proto-text` output formats, plus Basic Auth, custom headers, gzip compression, timeout, and retry support.
63+
Example:
64+
```toml
65+
[sink_group.sinks.params]
66+
endpoint = "http://localhost:8080/webhook" # Target HTTP endpoint
67+
method = "POST" # HTTP method
68+
username = "" # Use HTTP Basic authentication
69+
password = "" # Use HTTP Basic authentication
70+
fmt = "json" # Output format: `json`, `ndjson`, `csv`, `kv`, `raw`, `proto-text` (default `json`)
71+
batch_size = 1_0000 # Batch send size
72+
compression = "none" # Whether to enable gzip compression
73+
74+
# Optional: custom header parameters
75+
[connectors.params.headers]
76+
# content-type = "application/json"
77+
```
2778
- Usage doc: [HTTP Sink](../10-user/05-connectors/02-sinks/22-http.md)
2879
- Example index: [wp-examples/extensions/http/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/http/README.md)
2980

docs-zh/00-release/0.20.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,25 +5,80 @@
55

66
### 新增支持 ElasticSearch Sink。
77
支持通过 HTTP/HTTPS 将解析后的记录批量写入 Elasticsearch的`/_bulk`接口,支持超时、重试与密码认证配置。
8+
示例:
9+
```toml
10+
[connectors.params]
11+
protocol = "http" # 连接协议,按需改为 https
12+
host = "localhost" # Elasticsearch 服务地址
13+
port = "9200" # Elasticsearch 服务端口
14+
index = "wp_nginx" # 写入的目标索引名
15+
username = "elastic" # 用户名
16+
password = "" # 密码
17+
batch_size = 1_0000 # 单批次写入记录数
18+
```
819
- 使用文档:[Elasticsearch Sink](../10-user/05-connectors/02-sinks/20-elasticsearch_sink.md)
920
- 示例索引:[wp-examples/extensions/elasticsearch/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/elasticsearch/README.md)
1021

22+
1123
### 新增支持 Postgres Sink。
1224
- 使用文档:[PostgreSQL Sink](../10-user/05-connectors/02-sinks/23-postgresql_sink.md)
1325
- 示例索引:[wp-examples/extensions/tcp_postgres/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/tcp_postgres/README.md)
1426

1527
### 新增支持 Doris Sink。
1628
支持通过 Stream Load API 将数据批量写入 Apache Doris,支持自定义 Stream Load headers、超时、重试与密码认证。
29+
示例:
30+
```toml
31+
[connectors.params]
32+
endpoint = "http://localhost:8040" # 使用 BE 的 HTTP 端口
33+
user = "root" # 用户名
34+
password = "" # 密码
35+
database = "" # 数据库
36+
table = "" # 表名
37+
batch_size = 1_0000 # 请求批量大小
38+
39+
# 可选:自定义 Stream Load 参数
40+
[connectors.params.headers]
41+
strip_outer_array = "false"
42+
max_filter_ratio = "0.1"
43+
strict_mode = "false"
44+
```
1745
- 使用文档:[Doris Sink](../10-user/05-connectors/02-sinks/17-doris.md)
1846
- 示例索引:[wp-examples/extensions/doris/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/doris/README.md)
1947

48+
2049
### 新增支持 ClickHouse Sink。
2150
支持通过 使用 HTTP 传输的 RowBinary 格式写入Clickhouse接口,支持超时、重试与密码认证配置。
51+
示例:
52+
```toml
53+
[connectors.params]
54+
endpoint= "http://localhost:8123"
55+
username = "" # 用户名
56+
password = "" # 密码
57+
database = "" # 数据库
58+
table = "" # 表名
59+
batch_size = 1_0000 # 请求批量大小
60+
```
2261
- 使用文档:[ClickHouse Sink](../10-user/05-connectors/02-sinks/21-clickhouse.md)
2362
- 示例索引:[wp-examples/extensions/clickhouse](https://github.com/wp-labs/wp-examples/tree/main/extensions/clickhouse)
2463

64+
2565
### 新增支持 HTTP Sink。
2666
支持将解析后的记录发送到 HTTP/HTTPS 端点,适用于 webhook、API 集成和自定义数据管道,支持将数据以 `json``ndjson``csv``kv``raw``proto-text` 格式输出,以及 Basic Auth认证、自定义请求头与 gzip 压缩,超时、重试配置。
67+
示例:
68+
```toml
69+
[sink_group.sinks.params]
70+
endpoint = "http://localhost:8080/webhook" # 目标 HTTP 端点
71+
method = "POST" # 请求方法
72+
username = "" # 使用HTTP Basic 认证
73+
password = "" # 使用HTTP Basic 认证
74+
fmt = "json" # 输出格式:`json`、`ndjson`、`csv`、`kv`、`raw`、`proto-text`(默认 `json`)
75+
batch_size = 1_0000 # 批量发送大小
76+
compression = "none" # 是否启用 gzip 压缩
77+
78+
# 可选:自定义 header 参数
79+
[connectors.params.headers]
80+
# content-type = "application/json"
81+
```
2782
- 使用文档:[HTTP Sink](../10-user/05-connectors/02-sinks/22-http.md)
2883
- 示例索引:[wp-examples/extensions/http/README.md](https://github.com/wp-labs/wp-examples/blob/main/extensions/http/README.md)
2984

0 commit comments

Comments
 (0)