Skip to content

Commit 3122ed2

Browse files
leto-bleto-b
authored
add write back sink to table model data sync (#671)
* add write back sink to table model data sync * add opc ua sink and delete apache
1 parent d8f3d14 commit 3122ed2

8 files changed

Lines changed: 154 additions & 42 deletions

File tree

‎src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -401,7 +401,7 @@ WITH SINK (
401401
```
402402

403403

404-
## 4. Reference: Notes
404+
## Reference: Notes
405405

406406
You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows:
407407

@@ -464,9 +464,9 @@ pipe_sink_max_client_number=16
464464
pipe_all_sinks_rate_limit_bytes_per_second=-1
465465
```
466466

467-
## 5. Reference: Parameter Description
467+
## Reference: Parameter Description
468468

469-
### 5.1 source parameter
469+
### source parameter
470470

471471
| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** |
472472
| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- |
@@ -485,9 +485,9 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
485485
> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput.
486486
> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput.
487487
488-
### 5.2 sink parameter
488+
### sink parameter
489489

490-
#### 5.2.1 iotdb-thrift-sink
490+
#### iotdb-thrift-sink
491491

492492
| **Parameter** | **Description** | Value Range | Required | Default Value |
493493
| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ |
@@ -502,7 +502,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
502502
| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 |
503503
| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 |
504504

505-
#### 5.2.2 iotdb-thrift-ssl-sink
505+
#### iotdb-thrift-ssl-sink
506506

507507

508508
| **Parameter** | **Description** | Value Range | Required | Default Value |

‎src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md‎

Lines changed: 38 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -189,8 +189,8 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param
189189
<td>Default processor plugin that does not process incoming data.</td>
190190
</tr>
191191
<tr>
192-
<td rowspan="4">sink Plugin</td>
193-
<td rowspan="4">Supported</td>
192+
<td rowspan="6">sink Plugin</td>
193+
<td rowspan="6">Supported</td>
194194
<td>do-nothing-sink</td>
195195
<td>Does not process outgoing data.</td>
196196
</tr>
@@ -206,6 +206,14 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param
206206
<td>iotdb-thrift-ssl-sink</td>
207207
<td>Used for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded sync blocking IO model, suitable for high-security scenarios.</td>
208208
</tr>
209+
<tr>
210+
<td>write-back-sink</td>
211+
<td>A data write-back plugin for IoTDB (V2.0.2 and above) to achieve the effect of materialized views.</td>
212+
</tr>
213+
<tr>
214+
<td>opc-ua-sink</td>
215+
<td>An OPC UA protocol data transfer plugin for IoTDB (V2.0.2 and above), supporting both Client/Server and Pub/Sub communication modes. </td>
216+
</tr>
209217
</tbody>
210218
</table>
211219

@@ -418,7 +426,7 @@ WITH SINK (
418426
)
419427
```
420428

421-
## 4. Reference: Notes
429+
## Reference: Notes
422430

423431
You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows:
424432

@@ -491,9 +499,9 @@ pipe_air_gap_receiver_port=9780
491499
pipe_all_sinks_rate_limit_bytes_per_second=-1
492500
```
493501

494-
## 5. Reference: Parameter Description
502+
## Reference: Parameter Description
495503

496-
### 5.1 source parameter
504+
### source parameter
497505

498506
| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** |
499507
| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- |
@@ -512,9 +520,9 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
512520
> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput.
513521
> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput.
514522
515-
### 5.2 sink parameter
523+
### sink parameter
516524

517-
#### 5.2.1 iotdb-thrift-sink
525+
#### iotdb-thrift-sink
518526

519527
| **Parameter** | **Description** | Value Range | Required | Default Value |
520528
| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ |
@@ -529,7 +537,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
529537
| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 |
530538
| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 |
531539

532-
#### 5.2.2 iotdb-air-gap-sink
540+
#### iotdb-air-gap-sink
533541

534542
| **Parameter** | **Description** | Value Range | Required | Default Value |
535543
| :--------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ |
@@ -542,7 +550,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
542550
| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 |
543551
| air-gap.handshake-timeout-ms | The timeout duration for the handshake requests when the sender and receiver attempt to establish a connection for the first time, in milliseconds. | Integer | No | 5000 |
544552

545-
#### 5.2.3 iotdb-thrift-ssl-sink
553+
#### iotdb-thrift-ssl-sink
546554

547555
| **Parameter** | **Description** | Value Range | Required | Default Value |
548556
| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ |
@@ -557,4 +565,24 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
557565
| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 |
558566
| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 |
559567
| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - |
560-
| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - |
568+
| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - |
569+
570+
#### write-back-sink
571+
572+
| **Parameter** | **Description** | **value Range** | **Required** | **Default Value** |
573+
| ---------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |--------------|
574+
| sink | write-back-sink | String: write-back-sink | Yes | - |
575+
576+
#### opc-ua-sink
577+
578+
| **Parameter** | **Description** |Value Range | Required | Default Value |
579+
|:-------------------------------------|:-------------------------------------------------------------------------------------------------------| :------------------------------------- |:-------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
580+
| sink | opc-ua-sink | String: opc-ua-sink | Yes | - |
581+
| sink.opcua.model | OPC UA model used | String: client-server / pub-sub | No | pub-sub |
582+
| sink.opcua.tcp.port | OPC UA's TCP port | Integer: [0, 65536] | No | 12686 |
583+
| sink.opcua.https.port | OPC UA's HTTPS port | Integer: [0, 65536] | No | 8443 |
584+
| sink.opcua.security.dir | Directory for OPC UA's keys and certificates | String: Path, supports absolute and relative directories | No | Opc_security folder`<httpsPort: tcpPort>`in the conf directory of the DataNode related to iotdb <br> If there is no conf directory for iotdb (such as launching DataNode in IDEA), it will be the iotdb_opc_Security folder`<httpsPort: tcpPort>`in the user's home directory |
585+
| sink.opcua.enable-anonymous-access | Whether OPC UA allows anonymous access | Boolean | No | true |
586+
| sink.user | User for OPC UA, specified in the configuration | String | No | root |
587+
| sink.password | Password for OPC UA, specified in the configuration | String | No | root |
588+
| sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | Optional | "null" |

‎src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -401,7 +401,7 @@ WITH SINK (
401401
```
402402

403403

404-
## 4. Reference: Notes
404+
## Reference: Notes
405405

406406
You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows:
407407

@@ -464,9 +464,9 @@ pipe_sink_max_client_number=16
464464
pipe_all_sinks_rate_limit_bytes_per_second=-1
465465
```
466466

467-
## 5. Reference: Parameter Description
467+
## Reference: Parameter Description
468468

469-
### 5.1 source parameter
469+
### source parameter
470470

471471
| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** |
472472
| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- |
@@ -485,9 +485,9 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
485485
> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput.
486486
> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput.
487487
488-
### 5.2 sink parameter
488+
### sink parameter
489489

490-
#### 5.2.1 iotdb-thrift-sink
490+
#### iotdb-thrift-sink
491491

492492
| **Parameter** | **Description** | Value Range | Required | Default Value |
493493
| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ |
@@ -502,7 +502,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
502502
| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 |
503503
| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 |
504504

505-
#### 5.2.2 iotdb-thrift-ssl-sink
505+
#### iotdb-thrift-ssl-sink
506506

507507

508508
| **Parameter** | **Description** | Value Range | Required | Default Value |

0 commit comments

Comments
 (0)