Skip to content

Commit e465c3d

Browse files
W1y1rW1y1r
authored
Multi level storage adapted to non AWS object storage systems compliant with S3 protocol (#676)
1 parent ce1a1d5 commit e465c3d

8 files changed

Lines changed: 54 additions & 22 deletions

File tree

‎src/UserGuide/Master/Table/User-Manual/Tiered-Storage_timecho.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,13 @@ The specific parameter definitions and their descriptions are as follows.
4141
| `dn_data_dirs` | `data/datanode/data` | Specifies storage directories grouped into tiers. | Tiers are separated by `;`, directories within the same tier are separated by `,`. <br>Cloud storage (e.g., AWS S3) can only be the last tier. <br>Use `OBJECT_STORAGE` to denote cloud storage. <br>Only one cloud storage bucket is allowed. |
4242
| `tier_ttl_in_ms` | `-1` | Defines the TTL (in milliseconds) for each tier to determine the data range it manages. | Tiers are separated by `;`. <br>The number of tiers must match `dn_data_dirs`. <br>`-1` means "no limit". |
4343
| `dn_default_space_usage_thresholds` | `0.85` | Defines the minimum remaining space threshold (as a ratio) for each tier. When a tier’s remaining space falls below this threshold, data is migrated to the next tier. <br>The last tier triggers `READ_ONLY` mode. | -Tiers are separated by `;`.The number of tiers must match `dn_data_dirs`. |
44-
| `object_storage_type` | `AWS_S3` | Cloud storage type. | Only `AWS_S3` is supported. |
44+
| `object_storage_type` | `AWS_S3` | Cloud storage type. | all `AWS_S3` is supported. |
4545
| `object_storage_bucket` | `iotdb_data` | Cloud storage bucket name. | Required only if cloud storage is used. |
4646
| `object_storage_endpoiont` | (Empty) | Cloud storage endpoint. | Required only if cloud storage is used. |
47+
| `object_storage_region` | (Empty) | Cloud storage Region. | Required only if cloud storage is used. |
4748
| `object_storage_access_key` | (Empty) | Cloud storage access key. | Required only if cloud storage is used. |
4849
| `object_storage_access_secret` | (Empty) | Cloud storage access secret. | Required only if cloud storage is used. |
50+
| `enable_path_style_access` | false | Whether to enable path style access for object storage service. | Required only if cloud storage is used. |
4951
| `remote_tsfile_cache_dirs` | `data/datanode/data/cache` | Local cache directory for cloud storage. | Required only if cloud storage is used. |
5052
| `remote_tsfile_cache_page_size_in_kb` | `20480` | Page size (in KB) for cloud storage local cache. | Required only if cloud storage is used. |
5153
| `remote_tsfile_cache_max_disk_usage_in_mb` | `51200` | Maximum disk space (in MB) allocated for cloud storage local cache. | Required only if cloud storage is used. |
@@ -79,11 +81,13 @@ tier_ttl_in_ms=86400000;864000000;-1
7981
dn_default_space_usage_thresholds=0.2;0.15;0.1
8082
object_storage_type=AWS_S3
8183
object_storage_bucket=iotdb
84+
object_storage_region=<your_region>
8285
object_storage_endpoiont=<your_endpoint>
8386
object_storage_access_key=<your_access_key>
8487
object_storage_access_secret=<your_access_secret>
8588

8689
# Optional configurations
90+
enable_path_style_access=false
8791
remote_tsfile_cache_dirs=data/datanode/data/cache
8892
remote_tsfile_cache_page_size_in_kb=20971520
8993
remote_tsfile_cache_max_disk_usage_in_mb=53687091200
@@ -95,4 +99,4 @@ remote_tsfile_cache_max_disk_usage_in_mb=53687091200
9599
| :------- | :--------------------------- | :----------------------------- | :---------------------------- |
96100
| Tier 1 | `/data1/data` | Last 1 day of data | 20% |
97101
| Tier 2 | `/data2/data`, `/data3/data` | Data from 1 day to 10 days ago | 15% |
98-
| Tier 3 | AWS S3 Cloud Storage | Data older than 10 days | 10% |
102+
| Tier 3 | S3 Cloud Storage | Data older than 10 days | 10% |

‎src/UserGuide/Master/Tree/User-Manual/Tiered-Storage_timecho.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,11 +39,13 @@ The specific parameter definitions and their descriptions are as follows.
3939
| dn_data_dirs | data/datanode/data | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
4040
| tier_ttl_in_ms | -1 | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs；"-1" means "unlimited". |
4141
| dn_default_space_usage_thresholds | 0.85 | Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
42-
| object_storage_type | AWS_S3 | Cloud Storage Type | IoTDB currently only supports AWS S3 as a remote storage type, and this parameter can't be modified |
42+
| object_storage_type | `AWS_S3` | Cloud storage type. | all `AWS_S3` is supported. |
4343
| object_storage_bucket | iotdb_data | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used |
4444
| object_storage_endpoiont | | endpoint of cloud storage | endpoint of AWS S3；If remote storage is not used, no configuration required |
45+
| object_storage_region | (Empty) | Cloud storage Region. | Required only if cloud storage is used. |
4546
| object_storage_access_key | | Authentication information stored in the cloud: key | AWS S3 credential key；If remote storage is not used, no configuration required |
4647
| object_storage_access_secret | | Authentication information stored in the cloud: secret | AWS S3 credential secret；If remote storage is not used, no configuration required |
48+
| enable_path_style_access | false | Whether to enable path style access for object storage service. | Required only if cloud storage is used. |
4749
| remote_tsfile_cache_dirs | data/datanode/data/cache | Cache directory stored locally in the cloud | If remote storage is not used, no configuration required |
4850
| remote_tsfile_cache_page_size_in_kb | 20480 |Block size of locally cached files stored in the cloud | If remote storage is not used, no configuration required |
4951
| remote_tsfile_cache_max_disk_usage_in_mb | 51200 | Maximum Disk Occupancy Size for Cloud Storage Local Cache | If remote storage is not used, no configuration required |
@@ -75,13 +77,15 @@ The following takes three-level storage as an example:
7577
dn_data_dirs=/data1/data;/data2/data,/data3/data;OBJECT_STORAGE
7678
tier_ttl_in_ms=86400000;864000000;-1
7779
dn_default_space_usage_thresholds=0.2;0.15;0.1
78-
object_storage_name=AWS_S3
80+
object_storage_type=AWS_S3
7981
object_storage_bucket=iotdb
82+
object_storage_region=<your_region>
8083
object_storage_endpoiont=<your_endpoint>
8184
object_storage_access_key=<your_access_key>
8285
object_storage_access_secret=<your_access_secret>
8386

8487
// Optional configuration items
88+
enable_path_style_access=false
8589
remote_tsfile_cache_dirs=data/datanode/data/cache
8690
remote_tsfile_cache_page_size_in_kb=20971520
8791
remote_tsfile_cache_max_disk_usage_in_mb=53687091200
@@ -93,4 +97,4 @@ In this example, a total of three levels of storage are configured, specifically
9397
| -------- | -------------------------------------- | ---------------------------- | ------------------------ |
9498
| tier1 | path 1：/data1/data | data for last 1 day | 20% |
9599
| tier2 | path 1：/data2/data path 2：/data3/data | data from past 1 day to past 10 days | 15% |
96-
| tier3 | Remote AWS S3 Storage | data from 10 days ago | 10% |
100+
| tier 3 | S3 Cloud Storage | Data older than 10 days | 10% |

0 commit comments

Comments
 (0)