Collector: Add collector-distribution packaging module & Migrate IoTDBDataRegionSyncConnector & Add scheduled triggering for batch accumulation operations & Add subscribe source demo & Update iotdb dependence version #61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
YC27
commented
Apr 23, 2025
YC27
commented
- Add collector-distribution packaging module
- Migrate IoTDBDataRegionSyncConnector
- Add scheduled triggering for batch accumulation operations
- Add subscribe source demo
- Update iotdb dependence version
…SyncConnector & Add scheduled triggering for batch accumulation operations & Add subscribe source demo
(cherry picked from commit 81b635a)
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces several improvements, including the addition of a collector-distribution packaging module, migration of the IoTDBDataRegionSyncConnector, enhancements for scheduled batch accumulation operations, a new subscribe source demo, and an update to the IoTDB dependency version.
- Updated file paths and configurations to use runtime options (e.g., PluginRuntimeOptions, TaskRuntimeOptions)
- Added new runtime options via a comprehensive PipeRuntimeOptions configuration file
- Registered a new PeriodicalJobService in the Application and updated the CodeQL queries in the CI workflow
Reviewed Changes
Copilot reviewed 114 out of 124 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| collector-core/src/main/java/org/apache/iotdb/collector/persistence/PluginPersistence.java | Updated import order and file path resolution via PluginRuntimeOptions |
| collector-core/src/main/java/org/apache/iotdb/collector/persistence/Persistence.java | Minor formatting update surrounding database initialization |
| collector-core/src/main/java/org/apache/iotdb/collector/persistence/DBConstant.java | Changed default database file paths and URL construction to align with runtime options |
| collector-core/src/main/java/org/apache/iotdb/collector/config/TaskRuntimeOptions.java | Introduced new TASK_DATABASE_FILE_PATH option with home directory integration |
| collector-core/src/main/java/org/apache/iotdb/collector/config/PluginRuntimeOptions.java | Modified plugin library directory and added home directory resolution for plugin DB file path |
| collector-core/src/main/java/org/apache/iotdb/collector/config/PipeRuntimeOptions.java | Added comprehensive new runtime options for pipe configuration |
| collector-core/src/main/java/org/apache/iotdb/collector/config/Options.java | Added helper method addHomeDir for relative directory resolution |
| collector-core/src/main/java/org/apache/iotdb/collector/config/Configuration.java | Enhanced configuration loading with COLLECTOR_CONF and COLLECTOR_HOME support |
| collector-core/src/main/java/org/apache/iotdb/collector/Application.java | Registered PeriodicalJobService for scheduled operations |
| .github/workflows/code-analysis.yml | Updated CodeQL query set to include security and quality checks |
Files not reviewed (10)
- iotdb-collector/collector-core/pom.xml: Language not supported
- iotdb-collector/collector-core/src/assembly/core.xml: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/conf/application.properties: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/conf/logback.xml: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/collector-env.sh: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/common.sh: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/start-collector.bat: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/start-collector.sh: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/stop-collector.bat: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/stop-collector.sh: Language not supported
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces several improvements including packaging, migration of connectors, scheduled triggering for batch operations, and a new subscribe source demo while updating the iotdb dependency version.
- Added a collector-distribution packaging module.
- Migrated IoTDBDataRegionSyncConnector and updated database file paths & URLs.
- Introduced PipeRuntimeOptions and PeriodicalJobService, along with updates to configuration and CI workflows.
Reviewed Changes
Copilot reviewed 115 out of 125 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| collector-core/src/main/java/org/apache/iotdb/collector/persistence/Persistence.java | Added an extra initialization step by invoking initDatabaseFileIfPossible() after saving the database URL. |
| collector-core/src/main/java/org/apache/iotdb/collector/persistence/DBConstant.java | Updated file path constants and adjusted database URL construction using runtime options. |
| collector-core/src/main/java/org/apache/iotdb/collector/config/TaskRuntimeOptions.java | Refactored setValue method signatures and added an option for TASK_DATABASE_FILE_PATH. |
| collector-core/src/main/java/org/apache/iotdb/collector/config/PluginRuntimeOptions.java | Updated plugin library directory and added PLUGIN_DATABASE_FILE_PATH with home directory resolution. |
| collector-core/src/main/java/org/apache/iotdb/collector/config/PipeRuntimeOptions.java | Introduced a new configuration file with various pipeline related options. |
| collector-core/src/main/java/org/apache/iotdb/collector/config/Options.java | Added a helper method addHomeDir to support relative directory resolution. |
| collector-core/src/main/java/org/apache/iotdb/collector/config/Configuration.java | Enhanced configuration file lookup by incorporating system properties for collector home/conf. |
| collector-core/src/main/java/org/apache/iotdb/collector/Application.java | Added PeriodicalJobService to the list of services. |
| .github/workflows/compile-check.yml | Changed the Maven command from compile to package with integration tests. |
| .github/workflows/code-analysis.yml | Updated CodeQL queries configuration from security-extended to security-and-quality. |
Files not reviewed (10)
- iotdb-collector/collector-core/pom.xml: Language not supported
- iotdb-collector/collector-core/src/assembly/core.xml: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/conf/application.properties: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/conf/logback.xml: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/collector-env.sh: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/common.sh: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/start-collector.bat: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/start-collector.sh: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/stop-collector.bat: Language not supported
- iotdb-collector/collector-core/src/assembly/resources/sbin/stop-collector.sh: Language not supported
SteveYurongSu
approved these changes
Apr 24, 2025
SteveYurongSu
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.