Adds support for out of order processing of events arriving with a delay in the RSP Engine. #12
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ails currently.
This commit adds "CSPARQLWindow" to the "classes_to_log" array in the log_config.json file. It also introduces a new IntervalManager class in the IntervalManager.ts file, which is responsible for managing intervals and clearing them when necessary.
This commit refactors the CSPARQLWindow class to improve its functionality and readability. It also adds support for watermarking, allowing for better handling of out-of-order events. The changes include: - Improved window management and event processing - Added methods for checking if an event is late or not - Implemented buffering of late events for out-of-order processing - Eviction and triggering of windows based on the watermark - Updated tests to cover the new functionality
…nd late element processing This commit refactors the CSPARQLWindow class to improve its functionality and readability. It also adds support for setting the maximum delay for out-of-order events and processing late elements. The changes include: - Updated constructor to accept options object for max delay and time to trigger processing of late elements - Modified RSPEngine constructor to pass the options object to CSPARQLWindow - Updated tests to cover the new functionality
…nd late element processing
…nd late element processing
…nd late element processing
…window This commit refactors the `CSPARQLWindow` class to add a conditional check for the length of the `content` before triggering the window. If the `content` has a length greater than 0, the window is triggered and the `RStream` event is emitted. Otherwise, a log message is added to indicate that the window has no data.
refactor: Add simple-statistics npm dependency and update log_config.json This commit adds the "simple-statistics" npm dependency to the project, which is required for performing statistical calculations. It also updates the "log_config.json" file in both the source and distribution directories to include the "R2ROperator" class in the "classes_to_log" array. This ensures that log messages are generated for the "R2ROperator" class as well.
…window This commit adds a conditional check for the length of the `content` before triggering the window in the `CSPARQLWindow` class. If the `content` has a length greater than 0, the window is triggered and the `RStream` event is emitted. Otherwise, a log message is added to indicate that the window has no data.
…window This commit refactors the `CSPARQLWindow` class to add a conditional check for the length of the `content` before triggering the window. If the `content` has a length greater than 0, the window is triggered and the `RStream` event is emitted. Otherwise, a log message is added to indicate that the window has no data.
…mp equal to the current time of the window.
…ulation in CSPARQLWindow
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello,
I am requesting a possible review of the logic and implementation.
With this PR, events arriving late and out of order are processed and added to the windows.
A
max_delayvariable decides the acceptance criteria of an out of order event arriving to the engine. Post themax_delaytime, the event is not added to the engine and is discarded. In case the event is lesser than themax_delayset, it is processed.The
process_late_elements()function is triggered at regular intervals, based on the slide of the CSPARQL Window to process and subsequently clear the buffer of the late elements.I will work towards having 100% coverage of the tests.