Introduce RetryHeap and RetryHeapProcessor #1972
Open
+381
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
agarakan
added
the
ready for testing
agarakan
changed the base branch from
enable-multithreaded-logging-by-default
to
poison-pill
sky333999
reviewed
Dec 31, 2025
| ) | ||
|
|
||
| // retryHeapImpl implements heap.Interface for logEventBatch sorted by nextRetryTime | ||
| type retryHeapImpl []*logEventBatch |
Contributor
There was a problem hiding this comment.
nit: Add var _ heap.Interface = (*retryHeapImpl)(nil)
Contributor
Author
There was a problem hiding this comment.
Ack will update as follow up
| } | ||
|
|
||
| // RetryHeap manages failed batches during their retry wait periods | ||
| type RetryHeap interface { |
Contributor
There was a problem hiding this comment.
nit: Add var _ RetryHeap = (*retryHeap)(nil)
Contributor
Author
There was a problem hiding this comment.
Ack will update as follow up
| for _, batch := range readyBatches { | ||
| // Check if batch has expired | ||
| if batch.isExpired(p.maxRetryDuration) { | ||
| p.logger.Debugf("Dropping expired batch for %s/%s", batch.Group, batch.Stream) |
Contributor
There was a problem hiding this comment.
nit: Make it match https://github.com/aws/amazon-cloudwatch-agent/pull/1970/changes#diff-d38a5f3264bf20ef00aa1acd29a08512a7d92c63384dcacfd3737b70cdf7311cR119 for consistency with our logging.
Contributor
Author
There was a problem hiding this comment.
Ack will update as follow up
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the issue
Introduce RetryHeap and RetryHeapProcessor implementations for cloudwatchlogs log publishing
Description of changes
RetryHeap is implemented as a min-heap sorted on retryTimestamp for a given batch PutLogEvent request that has already been attempted once and failed.
RetryHeapProcessor will periodically scan for batches that are ready to be retried based on retryTimestamp and move them to the SenderPool tasks queue for processing by the sender pool.
Assumptions: The RetryHeap and RetryHeapProcessor will only be instantiated and used when concurrency is enabled.
No functionality changes are introduced in this PR as this sets up RetryHeap and related components, but does not instantiate them in the code yet.
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
New unit tests are passing
Requirements
Before commiting your code, please do the following steps.
make fmtandmake fmt-shmake lintIntegration Tests
To run integration tests against this PR, add the
ready for testinglabel.