Skip to content

Conversation

@agarakan
Copy link
Contributor

Description of the issue

Track putlogevents retry metadata as part of the batch struct to persist retry data.

Description of changes

  1. Update batch struct to include retry metadata
  2. Introduce helper methods to initialize and update retry metadata
  3. Move retry metadata update logic from Sender to batch, where Sender will call the helper methods to maintain existing functionality.

Changes include structural changes, no functional changes. This is in preparation for subsequent changes to the PLE retry processing logic.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Unit testing passes

make fmt 
make fmt-sh
make lint 
make test

Requirements

Before commiting your code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@agarakan agarakan requested a review from a team as a code owner December 30, 2025 05:17
@agarakan agarakan added the ready for testing Indicates this PR is ready for integration tests to run label Dec 30, 2025
@agarakan agarakan changed the base branch from main to enable-multithreaded-logging-by-default December 30, 2025 16:31
s.logger.Errorf("All %v retries to %v/%v failed for PutLogEvents, request dropped.", retryCountShort+retryCountLong-1, batch.Group, batch.Stream)
// Check if retry would exceed max duration
totalRetries := batch.retryCountShort + batch.retryCountLong - 1
if batch.isExpired(s.RetryDuration()) || batch.nextRetryTime.After(batch.startTime.Add(s.RetryDuration())) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wont the 2nd condition in this line always be triggered before isExpired returns true?

Also, can the second part be moved into isExpired()? Would the following achieve the same?

maxWaitTime := b.startTime.Add(maxRetryDuration)
b.nextRetryTime.After(maxWaitTime) || time.Now().After(maxWaitTime)

If this is equivalent to your logic, then even here we can see that the first part will always be true before the second part can be true. Since nextRetryTime in n-1 iteration will be more than time.Now() in the nth iteration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the time.Now() check in isExpired() is useful for the subsequent PRs when the RetryHeapProcessor comes into play?

In that case, should the condition here just be:
if batch.nextRetryTime.After(batch.startTime.Add(s.RetryDuration()))

}

// isExpired checks if the batch has exceeded the maximum retry duration.
func (b *logEventBatch) isExpired(maxRetryDuration time.Duration) bool {
Copy link
Contributor

@sky333999 sky333999 Dec 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can we just store a field called expireAfter on the logBatch which holds when that batch will expire? You can then just set that when the batch is created based on the start time and maxDuration. That way you dont need to pass it in each time here and more importantly, you dont need to pass it to the RetryHeapProcessor.
You also dont need to worry if diff batches have diff max durations for some reason down the line etc - as the RetryHeapProcessor is common.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it was an alternate option, i saw them as equvalent. The reason I kept this is because the maxRetryDuration seems to be a parameter in the existing code, so I wasnt clear if this can somehow change, and kept the same. Will consider revising in a follow up

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants