Replace job-iteration with ActiveJob::Continuable#1461
Open
kwent wants to merge 4 commits intoShopify:mainfrom
Open
Replace job-iteration with ActiveJob::Continuable#1461kwent wants to merge 4 commits intoShopify:mainfrom
kwent wants to merge 4 commits intoShopify:mainfrom
Conversation
Drop the job-iteration gem dependency in favor of Rails 8.1's built-in ActiveJob::Continuable for cursor-based iteration and resumption. - Bump minimum Rails to 8.1, remove job-iteration dependency - Rewrite TaskJobConcern to use a single Continuable `step :iterate` - Add custom enumerator classes (ActiveRecordRecordEnumerator, ActiveRecordBatchEnumerator, ArrayEnumerator, CsvRowEnumerator, CsvBatchEnumerator, OnceEnumerator) replacing job-iteration's enumerator builders - Override checkpoint! for time-based max_job_runtime interruption - Handle two interruption paths: queue stopping (Continuable Interrupt) and app stopping (pause/cancel via Run status) - Add MaintenanceTasks.max_job_runtime config (replaces JobIteration.max_job_runtime) - Update CI matrix to drop Rails 7.x/8.0 gemfiles - Update README references from job-iteration to Continuable Closes Shopify#1398
Verify Continuable actually re-enqueues via retry_job when interrupted.
- Extract ActiveRecordCursor module shared by both AR enumerators (eliminates 4 duplicated private methods) - Cache ordered scope in build_scope to avoid rebuilding Arel per batch - @collection_enum ivar → local var (only used within step block) - check_throttle: use find idiom instead of each + early return
Author
|
I have signed the CLA! |
1 similar comment
Author
|
I have signed the CLA! |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1398.
Drop the
job-iterationgem dependency in favor of Rails 8.1's built-inActiveJob::Continuablefor cursor-based iteration and resumption. This is a new minor version that bumps the minimum Rails to 8.1 while preserving all existing behavior.What changed
job-iterationdependency, replacedinclude JobIteration::Iterationwithinclude ActiveJob::ContinuableTaskJobConcernto use a single Continuablestep :iterateblock that handles all collection typesActiveRecordRecordEnumerator— cursor-based AR record iteration with multi-column/composite PK supportActiveRecordBatchEnumerator— cursor-based AR batch iterationArrayEnumerator,CsvRowEnumerator,CsvBatchEnumerator,OnceEnumeratorMaintenanceTasks.max_job_runtimeconfig (default 5 minutes) replacingJobIteration.max_job_runtimeInterruptexception → auto re-enqueue viaretry_job@run.stopping?check → manual state persistenceKey design insight
ActiveJob::Continuation::Interruptinherits fromException(notStandardError), so the existingrescue_from StandardErrorerror handling flows through cleanly without conflicting with Continuable's interrupt mechanism.checkpoint!overrideContinuable's
checkpoint!directly checksqueue_adapter.stopping?. To supportmax_job_runtime, we overridecheckpoint!on the class level (inside theincludedblock) to also check elapsed time. This is necessary becauseActiveJob::Continuablesits aboveTaskJobConcernin the MRO when included in theincludedblock.Test plan