DM-54879: Add --ignore-existing-metadata-for for when upstream outputs are selectively retained#561
Open
hsinfang wants to merge 4 commits into
Open
DM-54879: Add --ignore-existing-metadata-for for when upstream outputs are selectively retained#561hsinfang wants to merge 4 commits into
hsinfang wants to merge 4 commits into
Conversation
hsinfang
force-pushed
the
tickets/DM-54879
branch
from
d4f69bd to
4bf23e3
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #561 +/- ##
==========================================
+ Coverage 88.77% 88.80% +0.02%
==========================================
Files 159 160 +1
Lines 22053 22217 +164
Branches 2623 2631 +8
==========================================
+ Hits 19578 19729 +151
- Misses 1839 1849 +10
- Partials 636 639 +3 ☔ View full report in Codecov by Sentry. |
hsinfang
force-pushed
the
tickets/DM-54879
branch
from
4bf23e3 to
d6ef7aa
Compare
TallJimbo
approved these changes
May 22, 2026
| return zstandard.ZstdCompressionDict(b"") | ||
| self.comms.log.info("Training compression dictionary.") | ||
| training_inputs: list[bytes] = [] | ||
| training_inputs: list[bytes | bytearray | memoryview[int]] = [] |
Member
There was a problem hiding this comment.
I'm curious where this is coming from; AFAIK we don't use bytearray or memoryview[int] for any of these.
Contributor
Author
There was a problem hiding this comment.
I took it from mypy's suggestion directly, but this has now been fixed by ac0bcb1
| self.butler = butler.clone(collections=input_collections) | ||
| self.output_run = output_run | ||
| self.skip_existing_in = skip_existing_in | ||
| self.ignore_metadata_for: frozenset[str] = frozenset(ignore_metadata_for) |
Member
There was a problem hiding this comment.
We probably want to raise here if ignore_metadata_for is not empty but skip_existing_in empty (since then ignore_metadata_for would have no effect).
hsinfang
force-pushed
the
tickets/DM-54879
branch
2 times, most recently
from
7481f5b to
cf8760b
Compare
The skip_existing_in behavior of QuantumGraphBuilder was previously only covered through test_separable_pipeline_executor.py, where SeparablePipelineExecutor drives AllDimensionsQuantumGraphBuilder. No tests exercised the builder directly at the unit level.
hsinfang
force-pushed
the
tickets/DM-54879
branch
from
cf8760b to
3e4cff4
Compare
Daytime AP runs against data produced by Prompt Processing, which does not retain all output types. With --skip-existing-in, tasks whose metadata exists are skipped even when their outputs are absent, confusing downstream tasks. For tasks listed in the new retry_missing_outputs_for, the completion signal is changed from task metadata to all non-metadata, non-log outputs existing in skip_existing_in. If any are absent the quantum runs and regenerates them. When all outputs are present the quantum is skipped, and discard_output_in_the_way removes any copies already in output_run, so the behavior is identical to the standard skip path in that case. When outputs are absent, the quantum is not skipped and they cannot be "in the way" in a fresh output_run. _skip_quantum_if_metadata_exists is renamed to _skip_quantum_if_done, because the old name baked in the implementation detail (checking metadata). After retry_missing_outputs_for the function can also skip based on regular outputs, so the name no longer described the behavior.
SeparablePipelineExecutor is not used by pipetask, but we might as well extend the same option and get tested there.
hsinfang
force-pushed
the
tickets/DM-54879
branch
from
3e4cff4 to
52abd2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
package-docs builddoc/changes