Defnull/net 681 multi step scheduling 3#46
Draft
define-null wants to merge 20 commits into
Draft
Conversation
* feat(pg-storage): debug-only scheduler table row counts Add `PostgresStorage::table_sizes`, returning a `count(*)` for every table the scheduler touches (12 tables from 0001_sched_tables.sql, in a stable order). reshuffle-sim's multistep `step()` now logs table+count at debug level on every step; the count queries only run when debug logging is enabled, so it's free at the default log level. Un-gates `with_conn_ref` (was test-only) since it's now the read path for `table_sizes`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * remove comments from debug.rs * detach 'not yet wired' comment from mod debug * format table sizes as a single sorted, aligned log line * disable live table redraw when per-step debug logs are on The in-place redraw moves the cursor up and clears to end of screen, erasing log lines emitted between redraws. With the multistep table-size debug logs enabled, fall back to non-live rendering (stream logs, print table once at finish) so the logs survive. * simplify log_table_sizes to a single match * log table sizes after baseline ingestion too * remove log_table_sizes doc comment (review) --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/61 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
…stale capacity (#62) * reshuffle-sim: measure multistep shuffle on ideal placement + report stale capacity The multistep reshuffle metric diffed the published per-worker assignment between steps. That assignment is `ideal ∪ stale`, where stale = draining copies a worker still holds during a transition, so a chunk moved A→B appeared as a replication increase (the draining copy on A lingers) instead of a clean shuffle. Expose the ideal placement alone via WorkerAssignment.ideal_chunk_workers (populated by both storage backends) and diff ideal-vs-ideal in the tool, so true shuffles are isolated. used_capacity_bytes still measures the physical ideal∪stale union. Also add a "Stale %" report column: bytes held only by draining stale copies (used(union) − used(ideal)) as a share of total fleet capacity, plus a line on the capacity chart. Stateless path reports 0 (it rebuilds each step). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Query stale mappings from DB instead of propagating ideal via WorkerAssignment Per review: don't add an ideal_chunk_workers field to the published WorkerAssignment just for the sim. Instead expose PostgresStorage::stale_mappings() and have reshuffle-sim derive the ideal placement as published chunk_workers minus stale (the two are disjoint per chunk, so the subtraction is exact). The stale snapshot is taken inside run_cycle right after run_scheduling_cycle — before confirm/visibility expire or drain mappings — so it matches the ideal ∪ stale union in the returned assignment. stale_capacity_bytes is now the summed size of the stale copies directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Shrink reshuffle-sim diff: drop AssignmentOwners struct, flatten stale lookup owners_from_assignment goes back to a plain tuple return (matching the original 3-tuple style, now 4) instead of a one-off struct, and the per-chunk stale map becomes a flat HashSet<(ChunkPk, WorkerPk)>. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Use descriptive variable names in reshuffle-sim multistep path Rename terse bindings flagged in review: wa -> assignment, the worker-pk/ worker closure args (w/aw) -> worker_pk/worker, the duration match arm d -> duration, and the test's w*/aw* fixtures -> worker*_pk/worker*. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Trim comments to non-obvious rationale only Drop comments that restate the code (replication_by_weight, the stateless 0, the stale_pct formula, redundant test narration) and tighten the rest to the parts that aren't obvious from the code. --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/62 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
* Add stale % column to the live console table The HTML report had it but the console table (util.rs build_table) didn't — that's the table printed during a run. Insert a "Stale (% cap)" column after "Free cap (used %)" and bump the Workers span / single-col indices accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/63 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
…to Phase B (#64) * refactor(pg): make worker tombstone a tick, defer assignment-id mint to Phase B `sched_chunk_metadata.dropped_at_worker_assignment_id` was an FK to `sched_worker_assignments`, but every reader only tests its NULL-ness — no logic depends on which assignment it points to. The FK forced the cycle to mint the assignment id in Phase A (the GC transaction) so the tombstone UPDATE had a valid row to reference. On a Phase B shortage rollback that left a committed-but-unused assignment row (a phantom assignment) every cycle. Replace it with `dropped_from_worker_assignment_at BIGINT` — a bare drop tick, no FK (mirrors `marked_for_removal`). With nothing in Phase A needing an assignment id, `open_worker_assignment` moves into Phase B, after the algorithm succeeds, so a shortage no longer consumes an id. The in-memory oracle mirrors both changes (tick-valued tombstone; id allocated only after placement succeeds) to preserve pg/oracle parity under shortage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pg): trim cycle comments to declarative present tense Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/64 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
…trategy MultistepAlgorithm built the scheduler input from 's output — which drops uncovered chunks and may reorder — but built the pk and current-placement vectors from the original chunk order. As soon as the weight strategy dropped or reordered a chunk the two desynced: placements were attributed to the wrong chunk, and chunks the strategy doesn't cover were still scheduled.
* Share ring machinery + cache rings across cycles (#57, #60) Squashed ring-optimization history for replay onto the multistep base. --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/60
* perf(pg): push the scheduling cycle's set-diffing into the database via a twin-table swap
Replace the app-side reconcile (materialise prev_ideal in a BTreeMap, diff it
against the algorithm output in Rust, flatten back into parallel arrays) with a
twin `sched_future_ideal_chunk_workers` table and pure SQL set operations.
Each cycle stages the new ideal into the (empty) twin, then derives every delta
by joining the live ideal (prev) against the twin (new):
- stale minting : current holders the twin drops, known workers, not mid-drop
- routing diffs : full-join compare, IS DISTINCT FROM on canonical arrays
- flip-flops : stale rows whose pair is back in the twin
- entered stamps : metadata for chunks new to the twin
then TRUNCATE + 3-way RENAME promotes the twin to the live ideal and leaves the
twin empty for the next cycle. No inbound FKs reference the ideal table, so the
swap breaks nothing; DDL is transactional, so a shortage rollback restores both.
The only app-side marshalling left is writing the algorithm's output into the
twin. Unplaced chunks (no holder) are simply absent — the GROUP BY insert never
writes an empty holder set, matching the algorithm, which never emits one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* address review: push the current-placement union into the database
The algorithm's input placement (ideal ∪ stale) was fetched as two row sets and
unioned in Rust (merge_current_placement). Replace it with a single SQL query
that UNIONs the two sources and array_aggs each chunk's distinct, ordered holder
set (fetch_current_placement). Apply the same shape to build_worker_assignment's
published view (with the servable filter), retiring merge_current_placement and
the now-unused StaleWorkerRow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* address review: fetch active chunks and their placement in one query
The cycle fetched active chunks and the current placement (ideal ∪ stale) in two
separate queries. Unite them: fetch_active_chunks_with_placement LEFT JOINs the
placement union onto the active-chunk scan, returning an ActiveChunkRow (chunk +
holders) decoded in one pass. The algorithm only consults placement for chunks it
schedules, so restricting the placement to the active set is behaviour-preserving;
a never-placed active chunk comes back '{}' and is skipped, so current_placement
still mirrors the oracle (entries only where placement exists).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* address review: NULL-as-unplaced, and one query for the published assignment
#1328: the cycle's active-chunks query COALESCEd missing placement to '{}' and
Rust then filtered '{}' back out. Drop the COALESCE so an unplaced chunk comes
back NULL (ActiveChunkRow.worker_ids is now Option); Rust inserts only Some —
the "has placement?" decision lives in SQL, no empty-vs-absent dance.
build_worker_assignment: unite its two queries (placement union, then a second
chunk fetch by placed pk) into one placement-driven join returning ActiveChunkRow,
decoded in a single pass — one fewer round-trip, and the IdealChunkWorkersRow row
type retires.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(pg): batch the post-write deltas + swap into one round-trip
The four post-write delta phases (mint stale, record diffs, resolve flip-flops,
stamp entered) plus the 4-statement swap were 8 sequential statements = 8 round
trips on the single connection. They're mutually independent (disjoint rows,
different target tables) and the swap is always last, so collapse them into one
simple-query batch (sqlx::raw_sql) that runs them sequentially in the cycle
transaction — same semantics, one round-trip.
The simple protocol takes no bind parameters, so new_wa_id (self-minted, never
user input) is substituted as a literal; AssertSqlSafe acknowledges that. execute
sums rows_affected across the batch, so the phase timer still reports total rows
(no longer split per phase — the one observability cost).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(pg): don't fetch chunk files in the scheduling cycle
The algorithm never reads Chunk.files (it uses dataset/id/size/weight/version),
and the chunks handed to it are moved into schedule() and never read again — yet
the cycle's fetch selected the heavy files TEXT[] for every active chunk and
decoded it into a Vec<String> per chunk, all unused. build_worker_assignment
already re-fetches real files for the (smaller) placed subset where they're
needed.
Select an empty '{}'::text[] placeholder instead of c.files so ChunkRow still
decodes (now an empty Vec) with no struct change and no effect on the other
ChunkRow query sites, which keep selecting real files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(pg): write the future ideal via a composite type, dropping the regroup
write_future_ideal flattened each chunk's holder set into parallel (chunk, worker)
scalar arrays, then UNNEST + GROUP BY + ar...---------
Co-authored-by: Defnull <879658+define-null@users.noreply.github.com>
Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/66
Co-authored-by: claude <claude@example.com>
Co-committed-by: claude <claude@example.com>
* feat(reshuffle-sim): copy and replace whole datasets Two one-off dataset operations for modeling rare, high-impact events: - --copy-dataset SRC=DST: pre-run baseline transform that clones every chunk of SRC under a new dataset id (identical chunk id, block range, size), copies its initial placement, and inherits SRC's config weight. Works with both schedulers. - --replace-dataset BUCKET --replace-at-step N: at step N, supersede every live chunk of BUCKET with a same-range copy (id + "-replaced") via the storage correction path, so a full re-ingest's reshuffle is measured. Multistep only. To reach the real replace path from the tool, register_correction is un-gated from cfg(test) to the existing pg-testkit feature: the impl moves out of the test-only tests/ subtree into a sibling postgres/correction.rs and the trait method + Postgres impl are gated cfg(any(test, feature = "pg-testkit")). The trait stays the public entry point; the in-memory backend is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * address review: add --copy-at-step so copy mirrors replace Copy is now a step-triggered event (--copy-dataset SRC=DST --copy-at-step N), matching the --replace-dataset/--replace-at-step shape. Unlike replace (which needs the storage correction path), copy is purely additive, so it stays working for both schedulers: the clone is injected as ordinary new chunks at the step through the simulation loop. The multistep scheduler now registers a dataset the first time it appears, so the copy's new dataset works on the Postgres path too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/67 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
…le in-memory state (#71) * fix(reshuffle-sim): source replace/copy chunks from Postgres, not stale in-memory state Both whole-dataset operations on the multistep path read the target chunks from the wrong place, so neither reflects what storage actually holds. - replace: `replace_dataset` built its target set from `latest_chunks`, a clone of a prior published `WorkerAssignment.chunks`. That map is the *placed* subset (`build_worker_assignment`'s query keeps only chunks with an ideal or stale mapping), so any active-but-unplaced chunk of the target dataset was silently never superseded — a partial re-ingest, and skewed reshuffle metrics. - copy: `build()` does `mem::take(&mut baseline.chunks)`, so `Simulation` stores an empty `baseline_chunks` on the multistep path; `copy_chunks_due` then cloned almost nothing (a silent no-op when the copy fires before new chunks land). Fix: add `PostgresStorage::live_chunks_in_dataset`, returning every live chunk of a dataset (servable/supersedable state — matches `register_correction`'s acceptance — with real files). `replace_dataset` and a new `copy_dataset` read the target/source set fresh from storage at the step. Copy is now handled inside the multistep scheduler (clone from PG + register the dst dataset); the stateless path keeps injecting clones through the sim loop. This drops the `latest_chunks` retention and the `known_datasets` seeding machinery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * address review: source replace/copy from the retained assignment, not a new storage query Drops the PostgresStorage::live_chunks_in_dataset addition (and its test) and reverts correction.rs to baseline. Instead both operations resolve their targets from latest_chunks — the previous cycle's published WorkerAssignment.chunks, already retained in the scheduler and sourced from Postgres. On this branch that map carries real files and pks, and a validly-replaceable/copyable dataset (real, weight > 0, no version restriction) is fully placed, so the placed set is its live set. - replace_dataset: back to filtering latest_chunks (as before the regression fix). - copy_dataset: now also reads latest_chunks instead of the emptied sim baseline, which is what fixes the multistep copy — no storage/library change needed. - latest_chunks retention generalized to "any replace OR copy still ahead" (op_pending), and the copy is handled in the scheduler so it can see it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * address review: copy from the current step's assignment, placed the same step Copy no longer clones from the retained latest_chunks snapshot (state carried from an earlier cycle). It now runs after this step's scheduling cycle, clones src from the just-published assignment — so it reflects src as it stands when the copy fires, including this step's ingest — then runs a second cycle so the clones are placed in the same step. latest_chunks reverts to replace-only (replace needs the pre-cycle placement for its same-cycle swap); the op_pending helper is gone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * address review: stateless copy also reflects src as of the copy step The stateless copy runs through the sim loop (copy_chunks_due); it was computed before this step's ingest was folded into the accumulated set, so it cloned src as of the previous step. Reorder so the copy is taken after appending this step's new chunks — it now clones src as of the copy step, matching the multistep path (which clones from this step's fresh assignment). Stateless was never subject to the emptied-baseline bug (build() is multistep-only), so this is a consistency fix, not a correctness one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * address review: document why copy is handled per-scheduler Keep copy split between the two schedulers (no logic change) and add doc comments on both copy_chunks_due (stateless) and copy_dataset (multistep) explaining the rationale: multistep drops the in-memory baseline since Postgres is its source of truth, so it clones from the storage assignment rather than an in-memory set, which avoids forcing it to retain the full baseline in RAM just for copy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * address review: tighten comments Trim the verbose comments/doc-comments added across the tool to one or two lines each, keeping only the non-obvious rationale (why copy is per-scheduler, why replace reads latest_chunks, the two-cycle copy). No code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/71 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
* perf(pg): GiST range index for the non-overlap probe Registering a whole dataset in one batch (~147K chunks, reshuffle-sim --copy-dataset) drove the overlap probe past 8 minutes: the nearest-left btree walk liveness-checks every row it skips, and a brand-new dataset has no live row to stop at, so each candidate walks all its batch-mates -> O(N^2). Measured on production-shaped data (6.3M live chunks): 93s at just 20K candidates, ~85min extrapolated at 147K; the walk also costs ~1s per routine 2000-chunk append step. Replace the (dataset_id, first_block) btree with a composite GiST over (dataset_id, int8range(first_block, first_block+last_block_delta, '[]')) and probe with the matching && overlap, so the index itself yields only truly intersecting rows (~0-1 per candidate) and cost no longer depends on liveness: append 2000 probe ~1s -> 46ms, 147,560-chunk new dataset ~85min -> 4.8s (postgres:18.4-alpine, stock config). The probe keeps the LATERAL .. LIMIT 1 shape rather than a WHERE EXISTS: an EXISTS can flatten into a semi join, and a plan that sees the array cardinality picks a merge join sorting every live chunk (observed on the first five executions of each fresh connection); a LATERAL with LIMIT can't be flattened, so the plan stays on the index regardless of stats or plan-cache state. Verified: 231 tests pass incl. the PG-vs-in-memory oracle; the new query returns verdicts identical to the old on start/end-overlap, containment, exact-duplicate, contained-within, clear, and dead-but-overlapping cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/73 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
* perf: make 147K-chunk dataset replace/copy first-class fast paths Replacing or copying a whole production-size dataset (~147K chunks) hit four independent slow paths beyond the fixed overlap probe: - register_correction only existed per-chunk: a dataset replace was 147,560 sequential transactions (~900K round-trips), measured ~57s on a local socket and minutes over TCP. New batch register_corrections runs the same guards set-based and lands replacement chunks + correction links in one statement per 10K batch, one transaction total: measured ~8.8s for the full 147K (and independent of network latency). - apply_ready_corrections resolved readiness with an O(P^2) fixpoint (per-candidate rescan of the whole pending Vec): ~2.2e10 compares, tens of seconds of CPU inside the visibility transaction. An O(1) unfired-producer set lookup makes it O(P) per pass. - The sim placed a copied dataset by cloning 147K chunks out of the in-memory assignment, re-uploading them through jsonb, and running a SECOND full scheduling cycle (~162s at 6.36M chunks). Copy is now a server-side INSERT..SELECT of the source's live set before the step's single cycle, which also places the clones. - The tombstoned-stale DELETE re-scanned every previously tombstoned chunk each cycle, forever growing after a replace; it is now scoped to this cycle's tombstones and indexed. drop_marked_chunks similarly full- scanned 6.5M metadata rows each visibility cycle (~205ms idle) and now rides a matching partial index. The sim also stops cloning the whole 6.36M-entry assignment map each step while a replace is pending (only the target dataset's chunks are retained). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pg): single correction path — register_correction is a batch of one Review feedback: the batch API duplicated the per-item guard chain (availability, range, insert+link), two implementations to keep in lock-step. Delete the per-item chain and delegate register_correction to register_corrections(vec![..]). Batch errors carry the per-item detail: the range guard reports both ranges, unique violations name which constraint broke (replacement id vs one-correction-per-old), and the unknown-dataset path names the dataset (error-path diagnostic query). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pg): drop register_correction_impl; the trait method calls the batch API Review: the _impl indirection served only to host the per-item chain, which is gone — register_correction is register_corrections(vec![..]) inline in the trait impl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(pg): import HashSet/HashMap instead of full std::collections paths Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pg): PostgresStorage methods live in mod.rs, modules keep the meat Review: correction.rs and registration.rs carried impl PostgresStorage blocks; rewire them to the module convention — the inherent methods (register_corrections, copy_dataset_chunks) sit in mod.rs and delegate to free functions in their modules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: rustfmt reflow after HashSet import shortened the line Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(pg): note when the unknown-dataset reject can fire Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pg): gate copy_dataset_chunks to test/pg-testkit like the corrections API Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/74 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
* perf(pg): derive the routing fetch's visible set server-side fetch_confirmed_routing filtered sched_confirmed_chunk_workers through a ~6.5M-element chunk_pk array parameter — the client rebuilt, serialized (~50MB) and uploaded the full visible-pk list every visibility cycle, and the server hashed every element before probing. The visible set is already expressible as the same metadata predicate fetch_portal_visible_chunks reads (applied_at_portal NOT NULL, dropped_at_portal NULL), and evict_portal_overlaps writes its un-promotes back before the routing fetch, so a plain join sees exactly the set the in-memory assignment holds. Measured server-side at 6.3M visible chunks: 2.17s -> 0.74s, plus the client-side array build and 50MB upload gone entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/75 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
define-null
force-pushed
the
defnull/net-681-multi-step-scheduling-3
branch
3 times, most recently
from
7984ef7 to
24d38d3
Compare
define-null
force-pushed
the
defnull/net-681-multi-step-scheduling-3
branch
from
24d38d3 to
a2c0b2b
Compare
define-null
force-pushed
the
defnull/net-681-multi-step-scheduling-3
branch
from
f7758cb to
74dd9a3
Compare
…es, per-pipeline Chunk types (#77) * feat(scheduler-storage): per-dataset schema, schema-derived chunk files, per-pipeline Chunk types Squash of the claude/dataset-schema branch (review history in PRs #58 and #76) rebased onto a7186bd (benchmarking ingest/run/full paths). - schemas table: per-dataset content-hash dedup, one current schema per dataset, superseded rows kept for pinning. - chunks.schema_id NOT NULL: inserts stamp COALESCE(explicit pin, dataset's current schema); files derive from the schema at assignment construction (chunks.files is gone). - chunks.tables_present varbit bitmap (NULL = all present), same-dataset composite FK guard. - Per-pipeline chunk types: types::Chunk (legacy, files) / WorkerAssignmentChunk (published; schema_id + tables_present) / NewChunk (insert input) / AlgoChunk behind the weight::SchedulingChunk accessor trait; the hot cycle read decodes the slim AlgoChunkRow. - Batched inserts ship parallel typed arrays via UNNEST (bitmap binds as varbit[]); shared STALE_AGG_CTE/PLACEMENT_MERGE SQL fragments; row decoders consume their rows; load_schemas (all-or-ids) decodes jsonb directly. - In-memory oracle mirrors all of it; server-side test helpers live in postgres/testkit; pg test helpers deduped into tests/mod.rs. Rebase reconciliation: seeding moved inside a7186bd's Ingestion match — Run|Only seeds datasets with DatasetSchema::default() and converts baseline chunks via to_storage_chunk; Skip is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/network-scheduler/pulls/77 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.