feat: lazy shape store, native IFC pipeline, in-browser conversion, viewer mesh tools#235
Open
Krande wants to merge 140 commits into
Open
feat: lazy shape store, native IFC pipeline, in-browser conversion, viewer mesh tools#235Krande wants to merge 140 commits into
Krande wants to merge 140 commits into
Conversation
…metry Large STEP/IFC imports materialised as ada.geom trees cost ~10x their serialized size in resident memory (Munin crane: 2.6 GB settled vs 275 MB of NGEOM blobs). ShapeStore keeps each solid as one compact blob (NGEOM buffers retained zero-copy as they arrive from adacpp; Python-built geometry pickled losslessly incl. bool_operations) and hydrates the Geometry tree on demand behind a WeakValueDictionary, mirroring the FEM array-substrate design. ShapeProxy subclasses Shape, holds only (store, index), and exposes the raw blob for adacpp fast paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…irst auto reader - Config cad.lazy_shape_store (default on, ADA_CAD_LAZY_SHAPE_STORE=false to opt out) + cad.shape_store_compress: _read_step_streaming stores each solid as its NGEOM blob (native path, zero hydration at import) or pickled ada.geom tree (pure-Python path) and mints ShapeProxy objects. - reader='auto' now probes the native adacpp parser first (matching iter_from_step), with a first-solid hydrate probe and pure-Python fallback; StepReader.NATIVE + from_step Literal accept 'native' explicitly. - native_stream_read_step_blobs: per-solid (blob, meta) without hydration; native_stream_read_step reimplemented on top. - ConnectedFaceSet becomes a first-class B-rep root: solid_geom() accepts it, the OCC builder builds it, and NGEOM hydration promotes topologically closed face-sets back to ClosedShell (edge-pairing check, value-keyed). - AP242 stream writer shares EDGE_CURVEs by value signature instead of object identity, so NGEOM-hydrated trees (which decode one object per reference) re-emit watertight solids identical to Python-reader trees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- AdaCppBackend.tessellate_stream_buffer: tessellate a pre-encoded NGEOM buffer (a ShapeStore blob) directly — no hydration, no re-serialization; tessellate_stream now routes through it. - BatchTessellator: ShapeProxy solids tessellate straight from their stored blob when the stream kernel is selected; the bare-curve sniff skips proxies (it would hydrate the whole tree, and stored solids are never bare curves). - IFC writer: ShapeProxy accepted alongside Shape for parametric export; bare ConnectedFaceSet roots map to the ClosedShell face-set emit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
import_ifc_shape stores natively-read product geometry as one compact pickled blob per product (bool_operations, half-space operands and parametric profiles round-trip exactly) on a per-IfcStore ShapeStore and mints ShapeProxy objects; kernel-fallback products keep eager Shapes. Gated by the same cad.lazy_shape_store config (default on). Crane numbers for the STEP side of the store (native reader, 7291 solids): eager 2723/2556 MB peak/settled 96 s -> lazy 709/543 MB 56 s; all 7291 blobs hydrate cleanly and RSS returns to the blob floor after dropping the trees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ilter The exact-type filter excluded lazy proxies from by_type=Shape queries (topology from_assembly found no solids). A proxy normalises to its public type; Shape subclasses like PrimBox stay excluded as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Geometry.bool_operations were dropped by every stream-tessellation caller (the wrapper was stripped before serialization), so IFC clipping cuts and API booleans only evaluated on the OCC build path. Now: - serialize root() accepts core.Geometry wrappers and folds bool_operations into a nested BOOLEAN_RESULT (tag 52) chain — base as first operand, ops in order — which adacpp already decodes and evaluates via Manifold. - HalfSpaceSolid operands are lowered at encode time to a finite box on the material side of the plane, sized by the base solid's loose bbox — the exact lowering adacpp's native STEP/IFC readers apply (mk_halfspace), so no new NGEOM tag and no adacpp change is needed; the neutral path stays OCC-free (pure ada.geom bbox math). - BatchTessellator passes the wrapper through to tessellate_stream. Native-read CSG STEP roots (BOOLEAN_RESULT in an adacpp blob) still have no Python-side decode: the auto reader's hydrate-probe falls back to the pure- Python reader for such files, which parses BooleanResult natively. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Removes the per-instance __dict__ from every curve/surface/solid/placement/ boolean dataclass — the dominant per-object overhead when a B-rep model is hydrated (millions of instances). core.Geometry stays un-slotted: it is the weakref-cache value in the lazy ShapeStore (weakref_slot needs py>=3.11) and there is only one per solid. Two NGEOM serializer tests attached weights_data ad hoc to the non-rational B-spline classes; they now construct the Rational subclasses (real field). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
B-rep IFC products the Python-native readers can't resolve previously fell to the eager IfcOpenShell OCC kernel (unpicklable transient bodies, whole- model materialisation). They now try adacpp's dep-free IfcNgeomStream first: one guid-keyed scan per import, buffers retained zero-copy in the per-IfcStore ShapeStore, single-instance placement restored from the stream's composed world matrix. Color/props/spatial hierarchy still come from ifcopenshell (the stream's v1 carries geometry+guid+name+placement only); kernel fallback unchanged for products the native resolver skips, and adacpp builds without IfcNgeomStream degrade gracefully. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Audit regression: non-promoted native NGEOM roots hydrate as bare ConnectedFaceSet with FaceSurface members; the adacpp backend's build dispatch only knew the ClosedShell/OpenShell + AdvancedFace spellings and raised 'not yet ported'. ConnectedFaceSet sews like the other shells; FaceSurface is AdvancedFace's structurally-identical sibling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion LRU - SAT wire bodies round-tripped through IFC are bare-curve shapes; the proxy curve-sniff skip forced them down solid_geom() and they vanished from scenes (parity ifc 1->0). ShapeRecord now records curve-ness at store time and the tessellator asks the record — line rendering restored with zero hydration. - ~40% conversion slowdown: consumers read .geom 3-5x per call and each temporary died between property evaluations, so the weak cache re-decoded the blob per access. ShapeStore adds a small strong LRU window (hydration_cache_size=16, ~5 MB) over the weak cache; hydrate-all still returns to the blob floor (eviction test included). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
promote_closed_shell (edge-pairing closedness check) cost ~17% of native_stream_read_step on the crane, and the streaming exporters (obj/stl/step emit) that iterate it don't care about shell closedness. Promotion now happens only where it matters: lazy-store hydration (already there) and the eager from_step Assembly wrap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚀 Profiling Results (Top 20 most expensive calls)
|
…filer When the admin profile_conversions setting is on, the fork child gets ADACPP_STEP_PROFILE=1 alongside cProfile: adacpp prints [STEPPROF] phase wall times, RSS at phase boundaries, VmHWM peak, per-solid stats and parallelism/IO pressure to stderr, which the captured job Log keeps — the C++ sibling of the .prof artefact. Scoped per job like the other env overrides; nothing is baked into the container. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When profile_conversions is on, the worker parses the adacpp profiler's [STEPPROF-JSON] summaries out of the captured child log into convert_meta.cpp_profile, and the Metrics tab renders them next to the Python profile: per-pipeline phase wall/RSS breakdown with share bars, kernel-exact peak (VmHWM), per-solid stats, achieved parallelism, disk IO pressure and per-thread utilisation. Torn/interleaved lines are skipped; profiling off leaves no trace in meta or output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The audit log panel gains a state dropdown (queued / running / done / error) next to the action/target filters. Server-side: the /admin/audit route exposes a status param wired to list_audit's existing statuses filter, so it composes with keyset pagination instead of filtering one page client-side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ete-key + shift-arrow selection Corpus tab tree view gains the storage panel's organize toolkit via a shared FileTreeView mutations layer: inline rename of files/folders, client-side pending folders (New folder / New subfolder), kebab menus (download, move-to-folder picker, upload-here, delete with cascades), and drag-and-drop moves of files and whole folder subtrees with a move-to-root strip. Drag payloads embed the source scope in a distinct MIME so drags straying between panels/scopes are ignored, not mis-moved. Checkbox multi-select feeds a bulk Move/Copy/Delete toolbar and multi-key drags. Both panels: "Upload files" now prompts for the destination (existing folder / new path / top level, default top level) via FolderPickerModal allowRoot; keyboard lists support shift+Arrow selection extension and Delete-key deletes (selection first, else the focused file/folder) with confirm dialogs listing exactly which keys go (previewKeyList). Corpus files can be server-side copied into the caller's personal scope (Garage CopyObject, keys preserved, existing keys skipped) per file or per selection. InlineNameInput extracted from StorageBrowser into components/common so both panels share it. Read-only FileTreeView callers are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anel PositionedMenu and FolderPickerModal portal to document.body at z-50, but the admin Rnd panel host is a fixed z-[60] overlay — so the corpus tab's kebab/context menus and its move/upload destination prompts rendered behind the panel. Raise both to z-[70]. (The copy-from-scope modal was unaffected: it renders inline inside the panel's stacking context.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…escription
Both storage panels now show an in-panel spinner status line while a
batch file operation runs — previously a drag-drop move of many files
gave no feedback until the listing refreshed. Corpus tab: bar under the
button row covering moves, folder moves, bulk/folder deletes, and
copy-to-personal; storage panel: line under the header for drag-drop,
picker, and folder moves. Multi-key moves/copies are chunked (8 keys
per request) purely so the counter ticks between requests — every chunk
is still a server-side S3 rename/copy (CopyObject on Garage); no file
bytes pass through the browser. A busy ref rejects overlapping batches,
which would race on server-side collision checks.
Corpus name and description are now editable inline from the corpus
files header (pencil toggle) via a new PATCH /admin/corpora/{slug}
endpoint + db.update_corpus. The slug stays immutable — it is baked
into the storage prefix and scope URLs, so renaming it would orphan the
bucket bytes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…try dialog Corpus upload batches now pre-check destination keys against the listing and skip files already in the corpus instead of overwriting — same semantics as the copy flows; the outcome line reports uploaded/skipped counts. Failures no longer collapse into one error string: they open a dialog listing each failed file with its reason and Retry / Close actions. Retry re-attempts only the failed files (the File objects are held, no re-pick needed), and the skip pre-check makes retrying a partially-succeeded upload a safe no-op. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The corpus tree (and other FileTreeView selections) only supported per-row checkbox toggles and shift+arrow — shift+click did nothing, reading as a regression next to the storage panel's range select. Add the same anchor semantics: shift+click selects the visible file range from the last toggled row (adds, never removes); keyboard and checkbox paths keep the anchor in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mple set Ten files, five root causes, all fixed at the source: - Inch/foot units (scale 0.0254): the reader compared the file's length unit against the mm/m enum and raised for conversion-based units. store.py now compares raw scales and converts through the existing generic convert_file_length_units (6 files / 30 cells). - IfcTriangulatedFaceSet: imported fine but no backend had a build for it, so shapes silently tessellated to an empty scene. It IS a mesh — emit it directly via a new direct-mesh path in BatchTessellator (plus solid_geom() allowlist entries for TriangulatedFaceSet + FacetedBrep). - Type-library files (geometry only on an IfcTypeProduct RepresentationMap, no placed products — the texture samples): new instance-less type import, gated to files where no placed product carried geometry so normal models don't grow phantom shapes. - IfcRectangleProfileDef: parametric_profile_to_arbitrary now emits the centred rectangle outline (2 files, glb/obj/stl), and the AP242 stream writer converts parametric profiles through the same seam instead of assuming .outer_curve (ifc->step). - IfcTrimmedCurve parameter trims: the reader now preserves the IfcLine vector magnitude (parameterization scale) and normalizes conic trim angles to radians via the file's plane-angle unit (the degrees/radians sample pair now produce identical meshes). Both backends evaluate parameter trims: adacpp encodes line/circle natively and samples ellipse arcs; OCC builds trimmed line/circle/ellipse edges with the explicit XDirection frame. The 2D-placement ellipse reader crash is fixed the same way circle() handles it. - Products with no spatial container (IFC4x3 alignment-hosted signals on IfcLinearPlacement) attach to the assembly root instead of crashing on parent.Name. Regression tests over the (public) buildingSMART samples in files/ifc_files/bs_samples/, green on both CAD backends; full core suites pass (pyocc 1129, adacpp 1144). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An auto-validate run's total previously covered only the conversion grid; the parity cells were appended (extend_audit_run_total) when the run finished and the poller dispatched the validation pass, so the displayed total grew mid-run. The total now includes the parity cells from initial dispatch via a validate_total reservation column (migration 021): - set_audit_run_total reserves the parity count inside total, so the counter-bump finish check keeps the run 'running' through the gap between the last conversion cell and the validation dispatch - claim_audit_run_for_auto_validate claims reserved runs once their conversion cells alone have landed (still 'running'); legacy rows with validate_total=0 keep the finished+extend behaviour - consume_audit_run_validation_reserve swaps the reservation for the actual parity cell count at dispatch (total only moves on scope drift; zero actual finishes the run) — failures release the reserve so a run can't hang on cells that will never be enqueued - one storage listing feeds both the conversion grid and the parity reservation so the two counts can't disagree Manual Validate on a finished run still extends the total as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…osses) Audit run 63 (first run on the fixed worker image) confirmed 51 of the 54 conversion fixes and left 16 cells: 3 ifc re-exports the type-import fix unmasked, and 13 cross-format parity losses the newly-appended validation phase caught for the first time. ifc re-export (texture trio): - IfcStore.get_context creates the missing standard subcontext under the model context for imported files that carry none (the buildingSMART type-library samples have only bare '3D'/'2D' contexts) - TriangulatedFaceSet/FacetedBrep write parametrically (1:1 IfcTriangulatedFaceSet emit) instead of falling into the tessellation fallback, which needs a kernel build these kinds didn't have parity losses: - TriangulatedFaceSet builds on both backends (triangle faces sewn to a shell, mirroring PolygonalFaceSet) so mesh bodies reach STEP - GradientCurve directrix encodes as the shared alignment-evaluator's sampled polyline on both backends; OCC MakeSolid failure on an open sweep degrades to the swept shell instead of dropping the body - bare-curve shapes (SAT wire bodies) route through active_backend().build in the STEP export instead of a pythonocc-only wire builder, with build_wire/edge-record support on the adacpp backend - bare IfcClosedShell/IfcOpenShell representation items import again (write_shapes emits imported shells this way) - the DOM Genie-XML writer emits BeamRevolve/BeamSweep as straight chord beams like the streaming writer, instead of dropping them - the parity STEP counter falls back to a reload count when the native stream sees 0 roots (wireframe-only outputs are invisible to it) 8 new cross-format parity regression tests over the previously-failing sources; both core suites green (adacpp 1151, pyocc 1136; the one failure is the pre-existing plotly-to-file/kaleido issue, verified on a clean tree). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pty-model parity phantom Audit run 64 (image sha-b29eb92) cleared all 16 prior failures but flipped 12 parity cells to an inverted mismatch (source=0, step=1). Three root causes: - the parity element counter's reload fallback counted a zero-vertex phantom: a geometry-less STEP product materializes one empty Shape whose scene entry has no vertices. visualized_element_count skips renderably-empty entries, and tessellate_edges no longer passes an edge-less compound to discretize_edge (TopoDS_Edge assert) - wire-only STEP files read back as zero roots: GEOMETRIC_CURVE_SET was a referenced-entity builder but not a stream-reader ROOT, so adapy's own wireframe exports (SAT wire bodies) were invisible on re-read. It is now a yieldable root and a first-class curve body: CURVE_GEOM_TUPLE membership, native line discretization (incl. line-basis TrimmedCurve), per-element GL_LINES emit, OCC compound-of-wires build, and adacpp edge-record encode - the 12 files were passing vacuously (0=0=0): their products import as ZERO objects because a typed import failure (e.g. IfcBeam with a tessellated body and no material association) dropped the product entirely. Typed Beam/Plate/Wall imports now fall back to the generic Shape import on any failure, and the IfcTriangulatedFaceSet reader handles the optional Normals attribute. The tessellated I-beam sample now imports, renders and round-trips (parity 1=1=1) instead of silently showing an empty scene New regression test covers the empty-model zero count on every leg and the wireframe STEP round-trip. Both core suites green (adacpp 1152, pyocc 1137; the single failure is the pre-existing plotly-to-file/kaleido issue). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cProfile of the JacketHybrid parity cell (the audit's slowest Python cell, ~99s on the worker) showed round_array at ~28s cumulative: every call built a fresh np.vectorize(roundoff) to round a 3-vector, 637k times through compute_orientation_vec in FEM-shell->Plate conversion and the IFC/XML plate emitters. A plain per-element loop keeps roundoff's exact Decimal HALF_EVEN values (swapping in np.round shifted borderline vectors enough to drop 59 of 104k shells in plate conversion — rejected) and cuts FEM->concept conversion on that model from 59s to 46.5s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mbled scene cProfile of the hullskin parity cell showed ~44% of its wall time inside assembly_element_count: to_trimesh_scene tessellates the whole model at viewer quality and then assembles a full trimesh scene (trimesh objects, materials, normals) three times per cell — once for the source baseline and once per reloaded format leg — only for the entries to be counted. The count now consumes BatchTessellator.batch_tessellate directly over the same object set tessellate_part feeds it (physical objects with pipes as segments, plus welds): meshes_to_trimesh turns exactly one MeshStore into one scene entry, so counting non-point, non-empty stores is the same metric with the scene-assembly third of the cost removed. Exclusions unchanged: point clouds (the empty-scene placeholder) and zero-vertex stores. Measured on the two slowest audit parity cells, counts identical to the production baselines: hullskin.xml 130s -> 80s (5410 on every leg), JacketHybrid.FEM 86s local vs 98.6s worker (104188 on every leg). The parity regression suite (incl. the wire-STEP and empty-model edge cases that flow through this counter) is green on both backends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
convert_shell_elem_to_plates ran the full orientation/projection chain per element (normal_to_points_in_plane -> calc_yvec -> compute_orientation_vec x2 -> 2D projection -> winding -> back-transform): on a 100k-shell model that meant 1.5M Direction / 1.3M Point constructions, 2.9M weakref-cache ops and 2M Decimal roundings — ~2/3 of FEM->concept wall time. The math now runs once over (m, k, 3) arrays: - vector_transforms.shell_orientations_bulk replicates the scalar chain array-wise in the same floating-point operation order, Decimal-rounding via round_array semantics deduplicated over unique orientation rows (co-planar plate patches share frames), and escapes degenerate rows (duplicate corners, near-parallel edges, sub-1e-9 yvec) to the scalar path - CurvePoly2d.from_fem_shells_batch consumes it: batched matmul projection, the verbatim is_clockwise winding rule (its closing term is NOT the wrapped shoelace edge — replicated bit for bit), batched back-transform; only the output objects (interned Points, LineSegments, Nodes) are built per element - convert_part_shell_elements_to_plates gathers entries in element order with the exact scalar guards/warnings, coplanarity split, tri-branch material quirk and try/except, then batches per polygon arity Gate: all 109,949 JacketHybrid plates BITWISE-IDENTICAL to the scalar chain (points2d/3d, orientation vectors, segment indices, thickness, material). Conversion 39.0s -> 11.9s on the plates alone; create_objects_from_fem 59s (session start) -> 15.4s; the Jacket parity audit cell 98.6s (worker baseline) -> 59.2s locally with identical counts. Streaming exporters keep the per-element scalar path (iter_objects_from_fem builds detached plates one at a time by design). Core suites green on both backends; FEM suite green in the fem env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
If the JetStream pull loop stalls — e.g. the durable consumer wedges after a NATS restart, leaving the pod "Running" but no longer fetching (num_waiting=0) — the worker sits silently broken while jobs pile up unconsumed (exactly the incident that needed a manual rollout restart today). Touch a heartbeat file (WORKER_LIVENESS_FILE, default /tmp/worker-alive) on every pull-loop iteration (idle/between jobs, <=5s), on _on_sample (~2s during a subprocess conversion), and on _on_progress (stage/heartbeat, covers FEA/parity), plus once at startup. A k8s livenessProbe (added in gitops once this image is live) checks the file is fresh and restarts a stalled worker automatically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restart a worker whose JetStream pull loop has stalled (durable consumer wedged after a NATS restart — pod Running but num_waiting=0, jobs unconsumed) instead of leaving it silently broken. Exec probe checks /tmp/worker-alive (touched every pull round / conversion progress by the worker) is <120s fresh; 120s init + 4x30s failure grace tolerates long conversions and cold NATS starts. Overridable via .worker.livenessProbe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/config.js and /api/config are fetched on the SPA's critical startup path, and each derived its exts / conversion-matrix / utilities by calling queue.list_workers() 2-3x per request. list_workers() is an N+1 over NATS KV (list keys, then one round-trip per worker key), so every page load ate 2.5s (config.js) to 4s (api_config) of blocking NATS latency before the viewer could render. Measured on the homelab deploy: config.js ttfb ~2.5s consistent, api_config ~4s, /healthz 22ms. Fix: a background task (_worker_registry_refresh_loop) snapshots the worker registry + worker image tag into an in-memory cache every ~3s, primed once at startup. The config endpoints and _worker_advertised_* helpers now read that snapshot synchronously — zero NATS on the request path. Staleness is bounded by the refresh interval, immaterial for the picker / capability matrix (workers change on heartbeat timescales). Job routing (q._capability_for_ext) still uses live list_workers(). Also defer /config.js in index.html: a classic <script src> without defer stalls HTML parsing until it downloads+runs, so a slow config.js froze the whole page. Deferred classic + module scripts run in document order, so config.js still sets its window.* globals before index.tsx — ordering intact, but the browser parses/paints while it's in flight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docs image (Sphinx + paradoc FEA verification report) is the slowest build in the pipeline and is rarely what a given push needs. Flip it from "auto-build on docs/src/env changes, skippable with [nodocs]" to pure opt-in: docs build only when the head commit message carries a [docs] marker. The fast/full distinction is preserved for when docs IS requested (env/Dockerfile/workflow changes force a full rebuild, else fast). Viewer and worker builds are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The defer added in c62fb1d broke sign-in on the hosted viewer: the "signed in as" row went empty and the admin-panel button disappeared. Root cause: Vite emits the SPA entry as a module in <head> (<script type="module" src="/assets/index-*.js">), which is implicitly deferred. Deferred scripts — modules and `defer` classics — execute in document order after parsing. `defer` on the body's /config.js therefore made it run AFTER the head entry module, so the SPA read window.AUTH_ISSUER/AUTH_CLIENT_ID/COMMS_MODE before config.js defined them and bootstrapped with no auth config: no decoded token → empty identity and is_admin=false → no admin button. A blocking (non-defer) classic script runs during parse, in the body, before the deferred head module executes — so the globals are set when the SPA reads them. That was the pre-c62fb1d0 behavior. The backend cached-config change in c62fb1d is unaffected and stays; /config.js is served in ~20ms so blocking it costs nothing meaningful. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Profiled the storage-browser boot sequence. On connect, loadInitialServerState fired requestServerInfo (GET_SERVER_INFO) BEFORE the file listing, and the /rpc route computed the worker-advertised extension set for EVERY command — including the ones that ignore it — so worker-listing sat in front of the op the storage browser is actually waiting on. Measured on the live API: /api/scopes/<scope>/files is the slow leg (cold ~3s, warm 0.2-0.8s for 152 files) while /api/config is ~20ms warm (cached worker snapshot). Two changes, per "listing workers should be prioritised after finding files": - Frontend: loadInitialServerState now fires request_list_of_files_from_server FIRST, then requestServerInfo — the file listing goes out immediately instead of queueing behind server-info. - Backend: dispatch() takes a lazy exts_provider and resolves the worker-advertised set only inside the LIST_FILE_OBJECTS branch. Every other /rpc command (server-info, view-file, ...) no longer touches the worker registry at all. Single caller (the /rpc route); handler tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lity workers The pull-loop heartbeat livenessProbe (task #74) was emitted for EVERY worker pool, but only the adapy worker image writes /tmp/worker-alive. The abaqus and weld-gen capability pools run foreign images (adapy-viewer-worker-abaqus, an old base; asa-weld-gen-runner) that never write it, so the probe failed every cycle and SIGKILL-crashlooped them (exit 137 ~4.5min after start) — which in turn spammed PodCrashLooping/PodCrashLoopingSlow alerts all night. Make the heartbeat probe opt-in: the helper emits it only when a pool sets `heartbeatLiveness: true` (now the default on the adapy `worker` pool) or supplies its own `livenessProbe:`. Capability pools that do neither get no liveness probe. helm template verified: main worker keeps the heartbeat probe, abaqus + weld-gen render with none. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SPA's loadDiscovery() cached the .well-known/openid-configuration doc only in a module variable, so every page reload re-fetched it from authentik on the token-refresh path — a ~260ms round-trip that also eats an occasional slow TLS handshake when the ingress node is under load. The endpoints are public and static, so cache the doc in localStorage (keyed by issuer, 24h TTL) and reuse it across reloads; wrapped in try/catch so private-mode / embedded contexts fall back to fetching. Removes one of the sequential authentik round-trips from sign-in / reload. The backend already caches discovery + JWKS (600s TTL, singleton verifier), so no server change is needed — this closes the frontend gap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completed job-status entries were never removed from the shared NATS KV
bucket, so it grew unbounded (observed: 47k entries). list_workers() does a
kv.keys() over that whole bucket, and the worker-registry refresh loop called
it every 3s — so every 3s the API replayed ~48k messages through NATS. Result:
NATS pushed ~9000 msgs/s (522M in 16h) and both NATS and the API sat at ~0.25
core each, permanently, with zero actual work.
The KV is only a transient in-flight progress cache; the durable record lives
in Postgres (audit_runs / audit_log) and S3 (derived blobs, profiles), and the
audit read endpoints (/admin/audit/active,/runs,/runs/{id}) already query
Postgres, so history is unaffected.
- queue.purge_completed_jobs(grace_s): drops terminal (done/error/cancelled)
job entries older than the grace; never touches __meta_* (worker/registry)
keys.
- _job_cleanup_loop: runs it every 5m with a 15m grace, so a polling client
still sees a job's final state before it's reaped. Frontend behaviour
unchanged (it only polls active jobs; a purged old job 404s cleanly).
- Worker-registry refresh interval 3s -> 15s (capabilities change on the ~15s
heartbeat; no reason to scan faster), and it's cheap now that the KV is lean.
Companion to the runtime purge that already took the bucket 47k -> 10 entries
and dropped NATS out-msgs from ~9000/s to ~9/s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the Dockerfile.worker rebuild marker to re-pull the ADACPP_BRANCH overlay so the deployed worker picks up adacpp @ ae8fb6b: 469826 STEP->GLB peak RSS 2840->1286 MB (phase-A parse_cache_ bound) + CSR weld_mesh. Worker only — no viewer/frontend change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…logs visible
Two worker-startup fixes prompted by an audit slowdown after a worker roll.
1. Pre-warm (perf). Each convert runs in an os.fork()ed child. IFC children
re-imported ifcopenshell(.geom) and SAT children re-imported OCC.Core on
EVERY job — hundreds of MB of .so faulted from a cold/pressured page cache,
measured at ~20-40s of near-zero-CPU I/O-wait per conversion (STEP's native
adacpp reader was unaffected; cProfile showed the child's whole runtime in
import machinery). Import them once in the parent so forks inherit them
copy-on-write. Base pool only; per-module guarded so a slim/scoped pool skips
what it lacks. Runs in an executor so a cold import can't stall NATS keepalive.
Durable follow-up (noted in the code): route IFC->{stl,obj,xml} and SAT off
ada.from_ifc / ada.from_acis onto adacpp + the native C++ IFC reader/writer,
so these deps aren't loaded at all — native STEP already is.
2. Logs (observability). configure_logger() never set a level, so the "ada"
logger inherited root's WARNING and every worker: INFO line (booting /
connected / registered / subscribing / ready / warm-imported) was silently
dropped — a healthy idle worker looked identical to a hung one in kubectl
logs. Set INFO for the worker process (ADA_WORKER_LOG_LEVEL overrides) and
add explicit NATS connect lines. Worker process only; the viewer is untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the rebuild marker so the worker re-pulls the ADACPP_BRANCH overlay at 5fe25ba: phase-A parse_cache_ bound extended to the STEP->mesh (obj/stl) streamer, which was OOMing on 469826's 61k-face solid (GLB was already fixed). Worker only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…i_stats)
Audit rows carried peak RSS, timings and I/O but no tessellation-output size, so a
run-to-run change in triangle volume — a density regression, an adaptive-toggle
drift, the adaptive-density thread bug that scaled tris with the worker count, or a
silently dropped solid — was invisible without re-downloading and re-parsing the
artefact.
Native mesh conversions record their triangle count directly (tess_stats.record_tri_stats);
for GLB (the native return is solids, not tris) parse the output's JSON chunk for total
triangles + primitive count. subprocess_convert emits a [TRISTATS-JSON] marker the worker
parent folds into audit_log.convert_meta["tri_stats"] = {n_tris, engine?, n_primitives?}.
Best-effort — a tally failure never fails an otherwise-good conversion. Primary signal is
n_tris; n_solids/n_primitives catches "no geometry left behind" drops. Per-solid max is a
follow-up (needs the adacpp profiler counters surfaced).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ited RSS) The obj/stl OOM on 469826 (audit run 90, per-job watchdog SIGKILL at ~3.2 GB, then a pod-level Exit 137) is NOT tessellation volume: the conversion's intrinsic peak is ~1.3 GB (verified — a fresh process, and a faithful fork through run_isolated_convert with warmup + cProfile, both land ~1.0-1.1 GB with the same ~14 M triangles). The extra memory comes from the long-lived worker PARENT: measured on a 23h prod worker it had grown from ~218 MB fresh to 540-840 MB idle (anonymous private-dirty heap, no unbounded Python accumulator found — glibc arena retention from per-job transients: reading the child's captured log, marker/cpp-profile JSON parsing, convert_meta, the sample lists). Every conversion is a fork, so the child inherits that bloat (COW, counted in child RSS), lifting each conversion's baseline; late in a run a big-model fork clears the watchdog and parent+child clears the 6 GiB pod limit. Return the retained arena memory to the OS after each job (glibc malloc_trim(0), best-effort, disable with ADA_WORKER_NO_MALLOC_TRIM) so the parent stays flat and forks don't inherit the accumulation. Effect to be confirmed by measuring parent RSS across the next audit run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng + parent malloc_trim Bump the Dockerfile.worker rebuild marker to force a FULL worker build that re-clones the adacpp feat/lazy-shape-store overlay at b83a8f9 (thread_local model_scale fix + face-merge batching) and rebuilds src with the tri-count audit logging and the worker-parent malloc_trim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…vme write pressure) Bump the Dockerfile.worker marker to force a FULL worker build that re-clones the adacpp feat/lazy-shape-store overlay at fb9284a (GlbSpillWriter buffers pos/idx in RAM up to a lane-total threshold; small/medium models do zero spill disk writes). Targets the audit speed regression root-caused to nvme write saturation (worker spill 72% / garage 28%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the 8 CodeQL py/stack-trace-exposure alerts (medium) in the REST API: the
blob delete/rename/move cascades and the metrics-clear + scope-copy handlers packed
raw `str(exc)` / `f"{key}: {exc}"` into the `errors`/`failed`/`reason` fields returned
via JSONResponse, leaking backend/stack-trace text to clients. Report only the failed
KEY (still actionable) or a generic reason, and log the full exception server-side
(the logger.exception/.warning calls were already there). Behaviour otherwise unchanged
— callers still learn which keys failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Folds in dependabot PR #236 (dependencies group) across all workflows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Krande
changed the title
…-of-hooks crash)
The nameCopied useState + copyName useCallback were declared AFTER `if (!enabled)
return null`, so toggling gallery mode on/off changed the render's hook count and
crashed the app ("rendered more hooks than during the previous render" — the minified
React error seen when toggling gallery mode). Move both hooks above the early return.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clicking the "N / M" counter in gallery mode now opens the shared single-file picker (FilePickerModal — the same storage-browser tree the rest of the app uses) so you can jump straight to a file instead of stepping Prev/Next through a long scope. Single selection; the picked blob key maps back to the gallery file list (by name or basename) and loads via `go`, which works in both the files and geoms walks (in the geoms walk, loading a new file rebuilds the geom list). Restricted to loadable files, matching the gallery's own filter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… unit fix) + gallery UI Bump the Dockerfile.worker marker to force a FULL worker build that re-clones the adacpp feat/lazy-shape-store overlay at 50e8148 (IfcLinearPlacement CartesianPosition, IfcRelVoids opening subtraction, bogus large-SI-prefix guard for the empty storage-tank scenes). The push also ships the frontend gallery jump-to-file picker + hooks-crash fix and the storage-op exception-sanitisation security fixes via the viewer build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…onverting…" toast) convertViaServer's poll loop only terminated on 'done'/'error', so a user-killed job — whose status becomes 'cancelled' — was ignored and it kept polling until the ~30 min ceiling. The gallery reconvert (and anything awaiting the conversion) stayed stuck on "Re-converting…". Treat 'cancelled' as terminal and reject so the caller's finally runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parse the adacpp [GEOMHEALTH-JSON] marker (faces with a boundary that tessellated to zero
triangles) into audit_log.convert_meta["geom_health"] = {dropped_faces, total_faces}, so
silently-dropped geometry is catchable from the audit overview instead of visual inspection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface convert_meta.geom_health.dropped_faces as a red per-source badge in the audit run overview, so a conversion that silently dropped faces (a trim boundary meshing to zero triangles — e.g. the Ventilator's swept-extrusion faces) is caught at a glance instead of by visual inspection. Uses the max across the row's target formats (each drops the same source faces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…udit flag Bump the marker to force a full worker build that re-clones the adacpp overlay at 3baa1fe (dropped-face counter emitting [GEOMHEALTH-JSON]) and rebuilds src with the worker-side geom_health parsing, so live audit runs flag silently-dropped faces. Also carries the IFC placement/voids + storage-tank unit fixes not yet on the deployed worker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
👋 Hi there! I have checked your PR and found the following: PR Review:I found no pr-related issues.
Python Review:I found some python-related issues: Python Linting results:
Python Packaging results:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Started as a lazy blob-backed shape store for CAD imports; grew to carry the pure-C++ IFC pipeline (via adacpp Krande/adacpp#35), in-browser (wasm) conversion routes, viewer mesh tooling, and streaming-conversion performance + memory work. Sections below.
Lazy shape store (headline)
Large CAD imports materialised as
ada.geomobject trees cost ~10x their serialized size in resident memory (millions of small dataclass instances, each with a__dict__). This keeps each imported solid as one compact blob in a sharedShapeStoreand mintsShapeProxyobjects that hydrate the geometry tree on demand — mirroring the FEM array-substrate design (ada.api.mesh).Measured on a 778 MB / 7291-solid STEP assembly (
from_step, native reader):shape_store_compressada.api.shapes.ShapeStore: NGEOM buffers from the native readers are retained zero-copy as they arrive and double as the tessellation/export fast path (no hydrate, no re-serialize). Python-built geometry stores as lossless pickle blobs.ShapeProxy(Shape)holds(store, index);.geomhydrates on demand,.geom =pins,ngeom_blob()feeds adacpp directly. Weakref cache + bounded strong LRU window for back-to-back access; RSS returns to the blob floor when hydrated trees drop.Native pure-C++ IFC reader/writer
from_ifc/to_ifccan select the pure-C++ adacpp reader/writer (no ifcopenshell / OCC) for a full round-trip, honouringshape_store_compress(zero-copy + compression, matching STEP)..ifc → glbdefaults to the native adacpp path (stream_ifc_to_glb), falling back tofrom_ifcfor products it can't yet resolve; cgroup-aware worker count passed through.In-browser (wasm, no pyodide)
Viewer
Streaming conversion — performance & memory
convert_meta.tri_stats: total tris + primitive/engine), so a change in tessellation output (density drift, a dropped solid) is visible run-to-run.malloc_trimafter each job so the long-lived forking parent doesn't accumulate arena RSS that every fork inherits (behind an opt-out env). Pre-warm of heavy CAD imports in the parent for COW-shared forks; visible worker startup logs.parse_cache_bounding — cut streaming-conversion peak RSS and disk write pressure.Server / infra
livenessProbe;config.jsmust not be deferred (SPA auth bootstrap); docs build opt-in via[docs].Security / deps
py/stack-trace-exposurealerts): return the failed key or a generic reason, log full detail server-side.prefix-dev/setup-pixi0.9.6 → 0.10.0 (folds in dependabot chore: bump prefix-dev/setup-pixi from 0.9.6 to 0.10.0 in the dependencies group across 1 directory #236).