HyperTools 1.0: architecture refactor + bug hunt

[jeremymanning]

HyperTools 1.0 — architecture refactor + bug hunt

This PR merges the completed HyperTools 1.0 class-based refactor into dev-1.0, together with a broad bug hunt (open-issue triage + two animation-rendering fixes you reported + a legend-clipping fix) and the CI fixes needed to get all platforms green.

Note: originally developed under the working name "HyperTools 2.0"; renumbered to 1.0 (package version 1.0.0.dev0, branches dev-1.0/dev-1.0-refactor) per project decision. Older commit messages/notes may still say 2.0.

⚠️ Please do not merge. Opened for your review; I'll proceed however you decide. Base: dev-1.0 ← Head: dev-1.0-refactor.

Supersedes #271, which GitHub auto-closed when its head branch was renamed dev-2.0-refactor → dev-1.0-refactor; all commits, evidence, and CI history carry over unchanged.

---

1. Architecture refactor (Plans 1–8)

Reorganized the working dev-1.0 code into the class-based structure, on modern deps, with DataGeometry removed from the public API:

• Deps modernized: numpy ≥2, pandas ≥3, scikit-learn ≥1.4, matplotlib ≥3.8, plotly 6.x, via datawrangler 0.5 (funnel/stack/unstack/format-detection/model-dispatch). Supports pandas 3.0+.
• Class-based modules: core (eval-free apply_model), external (vendored PPCA + brainiak SRM/DetSRM/RSRM), manip (Normalize/ZScore/Smooth/Resample), reduce/align/cluster (generic sklearn dispatch by name), io, plot (matplotlib + plotly backends). Old tools/ names remain as shims.
• DataGeometry removed from the public API (kept only as a hidden unpickle shim so legacy hosted .geo datasets still load). plot() returns a Figure / (fig, ani); plot(..., return_model=True) returns {'fig','xform_data','models','animation'}; load() returns raw data.
• Docs/gallery/tutorials migrated to the 1.0 API and rebuilt.

---

2. Reported animation fixes

Animated bounding box "crowded / cut off on the right." Animated 3D plots zoomed the cube in too far. Fix: set_box_aspect zoom 1.25→1.125 + full-canvas axes. Min cube-to-edge margin over the full save_movie rotation: right 80→96px, bottom 51→72px.

before (crowded)	after (margin)

Duplicate animation-legend entries (GH #207). Faint "trail" artists carried the dataset label, so each label appeared twice. Fix: trails tagged _nolegend_; legend is now the static union of in-focus datasets. Verified ['first','second'].

Clipped static gallery legends. Wide legends (long labels / many entries) clipped off the right edge. Root cause: the fit routine measured the legend under seaborn's (narrower) font, but the figure is saved downstream under the default (wider) font. Fix: _fit_right_legend now measures the rasterized pixels under default rcParams and widens the figure (keeping the plot's size) until the legend has a margin. Regenerated plot_legend/plot_PPCA/plot_missing_data; pixel-based regression test added.

---

3. Open-issue triage → close-on-merge

Triaged all 67 open issues against this branch with real repros (see notes/issues-to-close-on-merge.md): 31 addressed/obsolete + 15 fixed or implemented on this branch (incl. §5's #169/#132 and §6's seven features) = 46 to close on merge; 21 stay open (feature requests / design decisions), each documented.

Bugs fixed from triage (all regression-tested):

#	bug	fix
#259	import hypertools mutated global rcParams['pdf.fonttype']	removed import-time mutation; editable-PDF default set inside manage_backend's snapshot/restore
#223	get_proj crash on 2D labeled plots	guard get_proj; match annotate/update tuple shapes for 2D vs 3D
#146/#190	DBSCAN/MeanShift/OPTICS crash on n_clusters	inject n_clusters only when the model's signature accepts it; register those clusterers
#148	show=False leaked the figure into pyplot	plt.close(fig) for static figures (skipped when the user passed ax, and for animated figures, whose timer must stay alive)
#132	DataFrame columns consumed positionally across datasets	format_data aligns named columns BY NAME to the first dataset's order (warns on reorder); mismatched column sets raise a clear ValueError
#214	wiki-model docstring vs wiki_model key	docstring corrected
—	reduce=<class/instance> → UnboundLocalError	initialize model_params in the custom-estimator branch

---

4. CI fixes (get all platforms green)

The first CI run surfaced two platform issues (unrelated to the features above); both fixed:

• Windows (all Pythons) — collection error. datawrangler 0.5.0 evaluates os.getenv('HOME') at import time to build its data dir; HOME is unset on Windows, so os.path.join(None, …) crashed dw's (and hypertools') import. Fixed hypertools-side by setting HOME=expanduser('~') before importing dw; filed upstream as Import crashes on Windows: config datadir evals os.getenv('HOME') which is None data-wrangler#32, fixed in dw 0.5.1 (released; verified import datawrangler works with HOME unset). The pydata-wrangler pin is bumped to >=0.5.1; the one-line HOME guard stays as belt-and-suspenders for environments still on 0.5.0.
• macOS/Ubuntu Python 3.11+ — 3 tests. matplotlib 3.11 (Python 3.11+ only) resets a figure's canvas after plt.close() (the disabling the figure doesn't work as intended #148 fix), so tests/callbacks that read fig.canvas.renderer/buffer_rgba() failed. Fixed by guarding the renderer in update_position and rendering the affected tests through an explicit Agg canvas. (savefig after close still works, so users are unaffected.)
• Windows — animated-figure draw crash. On Windows the animate backend switch to TkAgg actually succeeds (headless Linux/mac fall back to Agg), so the disabling the figure doesn't work as intended #148 plt.close() destroyed the FuncAnimation's real Tk timer and any later draw of the returned figure crashed ('NoneType' object has no attribute 'start'). Animated figures are now exempt from the show=False close — the disabling the figure doesn't work as intended #148 complaint was about static figures, and animations need their timer alive for playback.
• Ubuntu 3.12 — screenshot-verification step. The screenshot harness discovered figures via plt.get_fignums(), which the disabling the figure doesn't work as intended #148 close empties; it now uses the figure(s) returned by plot() (13/13 cases pass).

---

5. New: hyp.predict + hyp.impute (resolves GH #169)

Two new modules in the established class-based style (base class + one file per model + funnel dispatcher), integrated into hyp.plot/hyp.analyze like cluster/align:

• hyp.predict(data, model=..., t=...) — timeseries forecasting: Kalman, GaussianProcess, AutoRegressor (any sklearn regressor, recursive multi-step), ARIMA, Laplace (skaters ensemble), Chronos (HuggingFace foundation model, real chronos-t5-tiny test). t follows Use Kalman filter to fill in missing data #169's spec (int steps, or a datetime on time-indexed data — including past-date truncation). One forecast per input dataset, same dimensions, continued index.
• hyp.impute(data, model=...) — missing data: PPCA (default; clean interface over the vendored implementation — format_data's fill now routes through it, behavior-preserving), SimpleImputer/KNNImputer/IterativeImputer, and Kalman, which fills rows where every feature is NaN — the exact gap Use Kalman filter to fill in missing data #169 describes (PPCA cannot).
• return_model=True on both → (result, fitted) matching apply_model's convention; the fitted model can be passed back as model= on new data and is applied without re-estimation (verified: fit on A, forecast/impute B).
• hyp.plot(data, predict='Kalman', t=30) overlays one dashed, low-opacity, same-color forecast tail per dataset (2D + 3D, both backends, no legend duplication, frame always contains the forecasts). impute= selects the missing-data model in the plot/analyze pipeline.
• Dependencies: new [predict] extra (pykalman, statsmodels, skaters) and [predict-hf] (chronos-forecasting); GaussianProcess/AutoRegressor/sklearn imputers work on the base install; friendly ImportErrors otherwise (fresh-venv verified). yfinance is not a dependency — the tutorial self-installs it.
• Tutorials (executed, real data):
  • stock_forecasting.ipynb — scrapes 2y of real Yahoo Finance prices for 4 tickers, backtests all models against a 30-day holdout with an honest MAE/MAPE table (spoiler: ARIMA/Kalman ≈ the naive baseline, as efficient-market theory predicts — the tutorial says so).
  • projectile_kalman.ipynb — a real NBA SportVU jump-shot arc (25 Hz optical tracking): Kalman imputation of 5 fully-occluded frames recovers them to RMSE 0.20 ft vs the recorded truth; forecasting the arc's final 20 frames from the first 30 lands within MAE 4.2 ft.
• Forecast direction fix (review follow-up): the GaussianProcess default kernel is now DotProduct + RBF + WhiteKernel — the old stationary RBF reverted forecasts to the training mean beyond the data (drift −0.026/pt vs observed +0.0019/pt: reversed); the linear term extrapolates trends (+0.002..+0.010/pt: continues). Before/after renders + measurements in this comment.
• Gallery: plot_predict (helical forecasts) + plot_impute (PPCA-vs-Kalman panels on the Use Kalman filter to fill in missing data #169 case); API reference sections added.

---

6. Seven long-standing feature requests (GH #95, #100, #108, #109, #127, #142, #177, #191)

All seven implemented in the 1.0 design language, in both rendering backends, each with numeric + screenshot evidence in this comment:

• support multiindex Pandas DataFrames #95 MultiIndex DataFrames → leaf trajectories + per-level group means (thicker/opaquer means per level, colors by top-level index, exactly per the issue's spec).
• colorbar option #100 colorbar=True/dict — continuous hue (same palette as the lines, real value range) and discrete groups (segmented, labeled); coexists with legends without clipping.
• triangular meshes #109 surface=True/dict — smooth, lit convex-hull surfaces (3D, Blinn-Phong + plotly vertex shading) and smooth filled shapes (2D), dict-controlled properties, animatable (per-frame hull recompute); the axes cube auto-expands to contain the hull. New animate_surface_morph gallery demo.
• support kernel density plots in 2d and 3d #108/enhancement: replace matplotlib with ipyvolume #191 density=True/dict — subtle per-group KDE clouds (off by default; 2D alpha-ramp, 3D iso-surfaces/go.Volume with small-cluster auto-boost); enhancement: replace matplotlib with ipyvolume #191's ipyvolume ask is superseded by the plotly backend.
• feature request: support different selections of bullettime, precog, and chemtrails for different data matrices #127 chemtrails/precog/bullettime accept per-dataset lists (mixed trail styles in one animation); spin/serial warn instead of silently ignoring.
• BUG: legend isn't updated when color kwarg is specified #142 original legend/color symptom already fixed by the refactor (verified); re-verification exposed colors=/linestyles=/markers= were silent no-ops without their singular twins — fixed.
• Add support for Google Drive and other easy-to-add filetypes to hyp.load #177 hyp.load completed: Drive large-file interstitial (verified against a real 498MB public file), .xlsx/.xls, Google Sheets→CSV, remote-pickle trust policy.

Every graphical feature was adversarially screenshot-reviewed by fresh agents across a {2D,3D}×{mpl,plotly}×{static,animated} grid; their findings drove 6 further fix commits (plotly WebGL surface artifacts, bounding-box containment, MultiIndex colorbars, legend fitting, 3D density visibility, trail-mode warnings). New gallery examples: plot_surface, plot_density, plot_colorbar, plot_multiindex, animate_trails_mix, animate_surface_morph.

Maintainer-feedback round (evidence): hulls now hug the observations (hull-hugging smoothing: post-Taubin pull-back to the hull; cube-cloud oversize 1.63×→1.13×, ≥99% containment; axes box sized from actual meshes incl. mid-morph union bound, both backends) and morphing is a first-class animation style — animate='morph' (Hungarian point-cloud morphs between datasets, tagged-list form for static backdrops) with per-segment rotations lists ([1, 0.25, 2, ...] = per hold/transition). Both shape-morph gallery demos collapsed to single hyp.plot calls.

---

Testing

• Local suite: 765 passed, 0 failed (+310 tests across both rounds: surface/density/colorbar/multiindex/trails/meshutil/load/color-alias/morph-animation/hull-tightness suites); 6 plotly→kaleido image-export tests deselected locally only — they deadlock Chromium in this sandbox but run fine in CI.
• CI: all 12 jobs green (ubuntu/macos/windows × Python 3.10–3.13) — see this PR's checks for the head commit.
• Docs rebuilt (make html succeeds); gallery regenerated with the 6 new example pages (including two captured animations).

🤖 Generated with Claude Code

[jeremymanning]

Seven long-standing feature requests: implemented with evidence

This round addresses GH #95, #100, #108, #109, #127, #142, #177 (and #191 via the plotly backend), all in the 1.0 API design language, working in both rendering backends (backend='matplotlib' / backend='plotly'). Commit range 8d424e08..3861ca6e. Every graphical feature was screenshot-reviewed by fresh reviewer agents across a {2D,3D} × {mpl,plotly} × {static,animated} grid, and every bug they found was fixed and re-verified (details per section). Full suite: 681 tests passing (was 455 at the start of this round).

(raw image links below render from the dev-1.0-refactor branch)

#95 — MultiIndex DataFrames

A DataFrame with a row MultiIndex now expands into leaf trajectories plus per-level group averages, styled exactly per the issue spec: leaves lw=1, level-k means lw=k+1; alpha = min(1, 1/(depth_below_top+1) + 0.2) (top-level means opaque, leaves faint); colors shared by top-level index value; linestyles list length = number of top-level values; legend shows one entry per top-level value. Means are computed on transformed trajectories (post normalize/reduce/align) and equal np.mean of member leaves exactly (asserted in tests). 25 tests.

matplotlib	plotly

#100 — Colorbars

hyp.plot(..., colorbar=True) (or a dict: label, ticks, location). Continuous hue → colorbar over the actual hue value range using the same palette as the lines (endpoint RGB equality asserted numerically); discrete groups (cluster / lists / categorical hue) → segmented colorbar with legend/group labels (the groupvals question from the issue is answered by the hue/group values themselves). Legend + colorbar coexist without overlap (rasterized-pixel fitting, verified for location='left'/'top', long labels, and animated plots). Found + fixed a pre-existing bug along the way: legend=True with a string categorical hue showed integer ids instead of category names. 22 tests.

continuous (mpl)	discrete + legend (mpl)	continuous (plotly)

#109 — Smooth convex-hull surfaces (redirected per maintainer comment)

Instead of sliding-window triangle strips, surface=True (or a dict API: alpha, color, lighting={...}, smoothing, pre_inflate, keep_points) computes each dataset's convex hull and renders it as a smooth, lit surface (3D) or smooth filled shape (2D):

• Geometry: scipy.spatial.ConvexHull → outward orientation → interleaved rounds of midpoint subdivision + Taubin smoothing (pure numpy/scipy, no new dependency; 13.4 ms/mesh for 200 points, animation-friendly). 2D outlines use a centripetal Catmull-Rom closed spline (100% hull-vertex containment). Tiny clouds get a grow-only rescale so containment stays ≥96% (measured 100% on test blobs).
• matplotlib: per-face Blinn-Phong shading + backface culling (eliminates z-sorting cracks); plotly: Mesh3d with precomputed vertex shading (same Blinn-Phong model), double-winding + priority face-trimming to work around three verified upstream plotly.js WebGL defects (self-culling holes, translucency speckle, intersecting-mesh punch-through — each reproduced in a live Chromium tab before being worked around).
• The axes cube auto-expands to contain the full hull (both backends, static + all animation modes) — surfaces are never cut off by the bounding box.
• Animations recompute the hull per frame from the visible window. New gallery demo: animate_surface_morph.py — the shapes-zoo morph rendered as a morphing lit surface.
  62 surface+mesh tests.

mpl (translucent, lit)	plotly (vertex-shaded)	morph demo frames

#108 (+#191) — Subtle density shading

density=True (off by default; dict: alpha [default 0.2], levels, grid, per_group) renders each group's KDE as a soft cloud: 2D mpl imshow with an alpha-ramped colormap (contourf rejected — hard edges); 3D mpl nested marching-cubes iso-surfaces (scikit-image as optional density3d extra, with a real scatter-fog fallback + install hint); plotly go.Volume/go.Contour. Shell/volume opacity auto-boosts for small-in-scene clusters (separated datasets stayed invisible at fixed alpha — found by the review wave, fixed, re-verified at separations 0/5/10). Density is computed once from the full data for animations (measured KDE eval 536 ms @ 50³ ≫ 33 ms frame budget) and drawn as a static background. #191's ipyvolume proposal is superseded by the plotly backend, which provides the fast interactive 3D rendering it asked for. 57 tests.

2D (mpl)	3D iso-surfaces (mpl)	3D volume (plotly)

#127 — Per-dataset animation trail styles

chemtrails, precog, and bullettime each accept a bool or a list of bools (one per dataset), so different datasets can use different trail styles in one animation (validated per-frame in tests: at frame k the chemtrails dataset's trail ends at k, the precog dataset's covers the future, bullettime stays full — both backends). The animation mode (parallel/spin/serial) stays global (one camera, one frame loop — documented); setting trail flags under spin/serial now warns naming the affected datasets instead of silently no-opping. 19 tests.

mpl (chemtrails / precog / bullettime per dataset)

#142 — Legend/color kwarg bug

The 2017 symptom (legend swatches not tracking explicit colors) is already fixed by the 1.0 refactor — verified with renders in both backends. But re-verification exposed that colors=, linestyles=, and markers= (the plural aliases) were silent no-ops unless the singular kwarg was also passed — all three hoisted and fixed, with conflict warnings only when both forms are supplied. 10 regression tests.

before (colors=['red','green','blue'] ignored)	after

#177 — hyp.load completion

Most of the issue was already implemented on this branch (Drive share-links/ids, Dropbox, HF datasets, npy/npz/csv/tsv/txt/json/parquet/mat/pickle + content sniffing). This round closed the gaps, each verified against live services:

• Large Drive files: the virus-scan interstitial is now parsed and followed — verified end-to-end against a public 498 MB file (Content-Length 498,881,336 bytes, byte count matched exactly; full download in ~25–31 s). The interstitial parser is tested against the real captured HTML (committed fixture); the live test is @pytest.mark.bigdata (excluded from default/CI runs).
• Excel: .xlsx via openpyxl (new io extra), exact round-trip test; .xls raises a friendly install hint for xlrd.
• Google Sheets: spreadsheet URLs rewrite to CSV export — live-verified against Google's public sample sheet (loads as a (30, 6) DataFrame).
• Remote pickle policy: unpickling from remote sources warns unless trust=True; remote .npy/.npz load with allow_pickle=False by default (a dedicated HypertoolsTrustError keeps policy errors distinct from parse errors); builtin example datasets exempt. Tested over a real local HTTP server.
  15 new tests.

Process notes

• Every graphical feature was adversarially screenshot-reviewed by fresh agents; their findings drove 6 additional fix commits (plotly WebGL surface artifacts, bounding-box containment, MultiIndex colorbar segments, legend fitting under animate=/non-default colorbar locations, 3D density visibility, spin/serial trail warnings).
• New gallery examples: plot_surface.py, plot_density.py, plot_colorbar.py, plot_multiindex.py, animate_trails_mix.py, animate_surface_morph.py.
• Issues to close on merge updated: support multiindex Pandas DataFrames #95, colorbar option #100, support kernel density plots in 2d and 3d #108, triangular meshes #109, feature request: support different selections of bullettime, precog, and chemtrails for different data matrices #127, BUG: legend isn't updated when color kwarg is specified #142, Add support for Google Drive and other easy-to-add filetypes to hyp.load #177, enhancement: replace matplotlib with ipyvolume #191 added to the close list (notes/issues-to-close-on-merge.md).

🤖 Generated with Claude Code

[jeremymanning]

Refined surface rendering + first-class animate='morph' (maintainer feedback round)

Addresses the three follow-up requests: tighter hulls, morphing as a real animation style, and per-segment rotation control — with the shape demos migrated and evidence for both backends below. Commit range c0eff29a..HEAD.

1. Hulls now hug the observations

Root causes found and fixed:

• The containment rescale over-grew meshes (its nearest-by-angle ray proxy overstated the needed growth — measured up to 1.63× cloud extent on the demo's cube hold), compounding with a blanket pre_inflate=1.15.
• Fix: pre_inflate → 1.0 (no blanket padding); the rescale now uses the exact ray-vs-hull exit distance and is hard-capped; and — the key change — hull-hugging smoothing: after each Taubin pass, vertices that fell inside the original hull are pulled back toward the hull surface, so the mesh hugs the data by construction instead of shrinking-then-ballooning.

Measured on the demo's five shapes (mesh/cloud extent ratio, ≥99% containment everywhere):

shape	before	after
bunny	1.40	1.09
cube	1.63	1.13
sphere	~1.15	1.02
teapot	~1.36	1.01
vase	~1.36	1.09

The axes box is also now sized from the actual built meshes including mid-morph geometry (union-hull bound), on both backends, so surfaces never cross the wireframe:

before (loose, box escapes)	after (snug, contained — one panel per morph segment)

2. animate='morph' — Hungarian point-cloud morphs as a library animation style

hyp.plot([ds1, ds2, ...], animate='morph') treats the datasets as point clouds and morphs ds1→ds2→…→dsN (hold/transition segments, 2N−1 total): equal-size samples, consecutive clouds matched point-for-point with scipy.optimize.linear_sum_assignment, smoothstep easing — exactly the shapes-demo recipe, now built in (hypertools/plot/morph.py), for both backends. Colors interpolate between the datasets' palette colors during transitions. animate also accepts a per-dataset list (['morph', None, 'morph']): tagged datasets morph, untagged ones stay as static backdrops. Trail flags warn+skip (as with spin/serial); predict= is guarded.

3. Per-segment rotations list

rotations=[1, 0.25, 2, 0.25, 1] gives dataset 1 a full camera rotation, the first transition a quarter turn, dataset 2 two rotations, etc. (list length = 2N−1; only valid with the morph style — scalar rotations works everywhere and keeps the old default). Azimuth accumulates continuously across segments (verified numerically: hold start/end azimuth identical when its rotation count is 0; no jumps at boundaries — adjacent-frame camera deltas smooth in both backends).

matplotlib (6 checkpoints, rotations=[1,0.25,2,0.25,1])	plotly (same)

4. Demos migrated + docs

• examples/plot_shape_morph.py: the hand-rolled normalize/sample/Hungarian/FuncAnimation machinery (≈90 lines) collapses to one hyp.plot(clouds, animate='morph', rotations=[...]) call; holds spin a slow full rotation, transitions a brisk quarter-turn.
• examples/animate_surface_morph.py: same + surface=True — all manual mesh/shading/axes-sizing code deleted.
• hyp.plot docstring documents animate='morph', the tagged-list form, rotations, and morph_samples; README updated.

Verification

• Suite: 765 passed (+84 this round: morph schedule/matching/rotations math with exact-value asserts, hull tightness/containment metrics, box-containment regressions incl. mid-morph, mixed-tagging, both backends).
• Independent code review (math verification of the ray-exit/pull-back formulas, Hungarian chaining, azimuth continuity) and a fresh visual review across both backends drove three additional fixes (sampled-only box sizing for large clouds, early validation, untagged-dataset rendering in mpl mixed tagging) — all re-verified.

🤖 Generated with Claude Code