HyperTools 1.0: architecture refactor + bug hunt

.github/workflows/test.yml

@@ -2,7 +2,7 @@ name: Tests
 
 on:
   push:
-    branches: [ master, dev, dev-2.0 ]
+    branches: [ master, dev, dev-1.0, dev-1.0-refactor ]
   pull_request:
     branches: [ master, dev ]
 
@@ -13,6 +13,10 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest, macos-latest]
+        # datawrangler 0.5.0+ supports pandas 3.0 (dw#30/#31) and py3.10-3.13. The pandas<3
+        # ceiling is lifted, so the resolver naturally picks pandas 3.x on py3.11-3.13 and
+        # pandas 2.x on py3.10 (pandas 3.0 requires py>=3.11) -- the matrix exercises both.
+        # A dedicated pinned-pandas-3 gate runs below on ubuntu/py3.12.
         python-version: ['3.10', '3.11', '3.12', '3.13']
 
     steps:
@@ -69,6 +73,16 @@ jobs:
       run: |
         pytest -v --tb=short
 
+    - name: Acceptance gate — pandas 3.0 (Ubuntu Python 3.12 only)
+      # Guarantees a pandas-3 run regardless of resolver choices (dw#30/#31 fix).
+      if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
+      env:
+        MPLBACKEND: Agg
+      run: |
+        pip install "pandas>=3.0"
+        python -c "import pandas as pd; assert pd.__version__.split('.')[0] == '3', pd.__version__; print('pandas', pd.__version__)"
+        pytest -q
+
     - name: Run pytest with coverage (Ubuntu Python 3.12 only)
       if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
       env:

dev/hypertools_2.0_dev.ipynb → dev/hypertools_1.0_dev.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# HyperTools 2.0 dev/testing notebook\n\nInteractive companion to the `dev-2.0` modernization branch (see `notes/hypertools_2.0_roadmap.md`).\n\n**Purpose:** exercise EVERY public function across the full use-case matrix, visually verify output, and compare backends. Each section corresponds to one public API function. Run top-to-bottom in Jupyter, Colab, or Kaggle — the three environments we must support.\n\n**Use-case matrix** (applied to each function where relevant):\n\n| dimension | cases |\n|-|-|\n| input type | single ndarray, list of ndarrays, DataFrame, list of DataFrames, nested lists (→ MultiIndex), text |\n| dimensionality | 2D, 3D, high-D (reduced), 1D edge case |\n| missing data | none, NaNs (PPCA fill) |\n| styling | fmt strings, hue/group, legend, labels, title, multilevel-index color/thickness/opacity |\n| coloring | categorical labels, continuous values, mixture proportions, user matrices, multicolored lines |\n| models | reduce (PCA/IncrementalPCA/TSNE/UMAP), cluster (KMeans/HDBSCAN), mixtures (GaussianMixture/BayesianGaussianMixture/LDA/NMF), align (hyper/SRM) |\n| animation | static, animate=True, spin, sliding window |\n| backend | matplotlib (default), plotly (interactive) — 2.0 feature |"
+   "source": "# HyperTools 1.0 dev/testing notebook\n\nInteractive companion to the `dev-1.0` modernization branch (see `notes/hypertools_1.0_roadmap.md`).\n\n**Purpose:** exercise EVERY public function across the full use-case matrix, visually verify output, and compare backends. Each section corresponds to one public API function. Run top-to-bottom in Jupyter, Colab, or Kaggle — the three environments we must support.\n\n**Use-case matrix** (applied to each function where relevant):\n\n| dimension | cases |\n|-|-|\n| input type | single ndarray, list of ndarrays, DataFrame, list of DataFrames, nested lists (→ MultiIndex), text |\n| dimensionality | 2D, 3D, high-D (reduced), 1D edge case |\n| missing data | none, NaNs (PPCA fill) |\n| styling | fmt strings, hue/group, legend, labels, title, multilevel-index color/thickness/opacity |\n| coloring | categorical labels, continuous values, mixture proportions, user matrices, multicolored lines |\n| models | reduce (PCA/IncrementalPCA/TSNE/UMAP), cluster (KMeans/HDBSCAN), mixtures (GaussianMixture/BayesianGaussianMixture/LDA/NMF), align (hyper/SRM) |\n| animation | static, animate=True, spin, sliding window |\n| backend | matplotlib (default), plotly (interactive) — 2.0 feature |"
   },
   {
    "cell_type": "code",
@@ -19,7 +19,7 @@
     "\n",
     "t0 = time.time()\n",
     "import hypertools as hyp\n",
-    "print(f'hypertools {hyp.__version__} imported in {time.time()-t0:.2f}s')  # 2.0 target: < 1s\n",
+    "print(f'hypertools {hyp.__version__} imported in {time.time()-t0:.2f}s')  # 1.0 target: < 1s\n",
     "\n",
     "ENV = 'colab' if 'google.colab' in sys.modules else ('kaggle' if os.path.exists('/kaggle') else 'local')\n",
     "print('environment:', ENV)"
@@ -78,7 +78,7 @@
    "source": [
     "## 3. `hyp.plot` — animation\n",
     "\n",
-    "Known-broken on master: numpy>=2 Jupyter animations (#265), Colab `animate=True` (#235), figures in loops (#264). These cells are the acceptance tests for the 2.0 fixes."
+    "Known-broken on master: numpy>=2 Jupyter animations (#265), Colab `animate=True` (#235), figures in loops (#264). These cells are the acceptance tests for the 1.0 fixes."
    ]
   },
   {
@@ -91,14 +91,14 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "## 4. `hyp.plot` — 2.0 features (IMPLEMENTED on dev-2.0)\n\nBackend switching, nested-list multilevel styling, mixture-model soft clustering, and matrix-valued hue. Each of these also has pytest coverage (`test_interactive.py`, `test_nested.py`, `test_colors.py`, `test_cluster.py`) and screenshot-harness cases."
+   "source": "## 4. `hyp.plot` — 1.0 features (IMPLEMENTED on dev-1.0)\n\nBackend switching, nested-list multilevel styling, mixture-model soft clustering, and matrix-valued hue. Each of these also has pytest coverage (`test_interactive.py`, `test_nested.py`, `test_colors.py`, `test_cluster.py`) and screenshot-harness cases."
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": "# --- backend switching (approved policy: plotly only on Colab/Kaggle) ---\nhyp.plot(walk, backend='matplotlib')      # explicit classic renderer\nfig = hyp.plot(walk, backend='plotly')    # interactive plotly renderer\nhyp.plot(walk, backend='auto')            # plotly on Colab/Kaggle, else matplotlib\n\n# --- plotly ANIMATIONS: interactive frames in the notebook, exportable to\n# gif / animated png / mp4 exactly like the matplotlib backend ---\ngeo = hyp.plot(walk, animate=True, duration=3, backend='plotly')  # play button\nhyp.plot(walk, animate='spin', duration=3, backend='plotly',\n         save_path='plotly_spin_demo.gif', show=False)\nfrom IPython.display import Image as IPyImage\ndisplay(IPyImage('plotly_spin_demo.gif'))\n\n# --- nested lists -> multilevel styling (color by outer group; deeper\n# leaves render thinner + fainter) ---\nhyp.plot([[walk, walk2], [np.cumsum(np.random.default_rng(7).standard_normal((200, 10)), axis=0)]])\n\n# --- mixture models on OVERLAPPING clusters: multi-class membership shows\n# as blended (intermediate) colors between components ---\nprops = hyp.cluster(oclusters, cluster='GaussianMixture', n_clusters=3)\nsoft = np.mean(props.max(axis=1) < 0.9)\nprint(f'mixture proportions: {props.shape}; {soft:.0%} of points have genuinely mixed membership')\nhyp.plot(oclusters, 'o', cluster='GaussianMixture', n_clusters=3)\n\n# --- robust coloring: matrix-valued hue (proportions, weights, any matrix) ---\nhyp.plot(oclusters, 'o', hue=props)\nhyp.plot(walk, 'o', hue=np.arange(len(walk), dtype=float))  # continuous hue\n\n# --- MULTICOLORED LINES: continuous/matrix hue + line fmt colors each\n# trajectory continuously along its length (both backends) ---\nhyp.plot(walk, hue=np.arange(len(walk), dtype=float))\nhyp.plot(walk, hue=np.arange(len(walk), dtype=float), backend='plotly')\n\n# --- apply_model: stack -> fit once -> unstack core ---\nembedded = hyp.apply_model([walk, walk2], 'PCA', ndims=3)\nprint('shared embedding:', [e.shape for e in embedded])\n\n# --- retired in 2.0 (raise clear errors now): plot(group=...), plot(model=...),\n# reduce(model=/align=/normalize=), align(method=/align=True), cluster(ndims=) ---"
+   "source": "# --- backend switching (approved policy: plotly only on Colab/Kaggle) ---\nhyp.plot(walk, backend='matplotlib')      # explicit classic renderer\nfig = hyp.plot(walk, backend='plotly')    # interactive plotly renderer\nhyp.plot(walk, backend='auto')            # plotly on Colab/Kaggle, else matplotlib\n\n# --- plotly ANIMATIONS: interactive frames in the notebook, exportable to\n# gif / animated png / mp4 exactly like the matplotlib backend ---\ngeo = hyp.plot(walk, animate=True, duration=3, backend='plotly')  # play button\nhyp.plot(walk, animate='spin', duration=3, backend='plotly',\n         save_path='plotly_spin_demo.gif', show=False)\nfrom IPython.display import Image as IPyImage\ndisplay(IPyImage('plotly_spin_demo.gif'))\n\n# --- nested lists -> multilevel styling (color by outer group; deeper\n# leaves render thinner + fainter) ---\nhyp.plot([[walk, walk2], [np.cumsum(np.random.default_rng(7).standard_normal((200, 10)), axis=0)]])\n\n# --- mixture models on OVERLAPPING clusters: multi-class membership shows\n# as blended (intermediate) colors between components ---\nprops = hyp.cluster(oclusters, cluster='GaussianMixture', n_clusters=3)\nsoft = np.mean(props.max(axis=1) < 0.9)\nprint(f'mixture proportions: {props.shape}; {soft:.0%} of points have genuinely mixed membership')\nhyp.plot(oclusters, 'o', cluster='GaussianMixture', n_clusters=3)\n\n# --- robust coloring: matrix-valued hue (proportions, weights, any matrix) ---\nhyp.plot(oclusters, 'o', hue=props)\nhyp.plot(walk, 'o', hue=np.arange(len(walk), dtype=float))  # continuous hue\n\n# --- MULTICOLORED LINES: continuous/matrix hue + line fmt colors each\n# trajectory continuously along its length (both backends) ---\nhyp.plot(walk, hue=np.arange(len(walk), dtype=float))\nhyp.plot(walk, hue=np.arange(len(walk), dtype=float), backend='plotly')\n\n# --- apply_model: stack -> fit once -> unstack core ---\nembedded = hyp.apply_model([walk, walk2], 'PCA', ndims=3)\nprint('shared embedding:', [e.shape for e in embedded])\n\n# --- retired in 1.0 (raise clear errors now): plot(group=...), plot(model=...),\n# reduce(model=/align=/normalize=), align(method=/align=True), cluster(ndims=) ---"
   },
   {
    "cell_type": "markdown",

dev/hypertools_2.0_dev_executed.ipynb → dev/hypertools_1.0_dev_executed.ipynb

@@ -5,9 +5,9 @@
    "id": "a788b512",
    "metadata": {},
    "source": [
-    "# HyperTools 2.0 dev/testing notebook\n",
+    "# HyperTools 1.0 dev/testing notebook\n",
     "\n",
-    "Interactive companion to the `dev-2.0` modernization branch (see `notes/hypertools_2.0_roadmap.md`).\n",
+    "Interactive companion to the `dev-1.0` modernization branch (see `notes/hypertools_1.0_roadmap.md`).\n",
     "\n",
     "**Purpose:** exercise EVERY public function across the full use-case matrix, visually verify output, and compare backends. Each section corresponds to one public API function. Run top-to-bottom in Jupyter, Colab, or Kaggle — the three environments we must support.\n",
     "\n",
@@ -42,7 +42,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "hypertools 2.0.0.dev0 imported in 3.86s\n",
+      "hypertools 1.0.0.dev0 imported in 3.86s\n",
       "environment: local\n"
      ]
     }
@@ -56,7 +56,7 @@
     "\n",
     "t0 = time.time()\n",
     "import hypertools as hyp\n",
-    "print(f'hypertools {hyp.__version__} imported in {time.time()-t0:.2f}s')  # 2.0 target: < 1s\n",
+    "print(f'hypertools {hyp.__version__} imported in {time.time()-t0:.2f}s')  # 1.0 target: < 1s\n",
     "\n",
     "ENV = 'colab' if 'google.colab' in sys.modules else ('kaggle' if os.path.exists('/kaggle') else 'local')\n",
     "print('environment:', ENV)"
@@ -163,7 +163,7 @@
    "source": [
     "## 3. `hyp.plot` — animation\n",
     "\n",
-    "Known-broken on master: numpy>=2 Jupyter animations (#265), Colab `animate=True` (#235), figures in loops (#264). These cells are the acceptance tests for the 2.0 fixes."
+    "Known-broken on master: numpy>=2 Jupyter animations (#265), Colab `animate=True` (#235), figures in loops (#264). These cells are the acceptance tests for the 1.0 fixes."
    ]
   },
   {
@@ -37291,7 +37291,7 @@
    "id": "1ca254ea",
    "metadata": {},
    "source": [
-    "## 4. `hyp.plot` — 2.0 features (IMPLEMENTED on dev-2.0)\n",
+    "## 4. `hyp.plot` — 1.0 features (IMPLEMENTED on dev-1.0)\n",
     "\n",
     "Backend switching, nested-list multilevel styling, mixture-model soft clustering, and matrix-valued hue. Each of these also has pytest coverage (`test_interactive.py`, `test_nested.py`, `test_colors.py`, `test_cluster.py`) and screenshot-harness cases."
    ]
@@ -44449,7 +44449,7 @@
     "embedded = hyp.apply_model([walk, walk2], 'PCA', ndims=3)\n",
     "print('shared embedding:', [e.shape for e in embedded])\n",
     "\n",
-    "# --- retired in 2.0 (raise clear errors now): plot(group=...), plot(model=...),\n",
+    "# --- retired in 1.0 (raise clear errors now): plot(group=...), plot(model=...),\n",
     "# reduce(model=/align=/normalize=), align(method=/align=True), cluster(ndims=) ---"
    ]
   },

docs/api.rst

@@ -7,16 +7,6 @@
 API reference
 =============
 
-.. _plot_api:
-
-DataGeometry class
-------------------
-
-.. autosummary::
-  :toctree:
-
-  hypertools.DataGeometry
-
 Load
 ------------------
 
@@ -73,6 +63,22 @@ Cluster
 
   hypertools.cluster
 
+Predict
+------------------
+
+.. autosummary::
+  :toctree:
+
+  hypertools.predict
+
+Impute
+------------------
+
+.. autosummary::
+  :toctree:
+
+  hypertools.impute
+
 Apply model
 ------------------
 
@@ -99,7 +105,7 @@ Tools
 .. autosummary::
   :toctree:
 
-  hypertools.tools.procrustes
+  hypertools.align.procrustes
 
 .. autosummary::
   :toctree:

docs/auto_examples/analyze.codeobj.json

@@ -8,38 +8,6 @@
    "name": "list"
   }
  ],
- "geo": [
-  {
-   "is_class": false,
-   "is_explicit": false,
-   "module": "hypertools.datageometry",
-   "module_short": "hypertools",
-   "name": "DataGeometry"
-  },
-  {
-   "is_class": false,
-   "is_explicit": false,
-   "module": "hypertools",
-   "module_short": "hypertools",
-   "name": "DataGeometry"
-  }
- ],
- "geo.get_data": [
-  {
-   "is_class": false,
-   "is_explicit": false,
-   "module": "hypertools.datageometry",
-   "module_short": "hypertools",
-   "name": "DataGeometry.get_data"
-  },
-  {
-   "is_class": false,
-   "is_explicit": false,
-   "module": "hypertools",
-   "module_short": "hypertools",
-   "name": "DataGeometry.get_data"
-  }
- ],
  "hyp.analyze": [
   {
    "is_class": false,

docs/auto_examples/analyze.ipynb

@@ -1,58 +1,54 @@
 {
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "# Install hypertools (dev-2.0 preview) -- run this first on Colab.\n",
-    "# On release this becomes: %pip install hypertools\n",
-    "%pip install -q \"hypertools[interactive] @ git+https://github.com/ContextLab/hypertools.git@dev-2.0\"\n",
-    "\n",
-    "%matplotlib inline"
-   ]
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "# Install hypertools (dev-1.0-refactor preview) -- run this first on Colab.\n# On release this becomes: %pip install hypertools\n%pip install -q \"hypertools[interactive] @ git+https://github.com/ContextLab/hypertools.git@dev-1.0-refactor\"\n\n%matplotlib inline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "\n# Analyze data and then plot\n\nThis example demonstrates how to use the `analyze` function to process data\nprior to plotting. The data is a list of numpy arrays representing\nmulti-voxel activity patterns (columns) over time (rows).  First, analyze function\nnormalizes the columns of each matrix (within each matrix). Then the data is\nreduced using PCA (10 dims) and finally it is aligned with hyperalignment. We can\nthen plot the data with hyp.plot, which further reduces it so that it can be\nvisualized.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "# sphinx_gallery_thumbnail_path = '_static/thumbnails/sphx_glr_analyze_thumb.png'\n\n# Code source: Andrew Heusser\n# License: MIT\n\n# load hypertools\nimport hypertools as hyp\n\n# load the data\ndata = hyp.load('weights')\n\n# process the data\ndata = hyp.analyze(data, normalize='within', reduce='PCA', ndims=10,\n                align='hyper')\n\n# plot it\nhyp.plot(data)"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.12.10"
+    }
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n# Analyze data and then plot\n\nThis example demonstrates how to use the `analyze` function to process data\nprior to plotting. The data is a list of numpy arrays representing\nmulti-voxel activity patterns (columns) over time (rows).  First, analyze function\nnormalizes the columns of each matrix (within each matrix). Then the data is\nreduced using PCA (10 dims) and finally it is aligned with hyperalignment. We can\nthen plot the data with hyp.plot, which further reduces it so that it can be\nvisualized.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "# sphinx_gallery_thumbnail_path = '_static/thumbnails/sphx_glr_analyze_thumb.png'\n\n# Code source: Andrew Heusser\n# License: MIT\n\n# load hypertools\nimport hypertools as hyp\n\n# load the data\ngeo = hyp.load('weights')\ndata = geo.get_data()\n\n# process the data\ndata = hyp.analyze(data, normalize='within', reduce='PCA', ndims=10,\n                align='hyper')\n\n# plot it\nhyp.plot(data)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.12.10"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

docs/auto_examples/analyze.py

@@ -22,8 +22,7 @@
 import hypertools as hyp
 
 # load the data
-geo = hyp.load('weights')
-data = geo.get_data()
+data = hyp.load('weights')
 
 # process the data
 data = hyp.analyze(data, normalize='within', reduce='PCA', ndims=10,

docs/auto_examples/analyze.py.md5

@@ -1 +1 @@
-b188d6db61e339db0651634314ffdc5a
+68992cc2797469b2fefb43fcf35a16d6

docs/auto_examples/analyze.rst

@@ -30,7 +30,7 @@ reduced using PCA (10 dims) and finally it is aligned with hyperalignment. We ca
 then plot the data with hyp.plot, which further reduces it so that it can be
 visualized.
 
-.. GENERATED FROM PYTHON SOURCE LINES 15-34
+.. GENERATED FROM PYTHON SOURCE LINES 15-33
 
 
 
@@ -55,8 +55,7 @@ visualized.
     import hypertools as hyp
 
     # load the data
-    geo = hyp.load('weights')
-    data = geo.get_data()
+    data = hyp.load('weights')
 
     # process the data
     data = hyp.analyze(data, normalize='within', reduce='PCA', ndims=10,
@@ -68,7 +67,7 @@ visualized.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 0.389 seconds)
+   **Total running time of the script:** (0 minutes 0.374 seconds)
 
 
 .. _sphx_glr_download_auto_examples_analyze.py: