From 20904482eb5df920740b0c68e62e21b324559ffd Mon Sep 17 00:00:00 2001 From: Ofer Shaal Date: Wed, 1 Jul 2026 02:11:10 -0400 Subject: [PATCH] docs(adr-272): propose ruvector-tabfm zero-shot tabular foundation model Research + plan (no implementation) for adding a zero-shot tabular FM to RuVector, framed as the tabular sibling of ruvector-timesfm (port + point inward at the DB's own index/SONA tuning). Key findings: - License gate: TabFM weights are non-commercial/non-redistributable -> cannot ship. TabPFN 2.5/2.6/3 likewise. TabPFN-v2 (clf+reg) is Apache-2.0 + attribution (Prior Labs License) -> shippable Track-B base. - Track A benchmark (docs/research/tabfm/track_a_benchmark.py, license-clean): shippable TabPFN-v2 zero-shot on CPU is top-or-tied on 5/6 OpenML datasets vs tuned XGBoost/HistGBM (mean rank 1.33 vs 2.83); big multiclass win on vehicle (+0.034 AUC). Verdict: GO. - Highest-impact uses: (1) core candle port, (2) row-compression tabular-row embedder (row->vector, native to the vector DB), (3) inward zero-shot autotuner (predict HNSW/IVF/quant knobs), (4) Postgres tab_predict() = local AI.PREDICT. Adds ADR-272 + docs/research/tabfm/ (plan, benchmark script, results). No product code; direction left open for maintainer decision. --- ...ruvector-tabfm-tabular-foundation-model.md | 104 +++++++ docs/research/tabfm/PLAN.md | 260 ++++++++++++++++++ docs/research/tabfm/track_a_benchmark.py | 162 +++++++++++ docs/research/tabfm/track_a_results.json | 230 ++++++++++++++++ 4 files changed, 756 insertions(+) create mode 100644 docs/adr/ADR-272-ruvector-tabfm-tabular-foundation-model.md create mode 100644 docs/research/tabfm/PLAN.md create mode 100644 docs/research/tabfm/track_a_benchmark.py create mode 100644 docs/research/tabfm/track_a_results.json diff --git a/docs/adr/ADR-272-ruvector-tabfm-tabular-foundation-model.md b/docs/adr/ADR-272-ruvector-tabfm-tabular-foundation-model.md new file mode 100644 index 0000000000..8888ff143a --- /dev/null +++ b/docs/adr/ADR-272-ruvector-tabfm-tabular-foundation-model.md @@ -0,0 +1,104 @@ +# ADR-272: `ruvector-tabfm` — A Zero-Shot Tabular Foundation Model, and Pointing It Inward at the DB's Own Tuning + +- **Status**: Proposed — **gated, with a viable pivot found**. Gate #1 (weight license) resolved 2026-07-01: TabFM's own weights are **non-commercial / non-redistributable → NO-GO to ship**. But **`TabPFN-v2` weights (clf + reg) are Apache-2.0-with-attribution ("Prior Labs License", id `priorlabs-1-1`) → commercially shippable and redistributable.** So the product path is *port TabPFN-v2 timesfm-style* + adopt TabFM's Apache-2.0 architectural ideas, not "train from scratch." See "Gate #1 resolution" below and `docs/research/tabfm/PLAN.md`. +- **Date**: 2026-07-01 +- **External anchor**: Google Research, "Introducing TabFM: a zero-shot foundation model for tabular data" (research.google, 2026) +- **Mirrors**: ADR-191 line — `ruvector-timesfm` (candle port of TimesFM, wired inward into `sweep::EarlyStopper` / `rebuild` / `anomaly`) +- **Touches**: SONA (ADR-271), IVF/PQ tuning line (ADR-205/206), `ruvector-postgres` (143 SQL functions), `ruvector-attention` + +--- + +## Context + +**TabFM** is a foundation model for tabular classification/regression that predicts **zero-shot via in-context learning (ICL)**: hand it a table of labeled training rows + unlabeled test rows as one prompt, and it predicts in a **single forward pass** — no per-dataset training, hyper-parameter search, or feature engineering. Its architecture is three stages: (1) **alternating row↔column attention** over the raw grid (captures feature interactions natively, replacing manual feature crafting); (2) **row compression** — each contextualized row collapses to one dense vector; (3) an **ICL transformer over the compressed row-vectors** (cheap; scales to larger tables). It is pretrained on hundreds of millions of **synthetic tables** from structural causal models, is order-invariant, beats heavily-tuned XGBoost zero-shot on TabArena (38 classification + 13 regression datasets, 700–150k rows), and Google ships it in BigQuery as an `AI.PREDICT` SQL call. + +Two facts make this a high-leverage fit for RuVector, not a novelty: + +1. **We already did this for time series.** `crates/ruvector-timesfm` is a parity-validated candle port of TimesFM, and it is not merely a user feature — it is wired into the DB's *own* machinery: `sweep::EarlyStopper` (ADR-191) kills doomed tuning runs, `rebuild` forecasts recall-drift to time HNSW rebuilds, `anomaly` watches telemetry. **TabFM is the tabular sibling of TimesFM**, and the same "port it *and* point it inward" playbook applies. +2. **Every ingredient exists and no tabular work does.** A grep of `crates/`, `npm/`, `docs/` finds zero TabPFN/TabFM/tabular-ICL work — this is greenfield. Meanwhile the stack is already here: candle transformers (6 crates), a modular attention SDK (`ruvector-attention`: graph/sparse/MoE/training), quantization (RaBitQ/PQ), WASM builds, and a 143-function Postgres surface that is the natural home for a local `AI.PREDICT`. + +The open question this ADR answers: *which* TabFM-derived capabilities are worth building first, and in what order. + +## Decision + +Create **`crates/ruvector-tabfm`**, structured as a deliberate mirror of `ruvector-timesfm` (feature-gated `candle` numeric path; load-weights-once predictor; JSON-in/JSON-out CLI = MCP tool; WASM build). Scope the first release to **inference against published weights only** — porting the architecture, *not* training our own model. Prioritize the four highest-impact surfaces below; explicitly defer the two long-tail items. + +### 1. Core inference port (foundation — everything depends on it) +Port TabFM's three-stage forward pass into candle inside `ruvector-tabfm`: +- `predictor.rs`: `TabFm::load(weights, device)` → `predict(train_rows, train_labels, test_rows)` → class probabilities (classification) / point + p10..p90 quantiles (regression, reusing the quantile-band idiom already in `ruvector-timesfm::forecast_types`). +- The alternating row↔column attention block is the real engineering. Assess candle op coverage first; if a fused kernel is needed, add it under `ruvector-attention` (which already houses sparse/graph/custom attention) rather than in-crate, so it is reusable. +- `bin/predict.rs`: JSON-in/JSON-out CLI, registered as the `tab_predict` MCP tool (parity with `time_series_forecast`). +- Feature gating identical to timesfm: numeric path behind `candle` (+ `cuda`/`metal`); a stock `cargo build` stays light. + +### 2. Row-compression as a native tabular-row embedder (largest DB synergy) +TabFM's stage-2 already maps a heterogeneous row (mixed numeric/categorical) to **one dense vector**. That is exactly the "how do I embed a structured record?" problem a vector DB faces. Expose stage-2 as a standalone `encode_row(row) -> Vec` so a user can `INSERT` a SQL row and get an **indexable embedding** — no external embedding model, no feature engineering — that flows straight into the existing HNSW/DiskANN index **untouched**. This is a genuinely new retrieval modality (structured-record search) for near-zero marginal cost once §1 exists. + +### 3. Point TabFM inward — zero-shot index/SONA autotuner (the "make the DB smarter" play) +Choosing HNSW `M`/`efConstruction`, IVF `nprobe`, and quantization level from workload features **is a tabular prediction problem**. SONA (ADR-271) learns this online but cold-starts blind; the IVF/PQ tuning line (ADR-205/206) established that these knobs are workload-dependent and non-differentiable. Feed TabFM a small table of `(workload features → best config)` from tuning history and take its **zero-shot warm-start** prediction — the exact analog of `timesfm::sweep::EarlyStopper` gating optimization runs. This is the differentiator: a self-optimizing DB that cold-starts *near-optimal* instead of blind. + +### 4. Postgres `ruvector_tab_predict()` — a local, free `AI.PREDICT` (headline feature) +`SELECT ruvector_tab_predict('churn', ROW(...))` inside Postgres is BigQuery's `AI.PREDICT`, but local, free, and offline — a headline that fits the README's "local AI, PostgreSQL built in" story. A thin SQL wrapper over §1; low effort, high narrative value. + +### Explicitly deferred (out of scope for the first release) +- **Structured/tabular reranker** alongside `ruvector-gnn-rerank` (row-column attention over `(query, candidate-metadata)`): promising and complementary to the GNN, but adds an ICL pass to the per-query hot path — revisit after the core lands and latency is measured. + +### Gate #1 resolution (2026-07-01) — the split license reshapes everything above + +Verified against the primary sources: +- **Code** — `github.com/google-research/tabfm` is **Apache-2.0** ("not an officially supported Google product"; JAX + PyTorch backends; weights auto-download from HF). +- **Weights** — `google/tabfm-1.0.0-pytorch` ship under **TabFM Non-Commercial License v1.0**. Verbatim-confirmed terms: *"You may only access, use, or create Derivatives of the TabFM Model for Non-Commercial Purposes"*; commercial = *"any revenue-generating activity … interactions with end users or production systems … or to train, fine-tune, or distill other models for commercial use"*; *"You will not … Distribute the TabFM Model or a Derivative"*; *"Any restrictions set forth herein … also apply to any Derivative."* + +**Consequence for §1–§4 as written:** a candle port of the published weights is a *Derivative*, so it inherits the non-commercial + no-redistribution terms. That is incompatible with RuVector's identity (MIT/Apache, "free forever," bundled in `npx ruvector`, powering the commercial Cognitum product). Therefore the naive "port the weights" plan **cannot ship**. This splits the work into two tracks: + +- **Track A — Research (allowed now, published weights):** use TabFM strictly for **internal benchmarking/research** (the license's own non-commercial examples) to *quantify the ceiling* — does §3's TabFM-predicted config beat SONA? is §2's row embedding competitive? — **provided results do not feed commercial decision-making, paid products, or production**. No bundling, no redistribution, no production inference. This is a measurement exercise, not a feature. +- **Track B — Product (the shippable path):** **port `TabPFN-v2` weights** — the classifier and regressor are both under the **Prior Labs License = Apache-2.0 + attribution** (verbatim obligations: display *"Built with PriorLabs-TabPFN"* in docs/UI; if we distill/fine-tune into a *distributed* model, prefix its name with *"TabPFN"*; internal benchmarking needs no attribution). These weights are **commercial-OK and redistributable**, so we can bundle them in `npx ruvector`, Postgres, WASM, and Cognitum. Port them timesfm-style into candle, and separately adopt TabFM's **Apache-2.0 architectural** ideas (row compression, scale-friendly ICL-over-compressed-rows) as our own optimization on top. + +**This resurrects §1–§4 as shippable features** — just built on TabPFN-v2 weights (Apache-2.0+attr) instead of TabFM's non-commercial weights. "Train our own weights from scratch on a synthetic-SCM engine" reverts to a *genuinely optional* research bet (only if we later need to beat TabPFN-v2 or shed the attribution clause), **not** the critical path. + +**Licensing scorecard (weights):** + +| Model | Weights license | Ship in RuVector? | +|---|---|---| +| TabFM 1.0 | TabFM Non-Commercial v1.0 | ❌ research/benchmark only | +| TabPFN-2.5 / 2.6 / 3 | `tabpfn-2.5-license-v1.1` (non-commercial) | ❌ research/benchmark only | +| **TabPFN-v2 (clf + reg)** | **Prior Labs License (Apache-2.0 + attribution)** | ✅ **yes — Track B base** | + +**Revised sequencing:** Track A first (cheap — benchmark TabFM *and* the newer TabPFN weights internally to know the accuracy ceiling we'd forgo by shipping v2) → Track B (port TabPFN-v2 into `ruvector-tabfm`, timesfm-style; layer TabFM's Apache-2.0 architecture ideas) → §2/§4/§3 on the shipped v2 weights. + +## Consequences + +**Positive** +- Adds a whole capability class (zero-shot tabular ML + structured-record embedding) with **no new heavy runtime** — candle is already a workspace dependency, and the timesfm crate is a proven template. +- §3 turns TabFM into infrastructure the DB uses on itself, compounding the SONA/BET-line investment rather than sitting beside it. +- §4 gives a concrete, demoable, README-worthy feature (`AI.PREDICT` in local Postgres) that no pgvector competitor has. + +**Negative / risks** +- **Weight availability & license — RESOLVED, and it's a blocker for the naive path.** The published weights are **non-commercial + non-redistributable**, and derivatives inherit those terms (see "Gate #1 resolution"). We cannot bundle or ship them. Track B (Apache-2.0 architecture + our own weights) is the only commercial path; parity-validation would then be against TabFM's *behavior*, not a redistribution of its weights. +- **Candle op coverage.** The alternating row-column attention may need a custom kernel; budget for a `ruvector-attention` addition. +- **ICL context scaling.** Inference cost/memory grows with the number of in-context training rows; §3's tuning-history tables are small (safe), but §4 user tables need a documented row cap + a subsampling story. +- **§3 needs labeled history.** The autotuner is only as good as the `(features → best config)` table; it is strongest *after* the tuning-history logging exists, so it lands last. + +## Implementation status + +Not started. This ADR and `docs/research/tabfm/PLAN.md` are the research + plan deliverable. + +Gate #1 (weight license) is now **resolved with a viable pivot**: TabFM's own weights can't ship, but **TabPFN-v2 (Apache-2.0 + attribution)** can. + +**Track A ran (2026-07-01) → GO.** Empirical benchmark (license-clean; details in `docs/research/tabfm/PLAN.md` §5b): shippable TabPFN-v2, zero-shot on CPU, is **top-or-tied on 5/6** standard OpenML datasets vs **tuned** XGBoost/HistGBM (mean rank 1.33 vs 2.83), with one large multiclass win (`vehicle` +0.034 AUC / +8.5 pts acc) and binary margins small-but-consistent. The gated v2.5/v3 weights refused to load without license acceptance — corroborating the license finding. **Conclusion: the Apache-2.0 model we can ship is good enough to own → proceed to Track B.** Remaining steps: +1. **Track A (research):** benchmark harness to measure the accuracy ceiling of TabFM + newer TabPFN (non-commercial) weights, so we know exactly what we trade away by shipping v2. Cheap; de-risks the bet. +2. **Track B (product):** port TabPFN-v2 (clf + reg) into `crates/ruvector-tabfm` timesfm-style; satisfy the attribution clause ("Built with PriorLabs-TabPFN" in docs; name any distilled distributed model "TabPFN…"); layer TabFM's Apache-2.0 row-compression/ICL ideas. + +Naming note: the crate is provisionally `ruvector-tabfm` (the initiative's origin), but since the shipped base is TabPFN-v2, `ruvector-tabpfn` may be the more accurate name — and the attribution clause makes a "TabPFN" association *beneficial*, not just permitted. Decide at scaffold time. + +## References + +- Google Research — "Introducing TabFM: a zero-shot foundation model for tabular data" (research.google/blog, 2026). Builds on TabPFN and TabICL. +- Code (Apache-2.0): `github.com/google-research/tabfm`. Weights (non-commercial): `huggingface.co/google/tabfm-1.0.0-pytorch`; license text: `.../blob/main/LICENSE` = **TabFM Non-Commercial License v1.0**. +- **Track B base (Apache-2.0 + attribution):** `huggingface.co/Prior-Labs/TabPFN-v2-clf` + `TabPFN-v2-reg`; license = Prior Labs License (`priorlabs-1-1`), text mirrored at `github.com/PriorLabs/TabPFN/blob/main/LICENSE`. Attribution: display "Built with PriorLabs-TabPFN"; prefix any distilled/distributed derivative model name with "TabPFN"; internal benchmarking exempt. +- TabPFN newer weights (**non-commercial**, research-only): `Prior-Labs/tabpfn_2_5` etc., license `tabpfn-2.5-license-v1.1`. Commercial via Prior Labs Enterprise License (`sales@priorlabs.ai`). +- `crates/ruvector-timesfm` — the mirror template (ADR-191 line): `predictor`/`forecaster`, `sweep::EarlyStopper`, `rebuild`, `anomaly`, JSON CLI = MCP tool. +- ADR-271 — Metaharness-Darwin for SONA self-improvement (the online-tuning substrate §3 warm-starts). +- ADR-205/206 — IVF `nprobe` / IVFPQ tuning line (the workload-dependent knobs §3 predicts). +- `crates/ruvector-attention` — home for any custom row-column attention kernel. +- `crates/ruvector-postgres` — surface for §4's `ruvector_tab_predict()`. +- Full landscape, benchmarks, and reimplementation notes: `docs/research/tabfm/PLAN.md`. diff --git a/docs/research/tabfm/PLAN.md b/docs/research/tabfm/PLAN.md new file mode 100644 index 0000000000..d4173add66 --- /dev/null +++ b/docs/research/tabfm/PLAN.md @@ -0,0 +1,260 @@ +# TabFM → RuVector: Research & Improvement Plan + +Companion to **ADR-272**. Grounds the decision in TabFM's architecture, benchmarks, +and a reimplementation map onto existing RuVector crates. Scope: what to build, +in what order, ranked by impact. + +Source: Google Research, *"Introducing TabFM: a zero-shot foundation model for +tabular data"* (research.google/blog, 2026), which builds on TabPFN and TabICL. + +--- + +## 1. What TabFM is (technical) + +TabFM performs **classification and regression on tables** with **no per-dataset +training, tuning, or feature engineering**. It frames prediction as **in-context +learning (ICL)**: the whole table — labeled training rows *plus* unlabeled test +rows — is one prompt, and prediction is a **single forward pass**. + +### Architecture (three stages) + +| Stage | Mechanism | Purpose | +|---|---|---| +| 1 | **Alternating row↔column attention** over the raw grid | Attend across columns (feature interactions), then across rows (example comparisons), repeatedly. Replaces manual feature engineering. | +| 2 | **Row compression** | Collapse each contextualized row into a single dense vector. | +| 3 | **ICL transformer** over the compressed row-vectors | Cheap sequence model over one-vector-per-row; enables inference on larger tables. | + +Key properties: +- **Order-invariant** — tables are 2-D and *orderless* (rows and columns can be + permuted); the architecture is built for this, unlike 1-D ordered LM sequences. +- **Unified inference** — no dataset-specific parameter updates; predictions come + from interpreting column-row relationships *in context* at inference time. + +### Training + +Trained **only on synthetic data** — "hundreds of millions of synthetic datasets" +generated by **structural causal models** with varied random functions, to cover +distribution diversity and complex feature relationships. This sidesteps the +scarcity of diverse, non-proprietary real tables. + +### Benchmarks + +Evaluated on **TabArena**: 38 classification + 13 regression datasets, 700–150,000 +samples. Two configs: +- **TabFM** — out-of-the-box single forward pass, no tuning. +- **TabFM-Ensemble** — adds cross-features, SVD features, and calibration (Platt + scaling for classification; non-negative least squares for the ensemble weights). + +Result: competitive-to-winning ELO, **consistently outperforming heavily-tuned, +industry-standard supervised algorithms** (e.g. gradient-boosted trees / XGBoost) +zero-shot. + +### Deployment + +Available on Hugging Face + GitHub. Google integrates it into **BigQuery** as an +`AI.PREDICT` SQL command — classification/regression with no ML expertise. + +### Licensing (Gate #1 — resolved 2026-07-01) ⚠️ **SPLIT LICENSE** + +| Artifact | Location | License | +|---|---|---| +| **Code** | `github.com/google-research/tabfm` | **Apache-2.0** (JAX + PyTorch; weights auto-download from HF) | +| **Weights** | `google/tabfm-1.0.0-pytorch` | **TabFM Non-Commercial License v1.0** | + +Verbatim-confirmed weight terms: +- *"You may only access, use, or create Derivatives of the TabFM Model for **Non-Commercial Purposes**."* +- Commercial (prohibited) = *"any revenue-generating activity … interactions with end users or production systems … or to train, fine-tune, or **distill** other models for commercial use."* +- *"You will not … **Distribute** the TabFM Model or a Derivative."* +- *"Any restrictions … also apply to any **Derivative** you create."* +- Non-commercial carve-out: internal benchmarking / academic research, *provided results are not used in commercial decision-making, client deliverables, or paid products/services.* + +**Implication:** a candle port of the weights is a Derivative → inherits non-commercial + no-redistribution. **Incompatible** with RuVector (MIT/Apache, "free forever," bundled in `npx ruvector`, powering commercial Cognitum). The naive TabFM weight-port **cannot ship**. + +### The pivot — a shippable tabular FM exists (TabPFN-v2) + +Checked the obvious alternative and found a permissive escape hatch: + +| Model weights | License | Ship in RuVector? | +|---|---|---| +| TabFM 1.0 | TabFM Non-Commercial v1.0 | ❌ research/benchmark only | +| TabPFN-2.5 / 2.6 / 3 | `tabpfn-2.5-license-v1.1` (non-commercial) | ❌ research/benchmark only | +| **TabPFN-v2 (`clf` + `reg`)** | **Prior Labs License (`priorlabs-1-1`) = Apache-2.0 + attribution** | ✅ **yes** | + +**Prior Labs License attribution obligations (verbatim gist):** +1. Display *"Built with PriorLabs-TabPFN"* on related website / UI / docs / product pages. +2. If you use the weights/outputs to train/fine-tune/**distill** a model that you **distribute or make available**, prefix that model's name with *"TabPFN"*. +3. **Internal benchmarking/testing with no external communication is exempt** from attribution and does not count as distribution. + +Otherwise it *is* Apache-2.0 — **commercial use, redistribution, and derivatives are all permitted.** So Track B ships **TabPFN-v2 weights ported into candle**, plus TabFM's Apache-2.0 *architecture* ideas layered on top. Training our own from scratch reverts to optional. → see §3 (tracks) and §6. + +### Limitations (stated + implied) + +- Generalization depends on synthetic-data quality/diversity. +- Ensemble gains cost extra compute (SVD features, NNLS optimization). +- Primarily numeric/categorical structured data; mixed modalities not discussed. +- (Implied, from ICL) inference cost/memory scales with the number of in-context + training rows — a practical context ceiling. + +--- + +## 2. Why this fits RuVector specifically + +- **Direct precedent.** `crates/ruvector-timesfm` is a parity-validated candle + port of TimesFM (Google's *time-series* foundation model), wired inward into the + DB's own machinery: `sweep::EarlyStopper` (kill doomed tuning runs, ADR-191), + `rebuild` (forecast recall-drift → time HNSW rebuilds), `anomaly` (telemetry). + **TabFM is the tabular sibling of TimesFM** — same family, same playbook. +- **Greenfield.** No TabPFN/TabFM/tabular-ICL work exists anywhere in the repo. +- **Ingredients present.** candle (6 crates), a modular attention SDK + (`ruvector-attention`: graph/sparse/MoE/training), quantization (RaBitQ/PQ), + WASM builds, and `ruvector-postgres` (143 SQL functions) as an `AI.PREDICT` home. + +--- + +## 3. Ranked opportunities (by impact) + +Legend — **Effort**: S/M/L. **Dep**: prerequisite. Ordered by impact ÷ effort. + +> **Post-gate reframing (see Licensing above):** ①–④ against the *published* weights +> are **research-only (Track A)** — they cannot ship. To ship any of §2/§4/§3 as +> product, ⑥ (own weights on the Apache-2.0 architecture) becomes the **critical +> path (Track B)**, not a deferral. The rankings below are impact-if-owned; the +> gating reality is that owning requires ⑥. + +### ① Core inference port — `crates/ruvector-tabfm` · Effort M · Dep: none +The foundation; everything else hangs off it. Mirror the timesfm crate shape: +`predictor.rs` (load-once → `predict`), `bin/predict.rs` (JSON CLI = `tab_predict` +MCP tool), `candle`-gated numeric path, CPU/CUDA/Metal select, WASM build. +**Inference-only** against published weights — do *not* train. +*Risk:* weight license/availability; candle coverage for row-column attention +(may need a custom kernel in `ruvector-attention`). + +### ② Row-compression tabular-row embedder · Effort S (given ①) · Dep: ① +Expose stage-2 as `encode_row(row) -> Vec`. A heterogeneous SQL row → one +indexable dense vector, straight into the existing HNSW/DiskANN index. New +retrieval modality (structured-record search) for near-zero marginal cost. +**Highest DB synergy per unit effort.** + +### ③ Zero-shot index/SONA autotuner (point TabFM inward) · Effort M · Dep: ① +`(workload features → best config)` is a tabular prediction. Zero-shot warm-start +for HNSW `M`/`efConstruction`, IVF `nprobe`, quantization level — the analog of +`timesfm::sweep`. Compounds SONA (ADR-271) + the IVF/PQ line (ADR-205/206). +*Risk:* needs tuning-history logging to be useful → lands after ①/②. + +### ④ Postgres `ruvector_tab_predict()` — local `AI.PREDICT` · Effort S · Dep: ① +`SELECT ruvector_tab_predict('churn', ROW(...))`. Thin SQL wrapper over ①. +Headline feature; no pgvector competitor has it. *Risk:* document a row cap + +subsampling for large in-context tables (ICL context ceiling). + +### ⑤ Structured/tabular reranker (deferred) · Effort M · Dep: ① +Row-column attention over `(query, candidate-metadata)` beside +`ruvector-gnn-rerank`; reasons over structured columns, not just vector distance. +Complementary to the GNN but adds an ICL pass to the per-query hot path — +revisit after ①–④ once latency is measured. + +### ⑥ Synthetic SCM data engine + train-our-own — **now optional** (was critical path) · Effort L · Dep: architecture +With TabPFN-v2 shippable (Apache-2.0+attr), we no longer *need* to train our own to +have a commercial product. This reverts to an optional research bet — pursue only +if we later need to beat TabPFN-v2's accuracy or shed the attribution clause. The +SCM generator (seeded from `ruvector-graph-transformer` causal/physics modules) +remains the tool if we do. + +**Recommended path (post-gate):** +- **Track A (research, cheap):** benchmark harness over published **TabFM + newer + TabPFN (non-commercial)** weights, internal-only, to measure the accuracy ceiling + we forgo by shipping v2. De-risks the bet; license-clean (internal benchmarking + exempt). +- **Track B (product, the real build):** port **TabPFN-v2** (`clf`+`reg`) into + `ruvector-tabfm` timesfm-style; satisfy attribution; layer TabFM's Apache-2.0 + row-compression/scale ideas → then ②/④/③ on the shipped weights. Defer ⑤; ⑥ only + if Track A shows v2 leaves too much on the table. + +--- + +## 4. Reimplementation map (crate-by-crate) + +| TabFM component | Where it lands in RuVector | +|---|---| +| Load weights once, single-pass `predict` | New `ruvector-tabfm/src/predictor.rs`, mirroring `ruvector-timesfm/src/forecaster.rs` | +| Alternating row↔column attention | candle blocks in-crate; custom fused kernel (if needed) → `ruvector-attention` (sparse/graph SDK) | +| Row compression → dense vector | `ruvector-tabfm::encode_row`; output indexes via existing HNSW/DiskANN | +| ICL transformer over row-vectors | candle transformer in-crate | +| Regression point + quantile output | reuse quantile-band idiom from `ruvector-timesfm::forecast_types` (p10..p90) | +| Classification calibration (Platt) | small module in-crate; optional, matches TabFM-Ensemble | +| JSON-in/JSON-out inference | `ruvector-tabfm/src/bin/predict.rs` = `tab_predict` MCP tool (parity with `time_series_forecast`) | +| `AI.PREDICT` surface | `ruvector_tab_predict()` SQL fn in `ruvector-postgres` | +| Feature gating | `candle` (+ `cuda`/`metal`) feature, identical to timesfm — stock build stays light | +| WASM | `ruvector-tabfm-wasm` (follow existing `*-wasm` crate pattern) | + +--- + +## 5. Open questions / gates before coding + +1. **Weights.** ✅ **RESOLVED (2026-07-01): NO.** Published weights are + **non-commercial + non-redistributable**, derivatives inherit the terms (see + Licensing §1). TabFM naive port cannot ship. ✅ **Sub-question also RESOLVED: + TabPFN-v2 (clf+reg) is Apache-2.0+attribution → shippable Track-B base**; TabPFN + 2.5/2.6/3 are non-commercial like TabFM. → Track A (research) / Track B (port + TabPFN-v2) split. +2. **Reference parity.** What reference implementation do we validate the candle + port against, and what's the parity tolerance? (TimesFM set this precedent.) +3. **Attention kernel.** Does candle cover the alternating row-column attention, or + do we add a kernel to `ruvector-attention`? +4. **Context ceiling.** What in-context row cap + subsampling policy for ④'s + user-facing tables? (③'s tuning tables are small — safe.) +5. **Tuning-history logging.** Does the `(workload → best config)` table exist yet, + or must ③ ship a logger first? + +--- + +## 5b. Track A empirical results (2026-07-01) — RUN, license-clean + +Ran the shippable **TabPFN-v2 (Apache-2.0)** zero-shot vs **tuned** tree baselines +(nested 5-fold CV; XGBoost + HistGBM each get a 12–15-iter `RandomizedSearchCV` on +every train fold; TabPFN gets **no** tuning). Metric = ROC AUC (binary: `roc_auc`; +multiclass `vehicle`: macro `roc_auc_ovr`). 6 standard OpenML datasets. CPU only. +Script/artifacts: `scratchpad/track_a_benchmark.py`, `track_a_results.json`. + +| Dataset | n×feat | **TabPFN-v2** (ship, 0-tune) | XGB-tuned | HistGBM-tuned | RF | LogReg | +|---|---|---|---|---|---|---| +| credit-g | 1000×20 | 0.7863 | **0.7956** | 0.7689 | 0.7946 | 0.7862 | +| diabetes | 768×8 | **0.8422** | 0.8320 | 0.8171 | 0.8272 | 0.8304 | +| blood-transfusion | 748×4 | **0.7613** | 0.7450 | 0.7270 | 0.6848 | 0.7529 | +| breast-w | 699×9 | **0.9942** | 0.9921 | 0.9925 | 0.9929 | 0.9942 | +| qsar-biodeg | 1055×41 | **0.9435** | 0.9349 | 0.9328 | 0.9359 | 0.9276 | +| vehicle (4-class) | 846×18 | **0.9677** | 0.9342 | 0.9324 | 0.9309 | 0.9440 | +| **Mean AUC** | | **0.8825** | 0.8723 | 0.8618 | 0.8611 | 0.8725 | +| **Mean rank** (1=best) | | **1.33** | 2.83 | 4.33 | 3.50 | 3.00 | +| **# top-or-tied /6** | | **5** | 1 | 0 | 0 | 0 | + +**Read:** +- TabPFN-v2 is **top-or-tied on 5/6**, mean rank **1.33** vs 2.83 for tuned XGBoost — + and it did it **zero-shot** (no tuning) in ~1–3 s/fold on CPU, while the trees ran a + full hyperparameter search. +- Per-dataset binary margins are **small** (+0.002 to +0.016 AUC, mostly within the ± + bands) — not individually significant. The one **large** win is multiclass + `vehicle`: **+0.034 AUC / +8.5 pts accuracy**. +- Only loss: `credit-g` (−0.009 vs XGB), within noise. +- **License empirically corroborated:** v2 downloaded frictionlessly; loading + v2.5/v3 raised `TabPFNLicenseError: requires a one-time license acceptance` — i.e. + the newer weights are gated exactly as the license text says. + +**Q2 ceiling (published, NOT reproduced here — the newer weights are gated):** on the +full TabArena suite the non-commercial models (TabPFN-2.5 per arXiv 2511.08667; TabFM +per its blog) rank **above** TabPFN-v2 and additionally **raise the size ceiling** +(more rows/features). So shipping v2 forgoes *some* aggregate accuracy and headroom — +but this run shows v2 **already clears the "competitive with tuned trees" bar**, which +is the bar that matters for the go/no-go. + +**Verdict: GO (Track B).** The Apache-2.0 model we can actually ship is good enough to +be worth owning. Proceed to port TabPFN-v2 into `ruvector-tabfm` (timesfm-style); +treat ⑥ (own weights) as optional headroom, not a prerequisite. + +## 6. Next actions + +- [x] Resolve gate #1 (weight license) — **DONE: non-commercial, non-redistributable → naive port NO-GO.** +- [ ] **Track A / go-no-go:** stand up a *research-only* harness (published weights, internal benchmarking) to measure whether ② (row-embedding recall) and ③ (TabFM-predicted config vs. SONA) clear a bar worth owning. **Do not scaffold a product crate yet.** +- [x] Gate-check TabPFN licensing — **DONE: TabPFN-v2 (clf+reg) Apache-2.0+attribution → shippable; 2.5/2.6/3 non-commercial.** +- [ ] **Track B (product):** port **TabPFN-v2** (`clf`+`reg`) into `crates/ruvector-tabfm` (or `ruvector-tabpfn`) timesfm-style — `predict` + `tab_predict` MCP + WASM; satisfy attribution ("Built with PriorLabs-TabPFN"); prototype/layer TabFM's Apache-2.0 row-compression ideas (in-crate vs. `ruvector-attention` kernel). +- [ ] Then ②/④ in parallel; add tuning-history logging → ③. +- [ ] ⑥ (own weights on synthetic SCM) only if Track A shows TabPFN-v2 leaves too much accuracy on the table. diff --git a/docs/research/tabfm/track_a_benchmark.py b/docs/research/tabfm/track_a_benchmark.py new file mode 100644 index 0000000000..5edb6351b4 --- /dev/null +++ b/docs/research/tabfm/track_a_benchmark.py @@ -0,0 +1,162 @@ +""" +Track A — TabFM/TabPFN research benchmark for RuVector (ADR-272). + +Question 1 (primary): Is the SHIPPABLE model (TabPFN-v2, Apache-2.0 weights) +competitive zero-shot with HEAVILY-TUNED GBDTs on standard tabular data? + +Question 2 (ceiling): How much accuracy do we forgo vs the newer, NON-COMMERCIAL +TabPFN weights (v2.5/v3)? Try to load them; if HF-gated, record and defer to +published numbers. + +Protocol: nested CV. Tree baselines get RandomizedSearchCV on each training fold +(fair "tuned" comparison). TabPFN gets NO tuning (its value proposition). +Metric: ROC AUC (binary: roc_auc; multiclass: macro roc_auc_ovr) + accuracy. +License note: internal benchmarking is explicitly exempt under both the TabFM and +TabPFN non-commercial licenses, so running gated weights here is permitted. +""" +import time, json, warnings, sys +warnings.filterwarnings("ignore") +import numpy as np +from sklearn.model_selection import StratifiedKFold, RandomizedSearchCV +from sklearn.datasets import fetch_openml +from sklearn.preprocessing import OrdinalEncoder, StandardScaler +from sklearn.pipeline import Pipeline +from sklearn.compose import ColumnTransformer +from sklearn.impute import SimpleImputer +from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier +from sklearn.linear_model import LogisticRegression +from sklearn.metrics import roc_auc_score, accuracy_score +import xgboost as xgb +from tabpfn import TabPFNClassifier + +RNG = 42 +N_OUTER = 5 +DATASETS = [ # (openml name/id, label) — small, standard, CPU-friendly + ("credit-g", 31), ("diabetes", 37), ("blood-transfusion-service-center", 1464), + ("breast-w", 15), ("qsar-biodeg", 1494), ("vehicle", 54), +] + +def load(did): + d = fetch_openml(data_id=did, as_frame=True, parser="auto") + X, y = d.data.copy(), d.target + y = y.astype(str) + classes = sorted(y.unique()) + y = y.map({c: i for i, c in enumerate(classes)}).to_numpy() + cat = [c for c in X.columns if str(X[c].dtype) in ("category", "object")] + num = [c for c in X.columns if c not in cat] + return X, y, num, cat, len(classes) + +def auc(y_true, proba, n_cls): + if n_cls == 2: + return roc_auc_score(y_true, proba[:, 1]) + return roc_auc_score(y_true, proba, multi_class="ovr", average="macro") + +def tree_prep(num, cat): # ordinal-encode categoricals, impute — for XGB/HistGBM/RF + return ColumnTransformer([ + ("num", SimpleImputer(strategy="median"), num), + ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")), + ("ord", OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1))]), cat), + ]) + +def lin_prep(num, cat): # one-hot + scale — for LogReg + from sklearn.preprocessing import OneHotEncoder + return ColumnTransformer([ + ("num", Pipeline([("imp", SimpleImputer(strategy="median")), ("sc", StandardScaler())]), num), + ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")), + ("oh", OneHotEncoder(handle_unknown="ignore"))]), cat), + ]) + +def tabpfn_prep(X, num, cat): # numeric matrix; ordinal-encode categoricals + Xn = X.copy() + for c in cat: + Xn[c] = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1 + ).fit_transform(Xn[[c]].astype(str)) + return Xn.apply(lambda s: s.fillna(s.median() if s.dtype != object else 0)).to_numpy(dtype=float) + +def tuned_xgb(n_cls): + est = xgb.XGBClassifier(tree_method="hist", eval_metric="logloss", + objective="multi:softprob" if n_cls > 2 else "binary:logistic", + n_jobs=4, verbosity=0, random_state=RNG) + grid = {"n_estimators": [100, 300, 600], "max_depth": [3, 4, 6, 8], + "learning_rate": [0.02, 0.05, 0.1, 0.2], "subsample": [0.7, 0.85, 1.0], + "colsample_bytree": [0.6, 0.8, 1.0], "min_child_weight": [1, 3, 5]} + return RandomizedSearchCV(est, grid, n_iter=15, cv=3, scoring="roc_auc_ovr", + random_state=RNG, n_jobs=4) + +def tuned_hist(): + est = HistGradientBoostingClassifier(random_state=RNG) + grid = {"max_iter": [200, 400], "learning_rate": [0.03, 0.06, 0.1, 0.2], + "max_depth": [None, 3, 6, 10], "l2_regularization": [0.0, 0.1, 1.0], + "max_leaf_nodes": [15, 31, 63]} + return RandomizedSearchCV(est, grid, n_iter=12, cv=3, scoring="roc_auc_ovr", + random_state=RNG, n_jobs=4) + +# try to load a ceiling (non-commercial) model once +def load_ceiling(): + for mp, tag in [("tabpfn-v2.5-classifier.ckpt", "v2.5"), (None, "v3-auto")]: + try: + c = TabPFNClassifier(device="cpu", **({"model_path": mp} if mp else {})) + from sklearn.datasets import load_breast_cancer + Xb, yb = load_breast_cancer(return_X_y=True) + c.fit(Xb[:100], yb[:100]); c.predict_proba(Xb[:3]) + return c, tag + except Exception as e: + print(f" [ceiling {tag}] unavailable: {type(e).__name__}: {str(e)[:90]}") + return None, None + +def main(): + print("Loading ceiling (non-commercial) model for Q2 ...") + ceiling_clf, ceiling_tag = load_ceiling() + results = {} + for name, did in DATASETS: + print(f"\n=== {name} (openml {did}) ===") + X, y, num, cat, n_cls = load(did) + print(f" n={len(y)} feat={X.shape[1]} ({len(num)} num/{len(cat)} cat) classes={n_cls}") + Xtab = tabpfn_prep(X, num, cat) + skf = StratifiedKFold(n_splits=N_OUTER, shuffle=True, random_state=RNG) + per = {m: {"auc": [], "acc": [], "t": []} for m in + ["TabPFN-v2(ship)", "TabPFN-"+(ceiling_tag or "ceiling"), "XGB-tuned", "HistGBM-tuned", "RandomForest", "LogReg"]} + for tr, te in skf.split(X, y): + ytr, yte = y[tr], y[te] + # --- TabPFN v2 (shippable, zero-shot) --- + t = time.time() + m = TabPFNClassifier(device="cpu", model_path="tabpfn-v2-classifier.ckpt") + m.fit(Xtab[tr], ytr); p = m.predict_proba(Xtab[te]) + per["TabPFN-v2(ship)"]["auc"].append(auc(yte, p, n_cls)) + per["TabPFN-v2(ship)"]["acc"].append(accuracy_score(yte, p.argmax(1))) + per["TabPFN-v2(ship)"]["t"].append(time.time() - t) + # --- ceiling (non-commercial), if available --- + ck = "TabPFN-"+(ceiling_tag or "ceiling") + if ceiling_clf is not None: + t = time.time() + ceiling_clf.fit(Xtab[tr], ytr); p = ceiling_clf.predict_proba(Xtab[te]) + per[ck]["auc"].append(auc(yte, p, n_cls)); per[ck]["acc"].append(accuracy_score(yte, p.argmax(1))) + per[ck]["t"].append(time.time() - t) + # --- tuned tree baselines (nested search on train fold) --- + for mname, mk, prep in [("XGB-tuned", tuned_xgb(n_cls), tree_prep(num, cat)), + ("HistGBM-tuned", tuned_hist(), tree_prep(num, cat))]: + t = time.time() + Xt = prep.fit_transform(X.iloc[tr]); Xv = prep.transform(X.iloc[te]) + mk.fit(Xt, ytr); p = mk.predict_proba(Xv) + per[mname]["auc"].append(auc(yte, p, n_cls)); per[mname]["acc"].append(accuracy_score(yte, p.argmax(1))) + per[mname]["t"].append(time.time() - t) + # --- untuned references --- + prep = tree_prep(num, cat) + Xt = prep.fit_transform(X.iloc[tr]); Xv = prep.transform(X.iloc[te]) + rf = RandomForestClassifier(n_estimators=300, n_jobs=4, random_state=RNG).fit(Xt, ytr) + p = rf.predict_proba(Xv); per["RandomForest"]["auc"].append(auc(yte, p, n_cls)); per["RandomForest"]["acc"].append(accuracy_score(yte, p.argmax(1))); per["RandomForest"]["t"].append(0) + lp = lin_prep(num, cat); Xt = lp.fit_transform(X.iloc[tr]); Xv = lp.transform(X.iloc[te]) + lr = LogisticRegression(max_iter=2000).fit(Xt, ytr) + p = lr.predict_proba(Xv); per["LogReg"]["auc"].append(auc(yte, p, n_cls)); per["LogReg"]["acc"].append(accuracy_score(yte, p.argmax(1))); per["LogReg"]["t"].append(0) + results[name] = {m: {"auc_mean": float(np.mean(v["auc"])) if v["auc"] else None, + "auc_std": float(np.std(v["auc"])) if v["auc"] else None, + "acc_mean": float(np.mean(v["acc"])) if v["acc"] else None, + "t_mean": float(np.mean(v["t"])) if v["t"] else None} for m, v in per.items()} + for m, r in results[name].items(): + if r["auc_mean"] is not None: + print(f" {m:22s} AUC {r['auc_mean']:.4f}±{r['auc_std']:.3f} acc {r['acc_mean']:.4f} {r['t_mean']:.1f}s") + json.dump(results, open("track_a_results.json", "w"), indent=2) + print("\nSaved track_a_results.json") + +if __name__ == "__main__": + main() diff --git a/docs/research/tabfm/track_a_results.json b/docs/research/tabfm/track_a_results.json new file mode 100644 index 0000000000..4d09a51974 --- /dev/null +++ b/docs/research/tabfm/track_a_results.json @@ -0,0 +1,230 @@ +{ + "credit-g": { + "TabPFN-v2(ship)": { + "auc_mean": 0.7862857142857143, + "auc_std": 0.01616794650940766, + "acc_mean": 0.75, + "t_mean": 6.368113851547241 + }, + "TabPFN-ceiling": { + "auc_mean": null, + "auc_std": null, + "acc_mean": null, + "t_mean": null + }, + "XGB-tuned": { + "auc_mean": 0.7956428571428572, + "auc_std": 0.02063677342109377, + "acc_mean": 0.766, + "t_mean": 5.840635585784912 + }, + "HistGBM-tuned": { + "auc_mean": 0.7689047619047619, + "auc_std": 0.01121056959963676, + "acc_mean": 0.7550000000000001, + "t_mean": 23.20904674530029 + }, + "RandomForest": { + "auc_mean": 0.7946071428571428, + "auc_std": 0.02232400302614011, + "acc_mean": 0.7610000000000001, + "t_mean": 0.0 + }, + "LogReg": { + "auc_mean": 0.7861666666666667, + "auc_std": 0.01977085282215196, + "acc_mean": 0.748, + "t_mean": 0.0 + } + }, + "diabetes": { + "TabPFN-v2(ship)": { + "auc_mean": 0.8421928721174006, + "auc_std": 0.020865040834372618, + "acc_mean": 0.7655631949749597, + "t_mean": 3.3816460609436034 + }, + "TabPFN-ceiling": { + "auc_mean": null, + "auc_std": null, + "acc_mean": null, + "t_mean": null + }, + "XGB-tuned": { + "auc_mean": 0.8319713487071978, + "auc_std": 0.02773092193260367, + "acc_mean": 0.773389355742297, + "t_mean": 5.097432851791382 + }, + "HistGBM-tuned": { + "auc_mean": 0.8170957372466805, + "auc_std": 0.028992325953223343, + "acc_mean": 0.753823953823954, + "t_mean": 20.00244426727295 + }, + "RandomForest": { + "auc_mean": 0.827211390635919, + "auc_std": 0.01962968543002592, + "acc_mean": 0.7655886597063069, + "t_mean": 0.0 + }, + "LogReg": { + "auc_mean": 0.8303878406708595, + "auc_std": 0.02228758281308102, + "acc_mean": 0.7746965452847806, + "t_mean": 0.0 + } + }, + "blood-transfusion-service-center": { + "TabPFN-v2(ship)": { + "auc_mean": 0.7612886382623224, + "auc_std": 0.028668189135599813, + "acc_mean": 0.7901118568232662, + "t_mean": 2.367714595794678 + }, + "TabPFN-ceiling": { + "auc_mean": null, + "auc_std": null, + "acc_mean": null, + "t_mean": null + }, + "XGB-tuned": { + "auc_mean": 0.7449617098301309, + "auc_std": 0.029529667218531484, + "acc_mean": 0.7740671140939597, + "t_mean": 5.5280537605285645 + }, + "HistGBM-tuned": { + "auc_mean": 0.7270488721804511, + "auc_std": 0.029615827820882228, + "acc_mean": 0.7767427293064877, + "t_mean": 18.80963635444641 + }, + "RandomForest": { + "auc_mean": 0.6848377889167363, + "auc_std": 0.03570374207280446, + "acc_mean": 0.7539686800894855, + "t_mean": 0.0 + }, + "LogReg": { + "auc_mean": 0.7529003063213588, + "auc_std": 0.029103589688212236, + "acc_mean": 0.7727069351230426, + "t_mean": 0.0 + } + }, + "breast-w": { + "TabPFN-v2(ship)": { + "auc_mean": 0.994202989963305, + "auc_std": 0.0028427986230985898, + "acc_mean": 0.968530318602261, + "t_mean": 3.564052629470825 + }, + "TabPFN-ceiling": { + "auc_mean": null, + "auc_std": null, + "acc_mean": null, + "t_mean": null + }, + "XGB-tuned": { + "auc_mean": 0.9921115005964165, + "auc_std": 0.004484393878015614, + "acc_mean": 0.9685200411099691, + "t_mean": 2.8613541603088377 + }, + "HistGBM-tuned": { + "auc_mean": 0.9925217907277579, + "auc_std": 0.003348442505065294, + "acc_mean": 0.9613669064748201, + "t_mean": 13.400657558441162 + }, + "RandomForest": { + "auc_mean": 0.9928603422974541, + "auc_std": 0.003797209937763077, + "acc_mean": 0.9656628982528263, + "t_mean": 0.0 + }, + "LogReg": { + "auc_mean": 0.994152652102408, + "auc_std": 0.003264273481045298, + "acc_mean": 0.9656628982528263, + "t_mean": 0.0 + } + }, + "qsar-biodeg": { + "TabPFN-v2(ship)": { + "auc_mean": 0.94351321357401, + "auc_std": 0.011023981727609914, + "acc_mean": 0.881516587677725, + "t_mean": 15.588684368133546 + }, + "TabPFN-ceiling": { + "auc_mean": null, + "auc_std": null, + "acc_mean": null, + "t_mean": null + }, + "XGB-tuned": { + "auc_mean": 0.9349176634547494, + "auc_std": 0.011069609548969035, + "acc_mean": 0.875829383886256, + "t_mean": 4.892862606048584 + }, + "HistGBM-tuned": { + "auc_mean": 0.9328311234312349, + "auc_std": 0.01259142939898824, + "acc_mean": 0.8777251184834123, + "t_mean": 23.324076318740843 + }, + "RandomForest": { + "auc_mean": 0.9358966870768979, + "auc_std": 0.005800180965197466, + "acc_mean": 0.8672985781990521, + "t_mean": 0.0 + }, + "LogReg": { + "auc_mean": 0.9276146728492458, + "auc_std": 0.008287176428106533, + "acc_mean": 0.8701421800947868, + "t_mean": 0.0 + } + }, + "vehicle": { + "TabPFN-v2(ship)": { + "auc_mean": 0.9677119700265553, + "auc_std": 0.0038010548118525187, + "acc_mean": 0.8451374869474417, + "t_mean": 7.253725194931031 + }, + "TabPFN-ceiling": { + "auc_mean": null, + "auc_std": null, + "acc_mean": null, + "t_mean": null + }, + "XGB-tuned": { + "auc_mean": 0.9341836030085432, + "auc_std": 0.005624096480557556, + "acc_mean": 0.7600556909154192, + "t_mean": 17.91141095161438 + }, + "HistGBM-tuned": { + "auc_mean": 0.9323841841450486, + "auc_std": 0.005307110294248247, + "acc_mean": 0.7659310824921686, + "t_mean": 66.08864455223083 + }, + "RandomForest": { + "auc_mean": 0.9309225393340197, + "auc_std": 0.006291466028438093, + "acc_mean": 0.7576888270100939, + "t_mean": 0.0 + }, + "LogReg": { + "auc_mean": 0.9439931314078269, + "auc_std": 0.004988866543059998, + "acc_mean": 0.7884441350504698, + "t_mean": 0.0 + } + } +} \ No newline at end of file