Skip to content

docs: explain VERIFICATION.md metrics in plain English (closes #35)#130

Closed
DsThakurRawat wants to merge 1 commit into
FastCrest:mainfrom
DsThakurRawat:docs/understanding-verification
Closed

docs: explain VERIFICATION.md metrics in plain English (closes #35)#130
DsThakurRawat wants to merge 1 commit into
FastCrest:mainfrom
DsThakurRawat:docs/understanding-verification

Conversation

@DsThakurRawat
Copy link
Copy Markdown
Collaborator

@DsThakurRawat DsThakurRawat commented May 14, 2026

Description

Adds docs/understanding_verification.md, a plain-English guide that demystifies the auto-generated VERIFICATION.md trust receipt.

Key Additions:

  • Export Metadata Breakdown: Explains opset, denoising steps, chunk size, and model types.
  • File Hashes: Details why SHA256 matters for integrity, reproducibility, and audit trails.
  • Parity Metrics Explained: Clearly defines max_abs_diff, mean_abs_diff, pass/fail thresholds, and random seed reproducibility.
  • Vertical-Specific Tolerances: A table providing acceptable max_abs_diff ranges based on use-case sensitivity:
    • Warehouse arms (< 1e-04)
    • Farm robotics (< 1e-03)
    • Aerial drones (< 1e-04)
    • Retail & Traffic AI (< 1e-03)
  • Remediation Steps: What to do if validation fails (e.g., opset downgrade, precision check).

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Infrastructure / CI update

How Has This Been Tested?

  • pytest tests/ passes locally
  • reflex doctor sanity check
  • Other (please specify): Verified formatting and cross-referenced with codebase/schema.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have kept the PR scoped to a single concern (one concern per PR)

@rylinjames
Copy link
Copy Markdown
Collaborator

Thanks @DsThakurRawat — the teaching structure of this guide (When is it created → Section-by-section → Key metrics → Vertical interpretation → Recovery) is exactly right, and the fixture numbers in your parity example look like real output from a reflex validate run on pi0.5 (the aggregate 2.384e-07 matches the README's pi0.5 first-action 2.38e-07 within 1 ULP — that's a real measurement, not a placeholder). Credit preserved via Co-Authored-By on the commit in the replacement PR.

What changed in the replacement, narrower than I'd initially planned:

  • Refreshed only the Reflex version field (0.2.1 → 0.9.6 — that label was the only thing that didn't line up with current state). All your fixture numbers retained verbatim.
  • Added a cross-reference to the actual parity ledger so readers can sanity-check their own runs against the shipped numbers (SmolVLA 5.96e-07, pi0 2.09e-07, pi0.5 2.38e-07, GR00T 8.34e-07).
  • Added the Total: N files, X size line above the Files table — the renderer (src/reflex/verification_report.py) emits it.
  • Renamed file to verification.md for consistency with sibling docs (eval.md, embodiment_schema.md).
  • Replaced the cross-ref to TROUBLESHOOTING.md (doesn't exist on main yet — still in PR docs: compile TROUBLESHOOTING.md with CUDA and drone errors (closes #63) #129) with a reflex doctor flow.

Closing in favor of the replacement PR. Thanks again — this was the kind of doc that needed grounding more than rewriting.

@rylinjames rylinjames closed this May 15, 2026
rylinjames added a commit that referenced this pull request May 15, 2026
Adds docs/verification.md walking users through the auto-generated
VERIFICATION.md artifact — the trust receipt proving an exported ONNX
model matches its PyTorch source at machine precision.

Structure verified against src/reflex/verification_report.py — every
section + field name in the doc maps to a real line of the renderer.

Adapted from #130:
- Kept the fixture numbers (1.192e-07, 2.384e-07, etc.) — they're
  float32 ULP multiples, characteristic of correctly-exported models,
  and 2.384e-07 lines up with pi0.5's actual first-action max_abs
  (2.38e-07) from the README parity ledger.
- Refreshed Reflex version (0.2.1 → 0.9.6) + added a note that the
  field auto-fills from reflex.__version__.
- Cross-referenced the parity ledger explicitly so readers can sanity-
  check their own runs against the shipped numbers (SmolVLA 5.96e-07,
  pi0 2.09e-07, pi0.5 2.38e-07, GR00T 8.34e-07).
- Renamed file from understanding_verification.md to verification.md
  for consistency with sibling docs (eval.md, embodiment_schema.md).
- Replaced broken TROUBLESHOOTING.md cross-ref with reflex doctor flow.
- Kept the 'Interpreting results by vertical' table — verticals match
  the FastCrest customer research vault's P0 tier.
- Added 'Files' section preamble (Total: N files, X size) that #130
  omitted but the live renderer emits.

Supersedes #130.

Co-authored-by: Divyansh Rawat <186957976+DsThakurRawat@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants