diff --git a/CLAUDE.md b/CLAUDE.md index 23d6a57..5f5565f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -117,24 +117,25 @@ cd web && npm run test:watch # Watch mode - Astro + Vue 3 static site for browser-only tools (no backend required) - npm package name: `babel-explorer` (directory stays `web/`) - Deployed to GitHub Pages - - Currently hosts: NodeNorm lookup (unified instance selection via checkboxes, expandable result table, shareable URLs) - - Calls NodeNorm API directly from the browser via `fetch()` (CORS-enabled) + - Currently hosts: + - **NodeNorm lookup** — unified instance selection, expandable result table, shareable URLs + - **NameRes lookup** — batch name → CURIE resolution with expected-CURIE validation via `[[CURIE]]` annotations + - **Autocomplete playground** — live-as-you-type NameRes with preset dropdown (Disease/Gene/SmallMolecule/Custom), debounce+abort, latency badges, expected-CURIE rank check, and side-by-side environment comparison + - Calls NodeNorm and NameRes APIs directly from the browser via `fetch()` (CORS-enabled) - Uses Bootstrap 5 (CDN) with the same dark-navbar styling as the Python frontend - CURIE link-outs via [biolink-model prefix map](https://github.com/biolink/biolink-model) (v4.3.7, fetched at runtime) - - Tested with Vitest + @vue/test-utils + happy-dom (99 tests); see `web/tests/README.md` + - Tested with Vitest + @vue/test-utils + happy-dom (276 tests); see `web/tests/README.md` - See `web/README.md` for development instructions and `web/FUTURE.md` for deferred features 7. **Shared Configuration** (`config/`): - - `config/translator-endpoints.json` — single source of truth for NodeNorm and NameRes deployment URLs across all environments (dev, exp, ci, test, prod) - - Consumed by Python CLI/frontend (`nodenorm.py`) and Astro frontend (`NodeNormApp.vue`) + - `config/translator-endpoints.json` — single source of truth for NodeNorm and NameRes deployment URLs across all environments + - Consumed by Python CLI/frontend (`nodenorm.py`) and Astro frontend (`NodeNormApp.vue`, `NameResApp.vue`, `AutocompleteApp.vue`) + - **NodeNorm envs**: `dev`, `exp`, `ci` (ElasticSearch-backed CI), `redis_ci` (Redis CI — kept for comparison, to be retired), `test`, `prod` + - **NameRes envs**: `dev`, `exp`, `ci` (Solr-backed CI — the stable default), `es_ci` (ElasticSearch CI — experimental), `test`, `prod` + - Note: NodeNorm and NameRes use the same `ci` key but point to different backends. NodeNorm CI is ES-based; NameRes CI is still Solr-based. The NameRes `es_ci` key is a separate experimental instance. ### Data Flow -1. User provides CURIEs via CLI -2. BabelDownloader ensures required Parquet files are downloaded -3. BabelXRefs queries files using DuckDB -4. If `--labels` or `--recurse` flags are set, NodeNorm is queried for additional metadata -5. Results are printed to stdout 1. User provides CURIEs via CLI, Python web UI, or Astro web UI 2. For DB-dependent tools (XRefs, IDs): BabelDownloader ensures required Parquet files are downloaded, BabelXRefs queries files using DuckDB 3. For API-only tools (NodeNorm): the Astro frontend calls the NodeNorm API directly from the browser; the Python frontend proxies through the server @@ -148,25 +149,32 @@ web/src/ layouts/BaseLayout.astro # Bootstrap 5 CDN + dark navbar components/ Navbar.astro # Dark navbar matching Python frontend - nodenorm/ - NodeNormApp.vue # Root Vue island (client:only="vue") - NodeNormForm.vue # Form: textarea, checkboxes + custom URL, API toggles - ComparisonView.vue # Results table (rows=CURIEs, cols=instances) with expandable rows - CurieDetailPanel.vue # Expandable detail body: description, types, IC, equiv IDs - CurieResultCard.vue # Accordion card wrapping CurieDetailPanel (single-instance use) - ResultsSummary.vue # Unified stat tiles: normalized count, disagreements, types - EquivalentIdTable.vue # Striped table with togglable columns - ColumnVisibility.vue # Page-wide column show/hide controls + nodenorm/ # NodeNorm tool components (App, Form, ComparisonView, CurieDetailPanel, CurieResultCard, ResultsSummary, EquivalentIdTable, ColumnVisibility) + nameres/ # NameRes tool components (App, Form, ComparisonView, DetailPanel, ResultsSummary) + autocomplete/ # Autocomplete tool components (App, Form, Results, ComparisonView, HighlightedFragment, ExpectedCuriePanel, LatencyBadge) shared/ + InstanceSelector.vue # Reusable env checkboxes + custom URL + localStorage prefs CurieLink.vue # CURIE → external URL link using biolink prefix map + BiolinkTypeLink.vue # Biolink type badge → link to biolink.github.io/biolink-model/{Type}; handles both prefixed ("biolink:Gene") and unprefixed ("Gene") types; used by NodeNorm, NameRes, and Autocomplete lib/ nodenorm-api.ts # fetch() wrapper for NodeNorm get_normalized_nodes (AbortSignal support) + nameres-api.ts # fetch() wrapper for NameRes /lookup; parseSearchTerms; validateExpectedCuries + nameres-types.ts # NameResResult, NameResApiOptions, DEFAULT_NAMERES_OPTIONS + nameres-url-state.ts # Encode/decode NameRes query state + autocomplete-url-state.ts # Encode/decode Autocomplete query state (q, preset, target, expected, debounce, highlight) + autocomplete-presets.ts # Disease/Gene/SmallMolecule/Custom preset definitions + detectPreset + autocomplete-diff.ts # computeInstanceDiffs (presence/rank/label/types mismatch) + classifyExpectedCurie + debounce.ts # debounce(fn, ms) with .cancel(); synchronous path when ms===0 + highlight-sanitize.ts # Whitelist sanitizer for Solr highlighting ( only) + instance-prefs.ts # sessionPrefs + localStorage helpers + sortInstances (ENV_ORDER canonical sort) — used by InstanceSelector and all App components curie-links.ts # Biolink prefix map loader + URL builder - url-state.ts # Encode/decode query state in URL params (readQueryState, buildQueryUrl) - types.ts # TypeScript interfaces + DEFAULT_API_OPTIONS + url-state.ts # Encode/decode NodeNorm query state + types.ts # NodeNorm TypeScript interfaces + DEFAULT_API_OPTIONS pages/ index.astro # Landing page with tool cards nodenorm.astro # NodeNorm tool page + nameres.astro # NameRes tool page + autocomplete.astro # Autocomplete playground page ``` Key patterns: @@ -195,6 +203,8 @@ The NodeNorm and Test Concordance pages include a dropdown to select from predef - **DuckDB for querying**: In-memory SQL queries against Parquet files for fast lookups - **Dual frontend**: DB-dependent tools on a server (FastAPI + htmx), API-only tools in the browser (Astro + Vue). Split follows data dependencies. - **Shared config**: `config/translator-endpoints.json` is the single source of truth for deployment URLs, consumed by both frontends and the CLI +- **Canonical instance ordering**: `sortInstances()` in `instance-prefs.ts` sorts by `ENV_ORDER` (Exp → Dev → CI → ES CI → Redis CI → Test → Prod → custom URLs). Called in every App component after setting `queriedInstances` so comparison tables always display in pipeline order. +- **Adding a new named instance**: requires (1) entry in `config/translator-endpoints.json`, (2) add env key to `PRIMARY_ENVS` in `InstanceSelector.vue` if it should appear in the main section rather than "Extended environments", (3) add to `ENV_ORDER` in `instance-prefs.ts` for sort position, (4) add label to `ENV_LABELS` in the relevant App component(s) and in `InstanceSelector.vue`, (5) add a `labelHtml()` case in `InstanceSelector.vue` if the label needs `` rendering (e.g. ES, Redis), (6) add to Python `env_labels` in `nodenorm.py` if it's a NodeNorm instance. ## Testing @@ -231,6 +241,7 @@ Tests live in `tests/` and are split into fast **unit tests** (mocked, no networ - **Data directory**: The `data/` directory is gitignored and contains downloaded Parquet files and generated DuckDB databases - **Babel versions**: The default Babel version is `2025nov19`, but this can be customized via `--local-dir` and `--babel-url` +- **Pre-existing Python test failures**: `tests/test_web.py::test_nodenorm_page` and `tests/test_web.py::test_test_concord_page` fail with `AttributeError: type object 'NodeNorm' has no attribute 'URLs'` — this is a pre-existing issue unrelated to the Astro frontend work. The `NodeNorm.URLs` attribute referenced in the Python web routes no longer matches the actual class interface. ## File Locations diff --git a/config/translator-endpoints.json b/config/translator-endpoints.json index acb974d..4c18629 100644 --- a/config/translator-endpoints.json +++ b/config/translator-endpoints.json @@ -1,15 +1,17 @@ { "nodenorm": { - "dev": "https://nodenormalization-sri.renci.org/", - "exp": "https://nodenormalization-exp.apps.renci.org/", - "ci": "https://nodenorm.ci.transltr.io/", - "test": "https://nodenorm.test.transltr.io/", - "prod": "https://nodenorm.transltr.io/" + "dev": "https://nodenormalization-sri.renci.org/", + "exp": "https://nodenormalization-exp.apps.renci.org/", + "ci": "https://nodenorm-es.ci.transltr.io/", + "redis_ci": "https://nodenorm.ci.transltr.io/", + "test": "https://nodenorm.test.transltr.io/", + "prod": "https://nodenorm.transltr.io/" }, "nameres": { "dev": "https://name-resolution-sri.renci.org/", "exp": "https://name-resolution-exp.apps.renci.org/", "ci": "https://name-lookup.ci.transltr.io/", + "es_ci": "https://namelookup-es.ci.transltr.io/", "test": "https://name-lookup.test.transltr.io/", "prod": "https://name-lookup.transltr.io/" } diff --git a/src/babel_explorer/core/nodenorm.py b/src/babel_explorer/core/nodenorm.py index 0bd6682..dd80431 100644 --- a/src/babel_explorer/core/nodenorm.py +++ b/src/babel_explorer/core/nodenorm.py @@ -16,11 +16,12 @@ def _load_nodenorm_urls() -> dict[str, str]: try: endpoints = json.loads(config_path.read_text()) env_labels = { - "dev": "Dev", - "exp": "Exp", - "ci": "CI", - "test": "Test", - "prod": "Production", + "dev": "Dev", + "exp": "Exp", + "ci": "CI", + "redis_ci": "Redis CI", + "test": "Test", + "prod": "Production", } return { f"NodeNorm {env_labels.get(env, env)}": url diff --git a/web/README.md b/web/README.md index be63d53..3880a78 100644 --- a/web/README.md +++ b/web/README.md @@ -8,7 +8,7 @@ babel-explorer has **two** web frontends: | Frontend | Stack | Deployment | Tools | Directory | |----------|-------|------------|-------|-----------| -| **This one** | Astro + Vue 3 | GitHub Pages | NodeNorm (more planned) | `web/` | +| **This one** | Astro + Vue 3 | GitHub Pages | NodeNorm, NameRes, Autocomplete | `web/` | | **Python frontend** | FastAPI + htmx | Kubernetes | XRefs, IDs, Test Concordance, NodeNorm | `src/babel_explorer/web/` | The split follows data dependencies: tools that only call external APIs (NodeNorm, NameRes) can run entirely in the browser. Tools that query multi-GB Parquet files via DuckDB need a server. @@ -20,7 +20,7 @@ Both frontends share the same Bootstrap 5 dark-navbar styling for visual consist ### NodeNorm Lookup (`/nodenorm`) - **Bulk normalization**: Enter multiple CURIEs, toggle API options (conflation, descriptions, individual types, taxa) -- **Unified instance selection**: Checkboxes for known NodeNorm deployments (Dev, Exp, CI, Test, Prod) plus a custom URL input; any combination of instances can be queried together +- **Unified instance selection**: Checkboxes for known NodeNorm deployments (Dev, Exp, ES CI, Redis CI, Test, Prod) plus a custom URL input; any combination of instances can be queried together - **Comparison table**: Results shown as a table — rows = CURIEs, columns = selected instances; rows highlighted amber when instances disagree on preferred ID - **Expandable row detail**: Click any CURIE row to reveal per-instance panels showing description, biolink types, IC score, and equivalent identifiers (prefix summary + expand/collapse for large cliques) - **Column visibility**: Toggle biolink type, taxa, description columns page-wide @@ -28,6 +28,27 @@ Both frontends share the same Bootstrap 5 dark-navbar styling for visual consist - **Shareable URLs**: Query state encoded in URL params (`?curie=`, `?target=`, non-default options); Share button copies link to clipboard; auto-submits on page load when URL contains CURIEs - **CURIE link-outs**: Identifiers link to external resources via [biolink-model prefix map](https://github.com/biolink/biolink-model) (v4.3.7) +### NameRes Lookup (`/nameres`) + +- **Batch name → CURIE resolution**: Enter multiple search terms (one per line), see ranked results per instance +- **Expected-CURIE validation**: Annotate any line with `[[CURIE]]` to mark expected results; validation reports success/partial/failure with the rank of the best match, configurable via a "fail if in top N" threshold +- **Multi-instance comparison**: Same unified instance selector as NodeNorm +- **API tuning**: `biolink_type`, `only_prefixes`, `exclude_prefixes`, `only_taxa` exposed in the form; limit and autocomplete mode toggles; shareable URL state and JSON export + +### Autocomplete Playground (`/autocomplete`) + +Purpose-built for evaluating NameRes as a real autocomplete (the primary way the Translator UI uses it). + +- **Live-as-you-type**: every keystroke re-queries after a configurable debounce (0/150/300/500 ms); in-flight requests are aborted via `AbortController` so stale responses never render +- **Preset dropdown**: one-click switches between the three Translator UI query shapes — Disease (`DiseaseOrPhenotypicFeature` + `only_prefixes=MONDO|HP`), Gene, Small Molecule — plus Custom. Presets only set `biolink_type` and `only_prefixes`; every other field stays user-editable +- **Advanced options**: debounce, autocomplete flag, highlighting toggle, exclude_prefixes, only_taxa (collapsed by default, auto-opens when any non-default is set) +- **Latency badges**: per-instance response time, with a tooltip noting parallel-contention when comparing multiple environments +- **Match-reason highlighting**: renders NameRes's Solr `highlighting` fragments (``-wrapped matches) behind a whitelist sanitiser — the only place `v-html` is used in the codebase +- **Single- vs multi-instance views**: one instance → rich ranked table with copy-API-URL per row; multiple → side-by-side comparison table with row/cell styling for missing CURIEs, rank drift, label/types mismatches; each column header has an `API↗` link to the raw NameRes JSON for that instance +- **Biolink type display**: the left-hand CURIE column shows the most specific Biolink type as a badge (linked to biolink.github.io/biolink-model); when instances disagree, all distinct types are shown and the type is repeated per-cell for easy comparison +- **Expected-CURIE panel**: paste CURIEs you expect to see; "Check" button fires a `limit=100` parallel lookup per instance and shows whether each expected CURIE appears in the top-N (green), top-100 (amber) or is missing (red). Round-trips through URL state for shareable review links +- **Shareable state**: `q`, `preset`, repeated `target`, repeated `expected`, per-field option overrides, non-default `debounce` and `highlight` all encoded in the URL + ## Development ```bash @@ -41,7 +62,7 @@ This starts a local dev server at `http://localhost:4321/babel-explorer/`. ## Testing ```bash -npm test # Run all 99 Vitest unit + component tests +npm test # Run all Vitest unit + component tests (~276 tests across lib and components) npm run test:watch # Watch mode ``` @@ -78,24 +99,31 @@ src/ pages/ index.astro # Landing page with tool cards nodenorm.astro # Hosts NodeNormApp Vue island + nameres.astro # Hosts NameResApp Vue island + autocomplete.astro # Hosts AutocompleteApp Vue island components/ Navbar.astro # Shared navbar (Astro component) - nodenorm/ # NodeNorm Vue components - NodeNormApp.vue # Root island: orchestrates form + results - NodeNormForm.vue # CURIE input, checkbox instance selection, custom URL, API options - ComparisonView.vue # Results table with expandable per-CURIE rows - CurieDetailPanel.vue # Detail body: description, types, IC, equiv IDs table - CurieResultCard.vue # Accordion card wrapping CurieDetailPanel - ResultsSummary.vue # Stat tiles: normalized count, disagreements, type badges - EquivalentIdTable.vue # Equiv ID table with togglable columns - ColumnVisibility.vue # Column show/hide controls + nodenorm/ # NodeNorm Vue components (App, Form, ComparisonView, CurieDetailPanel, CurieResultCard, ResultsSummary, EquivalentIdTable, ColumnVisibility) + nameres/ # NameRes Vue components (App, Form, ComparisonView, DetailPanel, ResultsSummary) + autocomplete/ # Autocomplete Vue components (App, Form, Results, ComparisonView, HighlightedFragment, ExpectedCuriePanel, LatencyBadge) shared/ + InstanceSelector.vue # Env checkboxes + custom URL + localStorage prefs (used by all tools) CurieLink.vue # CURIE → external URL link + BiolinkTypeLink.vue # Biolink type → link to biolink.github.io/biolink-model/{Type}; accepts both "biolink:Gene" and "Gene" lib/ nodenorm-api.ts # NodeNorm API fetch wrapper (supports AbortSignal) + nameres-api.ts # NameRes /lookup wrapper; parseSearchTerms; validateExpectedCuries + nameres-types.ts # NameRes TypeScript interfaces + DEFAULT_NAMERES_OPTIONS + nameres-url-state.ts # NameRes URL-state encode/decode + autocomplete-url-state.ts # Autocomplete URL-state encode/decode + DEFAULT_AUTOCOMPLETE_OPTIONS + parseExpectedCuries + autocomplete-presets.ts # Disease/Gene/SmallMolecule/Custom presets + detectPreset + autocomplete-diff.ts # Cross-instance diff signals + expected-CURIE classification + debounce.ts # debounce(fn, ms) with .cancel() and synchronous 0-ms path + highlight-sanitize.ts # Whitelist sanitizer — allow only / + instance-prefs.ts # sessionPrefs + localStorage helpers + sortInstances (ENV_ORDER canonical sort) curie-links.ts # Biolink prefix map loader - url-state.ts # Encode/decode query state in URL params - types.ts # TypeScript interfaces + url-state.ts # NodeNorm URL-state encode/decode + types.ts # NodeNorm TypeScript interfaces ``` ## Adding a New Tool diff --git a/web/src/components/Navbar.astro b/web/src/components/Navbar.astro index d503039..e898fff 100644 --- a/web/src/components/Navbar.astro +++ b/web/src/components/Navbar.astro @@ -25,6 +25,11 @@ const base = rawBase.endsWith('/') ? rawBase : rawBase + '/'; NameRes +