diff --git a/docs/PROVIDER_INTERFACE_EVOLUTION.md b/docs/PROVIDER_INTERFACE_EVOLUTION.md new file mode 100644 index 0000000..a575785 --- /dev/null +++ b/docs/PROVIDER_INTERFACE_EVOLUTION.md @@ -0,0 +1,300 @@ +# Faster ValueSet Expansion via `expandForValueSet` + +## The Problem + +The `CodeSystemProvider` interface (`tx/cs/cs-api.js`) treats each provider as an +**iterator + attribute oracle**: given a filter, produce matching codes one at a +time, then answer per-code questions (`isInactive`, `getStatus`, `designations`, +etc.). The expansion worker assembles the result by driving this loop. + +For SQL-backed providers like RxNorm and LOINC, this creates two performance +problems: + +1. **Per-code overhead.** The worker calls 8–10 methods per code. If contexts aren't + self-sufficient, each call can trigger a SQL query. For RxNorm, expanding 1100 + codes originally cost 3301 SQL queries (1 filter + 1100 × 3 per-code status + lookups). This was fixed independently by making contexts carry all attributes + at creation time — zero interface changes required. + +2. **No paging push-down.** To return page 11 (offset=1000, count=100), the worker + processes all 1100 codes, then discards the first 1000. It can't tell the + provider to skip ahead because paging is a ValueSet-level concern (multiple + includes, excludes, dedup across code systems). A SQL provider could do + `LIMIT 100 OFFSET 1000` and return in ~1ms instead of ~200ms. + +Similarly, excludes and `activeOnly` filtering happen code-by-code in the worker. +A SQL provider could handle all of these in the WHERE clause. + +We briefly considered adding `{ offset, count, activeOnly }` options to the +existing `executeFilters()` method, but this creates awkward partially-resolved +states — the provider handles some post-filters but not others, and the worker +has to know which. Piecemeal push-down doesn't compose well. + +## The Interface Change + +One new optional method on `CodeSystemProvider`: + +```js +async expandForValueSet(spec) { + // Returns an AsyncIterable or null (can't handle → fall back) + return null; +} +``` + +Instead of threading individual parameters through the 10-call-per-code loop, +**give the provider the full hull of includes and excludes that apply to its code +system** and let it handle everything in one shot. + +### Input + +The worker groups compose entries by code system and builds: + +```js +{ + includes: [ // compose.include blocks for this CS + { concepts: [{code, display?}], // explicit code list (may be null) + filters: [{property, op, value}] // property filters (may be empty) + }, ... + ], + excludes: [ // compose.exclude blocks for this CS + { concepts: [{code}], + filters: [{property, op, value}] + }, ... + ], + activeOnly: boolean, // exclude inactive codes + searchText: string | null, // expansion 'filter' parameter + includeDesignations: boolean, // whether designations are needed + properties: string[], // which properties to include + languages: Languages, // requested display languages + + // Paging — non-null only when safe (single code system); must apply if present + offset: number | null, + count: number | null, +} +``` + +### Output + +`AsyncIterable` where each entry is a fully-resolved code: + +```js +{ + code: string, + display: string, + isAbstract: boolean, + isInactive: boolean, + isDeprecated: boolean, + status: string | null, + definition: string | null, + designations: [{language, use, value}], + properties: [{code, valueType, value}], + extensions: [...] | null, +} +``` + +The worker iterates these entries via `includeCode()`, which still handles dedup +across code systems, import filtering, expansion limits, and FHIR object construction. + +### Paging contract + +`offset` and `count` are **non-null only when the worker can verify they're +safe** (currently: single code system in the compose). When provided: + +- The provider **must apply them** (via SQL `LIMIT/OFFSET`). The worker zeros + its own offset after a successful `expandForValueSet` call, so the provider's + SQL is the sole paging authority. +- If the provider can handle filters and excludes but **not** paging, it should + return `null` to fall back to the framework's iterator path, which applies + offset/count during finalization. + +When `null` (multi-system compose, or offset/count not requested): + +- The provider returns all matching codes. The framework accumulates them into + `fullList` and applies paging during finalization. Still faster than baseline + because the provider handles filters, excludes, and activeOnly in SQL — just + without the LIMIT/OFFSET shortcut. + +Each expansion request is fully stateless — there is no cross-request memory of +previous pages. A request for offset=1000 re-derives the full ordered result set +and skips the first 1000 entries. SQL `LIMIT/OFFSET` lets the provider do this skip in +the B-tree index instead of iterating through rows. + +### Why the provider gets excludes + +With paging push-down, SQL-side excludes are **15x faster** than JS-side filtering +(1.1ms vs 15.9ms at offset=10000). The B-tree index handles seek + exclude together. +The provider needs excludes to make paging accurate. + +### Fallback protocol + +1. Provider returns `null` → worker falls back to the existing iterator-oracle + pattern, completely unchanged. SNOMED and any provider that doesn't implement + this method are unaffected. +2. Provider returns an iterable → worker iterates entries, skips the framework's + manual `excludeCodes()` and `includeCodes()` paths for that code system. +3. `this.offset = 0` after expansion — the provider's SQL handled OFFSET; prevents + the framework's finalization from double-skipping. + +### Why `better-sqlite3` + +The async `sqlite3` package can't support this pattern: `db.each()` can't abort +early, so there's no way to stop after N rows. `better-sqlite3` provides +synchronous lazy cursors via `stmt.iterate()` — the provider can break after the +page is filled. It's also 2x faster for bulk loads. + +The sync API is fine here — SQLite operations are inherently single-threaded and +in-process. The async wrappers add overhead for no concurrency benefit. + +--- + +## RxNorm Implementation + +### SQL strategies + +The RxNorm provider maps each filter/option to SQL: + +| Filter | SQL | Index used | +|--------|-----|------------| +| TTY (e.g., SBD) | `WHERE TTY IN (...)` | `(SAB, TTY, RXCUI)` — covers ORDER BY | +| STY (semantic type) | `JOIN rxnsty ON rxnsty.RXCUI = rxnconso.RXCUI WHERE TUI = ?` | `X_RXNSTY_2(TUI)` drives, probes `X_RXNCONSO_1(RXCUI)` | +| Concepts | `WHERE RXCUI IN (...)` with `+SAB = @sab` (unary `+` suppresses SAB index) | `X_RXNCONSO_1(RXCUI)` | +| Excludes | `AND RXCUI NOT IN (@p1, @p2, ...)` | — | +| activeOnly | `AND SUPPRESS <> '1'` | — | +| searchText | `AND UPPER(STR) LIKE @pattern` | — | + +**GROUP BY for JOINs**: When STY joins are present, rxnconso has 1–8 rows per RXCUI +(different TTY values). `GROUP BY RXCUI` deduplicates at the SQL level so +LIMIT/OFFSET counts unique codes, not raw rows. + +**Index lesson**: Adding indexes can *hurt* SQLite — a composite `rxnsty(TUI, RXCUI)` +index caused the planner to switch to a worse strategy. Prefer query shaping (JOIN +order, unary `+`) over explicit index hints. + +### Results (13 tests) + +``` +Test | Opt (ms) | Base (ms) | Speedup | Result +------------------------------|----------|----------|---------|------- +filter-tty-sbd-10 | 6.8 | 248.7 | 36.5x | ✅ exact +concept-5 | 2.5 | 3.8 | 1.5x | ✅ exact +exclude-concepts-3 | 3.2 | 235.2 | 73.8x | ✅ exact +multi-include-2 | 63.9 | 471.0 | 7.4x | ✅ sets equal (40k) +activeonly-sbd | 2.2 | 194.9 | 82.3x | ✅ exact +filter-tty-in-multi | 1.4 | 476.0 | 341.6x | ✅ exact +filter-sty-t200 | 217.1 | 1521.6 | 7.0x | ✅ exact +paged-offset-100 | 1.3 | 228.2 | 177.4x | ✅ exact +text-aspirin | 1.8 | TIMEOUT | ∞ | ✅ opt works +exclude-filter | 3.9 | 320.5 | 83.2x | ✅ exact +multi-include-concept+filter | 140.2 | 189.5 | 1.4x | ✅ sets equal (23k) +combo-active-text-paged | 1.4 | TIMEOUT | ∞ | ✅ opt works +multi-include-multi-exclude | 99.7 | 673.7 | 6.8x | ✅ sets equal (40k) +``` + +**All 13 pass.** Median speedup ~37x. Best case 342x (multi-value IN with index). + +--- + +## LOINC Implementation + +### Baseline problem + +LOINC's existing provider loads **all 240k codes into memory** at startup. Filter +queries run SQL to find matching CodeKeys, then **materialize the entire result +set** into an array before iterating. For large filters this takes 2–5 seconds: +STATUS=ACTIVE materializes 163k rows, CLASSTYPE=Lab materializes 73k rows. + +The LOINC database is more normalized than RxNorm: +- `Codes` (240k) — CodeKey PK, Code, Type (1=Code, 2=Part, 3=AnswerList, 4=Answer), StatusKey +- `Relationships` (1.2M) — links codes to parts (COMPONENT, CLASS, SYSTEM, SCALE_TYP, etc.) +- `Properties` (347k) + `PropertyValues` — key-value attributes (CLASSTYPE, ORDER_OBS) + +### SQL strategies + +| Filter | SQL | Notes | +|--------|-----|-------| +| Relationship (COMPONENT, CLASS, etc.) | `JOIN Relationships r ON r.TargetKey = (SELECT CodeKey FROM Codes WHERE Code = ?) AND r.RelationshipTypeKey = ? AND r.SourceKey = c.CodeKey` | Naturally scopes to Type=1 codes | +| STATUS | `WHERE c.StatusKey = ?` | Direct column match | +| CLASSTYPE | `JOIN Properties p ... JOIN PropertyValues pv ... AND pv.Value = ?` | Value "1" → "Laboratory class" via lookup | +| LIST (answers-for) | `JOIN Relationships r ON r.SourceKey = (SELECT ...) AND r.RelationshipTypeKey = 40 AND r.TargetKey = c.CodeKey` | Reversed direction — list is source, answers are targets | +| activeOnly | `WHERE c.StatusKey = 1` | — | + +**Multi-include via UNION**: Each compose include becomes a separate SELECT. Multiple +includes are `UNION ALL`'d, with the outer query applying `GROUP BY Code` for dedup: + +```sql +SELECT Code, Description FROM ( + SELECT c.Code, c.CodeKey, d.Description FROM Codes c + JOIN Descriptions d ON d.CodeKey = c.CodeKey AND d.DescriptionTypeKey = 1 + JOIN Relationships r1 ON ... -- include 1 filters + UNION ALL + SELECT c.Code, c.CodeKey, d.Description FROM Codes c + JOIN Descriptions d ON d.CodeKey = c.CodeKey AND d.DescriptionTypeKey = 1 + JOIN Relationships r2 ON ... -- include 2 filters +) GROUP BY Code ORDER BY CodeKey LIMIT ? OFFSET ? +``` + +This avoids AND-semantics where JOINs from different includes would all need to +match simultaneously. + +**Key decisions:** +- **ORDER BY CodeKey** (not Code string) matches baseline iteration order +- **No blanket Type=1 filter** — relationship JOINs scope naturally; a Type=1 + restriction would break LIST queries (answers are Type=4) +- **Existing indexes sufficient** — `RelationshipsTarget`, `PropertiesCode1`, + `CodesCode` cover all patterns without new indexes +- **Concept-only includes fall back** — `expandForValueSet` returns `null`, lets + the framework handle via `locate()` (efficient for small lists) + +### Results (14 tests) + +``` +Test | Opt (ms) | Base (ms) | Speedup | Result +------------------------------|----------|-----------|---------|--------------------------- +filter-component-bacteria | 11.3 | 31.4 | 2.7x | ✅ exact +filter-class-chem | 13.8 | 941.0 | 68.2x | ✅ exact +filter-scale-qn | 50.8 | 2611.3 | 51.4x | ✅ exact +filter-system-ser | 21.0 | 680.4 | 32.4x | ✅ exact +concept-5 | 2.8 | 1.9 | 0.6x | ✅ sets equal (5) +exclude-concepts | 2.3 | 57.4 | 24.9x | ✅ exact +activeonly-class | 14.0 | 119.4 | 8.7x | ✅ exact +filter-list-ll150 | 2.3 | 14.7 | 6.4x | ✅ sets equal (255) +filter-classtype-lab | 82.2 | 2020.7 | 24.6x | ✅ exact +paged-class-offset-100 | 14.0 | 102.8 | 7.5x | ✅ exact +multi-filter-comp-scale | 2.4 | 80.0 | 34.6x | ✅ exact +filter-status-active | 50.1 | 5324.6 | 106.4x | ✅ exact +text-glucose | 1.3 | 1.4 | 1.1x | ✅ exact +multi-include-2-components | 4.2 | 87.6 | 25.0x | ✅ sets equal (743) +``` + +**All 14 pass.** Median speedup ~25x. Biggest wins: STATUS=ACTIVE 5.3s → 50ms, +SCALE_TYP=Qn 2.6s → 51ms, CLASSTYPE=Lab 2.0s → 82ms. + +--- + +## Validation + +**Unit tests:** LOINC 37/37 pass, RxNorm 45/45 skip (require raw import data). + +**Replay tests** (18 captured production queries): +- RxNorm: 3/3 ✅ (validate-code, all 200) +- LOINC: 4/4 functionally correct (1 exact match, 3 now return 200 where production + returned 422 — we handle queries production rejected) +- SNOMED: 5/5 expected failures (not loaded) +- Other: 6/6 ✅ (batch-validate, multi-system expansions) + +## Summary + +The entire interface change is **one optional method** — `expandForValueSet`. It's +additive: providers that don't implement it are completely unaffected. The method +gives SQL-backed providers the information they need (the full compose hull, +activeOnly, excludes, paging) to push everything into a single query with +LIMIT/OFFSET. + +| What | RxNorm | LOINC | +|------|--------|-------| +| Tests | 13/13 pass | 14/14 pass | +| Median speedup | ~37x | ~25x | +| Best speedup | 342x | 106x | +| Biggest absolute win | text+combo: TIMEOUT → <2ms | STATUS=ACTIVE: 5.3s → 50ms | +| New indexes needed | None | None | +| Existing tests broken | None | None | diff --git a/server.js b/server.js index 14cb176..4d79083 100644 --- a/server.js +++ b/server.js @@ -502,6 +502,41 @@ app.get('/health', async (req, res) => { res.json(healthStatus); }); +app.get('/debug/perf-counters', (req, res) => { + const perfCounters = require('./tx/perf-counters'); + res.json(perfCounters.snapshot()); +}); + +app.post('/debug/perf-counters/reset', (req, res) => { + const perfCounters = require('./tx/perf-counters'); + perfCounters.reset(); + res.json({ ok: true }); +}); + +app.post('/debug/perf-counters/enable', (req, res) => { + const perfCounters = require('./tx/perf-counters'); + perfCounters.enable(); + res.json({ ok: true }); +}); + +app.post('/debug/bypass-expand-for-valueset', (req, res) => { + const { RxNormServices } = require('./tx/cs/cs-rxnorm'); + const { LoincServices } = require('./tx/cs/cs-loinc'); + const { SqliteRuntimeV0FactoryProvider } = require('./tx/cs/cs-sqlite-runtime-v0'); + const bypass = req.query.bypass !== 'false'; + RxNormServices.bypassExpandForValueSet = bypass; + LoincServices.bypassExpandForValueSet = bypass; + SqliteRuntimeV0FactoryProvider.bypassExpandForValueSet = bypass; + res.json({ bypassExpandForValueSet: bypass }); +}); + +app.get('/debug/bypass-expand-for-valueset', (req, res) => { + const { RxNormServices } = require('./tx/cs/cs-rxnorm'); + const { LoincServices } = require('./tx/cs/cs-loinc'); + const { SqliteRuntimeV0FactoryProvider } = require('./tx/cs/cs-sqlite-runtime-v0'); + res.json({ bypassExpandForValueSet: !!(RxNormServices.bypassExpandForValueSet || LoincServices.bypassExpandForValueSet || SqliteRuntimeV0FactoryProvider.bypassExpandForValueSet) }); +}); + /** * Get log directory statistics: file count, total size, and disk space info * @returns {string} HTML table row(s) with log stats diff --git a/tx/cs/cs-api.js b/tx/cs/cs-api.js index ea6a050..65c6d76 100644 --- a/tx/cs/cs-api.js +++ b/tx/cs/cs-api.js @@ -620,6 +620,44 @@ class CodeSystemProvider { } + /** + * Expand all include/exclude blocks for this code system in one shot. + * + * The worker groups compose.include and compose.exclude blocks by system, + * then hands the full hull to the provider. SQL-backed providers can translate + * the entire spec into a single query with LIMIT/OFFSET, activeOnly, and + * exclude push-down. + * + * @param {Object} spec + * @param {Object[]} spec.includes - include blocks for this CS, each with + * { concepts: [{code, display?}]|null, filters: [{property, op, value}] } + * @param {Object[]} spec.excludes - exclude blocks for this CS, each with + * { concepts: [{code}]|null, filters: [{property, op, value}] } + * @param {boolean} spec.activeOnly - exclude inactive codes + * @param {string|null} spec.searchText - expansion 'filter' parameter + * @param {boolean} spec.includeDesignations - whether designations are needed + * @param {string[]} spec.properties - which properties to include + * @param {number|null} spec.offset - paging offset (must apply if non-null) + * @param {number|null} spec.count - paging count (must apply if non-null) + * + * @returns {AsyncIterable<{ + * code: string, + * display: string, + * system: string, + * version: string, + * isAbstract: boolean, + * isInactive: boolean, + * isDeprecated: boolean, + * status: string|null, + * definition: string|null, + * designations: Array<{language: string, use: Object|null, value: string}>, + * properties: Array<{code: string, value: *}>|null, + * extensions: Array|null + * }>|null} An async iterable of expanded entries, or null if this provider + * cannot handle the spec (worker falls back to per-code iteration). + */ + async expandForValueSet(spec) { return null; } + /** * register the concept maps that are implicitly defined as part of the code system * diff --git a/tx/cs/cs-loinc.js b/tx/cs/cs-loinc.js index 7b58f9b..48b9f0b 100644 --- a/tx/cs/cs-loinc.js +++ b/tx/cs/cs-loinc.js @@ -1,4 +1,6 @@ const sqlite3 = require('sqlite3').verbose(); +let BetterSqlite3; +try { BetterSqlite3 = require('better-sqlite3'); } catch (e) { /* optional */ } const assert = require('assert'); const { CodeSystem } = require('../library/codesystem'); const { Language, Languages} = require('../../library/languages'); @@ -109,9 +111,11 @@ class LoincPrep { } class LoincServices extends BaseCSServices { - constructor(opContext, supplements, db, sharedData) { + constructor(opContext, supplements, db, sharedData, dbPath = null) { super(opContext, supplements); this.db = db; + this.dbPath = dbPath; + this._syncDb = null; // Lazy better-sqlite3 connection // Shared data from factory this.langs = sharedData.langs; @@ -132,6 +136,10 @@ class LoincServices extends BaseCSServices { this.db.close(); this.db = null; } + if (this._syncDb) { + this._syncDb.close(); + this._syncDb = null; + } } // Metadata methods @@ -569,6 +577,15 @@ class LoincServices extends BaseCSServices { return { context: null, message: undefined }; } + async locateMany(codes, allAltCodes = false) { + const results = new Map(); + for (const code of codes) { + const context = this.codes.get(code); + results.set(code, context ? { context, message: null } : { context: null, message: undefined }); + } + return results; + } + // Iterator methods async iterator(context) { @@ -994,6 +1011,19 @@ class LoincServices extends BaseCSServices { return this.codeList[key]; } + async filterPage(filterContext, set, count) { + set.cursor = set.cursor || 0; + if (set.cursor >= set.keys.length) return []; + + const end = Math.min(set.cursor + count, set.keys.length); + const results = []; + for (let i = set.cursor; i < end; i++) { + results.push(this.codeList[set.keys[i]]); + } + set.cursor = end; + return results; + } + async filterLocate(filterContext, set, code) { const context = this.codes.get(code); if (!context) { @@ -1042,6 +1072,210 @@ class LoincServices extends BaseCSServices { isDisplay(designation) { return designation.use.code == "SHORTNAME" || designation.use.code == "LONG_COMMON_NAME" || designation.use.code == "LinguisticVariantDisplayName"; } + + #getSyncDb() { + if (!this._syncDb) { + if (!BetterSqlite3 || !this.dbPath) return null; + this._syncDb = new BetterSqlite3(this.dbPath, { readonly: true }); + } + return this._syncDb; + } + + /** + * Build SQL fragments for a single filter {property, op, value}. + * Returns { where, joins, params } or null if unsupported. + */ + #buildLoincFilterSql(filter, prefix) { + const prop = filter.property; + const op = filter.op; + const value = filter.value; + + // Relationship-based filters (COMPONENT, CLASS, SYSTEM, SCALE_TYP, METHOD_TYP, TIME_ASPCT, etc.) + if (this.relationships.has(prop) && op === '=') { + const relKey = this.relationships.get(prop); + // value is a part code — look up its CodeKey + const ctx = this.codes.get(value); + if (!ctx) return null; + const alias = `_rel_${prefix}`; + return { + where: '', + joins: ` JOIN Relationships ${alias} ON ${alias}.SourceKey = c.CodeKey AND ${alias}.RelationshipTypeKey = ${relKey} AND ${alias}.TargetKey = @${prefix}_target`, + params: { [`${prefix}_target`]: ctx.key }, + needsGroupBy: true + }; + } + + // LIST filter — answers for an answer list + if (prop === 'LIST' && op === '=') { + const ctx = this.codes.get(value); + if (!ctx) return null; + const answerRelKey = this.relationships.get('Answer'); + if (!answerRelKey) return null; + const alias = `_ans_${prefix}`; + // Answer relationships: SourceKey=AnswerList, TargetKey=Answer code + // We want the target (answer) codes, so join differently + return { + where: '', + joins: ` JOIN Relationships ${alias} ON ${alias}.TargetKey = c.CodeKey AND ${alias}.RelationshipTypeKey = ${answerRelKey} AND ${alias}.SourceKey = @${prefix}_list`, + params: { [`${prefix}_list`]: ctx.key }, + needsGroupBy: true + }; + } + + // STATUS filter + if (prop === 'STATUS' && op === '=') { + const statusKey = this.statusKeys.get(value); + if (statusKey === undefined) return null; + return { + where: ` AND c.StatusKey = @${prefix}_status`, + joins: '', + params: { [`${prefix}_status`]: parseInt(statusKey) }, + needsGroupBy: false + }; + } + + // CLASSTYPE property filter + if (prop === 'CLASSTYPE' && op === '=') { + const ptKey = this.propertyList.get('CLASSTYPE'); + if (!ptKey) return null; + // PropertyValues stores the raw value; CLASSTYPE '1' = Laboratory, '2' = Clinical, etc. + const alias = `_prop_${prefix}`; + return { + where: '', + joins: ` JOIN Properties ${alias} ON ${alias}.CodeKey = c.CodeKey AND ${alias}.PropertyTypeKey = ${ptKey}` + + ` JOIN PropertyValues ${alias}_pv ON ${alias}_pv.PropertyValueKey = ${alias}.PropertyValueKey AND ${alias}_pv.Value = @${prefix}_pval`, + params: { [`${prefix}_pval`]: classTypes[value] || value }, + needsGroupBy: true + }; + } + + // Unsupported filter — fall back + return null; + } + + async expandForValueSet(spec) { + if (LoincServices.bypassExpandForValueSet) return null; + + const syncDb = this.#getSyncDb(); + if (!syncDb) return null; + + const sys = this.system(); + const ver = this.version(); + const allParams = {}; + const conceptCodes = new Set(); + + // Build a separate SELECT for each include, UNION ALL them + const selectParts = []; + + for (let i = 0; i < spec.includes.length; i++) { + const inc = spec.includes[i]; + + if (inc.concepts && inc.concepts.length > 0) { + // Explicit code list — no Type restriction (concepts can be any type) + const placeholders = inc.concepts.map((_, j) => `@_ic${i}_${j}`).join(','); + selectParts.push(`SELECT c.CodeKey, c.Code, c.Description, c.StatusKey FROM Codes c WHERE c.Code IN (${placeholders})`); + inc.concepts.forEach((cc, j) => { + allParams[`_ic${i}_${j}`] = cc.code; + conceptCodes.add(cc.code); + }); + } else if (inc.filters && inc.filters.length > 0) { + // Filter-based — each include is an independent SELECT with its own JOINs + let joins = ''; + let where = ''; + let needsGroupBy = false; + + for (let j = 0; j < inc.filters.length; j++) { + const result = this.#buildLoincFilterSql(inc.filters[j], `i${i}f${j}`); + if (!result) return null; // Unsupported filter — fall back entirely + where += result.where; + if (result.joins) joins += result.joins; + if (result.needsGroupBy) needsGroupBy = true; + Object.assign(allParams, result.params); + } + + // Relationship JOINs naturally scope to the correct code types + // (e.g., COMPONENT relationship only links from Type=1 LOINC codes). + // No explicit Type filter needed — the data model handles it. + let typeSql = ''; + + const groupBy = needsGroupBy ? ' GROUP BY c.CodeKey' : ''; + selectParts.push(`SELECT c.CodeKey, c.Code, c.Description, c.StatusKey FROM Codes c${joins} WHERE 1=1${where}${typeSql}${groupBy}`); + } else { + return null; // Bare "whole code system" — fall back + } + } + + if (selectParts.length === 0) return null; + + // Build exclude clause + let excludeSql = ''; + for (let i = 0; i < spec.excludes.length; i++) { + const exc = spec.excludes[i]; + if (exc.concepts && exc.concepts.length > 0) { + const placeholders = exc.concepts.map((_, j) => `@_ec${i}_${j}`).join(','); + excludeSql += ` AND Code NOT IN (${placeholders})`; + exc.concepts.forEach((cc, j) => { allParams[`_ec${i}_${j}`] = cc.code; }); + } + } + + // activeOnly — exclude DEPRECATED (2) and DISCOURAGED (4) status codes + let activeSql = ''; + if (spec.activeOnly) { + const depKey = this.statusKeys.get('DEPRECATED'); + const discKey = this.statusKeys.get('DISCOURAGED'); + const excludeKeys = [depKey, discKey].filter(k => k !== undefined).map(k => parseInt(k)); + if (excludeKeys.length > 0) { + activeSql = ` AND StatusKey NOT IN (${excludeKeys.join(',')})`; + } + } + + // searchText + if (spec.searchText) { + activeSql += ` AND Description LIKE @_searchText COLLATE NOCASE`; + allParams._searchText = `%${spec.searchText}%`; + } + + // Wrap in outer query for dedup, ordering, excludes, paging + const inner = selectParts.length === 1 + ? selectParts[0] + : selectParts.join(' UNION ALL '); + + let sql = `SELECT CodeKey, Code, Description, StatusKey FROM (${inner})` + + ` WHERE 1=1` + + excludeSql + + activeSql + + ` GROUP BY Code` + + ` ORDER BY CodeKey`; + + // Paging + if (spec.count != null && spec.count > 0) { + const limit = spec.count; + const offset = (spec.offset != null && spec.offset > 0) ? spec.offset : 0; + sql += ` LIMIT ${limit} OFFSET ${offset}`; + } + + const self = this; + return (function* () { + const stmt = syncDb.prepare(sql); + for (const row of stmt.iterate(allParams)) { + const statusDesc = self.statusCodes.get(row.StatusKey.toString()); + yield { + code: row.Code, + display: row.Description, + system: sys, + version: ver, + isAbstract: false, + isInactive: statusDesc === 'DISCOURAGED', + isDeprecated: statusDesc === 'DEPRECATED', + status: statusDesc === 'NotStated' ? null : statusDesc, + definition: null, + designations: [], + properties: null, + extensions: null, + }; + } + })(); + } } class LoincServicesFactory extends CodeSystemFactoryProvider { @@ -1352,7 +1586,7 @@ class LoincServicesFactory extends CodeSystemFactoryProvider { }); }); - return new LoincServices(opContext, supplements, db, this._sharedData); + return new LoincServices(opContext, supplements, db, this._sharedData, this.dbPath); } useCount() { diff --git a/tx/cs/cs-rxnorm.js b/tx/cs/cs-rxnorm.js index cb095c4..778aab5 100644 --- a/tx/cs/cs-rxnorm.js +++ b/tx/cs/cs-rxnorm.js @@ -1,4 +1,6 @@ const sqlite3 = require('sqlite3').verbose(); +let BetterSqlite3; +try { BetterSqlite3 = require('better-sqlite3'); } catch (e) { /* optional */ } const assert = require('assert'); const { CodeSystem } = require('../library/codesystem'); const { CodeSystemProvider, CodeSystemFactoryProvider } = require('./cs-api'); @@ -12,6 +14,7 @@ class RxNormConcept { this.display = display; this.others = []; // Array of alternative displays (SY terms, etc.) this.archived = false; + this.suppress = false; // Eagerly loaded from locate() to avoid redundant queries } } @@ -54,10 +57,12 @@ class RxNormIteratorContext { } class RxNormServices extends CodeSystemProvider { - constructor(opContext, supplements, db, sharedData, isNCI = false) { + constructor(opContext, supplements, db, sharedData, isNCI = false, dbPath = null) { super(opContext, supplements); this.db = db; this.isNCI = isNCI; + this.dbPath = dbPath; + this._syncDb = null; // Lazy better-sqlite3 connection // Shared data from factory this.dbVersion = sharedData.version; @@ -71,6 +76,10 @@ class RxNormServices extends CodeSystemProvider { this.db.close(); this.db = null; } + if (this._syncDb) { + this._syncDb.close(); + this._syncDb = null; + } } // Metadata methods @@ -148,18 +157,11 @@ class RxNormServices extends CodeSystemProvider { return 'archived'; } - // Check suppress flag - return new Promise((resolve, reject) => { - const sql = `SELECT suppress FROM rxnconso WHERE ${this.getCodeField()} = ? AND SAB = ? AND TTY <> 'SY'`; - - this.db.get(sql, [ctxt.code, this.getSAB()], (err, row) => { - if (err) { - reject(err); - } else { - resolve(row ? row.suppress === '1' ? 'suppressed' : null : null); - } - }); - }); + // Use cached suppress flag from locate() if available + if (ctxt) { + return ctxt.suppress ? 'suppressed' : null; + } + return null; } async isInactive(context) { @@ -170,18 +172,8 @@ class RxNormServices extends CodeSystemProvider { return true; } - // Check suppress flag - return new Promise((resolve, reject) => { - const sql = `SELECT suppress FROM rxnconso WHERE ${this.getCodeField()} = ? AND SAB = ? AND TTY <> 'SY'`; - - this.db.get(sql, [ctxt.code, this.getSAB()], (err, row) => { - if (err) { - reject(err); - } else { - resolve(row ? row.suppress === '1' : false); - } - }); - }); + // Use cached suppress flag from locate() + return ctxt ? ctxt.suppress : false; } async isDeprecated(context) { @@ -233,7 +225,7 @@ class RxNormServices extends CodeSystemProvider { if (!code) return { context: null, message: 'Empty code' }; return new Promise((resolve, reject) => { - let sql = `SELECT STR, TTY FROM rxnconso WHERE ${this.getCodeField()} = ? AND SAB = ?`; + let sql = `SELECT STR, TTY, SUPPRESS FROM rxnconso WHERE ${this.getCodeField()} = ? AND SAB = ?`; this.db.all(sql, [code, this.getSAB()], (err, rows) => { if (err) { @@ -266,6 +258,10 @@ class RxNormServices extends CodeSystemProvider { }); } + // locateMany intentionally not overridden: SQLite's prepared-statement + // index lookups are faster than a single IN(...) query with many codes. + // The base class fallback (N individual locate() calls) wins here. + #createConceptFromRows(code, rows, archived) { const concept = new RxNormConcept(code); concept.archived = archived; @@ -276,6 +272,10 @@ class RxNormServices extends CodeSystemProvider { } else { concept.display = row.STR.trim(); } + // Cache suppress flag from locate() query to avoid redundant SQL + if (row.SUPPRESS !== undefined) { + concept.suppress = row.SUPPRESS === '1'; + } } return concept; @@ -487,7 +487,7 @@ class RxNormServices extends CodeSystemProvider { } } - const fullQuery = `SELECT ${this.getCodeField()}, STR ${sql2} WHERE SAB = $sab AND TTY <> 'SY' ${sql1}`; + const fullQuery = `SELECT ${this.getCodeField()}, STR, SUPPRESS ${sql2} WHERE SAB = $sab AND TTY <> 'SY' ${sql1} ORDER BY ${this.getCodeField()}`; allParams.sab = this.getSAB(); // Create a single filter holder with the combined query @@ -616,6 +616,217 @@ class RxNormServices extends CodeSystemProvider { return str.replace(/'/g, "''"); } + // --- expandForValueSet: single-query expansion for ValueSet operations --- + + #getSyncDb() { + if (!this._syncDb) { + if (!BetterSqlite3 || !this.dbPath) return null; + this._syncDb = new BetterSqlite3(this.dbPath, { readonly: true }); + } + return this._syncDb; + } + + /** + * Build SQL WHERE fragments from a spec's filter array. + * Returns { sql, params } or null if unsupported filter encountered. + */ + #buildFilterSql(filters, paramPrefix) { + let sql = ''; + let joins = ''; + const params = {}; + for (let i = 0; i < filters.length; i++) { + const f = filters[i]; + const prop = f.property.toUpperCase(); + const pfx = `${paramPrefix}_f${i}`; + + if (f.op === '=' && prop === 'TTY') { + sql += ` AND rxnconso.TTY = @${pfx}_tty`; + params[`${pfx}_tty`] = f.value; + } else if (f.op === 'in' && prop === 'TTY') { + const values = f.value.split(',').map(v => v.trim()).filter(v => v); + const placeholders = values.map((_, j) => `@${pfx}_tty${j}`).join(','); + sql += ` AND rxnconso.TTY IN (${placeholders})`; + values.forEach((val, j) => { params[`${pfx}_tty${j}`] = val; }); + } else if (f.op === '=' && prop === 'STY') { + // Use JOIN for STY — planner drives from rxnsty via X_RXNSTY_2(TUI) + // then probes rxnconso via X_RXNCONSO_1(RXCUI), much faster than + // IN-subquery which scans all RXNORM rows via X_RXNCONSO_2(SAB) + const alias = `_sty${pfx}`; + joins += ` JOIN rxnsty ${alias} ON ${alias}.RXCUI = rxnconso.${this.getCodeField()} AND ${alias}.TUI = @${pfx}_sty`; + params[`${pfx}_sty`] = f.value; + } else if (f.op === '=' && prop === 'SAB') { + sql += ` AND ${this.getCodeField()} IN (SELECT ${this.getCodeField()} FROM rxnconso WHERE SAB = @${pfx}_sab)`; + params[`${pfx}_sab`] = f.value; + } else { + return null; // Unsupported filter — fall back + } + } + return { sql, joins, params, needsGroupBy: joins.length > 0 }; + } + + async expandForValueSet(spec) { + // Bypass flag: set RxNormServices.bypassExpandForValueSet = true to skip + if (RxNormServices.bypassExpandForValueSet) return null; + + const syncDb = this.#getSyncDb(); + if (!syncDb) return null; + + const codeField = this.getCodeField(); + const sab = this.getSAB(); + const sys = this.system(); + const ver = this.version(); + + // Build include clauses — each include becomes a condition group, OR'd together + const includeGroups = []; + let joinClauses = ''; + const allParams = { _sab: sab }; + + // Track concept codes for archive fallback + const conceptCodes = new Set(); + let conceptOnly = true; // true if all includes are concept lists (no filters) + let needsGroupBy = false; + + for (let i = 0; i < spec.includes.length; i++) { + const inc = spec.includes[i]; + + if (inc.concepts && inc.concepts.length > 0) { + // Explicit code list + const placeholders = inc.concepts.map((_, j) => `@_ic${i}_${j}`).join(','); + includeGroups.push(`rxnconso.${codeField} IN (${placeholders})`); + inc.concepts.forEach((cc, j) => { + allParams[`_ic${i}_${j}`] = cc.code; + conceptCodes.add(cc.code); + }); + } else if (inc.filters && inc.filters.length > 0) { + conceptOnly = false; + // Filter-based + const result = this.#buildFilterSql(inc.filters, `_i${i}`); + if (!result) return null; // Unsupported filter — fall back entirely + // The filter SQL starts with " AND ..." so strip the leading AND for the group + includeGroups.push('(1=1' + result.sql + ')'); + if (result.joins) joinClauses += result.joins; + if (result.needsGroupBy) needsGroupBy = true; + Object.assign(allParams, result.params); + } else { + return null; // Bare "whole code system" — fall back + } + } + + if (includeGroups.length === 0) return null; + + // Build exclude clause + let excludeSql = ''; + for (let i = 0; i < spec.excludes.length; i++) { + const exc = spec.excludes[i]; + if (exc.concepts && exc.concepts.length > 0) { + const placeholders = exc.concepts.map((_, j) => `@_ec${i}_${j}`).join(','); + excludeSql += ` AND rxnconso.${codeField} NOT IN (${placeholders})`; + exc.concepts.forEach((cc, j) => { allParams[`_ec${i}_${j}`] = cc.code; }); + } else if (exc.filters && exc.filters.length > 0) { + const result = this.#buildFilterSql(exc.filters, `_e${i}`); + if (!result) { + // Can't push this exclude to SQL — worker's isExcluded will handle it + continue; + } + excludeSql += ` AND rxnconso.${codeField} NOT IN (SELECT rxnconso.${codeField} FROM rxnconso${result.joins || ''} WHERE rxnconso.SAB = @_sab AND rxnconso.TTY <> 'SY' ${result.sql})`; + Object.assign(allParams, result.params); + } + } + + // activeOnly + let activeSql = ''; + if (spec.activeOnly) { + activeSql = ` AND rxnconso.SUPPRESS <> '1'`; + } + + // searchText (basic LIKE match on STR) + if (spec.searchText) { + activeSql += ` AND rxnconso.STR LIKE @_searchText`; + allParams._searchText = `%${spec.searchText}%`; + } + + // Combine: SELECT ... WHERE SAB=? AND TTY<>'SY' AND (include1 OR include2) AND NOT excludes + const includeCondition = includeGroups.length === 1 + ? includeGroups[0] + : '(' + includeGroups.join(' OR ') + ')'; + + // For concept-only queries, hint the planner to use the RXCUI index + // (otherwise it picks the SAB index and scans all RXNORM rows) + const indexHint = conceptOnly ? ' INDEXED BY X_RXNCONSO_1' : ''; + + let sql = `SELECT rxnconso.${codeField}, rxnconso.STR, rxnconso.SUPPRESS FROM rxnconso${indexHint}${joinClauses}` + + ` WHERE rxnconso.SAB = @_sab AND rxnconso.TTY <> 'SY'` + + ` AND ${includeCondition}` + + excludeSql + + activeSql + + (needsGroupBy ? ` GROUP BY rxnconso.${codeField}` : '') + + ` ORDER BY rxnconso.${codeField}`; + + // Paging — SQL handles offset directly so the framework doesn't re-skip + if (spec.count != null && spec.count > 0) { + const limit = spec.count; + const offset = (spec.offset != null && spec.offset > 0) ? spec.offset : 0; + sql += ` LIMIT ${limit} OFFSET ${offset}`; + } + + // Return a generator backed by better-sqlite3's lazy cursor + const self = this; + return (function* () { + const stmt = syncDb.prepare(sql); + const seen = new Set(); + for (const row of stmt.iterate(allParams)) { + const code = row[codeField]; + if (seen.has(code)) continue; // dedup (e.g. STY JOIN produces duplicates) + seen.add(code); + yield { + code: row[codeField], + display: row.STR, + system: sys, + version: ver, + isAbstract: false, + isInactive: row.SUPPRESS === '1', + isDeprecated: false, + status: row.SUPPRESS === '1' ? 'inactive' : 'active', + definition: null, + designations: [], + properties: null, + extensions: null, + }; + } + // Archive fallback for explicit concept codes not found in rxnconso + if (conceptCodes.size > 0) { + const missing = [...conceptCodes].filter(c => !seen.has(c)); + if (missing.length > 0) { + const archPlaceholders = missing.map((_, i) => `@_arch${i}`).join(','); + const archParams = {}; + missing.forEach((c, i) => { archParams[`_arch${i}`] = c; }); + archParams._archSab = sab; + const archSql = `SELECT ${codeField}, STR FROM RXNATOMARCHIVE` + + ` WHERE ${codeField} IN (${archPlaceholders}) AND SAB = @_archSab AND TTY <> 'SY'` + + ` GROUP BY ${codeField}` + + ` ORDER BY ${codeField}`; + const archStmt = syncDb.prepare(archSql); + for (const row of archStmt.iterate(archParams)) { + yield { + code: row[codeField], + display: row.STR, + system: sys, + version: ver, + isAbstract: false, + isInactive: true, + isDeprecated: true, + status: 'retired', + definition: null, + designations: [], + properties: null, + extensions: null, + }; + } + } + } + })(); + } + // Subsumption testing async subsumesTest(codeA, codeB) { await this.#ensureContext(codeA); @@ -795,7 +1006,7 @@ class RxNormTypeServicesFactory extends CodeSystemFactoryProvider { // Create fresh database connection for this provider instance const db = new sqlite3.Database(this.dbPath); - return new RxNormServices(opContext, supplements, db, this._sharedData, this.isNCI); + return new RxNormServices(opContext, supplements, db, this._sharedData, this.isNCI, this.dbPath); } name() { diff --git a/tx/cs/cs-sqlite-runtime-v0.js b/tx/cs/cs-sqlite-runtime-v0.js new file mode 100644 index 0000000..e7903d6 --- /dev/null +++ b/tx/cs/cs-sqlite-runtime-v0.js @@ -0,0 +1,3129 @@ +'use strict'; + +const sqlite3 = require('sqlite3').verbose(); +let BetterSqlite3; +try { BetterSqlite3 = require('better-sqlite3'); } catch (_) { BetterSqlite3 = null; } +const { CodeSystem } = require('../library/codesystem'); +const { CodeSystemProvider, CodeSystemFactoryProvider, FilterExecutionContext } = require('./cs-api'); + +const SQLITE_RUNTIME_V0_FACTORY_REGISTRY = []; + +class SqliteRuntimeV0Context { + constructor(conceptId, code, display, definition, active) { + this.conceptId = conceptId; + this.code = code; + this.display = display; + this.definition = definition; + this.active = active; + } +} + +class SqliteRuntimeV0Iterator { + constructor(codes) { + this.codes = codes || []; + this.cursor = 0; + } +} + +class SqliteRuntimeV0QueryIterator { + constructor(mode, options = {}) { + this.mode = mode; + this.pageSize = Number(options.pageSize) > 0 ? Number(options.pageSize) : 512; + this.targetConceptId = options.targetConceptId || null; + this.rows = []; + this.cursor = 0; + this.lastCode = null; + this.done = false; + } +} + +class SqliteRuntimeV0FilterSet { + constructor(name, codes, closed = true) { + this.name = name; + this.summary = name; + this.codes = codes || []; + this.cursor = -1; + this.closed = closed; + this._set = null; + } + + has(code) { + if (!this._set) { + this._set = new Set(this.codes); + } + return this._set.has(code); + } +} + +class SqliteRuntimeV0PredicateFilter { + constructor(name, kind, details = {}, closed = true) { + this.name = name; + this.summary = name; + this.kind = kind; + this.closed = closed; + this.cursor = -1; + Object.assign(this, details || {}); + } +} + +class SqliteRuntimeV0PagedDescendantFilter { + constructor(name, ancestorId, includeSelf, pageSize = 512) { + this.name = name; + this.summary = name; + this.ancestorId = ancestorId; + this.includeSelf = includeSelf; + this.pageSize = pageSize; + this.closed = true; + this.cursor = -1; + this.rows = []; + this.done = false; + this.lastCode = null; + this.strategy = null; + this.descendantCount = null; + } +} + +class SqliteRuntimeV0Provider extends CodeSystemProvider { + constructor(opContext, supplements, db, metadata, runtime, options = {}) { + super(opContext, supplements); + this.db = db; + this.meta = metadata; + this.runtime = runtime || {}; + this.propertyDefs = new Map(); + this.sharedState = options.sharedState || null; + this.statusCache = null; + this.ownsDb = options.ownsDb === true; + this.dbPath = options.dbPath || null; + this._syncDb = null; + this.defaultIterationRegex = null; + const regexSource = this.runtime?.iteration?.defaultCodeRegex; + if (regexSource) { + try { + this.defaultIterationRegex = new RegExp(String(regexSource)); + } catch (_error) { + this.defaultIterationRegex = null; + } + } + } + + close() { + if (!this.db || !this.ownsDb) return; + this.statusCache = null; + this.db.close(); + this.db = null; + } + + // --- expandForValueSet: single-query expansion for ValueSet operations --- + + #getSyncDb() { + if (this._syncDb) return this._syncDb; + if (!BetterSqlite3 || !this.dbPath) return null; + this._syncDb = new BetterSqlite3(this.dbPath, { readonly: true }); + return this._syncDb; + } + + /** + * Build a SQL condition for a single filter {property, op, value}. + * Returns { sql, params, joins } or null if unsupported. + * @param {string} alias - concept table alias (e.g. 'c' or 'c2') + */ + #buildV0FilterSql(filter, paramPrefix, alias = 'c') { + const { property, op, value } = filter; + const csId = this.meta.csId; + const params = {}; + + if (property === 'concept') { + if (op === '=') { + params[`${paramPrefix}_code`] = value; + return { + sql: ` AND ${alias}.code = @${paramPrefix}_code`, + params, + joins: '', + }; + } + + if (op === 'is-a' || op === 'descendent-of') { + const includeSelf = op === 'is-a' + ? (this.runtime?.filters?.concept?.isAIncludesSelf !== false) + : false; + // Use closure table for hierarchy + params[`${paramPrefix}_anc_code`] = value; + params[`${paramPrefix}_cs`] = csId; + const selfClause = includeSelf ? '' : ` AND cl_${paramPrefix}.descendant_id != cl_${paramPrefix}.ancestor_id`; + return { + sql: selfClause, + params, + joins: ` JOIN closure cl_${paramPrefix} ON cl_${paramPrefix}.descendant_id = ${alias}.concept_id` + + ` AND cl_${paramPrefix}.ancestor_id = (SELECT concept_id FROM concept WHERE code = @${paramPrefix}_anc_code AND cs_id = @${paramPrefix}_cs)`, + }; + } + + if (op === 'in') { + const url = resolveInValueSetUrl(this.system(), value, this.runtime); + params[`${paramPrefix}_vs_url`] = url; + params[`${paramPrefix}_cs`] = csId; + return { + sql: '', + params, + joins: ` JOIN value_set_member vsm_${paramPrefix} ON vsm_${paramPrefix}.concept_id = ${alias}.concept_id AND vsm_${paramPrefix}.active = 1` + + ` JOIN value_set vs_${paramPrefix} ON vs_${paramPrefix}.vs_id = vsm_${paramPrefix}.vs_id AND vs_${paramPrefix}.cs_id = @${paramPrefix}_cs AND vs_${paramPrefix}.url = @${paramPrefix}_vs_url`, + }; + } + + return null; // Unsupported concept operator + } + + if (property === 'code' && op === 'regex') { + // Can't do regex in SQL — fall back + return null; + } + + // Property filter: resolve via property_def → concept_link or concept_literal + const syncDb = this.#getSyncDb(); + if (!syncDb) return null; + + const propDef = syncDb.prepare( + 'SELECT property_id, value_kind FROM property_def WHERE cs_id = ? AND property_code = ? LIMIT 1' + ).get(csId, property); + if (!propDef) return null; + + if (propDef.value_kind === 'concept') { + if (op === '=') { + params[`${paramPrefix}_prop`] = propDef.property_id; + params[`${paramPrefix}_val_code`] = value; + params[`${paramPrefix}_val_cs`] = csId; + params[`${paramPrefix}_eset`] = this.meta.hierarchyEdgeSetId || 1; + return { + sql: '', + params, + joins: ` JOIN concept_link lnk_${paramPrefix}` + + ` ON lnk_${paramPrefix}.source_concept_id = ${alias}.concept_id` + + ` AND lnk_${paramPrefix}.property_id = @${paramPrefix}_prop` + + ` AND lnk_${paramPrefix}.edge_set_id = @${paramPrefix}_eset` + + ` AND lnk_${paramPrefix}.active = 1` + + ` AND lnk_${paramPrefix}.target_concept_id = (SELECT concept_id FROM concept WHERE code = @${paramPrefix}_val_code AND cs_id = @${paramPrefix}_val_cs)`, + }; + } + return null; + } + + if (propDef.value_kind === 'string' || propDef.value_kind === 'literal') { + if (op === '=') { + params[`${paramPrefix}_prop`] = propDef.property_id; + params[`${paramPrefix}_val`] = value; + return { + sql: '', + params, + joins: ` JOIN concept_literal lit_${paramPrefix}` + + ` ON lit_${paramPrefix}.source_concept_id = ${alias}.concept_id` + + ` AND lit_${paramPrefix}.property_id = @${paramPrefix}_prop` + + ` AND lit_${paramPrefix}.value = @${paramPrefix}_val`, + }; + } + return null; + } + + return null; // Unsupported property type + } + + async expandForValueSet(spec) { + if (SqliteRuntimeV0FactoryProvider.bypassExpandForValueSet) return null; + + const syncDb = this.#getSyncDb(); + if (!syncDb) return null; + + const csId = this.meta.csId; + const sys = this.system(); + const ver = this.version(); + + // Build per-include SQL subqueries, UNION ALL'd together + const unionParts = []; + const allParams = { _csId: csId }; + let conceptCodes = null; // Track explicit codes for missing-concept fallback + + for (let i = 0; i < spec.includes.length; i++) { + const inc = spec.includes[i]; + + if (inc.concepts && inc.concepts.length > 0) { + // Explicit concept list + const placeholders = inc.concepts.map((_, j) => `@_ic${i}_${j}`).join(','); + inc.concepts.forEach((cc, j) => { allParams[`_ic${i}_${j}`] = cc.code; }); + unionParts.push( + `SELECT c.concept_id, c.code, c.display, c.definition, c.active` + + ` FROM concept c` + + ` WHERE c.cs_id = @_csId AND c.code IN (${placeholders})` + ); + if (!conceptCodes) conceptCodes = new Set(); + for (const cc of inc.concepts) conceptCodes.add(cc.code); + } else if (inc.filters && inc.filters.length > 0) { + // Filter-based include + let joins = ''; + let where = ''; + let unsupported = false; + + for (let fi = 0; fi < inc.filters.length; fi++) { + const result = this.#buildV0FilterSql(inc.filters[fi], `_i${i}f${fi}`); + if (!result) { + unsupported = true; + break; + } + joins += result.joins; + where += result.sql; + Object.assign(allParams, result.params); + } + if (unsupported) return null; // Fall back to per-code iteration + + unionParts.push( + `SELECT c.concept_id, c.code, c.display, c.definition, c.active` + + ` FROM concept c${joins}` + + ` WHERE c.cs_id = @_csId${where}` + ); + } else { + // Bare "whole code system" — fall back + return null; + } + } + + if (unionParts.length === 0) return null; + + // Build exclude clause + let excludeSql = ''; + for (let i = 0; i < spec.excludes.length; i++) { + const exc = spec.excludes[i]; + if (exc.concepts && exc.concepts.length > 0) { + const placeholders = exc.concepts.map((_, j) => `@_ec${i}_${j}`).join(','); + exc.concepts.forEach((cc, j) => { allParams[`_ec${i}_${j}`] = cc.code; }); + excludeSql += ` AND code NOT IN (${placeholders})`; + } else if (exc.filters && exc.filters.length > 0) { + // Build exclude subquery + let exJoins = ''; + let exWhere = ''; + let unsupported = false; + for (let fi = 0; fi < exc.filters.length; fi++) { + const result = this.#buildV0FilterSql(exc.filters[fi], `_e${i}f${fi}`, 'c2'); + if (!result) { unsupported = true; break; } + exJoins += result.joins; + exWhere += result.sql; + Object.assign(allParams, result.params); + } + if (!unsupported) { + excludeSql += ` AND code NOT IN (SELECT c2.code FROM concept c2${exJoins}` + + ` WHERE c2.cs_id = @_csId${exWhere})`; + } + // If unsupported exclude filter, skip — worker's isExcluded handles it + } + } + + // activeOnly + const activeSql = spec.activeOnly ? ' AND active = 1' : ''; + + // Wrap union in outer SELECT for dedup, filtering, and paging + const innerSql = unionParts.join(' UNION ALL '); + + let sql = `SELECT DISTINCT code, display, definition, active FROM (${innerSql})` + + ` WHERE 1=1${activeSql}${excludeSql}` + + ` ORDER BY code`; + + // Paging + if (spec.count != null && spec.count > 0) { + sql += ` LIMIT ${Math.max(0, spec.count)}`; + } + if (spec.offset != null && spec.offset > 0) { + sql += ` OFFSET ${Math.max(0, spec.offset)}`; + } + + // Return a synchronous generator backed by better-sqlite3's lazy cursor + const self = this; + return (function* () { + const stmt = syncDb.prepare(sql); + for (const row of stmt.iterate(allParams)) { + yield { + code: row.code, + display: row.display, + system: sys, + version: ver, + isAbstract: false, + isInactive: row.active !== 1, + isDeprecated: false, + status: row.active === 1 ? 'active' : 'inactive', + definition: row.definition || null, + designations: [], + properties: null, + extensions: null, + }; + } + })(); + } + + system() { + return this.meta.baseUri || this.meta.canonicalUri || ''; + } + + version() { + const outputMode = this.runtime?.versioning?.output || 'canonical'; + if (outputMode === 'version') { + return this.meta.version || this.meta.canonicalUri || null; + } + return this.meta.canonicalUri || this.meta.version || null; + } + + name() { + return this.meta.name || this.system(); + } + + description() { + return `${this.name()} (${this.meta.version || 'unknown version'})`; + } + + async totalCount() { + return this.meta.totalConcepts || 0; + } + + hasParents() { + return !!this.meta.hierarchyPropertyId; + } + + defLang() { + return this.runtime.languages?.default || this.meta.defaultLanguage || 'en'; + } + + versionAlgorithm() { + return this.runtime.versioning?.algorithm || 'string'; + } + + versionIsMoreDetailed(checkVersion, actualVersion) { + if (!checkVersion || !actualVersion) return false; + + const partialMatch = this.runtime.versioning?.partialMatch !== false; + if (!partialMatch) { + return checkVersion === actualVersion; + } + + return actualVersion.startsWith(checkVersion); + } + + async code(context) { + const ctxt = await this.#ensureContext(context); + return ctxt ? ctxt.code : null; + } + + async display(context) { + const ctxt = await this.#ensureContext(context); + if (!ctxt) return null; + + const supplementDisplay = this._displayFromSupplements(ctxt.code); + if (supplementDisplay) { + return supplementDisplay; + } + + return ctxt.display || ctxt.code; + } + + async definition(context) { + const ctxt = await this.#ensureContext(context); + return ctxt ? ctxt.definition : null; + } + + async isAbstract(context) { + await this.#ensureContext(context); + const abstractCfg = this.runtime.status?.abstract; + if (abstractCfg?.source === 'constant') { + return !!abstractCfg.value; + } + return false; + } + + async isInactive(context) { + const ctxt = await this.#ensureContext(context); + if (!ctxt) return false; + const inactiveCfg = this.runtime.status?.inactive; + if (inactiveCfg?.source === 'concept.active') { + return inactiveCfg.invert === true ? !ctxt.active : !!ctxt.active; + } + return !ctxt.active; + } + + async isDeprecated(context) { + await this.#ensureContext(context); + const depCfg = this.runtime.status?.deprecated; + if (depCfg?.source === 'constant') { + return !!depCfg.value; + } + return false; + } + + async getStatus(context) { + const ctxt = await this.#ensureContext(context); + if (!ctxt) return null; + + const statusValue = await this.#statusValueForConcept(ctxt.conceptId); + if (statusValue) { + return statusValue; + } + + return ctxt.active ? 'active' : 'inactive'; + } + + async #statusValueForConcept(conceptId) { + const statusPropertyCode = this.runtime?.status?.statusProperty; + if (!statusPropertyCode) { + return null; + } + + const propDef = await this.#resolvePropertyDef(statusPropertyCode); + if (!propDef?.property_id) { + return null; + } + + await this.#ensureStatusCache(propDef.property_id); + return this.statusCache?.values?.get(conceptId) || null; + } + + async #ensureStatusCache(propertyId) { + if (this.statusCache?.propertyId === propertyId && this.statusCache.values instanceof Map) { + return; + } + + if (this.sharedState && this.sharedState.statusByPropertyId instanceof Map) { + const existing = this.sharedState.statusByPropertyId.get(propertyId); + if (existing instanceof Map) { + this.statusCache = { propertyId, values: existing }; + return; + } + + if (!(this.sharedState.statusLoadPromises instanceof Map)) { + this.sharedState.statusLoadPromises = new Map(); + } + + let promise = this.sharedState.statusLoadPromises.get(propertyId); + if (!promise) { + promise = this.#loadStatusMap(propertyId) + .then((values) => { + this.sharedState.statusByPropertyId.set(propertyId, values); + this.sharedState.statusLoadPromises.delete(propertyId); + return values; + }) + .catch((error) => { + this.sharedState.statusLoadPromises.delete(propertyId); + throw error; + }); + this.sharedState.statusLoadPromises.set(propertyId, promise); + } + + const values = await promise; + this.statusCache = { propertyId, values }; + return; + } + + const values = await this.#loadStatusMap(propertyId); + this.statusCache = { propertyId, values }; + } + + async #loadStatusMap(propertyId) { + const rows = await all( + this.db, + `SELECT source_concept_id, + COALESCE(value_text, value_raw) AS value + FROM concept_literal + WHERE property_id = ? + AND active = 1 + AND COALESCE(value_text, value_raw) IS NOT NULL + ORDER BY source_concept_id, literal_id`, + [propertyId] + ); + + const values = new Map(); + for (const row of rows) { + if (!values.has(row.source_concept_id) && row.value != null) { + values.set(row.source_concept_id, row.value); + } + } + return values; + } + + async designations(context, displays) { + const ctxt = await this.#ensureContext(context); + if (!ctxt) return; + + // Keep legacy behavior where a primary display is always available as a designation. + displays.addDesignation( + true, + ctxt.active ? 'active' : 'inactive', + this.defLang(), + CodeSystem.makeUseForDisplay(), + ctxt.display || ctxt.code + ); + + const designationTableRef = this.meta?.designationOrderIndex + ? 'designation INDEXED BY idx_designation_concept_pref_term' + : 'designation'; + const rows = await all( + this.db, + `SELECT language_code, use_code, term, preferred, active + FROM ${designationTableRef} + WHERE concept_id = ? + ORDER BY preferred DESC, term`, + [ctxt.conceptId] + ); + + for (const row of rows) { + displays.addDesignation( + row.preferred === 1, + row.active === 1 ? 'active' : 'inactive', + row.language_code || this.defLang(), + useFromDesignation(row, this.runtime, this.system()), + row.term + ); + } + + this._listSupplementDesignations(ctxt.code, displays); + } + + async properties(context) { + const ctxt = await this.#ensureContext(context); + if (!ctxt) return []; + + const props = []; + props.push({ code: 'inactive', valueBoolean: !ctxt.active }); + + if (!this.meta.hierarchyPropertyId) { + return props; + } + + const parentRows = await all( + this.db, + `SELECT p.code AS target_code + FROM concept_link l + JOIN concept p ON p.concept_id = l.target_concept_id + WHERE l.source_concept_id = ? + AND l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1`, + [ctxt.conceptId, this.meta.hierarchyPropertyId, this.meta.hierarchyEdgeSetId] + ); + + const parentPropCode = this.runtime.hierarchy?.parentPropertyCode || 'parent'; + for (const row of parentRows) { + props.push({ code: parentPropCode, valueCode: row.target_code }); + } + + return props; + } + + async locate(code) { + if (!code) { + return { context: null, message: 'Empty code' }; + } + + const row = await get( + this.db, + `SELECT concept_id, code, display, definition, active + FROM concept + WHERE cs_id = ? AND code = ?`, + [this.meta.csId, code] + ); + + if (!row) { + return { context: null, message: undefined }; + } + + return { + context: new SqliteRuntimeV0Context(row.concept_id, row.code, row.display, row.definition, row.active === 1), + message: null + }; + } + + async locateMany(codes, allAltCodes = false) { + void allAltCodes; + const out = new Map(); + if (!Array.isArray(codes) || codes.length === 0) { + return out; + } + + const normalized = []; + const seen = new Set(); + for (const raw of codes) { + const code = String(raw || ''); + if (!code || seen.has(code)) { + continue; + } + seen.add(code); + normalized.push(code); + } + if (normalized.length === 0) { + return out; + } + + const batchSize = 500; + for (let i = 0; i < normalized.length; i += batchSize) { + const batch = normalized.slice(i, i + batchSize); + const placeholders = batch.map(() => '?').join(', '); + const rows = await all( + this.db, + `SELECT concept_id, code, display, definition, active + FROM concept + WHERE cs_id = ? + AND code IN (${placeholders})`, + [this.meta.csId, ...batch] + ); + + for (const row of rows) { + out.set(row.code, { + context: new SqliteRuntimeV0Context( + row.concept_id, + row.code, + row.display, + row.definition, + row.active === 1 + ), + message: null + }); + } + } + + return out; + } + + // Backward-compat alias while workers migrate to locateMany. + async locateBatch(codes, allAltCodes = false) { + return this.locateMany(codes, allAltCodes); + } + + async locateIsA(code, parent, disallowParent = false) { + const located = await this.locate(code); + if (!located.context) { + return located; + } + + const parentLocated = await this.locate(parent); + if (!parentLocated.context) { + return { context: null, message: `Parent concept '${parent}' not found` }; + } + + const isA = await this.#isA(parentLocated.context.conceptId, located.context.conceptId, !disallowParent); + if (!isA) { + return { context: null, message: `Code '${code}' is not in hierarchy of '${parent}'` }; + } + + return located; + } + + async iterator(code) { + if (!this.meta.hierarchyPropertyId) { + return this.iteratorAll(); + } + + if (!code) { + if (this.runtime?.iteration?.rootMode === 'all') { + return this.iteratorAll(); + } + return new SqliteRuntimeV0QueryIterator('roots'); + } + + const ctxt = await this.#ensureContext(code); + if (!ctxt) return null; + + if (this.runtime?.iteration?.children === false) { + return new SqliteRuntimeV0Iterator([]); + } + + const hasChildren = await this.#conceptHasHierarchyChildren(ctxt.conceptId); + if (!hasChildren) { + return new SqliteRuntimeV0Iterator([]); + } + + return new SqliteRuntimeV0QueryIterator('children', { targetConceptId: ctxt.conceptId }); + } + + async #conceptHasHierarchyChildren(conceptId) { + if (!conceptId || !this.meta.hierarchyPropertyId) { + return false; + } + const targets = await this.#getHierarchyChildTargetSet(); + return targets.has(conceptId); + } + + async #getHierarchyChildTargetSet() { + if (!this.meta.hierarchyPropertyId) { + return new Set(); + } + + const key = `${this.meta.hierarchyPropertyId}:${this.meta.hierarchyEdgeSetId}`; + if (this.sharedState && this.sharedState.childTargetSets instanceof Map) { + const existing = this.sharedState.childTargetSets.get(key); + if (existing instanceof Set) { + return existing; + } + + if (!(this.sharedState.childTargetLoadPromises instanceof Map)) { + this.sharedState.childTargetLoadPromises = new Map(); + } + + let promise = this.sharedState.childTargetLoadPromises.get(key); + if (!promise) { + promise = this.#loadHierarchyChildTargetSet( + this.meta.hierarchyPropertyId, + this.meta.hierarchyEdgeSetId + ).then((set) => { + this.sharedState.childTargetSets.set(key, set); + this.sharedState.childTargetLoadPromises.delete(key); + return set; + }).catch((error) => { + this.sharedState.childTargetLoadPromises.delete(key); + throw error; + }); + this.sharedState.childTargetLoadPromises.set(key, promise); + } + return promise; + } + + return this.#loadHierarchyChildTargetSet(this.meta.hierarchyPropertyId, this.meta.hierarchyEdgeSetId); + } + + async #loadHierarchyChildTargetSet(propertyId, edgeSetId) { + const rows = await all( + this.db, + `SELECT DISTINCT target_concept_id + FROM concept_link + WHERE property_id = ? + AND edge_set_id = ? + AND active = 1`, + [propertyId, edgeSetId] + ); + return new Set(rows.map((r) => r.target_concept_id)); + } + + async iteratorAll() { + return new SqliteRuntimeV0QueryIterator('all'); + } + + async nextContext(iteratorContext) { + if (!iteratorContext) { + return null; + } + + if (this.#isQueryIterator(iteratorContext)) { + while (iteratorContext.cursor >= iteratorContext.rows.length) { + if (iteratorContext.done) { + return null; + } + await this.#loadNextIteratorPage(iteratorContext); + } + + const context = iteratorContext.rows[iteratorContext.cursor]; + iteratorContext.cursor += 1; + return context || null; + } + + if (!(iteratorContext instanceof SqliteRuntimeV0Iterator)) { + return null; + } + + if (iteratorContext.cursor >= iteratorContext.codes.length) { + return null; + } + + const entry = iteratorContext.codes[iteratorContext.cursor]; + iteratorContext.cursor += 1; + if (entry instanceof SqliteRuntimeV0Context) { + return entry; + } + + const located = await this.locate(entry); + return located.context; + } + + async subsumesTest(codeA, codeB) { + const a = await this.#ensureContext(codeA); + const b = await this.#ensureContext(codeB); + if (!a || !b) return 'not-subsumed'; + + if (a.code === b.code) return 'equivalent'; + + if (await this.#isA(a.conceptId, b.conceptId, true)) return 'subsumes'; + if (await this.#isA(b.conceptId, a.conceptId, true)) return 'subsumed-by'; + return 'not-subsumed'; + } + + async doesFilter(prop, op, _value) { + void _value; + if (!prop || !op) return false; + + const propCfg = this.runtime.filters?.[prop]; + if (propCfg?.operators && Array.isArray(propCfg.operators)) { + return propCfg.operators.includes(op); + } + + if (prop === 'concept') { + return ['=', 'is-a', 'descendent-of', 'in'].includes(op); + } + if (prop === 'code' && op === 'regex') { + return true; + } + + const propertyCfg = await this.#resolvePropertyFilterConfig(prop); + if (propertyCfg?.operators && Array.isArray(propertyCfg.operators)) { + return propertyCfg.operators.includes(op); + } + + return false; + } + + async getPrepContext(iterate) { + return new FilterExecutionContext(iterate); + } + + async searchFilter(filterContext, filter, _sort) { + void _sort; + const searchText = typeof filter === 'string' + ? filter + : (filter && typeof filter.filter === 'string' ? filter.filter : null); + + if (!searchText || !searchText.trim()) { + throw new Error('Invalid search filter'); + } + + const searchCfg = normalizedSearchConfig(this.runtime.search); + let codes = []; + + if (this.#canUseFtsSearch(searchCfg)) { + try { + codes = await this.#searchCodesWithFts(searchText, searchCfg); + } catch (error) { + if (!searchCfg.likeFallback?.enabled) { + throw error; + } + codes = await this.#searchCodesWithLike(searchText, searchCfg); + } + } else { + codes = await this.#searchCodesWithLike(searchText, searchCfg); + } + + filterContext.filters.push( + new SqliteRuntimeV0FilterSet(`search:${searchText}`, codes, true) + ); + } + + async filter(filterContext, prop, op, value) { + if (prop === 'code' && op === 'regex') { + const re = new RegExp(`^${value}$`); + const rows = await all( + this.db, + `SELECT code + FROM concept + WHERE cs_id = ? + ORDER BY code`, + [this.meta.csId] + ); + const codes = rows.map(r => r.code).filter(c => re.test(c)); + filterContext.filters.push(new SqliteRuntimeV0FilterSet(`code-regex:${value}`, codes, true)); + return; + } + + if (prop !== 'concept') { + const propertyCfg = await this.#resolvePropertyFilterConfig(prop); + if (!propertyCfg) { + throw new Error(`Unsupported sqlite runtime filter property '${prop}'`); + } + if (!propertyCfg.operators.includes(op)) { + throw new Error(`Unsupported sqlite runtime filter operator '${op}' for property '${prop}'`); + } + await this.#filterByProperty(filterContext, propertyCfg, op, value); + return; + } + + if (op === '=') { + if (this.#useMembershipPredicate(filterContext)) { + filterContext.filters.push(new SqliteRuntimeV0PredicateFilter( + `concept=${value}`, + 'concept-equals', + { code: value }, + true + )); + } else { + const located = await this.locate(value); + const codes = located.context ? [value] : []; + filterContext.filters.push(new SqliteRuntimeV0FilterSet(`concept=${value}`, codes, true)); + } + return; + } + + if (op === 'is-a' || op === 'descendent-of') { + const includeSelf = op === 'is-a' + ? (this.runtime?.filters?.concept?.isAIncludesSelf !== false) + : false; + if (this.#useMembershipPredicate(filterContext)) { + const parent = await this.locate(value); + filterContext.filters.push(new SqliteRuntimeV0PredicateFilter( + `concept-${op}:${value}`, + 'concept-hierarchy', + { + parentCode: value, + ancestorId: parent.context ? parent.context.conceptId : null, + includeSelf, + missingMessage: parent.context ? null : `Parent concept '${value}' not found` + }, + true + )); + } else { + const parent = await this.locate(value); + if (parent.context && this.meta.closureRows > 0 && this.meta.useClosure) { + filterContext.filters.push(new SqliteRuntimeV0PagedDescendantFilter( + `concept-${op}:${value}`, + parent.context.conceptId, + includeSelf + )); + } else { + const codes = await this.#descendantCodes(value, includeSelf); + filterContext.filters.push(new SqliteRuntimeV0FilterSet(`concept-${op}:${value}`, codes, true)); + } + } + return; + } + + if (op === 'in') { + const url = resolveInValueSetUrl(this.system(), value, this.runtime); + if (this.#useMembershipPredicate(filterContext)) { + filterContext.filters.push(new SqliteRuntimeV0PredicateFilter( + `concept-in:${value}`, + 'concept-in', + { valueSetUrl: url, rawValue: value }, + true + )); + } else { + const rows = await all( + this.db, + `SELECT c.code + FROM value_set v + JOIN value_set_member m ON m.vs_id = v.vs_id + JOIN concept c ON c.concept_id = m.concept_id + WHERE v.cs_id = ? + AND v.url = ? + AND m.active = 1 + ORDER BY code`, + [this.meta.csId, url] + ); + + filterContext.filters.push(new SqliteRuntimeV0FilterSet(`concept-in:${value}`, rows.map(r => r.code), true)); + } + return; + } + + throw new Error(`Unsupported sqlite runtime filter operator '${op}' for concept`); + } + + async executeFilters(filterContext) { + return filterContext.filters || []; + } + + capabilities() { + return { + filterPage: true + }; + } + + async filterPage(filterContext, set, count = 256) { + void filterContext; + const pageSize = Math.max(1, Number.isFinite(count) ? Math.floor(count) : 256); + + if (this.#isPredicateFilter(set)) { + return []; + } + + if (this.#isPagedDescendantFilter(set)) { + let start = set.cursor + 1; + while (!set.done && (set.rows.length - start) < pageSize) { + await this.#loadNextDescendantPage(set); + start = set.cursor + 1; + } + if (start >= set.rows.length) { + return []; + } + const end = Math.min(set.rows.length, start + pageSize); + const page = set.rows.slice(start, end); + set.cursor = end - 1; + return page; + } + + if (set instanceof SqliteRuntimeV0FilterSet) { + const start = set.cursor + 1; + if (start >= set.codes.length) { + return []; + } + const end = Math.min(set.codes.length, start + pageSize); + const codes = set.codes.slice(start, end); + set.cursor = end - 1; + return this.#batchLoadContextsByCodes(codes); + } + + return null; + } + + async filterSize(_filterContext, set) { + if (this.#isPredicateFilter(set)) { + return 0; + } + if (this.#isPagedDescendantFilter(set)) { + return set.done ? set.rows.length : 0; + } + return set.codes.length; + } + + async filtersNotClosed(filterContext) { + return (filterContext.filters || []).some(f => !f.closed); + } + + async filterMore(_filterContext, set) { + if (this.#isPredicateFilter(set)) { + return false; + } + if (this.#isPagedDescendantFilter(set)) { + set.cursor += 1; + while (set.cursor >= set.rows.length) { + if (set.done) { + return false; + } + await this.#loadNextDescendantPage(set); + } + return true; + } + set.cursor += 1; + return set.cursor < set.codes.length; + } + + async filterConcept(_filterContext, set) { + if (this.#isPredicateFilter(set)) { + return null; + } + if (this.#isPagedDescendantFilter(set)) { + if (set.cursor < 0 || set.cursor >= set.rows.length) { + return null; + } + return set.rows[set.cursor]; + } + if (set.cursor < 0 || set.cursor >= set.codes.length) { + return null; + } + const located = await this.locate(set.codes[set.cursor]); + return located.context; + } + + async filterLocate(_filterContext, set, code) { + if (this.#isPredicateFilter(set)) { + return this.#filterLocatePredicate(set, code); + } + if (this.#isPagedDescendantFilter(set)) { + const located = await this.locate(code); + if (!located.context) { + return `Code '${code}' not found in filter set`; + } + const ok = await this.#isA(set.ancestorId, located.context.conceptId, !!set.includeSelf); + return ok ? located.context : `Code '${code}' not found in filter set`; + } + if (!set.has(code)) { + return `Code '${code}' not found in filter set`; + } + const located = await this.locate(code); + return located.context || `Code '${code}' not found`; + } + + async filterCheck(_filterContext, set, concept) { + const ctxt = await this.#ensureContext(concept); + if (!ctxt) return false; + if (this.#isPredicateFilter(set)) { + return this.#predicateMatchesContext(set, ctxt); + } + if (this.#isPagedDescendantFilter(set)) { + return this.#isA(set.ancestorId, ctxt.conceptId, !!set.includeSelf); + } + return set.has(ctxt.code); + } + + async buildKnownValueSet(_url, _version) { + void _url; + void _version; + return null; + } + + async #resolvePropertyDef(propertyCode) { + if (!propertyCode) return null; + if (this.propertyDefs.has(propertyCode)) { + return this.propertyDefs.get(propertyCode); + } + + const row = await get( + this.db, + `SELECT property_id, property_code, value_kind + FROM property_def + WHERE cs_id = ? + AND property_code = ? + LIMIT 1`, + [this.meta.csId, propertyCode] + ); + const result = row || null; + this.propertyDefs.set(propertyCode, result); + return result; + } + + async #resolvePropertyFilterConfig(propertyCode) { + if (!propertyCode) return null; + + const filtersCfg = this.runtime.filters?.properties; + if (!filtersCfg) { + const propertyDef = await this.#resolvePropertyDef(propertyCode); + if (!propertyDef) { + return null; + } + return { + propertyId: propertyDef.property_id, + propertyCode: propertyDef.property_code, + operators: ['=', 'in'], + sources: inferSourcesFromValueKind(propertyDef.value_kind), + linkMatch: 'code-only', + value: {}, + specialHandler: null + }; + } + + const aliases = filtersCfg.aliases || {}; + const rawCode = String(propertyCode); + const aliasTarget = aliases[rawCode] ?? aliases[rawCode.toLowerCase()]; + const resolvedCode = aliasTarget || rawCode; + + const byCode = filtersCfg.byCode || {}; + const specific = byCode[resolvedCode] || byCode[rawCode] || null; + if (!specific && filtersCfg.allPropertiesFilterable !== true) { + return null; + } + + const propertyDef = await this.#resolvePropertyDef(resolvedCode); + if (!propertyDef) { + return null; + } + + const operators = Array.isArray(specific?.operators) && specific.operators.length > 0 + ? specific.operators + : (Array.isArray(filtersCfg.defaultOperators) && filtersCfg.defaultOperators.length > 0 + ? filtersCfg.defaultOperators + : ['=']); + + const defaultSources = Array.isArray(filtersCfg.defaultSources) + ? filtersCfg.defaultSources + : inferSourcesFromValueKind(propertyDef.value_kind); + const sources = Array.isArray(specific?.sources) && specific.sources.length > 0 + ? specific.sources + : defaultSources; + + const cleanedSources = dedupSources(sources, propertyDef.value_kind); + const linkMatch = specific?.linkMatch || filtersCfg.defaultLinkMatch || 'code-only'; + const value = { + ...(filtersCfg.defaultValue || {}), + ...(specific?.value || {}) + }; + + return { + propertyId: propertyDef.property_id, + propertyCode: resolvedCode, + operators, + sources: cleanedSources, + linkMatch, + value, + specialHandler: specific?.specialHandler || null + }; + } + + async #filterByProperty(filterContext, propertyCfg, op, value) { + const filterName = `property-${propertyCfg.propertyCode}-${op}:${value}`; + + if (propertyCfg.specialHandler) { + const codes = await this.#runSpecialPropertyHandler(propertyCfg, op, value); + filterContext.filters.push(new SqliteRuntimeV0FilterSet(filterName, codes, true)); + return; + } + + if (this.#useMembershipPredicate(filterContext)) { + const predicate = await this.#buildPropertyPredicateFilter(filterName, propertyCfg, op, value); + if (predicate) { + filterContext.filters.push(predicate); + return; + } + } + + if (op === '=') { + const candidates = normalizedFilterCandidates(value, propertyCfg.value); + if (candidates.length === 0) { + filterContext.filters.push(new SqliteRuntimeV0FilterSet(filterName, [], true)); + return; + } + const codes = await this.#propertyEqualsCodes(propertyCfg, candidates); + filterContext.filters.push(new SqliteRuntimeV0FilterSet(filterName, codes, true)); + return; + } + + if (op === 'in') { + const members = splitFilterValueList(value); + const aggregate = new Set(); + for (const member of members) { + const candidates = normalizedFilterCandidates(member, propertyCfg.value); + if (candidates.length === 0) continue; + const codes = await this.#propertyEqualsCodes(propertyCfg, candidates); + for (const code of codes) { + aggregate.add(code); + } + } + filterContext.filters.push(new SqliteRuntimeV0FilterSet(filterName, Array.from(aggregate).sort(), true)); + return; + } + + if (op === 'exists') { + const codes = await this.#propertyExistsCodes(propertyCfg, value); + filterContext.filters.push(new SqliteRuntimeV0FilterSet(filterName, codes, true)); + return; + } + + if (op === 'regex') { + const codes = await this.#propertyRegexCodes(propertyCfg, value); + filterContext.filters.push(new SqliteRuntimeV0FilterSet(filterName, codes, true)); + return; + } + + throw new Error(`Unsupported sqlite runtime property operator '${op}'`); + } + + async #buildPropertyPredicateFilter(filterName, propertyCfg, op, value) { + const caseSensitive = propertyCfg.value?.caseSensitive === true; + + if (op === '=') { + const candidates = normalizedFilterCandidates(value, propertyCfg.value); + if (candidates.length === 0) { + return new SqliteRuntimeV0FilterSet(filterName, [], true); + } + return new SqliteRuntimeV0PredicateFilter( + filterName, + 'property-filter', + { propertyCfg, op, candidates, caseSensitive }, + true + ); + } + + if (op === 'in') { + const members = splitFilterValueList(value); + const aggregate = new Set(); + for (const member of members) { + const candidates = normalizedFilterCandidates(member, propertyCfg.value); + for (const candidate of candidates) { + aggregate.add(candidate); + } + } + const values = Array.from(aggregate); + if (values.length === 0) { + return new SqliteRuntimeV0FilterSet(filterName, [], true); + } + return new SqliteRuntimeV0PredicateFilter( + filterName, + 'property-filter', + { propertyCfg, op, candidates: values, caseSensitive }, + true + ); + } + + if (op === 'exists') { + const expectExists = String(value ?? 'true').toLowerCase() !== 'false'; + return new SqliteRuntimeV0PredicateFilter( + filterName, + 'property-filter', + { propertyCfg, op, expectExists }, + true + ); + } + + if (op === 'regex') { + try { + new RegExp(String(value || '')); + } catch (error) { + throw new Error(`Invalid regex '${value}': ${error.message}`); + } + return new SqliteRuntimeV0PredicateFilter( + filterName, + 'property-filter', + { propertyCfg, op, pattern: String(value || '') }, + true + ); + } + + return null; + } + + async #propertyEqualsCodes(propertyCfg, candidates) { + const codeSet = new Set(); + const caseSensitive = propertyCfg.value?.caseSensitive === true; + + if (propertyCfg.sources.includes('literal')) { + const rows = await this.#propertyLiteralEqualsRows(propertyCfg.propertyId, candidates, caseSensitive); + for (const row of rows) { + codeSet.add(row.code); + } + } + + if (propertyCfg.sources.includes('link')) { + const rows = await this.#propertyLinkEqualsRows(propertyCfg, candidates, caseSensitive); + for (const row of rows) { + codeSet.add(row.code); + } + } + + return Array.from(codeSet).sort(); + } + + async #propertyRegexCodes(propertyCfg, pattern) { + let regex; + try { + regex = new RegExp(String(pattern || '')); + } catch (error) { + throw new Error(`Invalid regex '${pattern}': ${error.message}`); + } + + const codeSet = new Set(); + + if (propertyCfg.sources.includes('literal')) { + const rows = await all( + this.db, + `SELECT c.code AS code, + COALESCE(cl.value_text, cl.value_raw) AS value + FROM concept_literal cl + JOIN concept c ON c.concept_id = cl.source_concept_id + WHERE cl.property_id = ? + AND cl.active = 1 + AND c.cs_id = ? + AND COALESCE(cl.value_text, cl.value_raw) IS NOT NULL`, + [propertyCfg.propertyId, this.meta.csId] + ); + for (const row of rows) { + if (regex.test(row.value)) { + codeSet.add(row.code); + } + } + } + + if (propertyCfg.sources.includes('link')) { + const rows = await all( + this.db, + `SELECT src.code AS code, + tgt.code AS target_code, + tgt.display AS target_display + FROM concept_link l + JOIN concept src ON src.concept_id = l.source_concept_id + JOIN concept tgt ON tgt.concept_id = l.target_concept_id + WHERE l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + AND src.cs_id = ?`, + [propertyCfg.propertyId, this.meta.hierarchyEdgeSetId, this.meta.csId] + ); + for (const row of rows) { + const codeMatch = row.target_code && regex.test(row.target_code); + const displayMatch = propertyCfg.linkMatch === 'code-or-display' && + row.target_display && regex.test(row.target_display); + if (codeMatch || displayMatch) { + codeSet.add(row.code); + } + } + } + + return Array.from(codeSet).sort(); + } + + async #propertyLiteralEqualsRows(propertyId, candidates, caseSensitive) { + if (!propertyId || !Array.isArray(candidates) || candidates.length === 0) { + return []; + } + + const normalized = candidates.map(v => String(v)); + const placeholders = normalized.map(() => '?').join(', '); + const textPredicate = caseSensitive + ? `cl.value_text IN (${placeholders})` + : `cl.value_text COLLATE NOCASE IN (${placeholders})`; + const rawPredicate = caseSensitive + ? `cl.value_raw IN (${placeholders})` + : `cl.value_raw COLLATE NOCASE IN (${placeholders})`; + + return all( + this.db, + `WITH matched_concepts AS ( + SELECT cl.source_concept_id AS concept_id + FROM concept_literal cl + WHERE cl.property_id = ? + AND cl.active = 1 + AND cl.value_text IS NOT NULL + AND ${textPredicate} + UNION + SELECT cl.source_concept_id AS concept_id + FROM concept_literal cl + WHERE cl.property_id = ? + AND cl.active = 1 + AND cl.value_text IS NULL + AND cl.value_raw IS NOT NULL + AND ${rawPredicate} + ) + SELECT DISTINCT c.code AS code + FROM matched_concepts m + JOIN concept c ON c.concept_id = m.concept_id + WHERE c.cs_id = ? + ORDER BY c.code`, + [propertyId, ...normalized, propertyId, ...normalized, this.meta.csId] + ); + } + + async #propertyLinkEqualsRows(propertyCfg, candidates, caseSensitive) { + if (!propertyCfg?.propertyId || !Array.isArray(candidates) || candidates.length === 0) { + return []; + } + + const normalized = candidates.map(v => String(v)); + const placeholders = normalized.map(() => '?').join(', '); + const codeExpr = caseSensitive + ? `code IN (${placeholders})` + : `code COLLATE NOCASE IN (${placeholders})`; + const displayExpr = caseSensitive + ? `display IN (${placeholders})` + : `display COLLATE NOCASE IN (${placeholders})`; + + const targetSql = [ + `SELECT concept_id + FROM concept + WHERE cs_id = ? + AND ${codeExpr}` + ]; + const params = [this.meta.csId, ...normalized]; + + if (propertyCfg.linkMatch === 'code-or-display') { + targetSql.push( + `SELECT concept_id + FROM concept + WHERE cs_id = ? + AND ${displayExpr}` + ); + params.push(this.meta.csId, ...normalized); + } + + return all( + this.db, + `WITH matched_targets AS ( + ${targetSql.join('\nUNION\n')} + ) + SELECT DISTINCT src.code AS code + FROM concept_link l + JOIN matched_targets mt ON mt.concept_id = l.target_concept_id + JOIN concept src ON src.concept_id = l.source_concept_id + WHERE src.cs_id = ? + AND l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + ORDER BY src.code`, + [...params, this.meta.csId, propertyCfg.propertyId, this.meta.hierarchyEdgeSetId] + ); + } + + async #propertyExistsCodes(propertyCfg, value) { + const expectExists = String(value ?? 'true').toLowerCase() !== 'false'; + const codeSet = new Set(); + + if (propertyCfg.sources.includes('literal')) { + const rows = await all( + this.db, + `SELECT DISTINCT c.code AS code + FROM concept_literal cl + JOIN concept c ON c.concept_id = cl.source_concept_id + WHERE cl.property_id = ? + AND cl.active = 1 + AND c.cs_id = ?`, + [propertyCfg.propertyId, this.meta.csId] + ); + for (const row of rows) { + codeSet.add(row.code); + } + } + + if (propertyCfg.sources.includes('link')) { + const rows = await all( + this.db, + `SELECT DISTINCT src.code AS code + FROM concept_link l + JOIN concept src ON src.concept_id = l.source_concept_id + WHERE l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + AND src.cs_id = ?`, + [propertyCfg.propertyId, this.meta.hierarchyEdgeSetId, this.meta.csId] + ); + for (const row of rows) { + codeSet.add(row.code); + } + } + + if (expectExists) { + return Array.from(codeSet).sort(); + } + + const allRows = await all( + this.db, + `SELECT code + FROM concept + WHERE cs_id = ?`, + [this.meta.csId] + ); + return allRows.map(r => r.code).filter(code => !codeSet.has(code)).sort(); + } + + async #runSpecialPropertyHandler(propertyCfg, op, value) { + const handler = propertyCfg.specialHandler; + if (handler && typeof handler === 'object' && handler.kind === 'derived-link-filter') { + return this.#runDerivedLinkPropertyHandler(propertyCfg, handler, op, value); + } + throw new Error(`Unsupported sqlite runtime special handler '${JSON.stringify(handler)}'`); + } + + async #runDerivedLinkPropertyHandler(propertyCfg, handler, op, value) { + if (!propertyCfg?.propertyId || !handler) { + return []; + } + + if (!['=', 'in'].includes(op)) { + throw new Error(`Unsupported sqlite runtime property operator '${op}' for derived-link-filter`); + } + + const values = this.#normalizedFilterValuesForSpecialHandler(op, value, propertyCfg.value); + if (values.length === 0) { + return []; + } + + const seedCfg = handler.seed || {}; + const seedCodes = new Set(); + const directPrefixes = Array.isArray(seedCfg.directCodePrefixes) + ? seedCfg.directCodePrefixes.map(v => String(v || '')).filter(Boolean) + : []; + const allowAnyDirect = seedCfg.allowAnyDirect === true; + + for (const raw of values) { + if (!raw) continue; + if (allowAnyDirect || directPrefixes.some(prefix => raw.startsWith(prefix))) { + seedCodes.add(raw); + } + } + + let inversePropertyId = null; + if (seedCfg.useCurrentPropertyAsInverse === true) { + inversePropertyId = propertyCfg.propertyId; + } else if (seedCfg.inversePropertyCode) { + const inversePropertyDef = await this.#resolvePropertyDef(seedCfg.inversePropertyCode); + inversePropertyId = inversePropertyDef?.property_id || null; + } + + if (inversePropertyId) { + const reverseMatches = await this.#sourceCodesForTargetCodes(inversePropertyId, values); + for (const code of reverseMatches) { + seedCodes.add(code); + } + } + + if (seedCodes.size === 0) { + return []; + } + + const projectionCfg = handler.projection || {}; + const projectionPropertyCode = projectionCfg.propertyCode; + if (!projectionPropertyCode) { + throw new Error('derived-link-filter handler requires projection.propertyCode'); + } + const projectionPropertyDef = await this.#resolvePropertyDef(projectionPropertyCode); + if (!projectionPropertyDef) { + return []; + } + + const side = projectionCfg.side === 'source' ? 'source' : 'target'; + return this.#codesFromSourceCodesViaProperty( + projectionPropertyDef.property_id, + Array.from(seedCodes), + side + ); + } + + #normalizedFilterValuesForSpecialHandler(op, value, valueCfg) { + const rawValues = op === 'in' + ? splitFilterValueList(value) + : [String(value ?? '').trim()]; + + const out = new Set(); + for (const raw of rawValues) { + const normalized = normalizedFilterCandidates(raw, valueCfg); + for (const entry of normalized) { + if (entry) out.add(entry); + } + } + return Array.from(out); + } + + async #sourceCodesForTargetCodes(propertyId, targetCodes) { + if (!propertyId || !Array.isArray(targetCodes) || targetCodes.length === 0) { + return []; + } + + const placeholders = targetCodes.map(() => '?').join(', '); + const rows = await all( + this.db, + `SELECT DISTINCT src.code AS source_code + FROM concept_link l + JOIN concept src ON src.concept_id = l.source_concept_id + JOIN concept tgt ON tgt.concept_id = l.target_concept_id + WHERE src.cs_id = ? + AND l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + AND tgt.code IN (${placeholders})`, + [this.meta.csId, propertyId, this.meta.hierarchyEdgeSetId, ...targetCodes] + ); + return rows.map(row => row.source_code).filter(Boolean); + } + + async #codesFromSourceCodesViaProperty(propertyId, sourceCodes, resultSide) { + if (!propertyId || !Array.isArray(sourceCodes) || sourceCodes.length === 0) { + return []; + } + + const placeholders = sourceCodes.map(() => '?').join(', '); + const rows = await all( + this.db, + `SELECT DISTINCT src.code AS source_code, + tgt.code AS target_code + FROM concept_link l + JOIN concept src ON src.concept_id = l.source_concept_id + JOIN concept tgt ON tgt.concept_id = l.target_concept_id + WHERE src.cs_id = ? + AND l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + AND src.code IN (${placeholders})`, + [this.meta.csId, propertyId, this.meta.hierarchyEdgeSetId, ...sourceCodes] + ); + + const picked = rows.map((row) => (resultSide === 'source' ? row.source_code : row.target_code)); + return Array.from(new Set(picked.filter(Boolean))).sort(); + } + + #canUseFtsSearch(searchCfg) { + if (searchCfg.mode !== 'fts-broad') { + return false; + } + const tables = searchCfg.ftsTables || {}; + const available = this.meta.searchFtsTables || {}; + + for (const source of searchCfg.sources) { + const table = tables[source]; + if (!table) { + return false; + } + if (available[table] !== true) { + return false; + } + } + return true; + } + + async #searchCodesWithFts(searchText, searchCfg) { + const sqlParts = []; + const params = []; + const activeClause = searchCfg.activeOnly ? ' AND c.active = 1' : ''; + const matchText = toFtsMatchText(searchText); + + for (const source of searchCfg.sources) { + if (source === 'display') { + const table = sqlIdentifier(searchCfg.ftsTables.display, 'search_fts_display'); + sqlParts.push( + `SELECT c.code AS code + FROM ${table} f + JOIN concept c ON c.concept_id = f.rowid + WHERE c.cs_id = ?${activeClause} + AND f.term MATCH ?` + ); + params.push(this.meta.csId, matchText); + continue; + } + + if (source === 'designation') { + const table = sqlIdentifier(searchCfg.ftsTables.designation, 'search_fts_designation'); + const designationClause = searchCfg.designationActiveOnly ? ' AND d.active = 1' : ''; + sqlParts.push( + `SELECT c.code AS code + FROM ${table} f + JOIN designation d ON d.designation_id = f.rowid + JOIN concept c ON c.concept_id = d.concept_id + WHERE c.cs_id = ?${activeClause}${designationClause} + AND f.term MATCH ?` + ); + params.push(this.meta.csId, matchText); + continue; + } + + if (source === 'literal') { + const table = sqlIdentifier(searchCfg.ftsTables.literal, 'search_fts_literal'); + const literalClause = searchCfg.literalActiveOnly ? ' AND cl.active = 1' : ''; + sqlParts.push( + `SELECT c.code AS code + FROM ${table} f + JOIN concept_literal cl ON cl.literal_id = f.rowid + JOIN concept c ON c.concept_id = cl.source_concept_id + WHERE c.cs_id = ?${activeClause}${literalClause} + AND f.term MATCH ?` + ); + params.push(this.meta.csId, matchText); + } + } + + if (sqlParts.length === 0) { + return []; + } + + const rows = await all( + this.db, + `SELECT DISTINCT code + FROM ( + ${sqlParts.join('\nUNION\n')} + ) + ORDER BY code`, + params + ); + return rows.map(r => r.code); + } + + async #searchCodesWithLike(searchText, searchCfg) { + const sqlParts = []; + const params = []; + const likeText = `%${searchText}%`; + const activeClause = searchCfg.activeOnly ? ' AND c.active = 1' : ''; + const likeExpr = searchCfg.likeFallback?.caseInsensitive === false + ? { display: 'c.display LIKE ?', designation: 'd.term LIKE ?', literal: 'COALESCE(cl.value_text, cl.value_raw) LIKE ?' } + : { + display: 'LOWER(c.display) LIKE LOWER(?)', + designation: 'LOWER(d.term) LIKE LOWER(?)', + literal: 'LOWER(COALESCE(cl.value_text, cl.value_raw)) LIKE LOWER(?)' + }; + + for (const source of searchCfg.sources) { + if (source === 'display') { + sqlParts.push( + `SELECT c.code AS code + FROM concept c + WHERE c.cs_id = ?${activeClause} + AND c.display IS NOT NULL + AND ${likeExpr.display}` + ); + params.push(this.meta.csId, likeText); + continue; + } + + if (source === 'designation') { + const designationClause = searchCfg.designationActiveOnly ? ' AND d.active = 1' : ''; + sqlParts.push( + `SELECT c.code AS code + FROM designation d + JOIN concept c ON c.concept_id = d.concept_id + WHERE c.cs_id = ?${activeClause}${designationClause} + AND d.term IS NOT NULL + AND ${likeExpr.designation}` + ); + params.push(this.meta.csId, likeText); + continue; + } + + if (source === 'literal') { + const literalClause = searchCfg.literalActiveOnly ? ' AND cl.active = 1' : ''; + sqlParts.push( + `SELECT c.code AS code + FROM concept_literal cl + JOIN concept c ON c.concept_id = cl.source_concept_id + WHERE c.cs_id = ?${activeClause}${literalClause} + AND COALESCE(cl.value_text, cl.value_raw) IS NOT NULL + AND ${likeExpr.literal}` + ); + params.push(this.meta.csId, likeText); + } + } + + if (sqlParts.length === 0) { + return []; + } + + const rows = await all( + this.db, + `SELECT DISTINCT code + FROM ( + ${sqlParts.join('\nUNION\n')} + ) + ORDER BY code`, + params + ); + return rows.map(r => r.code); + } + + async #ensureContext(code) { + if (!code) { + return null; + } + if (typeof code === 'string') { + const located = await this.locate(code); + return located.context; + } + if (code instanceof SqliteRuntimeV0Context) { + return code; + } + throw new Error(`Unknown context type: ${typeof code}`); + } + + #useMembershipPredicate(filterContext) { + return !!filterContext && filterContext.forIterate === false; + } + + #isPredicateFilter(set) { + return set instanceof SqliteRuntimeV0PredicateFilter; + } + + #isPagedDescendantFilter(set) { + return set instanceof SqliteRuntimeV0PagedDescendantFilter; + } + + #isQueryIterator(iteratorContext) { + return iteratorContext instanceof SqliteRuntimeV0QueryIterator; + } + + async #loadNextIteratorPage(iteratorContext) { + if (!this.#isQueryIterator(iteratorContext) || iteratorContext.done) { + return; + } + + const sql = []; + const params = []; + + if (iteratorContext.mode === 'all') { + sql.push( + `SELECT c.concept_id, c.code, c.display, c.definition, c.active`, + `FROM concept c`, + `WHERE c.cs_id = ?`, + ` AND c.active = 1` + ); + params.push(this.meta.csId); + } else if (iteratorContext.mode === 'roots') { + sql.push( + `SELECT c.concept_id, c.code, c.display, c.definition, c.active`, + `FROM concept c`, + `LEFT JOIN concept_link l`, + ` ON l.source_concept_id = c.concept_id`, + ` AND l.property_id = ?`, + ` AND l.edge_set_id = ?`, + ` AND l.active = 1`, + `WHERE c.cs_id = ?`, + ` AND c.active = 1`, + ` AND l.edge_id IS NULL` + ); + params.push(this.meta.hierarchyPropertyId, this.meta.hierarchyEdgeSetId, this.meta.csId); + } else if (iteratorContext.mode === 'children') { + if (!iteratorContext.targetConceptId) { + iteratorContext.done = true; + return; + } + sql.push( + `SELECT c.concept_id, c.code, c.display, c.definition, c.active`, + `FROM concept_link l`, + `JOIN concept c ON c.concept_id = l.source_concept_id`, + `WHERE l.target_concept_id = ?`, + ` AND l.property_id = ?`, + ` AND l.edge_set_id = ?`, + ` AND l.active = 1`, + ` AND c.cs_id = ?` + ); + params.push( + iteratorContext.targetConceptId, + this.meta.hierarchyPropertyId, + this.meta.hierarchyEdgeSetId, + this.meta.csId + ); + } else { + iteratorContext.done = true; + return; + } + + if (iteratorContext.lastCode !== null) { + sql.push(`AND c.code > ?`); + params.push(iteratorContext.lastCode); + } + + sql.push(`ORDER BY c.code`); + sql.push(`LIMIT ?`); + params.push(iteratorContext.pageSize); + + const rows = await all(this.db, sql.join('\n'), params); + iteratorContext.rows = []; + iteratorContext.cursor = 0; + + if (!rows.length) { + iteratorContext.done = true; + return; + } + + for (const row of rows) { + if (!this.#allowDefaultIterationCode(row.code)) { + continue; + } + iteratorContext.rows.push( + new SqliteRuntimeV0Context( + row.concept_id, + row.code, + row.display, + row.definition, + row.active === 1 + ) + ); + } + + iteratorContext.lastCode = rows[rows.length - 1].code; + if (rows.length < iteratorContext.pageSize) { + iteratorContext.done = true; + } + } + + async #loadNextDescendantPage(set) { + if (!this.#isPagedDescendantFilter(set) || set.done) { + return; + } + + if (!set.strategy) { + const countRow = await get( + this.db, + `SELECT COUNT(*) AS n + FROM closure + WHERE ancestor_id = ?`, + [set.ancestorId] + ); + const rawCount = Math.max(0, countRow?.n || 0); + set.descendantCount = set.includeSelf ? rawCount : Math.max(0, rawCount - 1); + const threshold = Number(this.runtime?.hierarchy?.closure?.conceptScanThreshold || 25000); + set.strategy = set.descendantCount >= threshold ? 'concept-scan' : 'closure-join'; + } + + let rows; + if (set.strategy === 'concept-scan') { + const sql = [ + `SELECT c.concept_id, c.code, c.display, c.definition, c.active`, + `FROM concept c`, + `WHERE c.cs_id = ?`, + ` AND EXISTS (`, + ` SELECT 1`, + ` FROM closure cl`, + ` WHERE cl.ancestor_id = ?`, + ` AND cl.descendant_id = c.concept_id`, + ` )` + ]; + const params = [this.meta.csId, set.ancestorId]; + + if (!set.includeSelf) { + sql.push(`AND c.concept_id <> ?`); + params.push(set.ancestorId); + } + if (set.lastCode !== null) { + sql.push(`AND c.code > ?`); + params.push(set.lastCode); + } + + sql.push(`ORDER BY c.code`); + sql.push(`LIMIT ?`); + params.push(set.pageSize); + rows = await all(this.db, sql.join('\n'), params); + } else { + const sql = [ + `SELECT c.concept_id, c.code, c.display, c.definition, c.active`, + `FROM closure cl`, + `JOIN concept c ON c.concept_id = cl.descendant_id`, + `WHERE cl.ancestor_id = ?` + ]; + const params = [set.ancestorId]; + + if (!set.includeSelf) { + sql.push(`AND cl.descendant_id <> ?`); + params.push(set.ancestorId); + } + if (set.lastCode !== null) { + sql.push(`AND c.code > ?`); + params.push(set.lastCode); + } + + sql.push(`ORDER BY c.code`); + sql.push(`LIMIT ?`); + params.push(set.pageSize); + rows = await all(this.db, sql.join('\n'), params); + } + + if (!rows.length) { + set.done = true; + return; + } + + for (const row of rows) { + set.rows.push( + new SqliteRuntimeV0Context( + row.concept_id, + row.code, + row.display, + row.definition, + row.active === 1 + ) + ); + } + + set.lastCode = rows[rows.length - 1].code; + if (rows.length < set.pageSize) { + set.done = true; + } + } + + async #filterLocatePredicate(set, code) { + if (!code) { + return `Code '${code}' not found in filter set`; + } + const located = await this.locate(code); + if (!located.context) { + return `Code '${code}' not found in filter set`; + } + const ok = await this.#predicateMatchesContext(set, located.context); + if (ok) { + return located.context; + } + return this.#predicateNotFoundMessage(set, code); + } + + async #predicateMatchesContext(set, context) { + const ctxt = await this.#ensureContext(context); + if (!ctxt) return false; + + if (set.kind === 'concept-equals') { + return ctxt.code === set.code; + } + + if (set.kind === 'concept-hierarchy') { + if (!set.ancestorId) return false; + return this.#isA(set.ancestorId, ctxt.conceptId, !!set.includeSelf); + } + + if (set.kind === 'concept-in') { + return this.#isConceptInValueSet(ctxt.conceptId, set.valueSetUrl); + } + + if (set.kind === 'property-filter') { + return this.#matchesPropertyPredicate(set, ctxt); + } + + throw new Error(`Unknown predicate filter kind '${set.kind}'`); + } + + #predicateNotFoundMessage(set, code) { + if (set.kind === 'concept-hierarchy') { + if (set.missingMessage) { + return set.missingMessage; + } + return `Code '${code}' is not in hierarchy of '${set.parentCode}'`; + } + + if (set.kind === 'concept-in') { + return `Code '${code}' not found in value set '${set.valueSetUrl}'`; + } + + if (set.kind === 'concept-equals') { + return `Code '${code}' does not equal '${set.code}'`; + } + + return `Code '${code}' not found in filter set`; + } + + async #matchesPropertyPredicate(set, context) { + const propertyCfg = set.propertyCfg; + if (!propertyCfg?.propertyId || !context?.conceptId) { + return false; + } + + if (set.op === '=') { + return this.#conceptMatchesPropertyEquals( + context.conceptId, + propertyCfg, + set.candidates || [], + set.caseSensitive === true + ); + } + + if (set.op === 'in') { + return this.#conceptMatchesPropertyEquals( + context.conceptId, + propertyCfg, + set.candidates || [], + set.caseSensitive === true + ); + } + + if (set.op === 'exists') { + return this.#conceptMatchesPropertyExists( + context.conceptId, + propertyCfg, + set.expectExists !== false + ); + } + + if (set.op === 'regex') { + return this.#conceptMatchesPropertyRegex( + context.conceptId, + propertyCfg, + set.pattern || '' + ); + } + + throw new Error(`Unsupported property predicate operator '${set.op}'`); + } + + async #conceptMatchesPropertyEquals(conceptId, propertyCfg, candidates, caseSensitive) { + if (!Array.isArray(candidates) || candidates.length === 0) { + return false; + } + + const values = candidates.map(v => String(v)); + if (propertyCfg.sources.includes('literal')) { + const literalHit = await this.#conceptHasLiteralEquals(conceptId, propertyCfg.propertyId, values, caseSensitive); + if (literalHit) { + return true; + } + } + + if (propertyCfg.sources.includes('link')) { + const linkHit = await this.#conceptHasLinkEquals(conceptId, propertyCfg, values, caseSensitive); + if (linkHit) { + return true; + } + } + + return false; + } + + async #conceptHasLiteralEquals(conceptId, propertyId, values, caseSensitive) { + if (!conceptId || !propertyId || !Array.isArray(values) || values.length === 0) { + return false; + } + + const placeholders = values.map(() => '?').join(', '); + const textPredicate = caseSensitive + ? `cl.value_text IN (${placeholders})` + : `cl.value_text COLLATE NOCASE IN (${placeholders})`; + const rawPredicate = caseSensitive + ? `cl.value_raw IN (${placeholders})` + : `cl.value_raw COLLATE NOCASE IN (${placeholders})`; + + const row = await get( + this.db, + `SELECT 1 AS found + FROM concept_literal cl + WHERE cl.source_concept_id = ? + AND cl.property_id = ? + AND cl.active = 1 + AND ( + (cl.value_text IS NOT NULL AND ${textPredicate}) + OR + (cl.value_text IS NULL AND cl.value_raw IS NOT NULL AND ${rawPredicate}) + ) + LIMIT 1`, + [conceptId, propertyId, ...values, ...values] + ); + return !!row; + } + + async #conceptHasLinkEquals(conceptId, propertyCfg, values, caseSensitive) { + if (!conceptId || !propertyCfg?.propertyId || !Array.isArray(values) || values.length === 0) { + return false; + } + + const placeholders = values.map(() => '?').join(', '); + const codePredicate = caseSensitive + ? `tgt.code IN (${placeholders})` + : `tgt.code COLLATE NOCASE IN (${placeholders})`; + const displayPredicate = caseSensitive + ? `tgt.display IN (${placeholders})` + : `tgt.display COLLATE NOCASE IN (${placeholders})`; + const where = propertyCfg.linkMatch === 'code-or-display' + ? `(${codePredicate} OR ${displayPredicate})` + : `(${codePredicate})`; + + const params = [ + conceptId, + propertyCfg.propertyId, + this.meta.hierarchyEdgeSetId, + this.meta.csId, + ...values + ]; + if (propertyCfg.linkMatch === 'code-or-display') { + params.push(...values); + } + + const row = await get( + this.db, + `SELECT 1 AS found + FROM concept_link l + JOIN concept tgt ON tgt.concept_id = l.target_concept_id + WHERE l.source_concept_id = ? + AND l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + AND tgt.cs_id = ? + AND ${where} + LIMIT 1`, + params + ); + return !!row; + } + + async #conceptMatchesPropertyExists(conceptId, propertyCfg, expectExists) { + let found = false; + if (propertyCfg.sources.includes('literal')) { + found = found || await this.#conceptHasLiteralAny(conceptId, propertyCfg.propertyId); + } + if (propertyCfg.sources.includes('link')) { + found = found || await this.#conceptHasLinkAny(conceptId, propertyCfg.propertyId); + } + return expectExists ? found : !found; + } + + async #conceptHasLiteralAny(conceptId, propertyId) { + const row = await get( + this.db, + `SELECT 1 AS found + FROM concept_literal + WHERE source_concept_id = ? + AND property_id = ? + AND active = 1 + LIMIT 1`, + [conceptId, propertyId] + ); + return !!row; + } + + async #conceptHasLinkAny(conceptId, propertyId) { + const row = await get( + this.db, + `SELECT 1 AS found + FROM concept_link + WHERE source_concept_id = ? + AND property_id = ? + AND edge_set_id = ? + AND active = 1 + LIMIT 1`, + [conceptId, propertyId, this.meta.hierarchyEdgeSetId] + ); + return !!row; + } + + async #conceptMatchesPropertyRegex(conceptId, propertyCfg, pattern) { + let regex; + try { + regex = new RegExp(String(pattern || '')); + } catch (error) { + throw new Error(`Invalid regex '${pattern}': ${error.message}`); + } + + if (propertyCfg.sources.includes('literal')) { + const rows = await all( + this.db, + `SELECT COALESCE(value_text, value_raw) AS value + FROM concept_literal + WHERE source_concept_id = ? + AND property_id = ? + AND active = 1 + AND COALESCE(value_text, value_raw) IS NOT NULL`, + [conceptId, propertyCfg.propertyId] + ); + for (const row of rows) { + if (row.value && regex.test(row.value)) { + return true; + } + } + } + + if (propertyCfg.sources.includes('link')) { + const rows = await all( + this.db, + `SELECT tgt.code AS target_code, + tgt.display AS target_display + FROM concept_link l + JOIN concept tgt ON tgt.concept_id = l.target_concept_id + WHERE l.source_concept_id = ? + AND l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1`, + [conceptId, propertyCfg.propertyId, this.meta.hierarchyEdgeSetId] + ); + for (const row of rows) { + const codeMatch = row.target_code && regex.test(row.target_code); + const displayMatch = propertyCfg.linkMatch === 'code-or-display' && + row.target_display && regex.test(row.target_display); + if (codeMatch || displayMatch) { + return true; + } + } + } + + return false; + } + + async #isConceptInValueSet(conceptId, valueSetUrl) { + if (!conceptId || !valueSetUrl) return false; + const row = await get( + this.db, + `SELECT 1 AS found + FROM value_set v + JOIN value_set_member m ON m.vs_id = v.vs_id + WHERE v.cs_id = ? + AND v.url = ? + AND m.active = 1 + AND m.concept_id = ? + LIMIT 1`, + [this.meta.csId, valueSetUrl, conceptId] + ); + return !!row; + } + + async #isA(ancestorId, descendantId, includeSelf) { + if (!this.meta.hierarchyPropertyId) return false; + if (!ancestorId || !descendantId) return false; + if (ancestorId === descendantId && includeSelf) return true; + + if (this.meta.closureRows > 0 && this.meta.useClosure) { + const row = await get( + this.db, + `SELECT 1 AS found + FROM closure + WHERE ancestor_id = ? + AND descendant_id = ? + LIMIT 1`, + [ancestorId, descendantId] + ); + if (!row) return false; + if (!includeSelf && ancestorId === descendantId) return false; + return true; + } + + if (!this.#allowRecursiveHierarchyFallback()) { + return false; + } + + const row = await get( + this.db, + `WITH RECURSIVE descendants(concept_id) AS ( + SELECT ? + UNION + SELECT l.source_concept_id + FROM concept_link l + JOIN descendants d ON d.concept_id = l.target_concept_id + WHERE l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + ) + SELECT 1 AS found + FROM descendants + WHERE concept_id = ? + LIMIT 1`, + [ancestorId, this.meta.hierarchyPropertyId, this.meta.hierarchyEdgeSetId, descendantId] + ); + + if (!row) return false; + if (!includeSelf && ancestorId === descendantId) return false; + return true; + } + + async #descendantCodes(ancestorCode, includeSelf) { + if (!ancestorCode) return []; + const ancestorContext = await this.#ensureContext(ancestorCode); + if (!ancestorContext) return []; + const ancestorId = ancestorContext.conceptId; + + if (this.meta.closureRows > 0 && this.meta.useClosure) { + const rows = await all( + this.db, + `SELECT c.code, cl.descendant_id + FROM closure cl + JOIN concept c ON c.concept_id = cl.descendant_id + WHERE cl.ancestor_id = ? + ORDER BY c.code`, + [ancestorId] + ); + return rows + .filter(r => includeSelf || r.descendant_id !== ancestorId) + .map(r => r.code); + } + + if (!this.#allowRecursiveHierarchyFallback()) { + return []; + } + + const rows = await all( + this.db, + `WITH RECURSIVE descendants(concept_id, depth) AS ( + SELECT ?, 0 + UNION + SELECT l.source_concept_id, descendants.depth + 1 + FROM concept_link l + JOIN descendants ON descendants.concept_id = l.target_concept_id + WHERE l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + ) + SELECT c.code, descendants.concept_id + FROM descendants + JOIN concept c ON c.concept_id = descendants.concept_id + ORDER BY c.code`, + [ancestorId, this.meta.hierarchyPropertyId, this.meta.hierarchyEdgeSetId] + ); + + return rows + .filter(r => includeSelf || r.concept_id !== ancestorId) + .map(r => r.code); + } + + async #batchLoadContextsByCodes(codes) { + if (!Array.isArray(codes) || codes.length === 0) { + return []; + } + + const placeholders = codes.map(() => '?').join(', '); + const rows = await all( + this.db, + `SELECT concept_id, code, display, definition, active + FROM concept + WHERE cs_id = ? + AND code IN (${placeholders})`, + [this.meta.csId, ...codes] + ); + + const byCode = new Map(); + for (const row of rows) { + byCode.set(row.code, row); + } + + const contexts = []; + for (const code of codes) { + const row = byCode.get(code); + if (!row) { + continue; + } + contexts.push( + new SqliteRuntimeV0Context( + row.concept_id, + row.code, + row.display, + row.definition, + row.active === 1 + ) + ); + } + + return contexts; + } + + #allowDefaultIterationCode(code) { + if (!this.defaultIterationRegex) { + return true; + } + return this.defaultIterationRegex.test(String(code || '')); + } + + #allowRecursiveHierarchyFallback() { + return this.runtime?.hierarchy?.closure?.fallbackRecursive !== false; + } +} + +class SqliteRuntimeV0FactoryProvider extends CodeSystemFactoryProvider { + static registerSpecializedFactory(definition) { + if (!definition || typeof definition !== 'object') { + throw new Error('registerSpecializedFactory requires an object definition'); + } + if (typeof definition.createFactory !== 'function') { + throw new Error('registerSpecializedFactory requires createFactory(context) function'); + } + const matchTags = Array.isArray(definition.matchTags) + ? definition.matchTags.map((tag) => String(tag || '').trim()).filter(Boolean) + : []; + SQLITE_RUNTIME_V0_FACTORY_REGISTRY.push({ + id: String(definition.id || `factory-${SQLITE_RUNTIME_V0_FACTORY_REGISTRY.length + 1}`), + priority: Number.isFinite(definition.priority) ? definition.priority : 0, + matchTags, + createFactory: definition.createFactory + }); + SQLITE_RUNTIME_V0_FACTORY_REGISTRY.sort((a, b) => { + if (b.matchTags.length !== a.matchTags.length) { + return b.matchTags.length - a.matchTags.length; + } + return b.priority - a.priority; + }); + } + + static listSpecializedFactories() { + return SQLITE_RUNTIME_V0_FACTORY_REGISTRY.map((entry) => ({ + id: entry.id, + priority: entry.priority, + matchTags: [...entry.matchTags] + })); + } + + static async createFromMetadata(i18n, dbPath, options = {}) { + const probe = new SqliteRuntimeV0FactoryProvider(i18n, dbPath, options); + await probe.load(); + + const tags = metadataTagsFromRuntime(probe._runtime); + let selected = null; + for (const entry of SQLITE_RUNTIME_V0_FACTORY_REGISTRY) { + if (entry.matchTags.every((tag) => tags.has(tag))) { + selected = entry; + break; + } + } + if (!selected) { + return probe; + } + + const resolved = await selected.createFactory({ + i18n, + dbPath, + options, + tags, + runtime: probe._runtime, + metadata: probe._meta, + baseFactory: probe + }); + + if (!resolved || resolved === probe) { + return probe; + } + + probe.close(); + if (typeof resolved.load === 'function' && !resolved._loaded) { + await resolved.load(); + } + return resolved; + } + + constructor(i18n, dbPath, options = {}) { + super(i18n); + this.dbPath = dbPath; + this.idPrefix = options.idPrefix || 'sqlite-runtime-v0'; + this._loaded = false; + this._loadPromise = null; + this._db = null; + this._meta = null; + this._runtime = null; + this._sharedState = { + statusByPropertyId: new Map(), + statusLoadPromises: new Map(), + childTargetSets: new Map(), + childTargetLoadPromises: new Map() + }; + } + + system() { + return this._meta?.baseUri || null; + } + + version() { + return this._meta?.canonicalUri || this._meta?.version || null; + } + + getPartialVersion() { + const v = this.version(); + if (!v) return null; + const idx = v.indexOf('/version/'); + if (idx === -1) return null; + return v.substring(0, idx); + } + + name() { + return this._meta?.name || this.system() || 'SQLite Runtime'; + } + + defaultVersion() { + return this._meta?.version || 'unknown'; + } + + async load() { + if (this._loaded) { + return; + } + if (!this._loadPromise) { + this._loadPromise = (async () => { + this._db = await openDb(this.dbPath, true); + const db = this._db; + + const codeSystem = await get( + db, + `SELECT cs_id, base_uri, canonical_uri, edition_code, version, name + FROM code_system + ORDER BY cs_id DESC + LIMIT 1`, + [] + ); + + if (!codeSystem) { + throw new Error(`No code_system rows found in ${this.dbPath}`); + } + + const totalRow = await get( + db, + 'SELECT COUNT(*) AS n FROM concept WHERE cs_id = ?', + [codeSystem.cs_id] + ); + + const cfgRows = await all( + db, + `SELECT key, value + FROM cs_config + WHERE cs_id = ?`, + [codeSystem.cs_id] + ); + const cfg = {}; + for (const row of cfgRows) { + cfg[row.key] = parseConfigValue(row.value); + } + + const runtime = buildRuntimeConfig(cfg, codeSystem.base_uri); + const searchCfg = normalizedSearchConfig(runtime.search); + + let hierarchyPropertyCode = runtime.hierarchy?.propertyCode || null; + if (!hierarchyPropertyCode) { + const hierarchyRow = await get( + db, + `SELECT property_code + FROM property_def + WHERE cs_id = ? AND is_hierarchy = 1 + ORDER BY property_id + LIMIT 1`, + [codeSystem.cs_id] + ); + hierarchyPropertyCode = hierarchyRow?.property_code || null; + } + + let hierarchyPropertyId = null; + if (hierarchyPropertyCode) { + const hierarchyPropRow = await get( + db, + `SELECT property_id + FROM property_def + WHERE cs_id = ? AND property_code = ?`, + [codeSystem.cs_id, hierarchyPropertyCode] + ); + hierarchyPropertyId = hierarchyPropRow?.property_id || null; + } + + const closureCountRow = await get(db, `SELECT COUNT(*) AS n FROM closure`, []); + const searchFtsTables = {}; + for (const source of searchCfg.sources) { + const configured = searchCfg.ftsTables?.[source]; + if (!configured) continue; + const table = sqlIdentifier(configured, configured); + if (!table) continue; + const exists = await get( + db, + `SELECT 1 AS found + FROM sqlite_master + WHERE type = 'table' AND name = ? + LIMIT 1`, + [table] + ); + searchFtsTables[table] = !!exists; + } + const designationOrderIndex = await get( + db, + `SELECT 1 AS found + FROM sqlite_master + WHERE type = 'index' AND name = 'idx_designation_concept_pref_term' + LIMIT 1`, + [] + ); + + this._meta = { + csId: codeSystem.cs_id, + baseUri: codeSystem.base_uri, + canonicalUri: codeSystem.canonical_uri, + editionCode: codeSystem.edition_code, + version: codeSystem.version, + name: codeSystem.name || codeSystem.base_uri, + totalConcepts: totalRow ? totalRow.n : 0, + defaultLanguage: runtime.languages?.default || 'en', + closureRows: closureCountRow ? closureCountRow.n : 0, + hierarchyPropertyId, + hierarchyEdgeSetId: runtime.hierarchy?.edgeSetId || 1, + useClosure: runtime.hierarchy?.closure?.enabled !== false, + searchFtsTables, + designationOrderIndex: !!designationOrderIndex + }; + this._runtime = runtime; + this._loaded = true; + })(); + } + await this._loadPromise; + } + + async build(opContext, supplements) { + if (!this._loaded) { + await this.load(); + } + + this.recordUse(); + return new SqliteRuntimeV0Provider(opContext, supplements, this._db, this._meta, this._runtime, { + ownsDb: false, + sharedState: this._sharedState, + dbPath: this.dbPath + }); + } + + async buildKnownValueSet(url, version) { + if (!this._loaded) { + await this.load(); + } + + if (!url || !this.system() || !url.startsWith(this.system())) { + return null; + } + + if (version && this._meta.canonicalUri && !this._meta.canonicalUri.startsWith(version)) { + return null; + } + + const qIndex = url.indexOf('?'); + if (qIndex < 0) { + return null; + } + + const query = url.substring(qIndex + 1); + const implicit = this._runtime.implicitValueSets || {}; + + if (Array.isArray(implicit.all?.queries) && implicit.all.queries.includes(query)) { + return { + resourceType: 'ValueSet', + url, + version: this._meta.version, + status: 'active', + name: `${sanitizeName(this.system())}All`, + title: `${this.name()} All Concepts`, + description: `All concepts from ${this.name()}`, + compose: { include: [{ system: this.system() }] } + }; + } + + for (const [name, cfg] of Object.entries(implicit)) { + if (!cfg || !cfg.queryPrefix || !cfg.filter) continue; + if (!query.startsWith(cfg.queryPrefix)) continue; + + const suffix = query.substring(cfg.queryPrefix.length); + const filterValue = cfg.filter.valueFromSuffix ? suffix : cfg.filter.value; + return { + resourceType: 'ValueSet', + url, + version: this._meta.version, + status: 'active', + name: `${sanitizeName(this.system())}${name}${suffix}`, + compose: { + include: [{ + system: this.system(), + filter: [{ + property: cfg.filter.property, + op: cfg.filter.op, + value: filterValue + }] + }] + } + }; + } + + return null; + } + + id() { + return `${this.idPrefix}:${this._meta?.version || 'unknown'}`; + } + + close() { + if (!this._db) { + return; + } + this._db.close(); + this._db = null; + this._loaded = false; + this._loadPromise = null; + } +} + +function buildRuntimeConfig(cfg, system) { + const searchCfg = normalizedSearchConfig(cfg['runtime.search']); + + const runtime = { + versioning: cfg['runtime.versioning'] || { algorithm: 'string', partialMatch: true }, + languages: cfg['runtime.languages'] || { default: 'en' }, + designations: cfg['runtime.designations'] || {}, + hierarchy: cfg['runtime.hierarchy'] || { + propertyCode: null, + edgeSetId: 1, + closure: { enabled: true, fallbackRecursive: false } + }, + filters: cfg['runtime.filters'] || { + concept: { operators: ['=', 'is-a', 'descendent-of', 'in'] }, + code: { operators: ['regex'] } + }, + implicitValueSets: cfg['runtime.implicitValueSets'] || defaultImplicitValueSets(system), + status: cfg['runtime.status'] || { + inactive: { source: 'concept.active', invert: true }, + deprecated: { source: 'constant', value: false }, + abstract: { source: 'constant', value: false } + }, + iteration: cfg['runtime.iteration'] || {}, + search: searchCfg, + behaviorFlags: cfg['runtime.behaviorFlags'] || {} + }; + + if (!runtime.hierarchy.edgeSetId) runtime.hierarchy.edgeSetId = 1; + if (!runtime.languages.default) runtime.languages.default = 'en'; + + return runtime; +} + +function normalizedSearchConfig(raw) { + const value = raw || {}; + + const sources = Array.isArray(value.sources) && value.sources.length > 0 + ? value.sources.filter(s => ['display', 'designation', 'literal'].includes(s)) + : ['designation']; + + return { + mode: value.mode || 'like', + activeOnly: value.activeOnly !== false, + designationActiveOnly: value.designationActiveOnly !== false, + literalActiveOnly: value.literalActiveOnly !== false, + sources, + ftsTables: { + display: value.ftsTables?.display || 'search_fts_display', + designation: value.ftsTables?.designation || 'search_fts_designation', + literal: value.ftsTables?.literal || 'search_fts_literal' + }, + likeFallback: { + enabled: value.likeFallback?.enabled !== false, + caseInsensitive: value.likeFallback?.caseInsensitive !== false + } + }; +} + +function defaultImplicitValueSets(system) { + return { + all: { queries: ['fhir_vs', 'fhir_vs=all'] }, + isa: { queryPrefix: 'fhir_vs=isa/', filter: { property: 'concept', op: 'is-a', valueFromSuffix: true } }, + refset: { queryPrefix: 'fhir_vs=refset/', filter: { property: 'concept', op: 'in', valueFromSuffix: true } }, + _system: system + }; +} + +function metadataTagsFromRuntime(runtime) { + const flags = runtime?.behaviorFlags || {}; + const tags = new Set(); + + if (Array.isArray(flags.tags)) { + for (const raw of flags.tags) { + const tag = String(raw || '').trim(); + if (tag) tags.add(tag); + } + } + + const legacyAdapter = String(flags.adapter || '').trim(); + if (legacyAdapter) { + tags.add(`adapter:${legacyAdapter}`); + tags.add(legacyAdapter); + if (legacyAdapter === 'loinc-v0') { + tags.add('loinc'); + tags.add('implicit-vs-path'); + } else if (legacyAdapter === 'snomed-v0') { + tags.add('snomed'); + } else if (legacyAdapter === 'rxnorm-v0') { + tags.add('rxnorm'); + } + } + + return tags; +} + +function inferSourcesFromValueKind(valueKind) { + if (valueKind === 'literal') { + return ['literal']; + } + if (valueKind === 'concept') { + return ['link']; + } + return ['literal', 'link']; +} + +function dedupSources(sources, valueKind) { + const input = Array.isArray(sources) && sources.length > 0 + ? sources + : inferSourcesFromValueKind(valueKind); + const cleaned = []; + for (const source of input) { + if ((source === 'literal' || source === 'link') && !cleaned.includes(source)) { + cleaned.push(source); + } + } + if (cleaned.length === 0) { + return inferSourcesFromValueKind(valueKind); + } + return cleaned; +} + +function normalizedFilterCandidates(value, valueCfg) { + const raw = String(value ?? '').trim(); + if (!raw) return []; + + const cfg = valueCfg || {}; + const normalizeCase = cfg.normalizeCase !== false; + const aliases = cfg.aliases || {}; + + const out = new Set(); + out.add(raw); + + const rawKey = normalizeCase ? raw.toLowerCase() : raw; + let alias = aliases[raw]; + if (alias === undefined) { + alias = aliases[rawKey]; + } + if (alias !== undefined && alias !== null && String(alias).trim() !== '') { + out.add(String(alias).trim()); + } + + return Array.from(out); +} + +function splitFilterValueList(value) { + if (Array.isArray(value)) { + return value.map(v => String(v ?? '').trim()).filter(Boolean); + } + return String(value ?? '') + .split(',') + .map(v => v.trim()) + .filter(Boolean); +} + +function resolveInValueSetUrl(system, value, runtime) { + if (typeof value === 'string' && value.startsWith('http://')) return value; + if (typeof value === 'string' && value.startsWith('https://')) return value; + + const refsetCfg = runtime.implicitValueSets?.refset; + if (refsetCfg?.urlTemplate) { + return refsetCfg.urlTemplate + .replace('{system}', system) + .replace('{value}', extractRefsetId(value)); + } + + return `${system}?fhir_vs=refset/${extractRefsetId(value)}`; +} + +function extractRefsetId(value) { + if (!value) return value; + const marker = 'refset/'; + const idx = value.indexOf(marker); + if (idx === -1) return value; + return value.substring(idx + marker.length); +} + +function useFromDesignation(row, runtime, system) { + const map = runtime.designations?.useMapping; + if (map && row.use_code && map[row.use_code]) { + return map[row.use_code]; + } + + if (row.use_code) { + return { + system: runtime.designations?.defaultSystem || system, + code: row.use_code, + display: row.use_code + }; + } + + return CodeSystem.makeUseForDisplay(); +} + +function sanitizeName(system) { + return (system || 'CS').replace(/[^A-Za-z0-9]/g, '').slice(0, 40) || 'CS'; +} + +function toFtsMatchText(text) { + // Use phrase syntax to avoid accidental MATCH operators from user input. + return `"${String(text || '').replace(/"/g, '""')}"`; +} + +function sqlIdentifier(name, fallback) { + const primary = typeof name === 'string' ? name : null; + if (primary && /^[A-Za-z_][A-Za-z0-9_]*$/.test(primary)) { + return primary; + } + if (typeof fallback === 'string' && /^[A-Za-z_][A-Za-z0-9_]*$/.test(fallback)) { + return fallback; + } + return null; +} + +function parseConfigValue(value) { + if (value === null || value === undefined) return null; + if (typeof value !== 'string') return value; + + try { + return JSON.parse(value); + } catch (_error) { + return value; + } +} + +function openDb(dbPath, readOnly) { + return new Promise((resolve, reject) => { + const flags = readOnly ? sqlite3.OPEN_READONLY : sqlite3.OPEN_READWRITE; + const db = new sqlite3.Database(dbPath, flags, (err) => { + if (err) reject(err); + else resolve(db); + }); + }); +} + +function closeDb(db) { + return new Promise((resolve, reject) => { + db.close((err) => { + if (err) reject(err); + else resolve(); + }); + }); +} + +function get(db, sql, params = []) { + return new Promise((resolve, reject) => { + db.get(sql, params, (err, row) => { + if (err) reject(err); + else resolve(row); + }); + }); +} + +function all(db, sql, params = []) { + return new Promise((resolve, reject) => { + db.all(sql, params, (err, rows) => { + if (err) reject(err); + else resolve(rows || []); + }); + }); +} + +module.exports = { + SqliteRuntimeV0FactoryProvider, + SqliteRuntimeV0Provider, + SqliteRuntimeV0Context, + SqliteRuntimeV0FilterSet +}; diff --git a/tx/cs/cs-sqlite-v0-specializers.js b/tx/cs/cs-sqlite-v0-specializers.js new file mode 100644 index 0000000..c5afe9d --- /dev/null +++ b/tx/cs/cs-sqlite-v0-specializers.js @@ -0,0 +1,87 @@ +'use strict'; + +const { SqliteRuntimeV0FactoryProvider } = require('./cs-sqlite-runtime-v0'); + +class LoincImplicitValueSetFactory extends SqliteRuntimeV0FactoryProvider { + async buildKnownValueSet(url, version) { + if (!this._loaded) { + await this.load(); + } + + const system = this.system(); + if (!url || !system || !url.startsWith(`${system}/vs`)) { + return super.buildKnownValueSet(url, version); + } + + if (version && this._meta.canonicalUri && !this._meta.canonicalUri.startsWith(version)) { + return null; + } + + const vsBase = `${system}/vs`; + if (url === vsBase || url === `${vsBase}/`) { + return { + resourceType: 'ValueSet', + url, + version: this._meta.version, + status: 'active', + name: `${sanitizeName(this.name())}All`, + description: `All concepts from ${this.name()}`, + compose: { include: [{ system }] } + }; + } + + if (!url.startsWith(`${vsBase}/`)) { + return super.buildKnownValueSet(url, version); + } + + const token = decodeURIComponent(url.substring(vsBase.length + 1)); + if (token.startsWith('LL')) { + return { + resourceType: 'ValueSet', + url, + version: this._meta.version, + status: 'active', + name: `LOINCAnswerList${sanitizeName(token)}`, + compose: { + include: [{ + system, + filter: [{ property: 'LIST', op: '=', value: token }] + }] + } + }; + } + + if (token.startsWith('LP')) { + return { + resourceType: 'ValueSet', + url, + version: this._meta.version, + status: 'active', + name: `LOINCPart${sanitizeName(token)}`, + compose: { + include: [{ + system, + filter: [{ property: 'concept', op: 'is-a', value: token }] + }] + } + }; + } + + return super.buildKnownValueSet(url, version); + } +} + +function sanitizeName(value) { + return String(value || 'CS').replace(/[^A-Za-z0-9]/g, '').slice(0, 60) || 'CS'; +} + +SqliteRuntimeV0FactoryProvider.registerSpecializedFactory({ + id: 'loinc-implicit-valuesets', + matchTags: ['loinc', 'implicit-vs-path'], + priority: 100, + createFactory: ({ i18n, dbPath, options }) => new LoincImplicitValueSetFactory(i18n, dbPath, options) +}); + +module.exports = { + LoincImplicitValueSetFactory +}; diff --git a/tx/importers/import-sct-sqlite-v0.module.js b/tx/importers/import-sct-sqlite-v0.module.js new file mode 100644 index 0000000..71a88ae --- /dev/null +++ b/tx/importers/import-sct-sqlite-v0.module.js @@ -0,0 +1,415 @@ +'use strict'; + +const fs = require('fs'); +const path = require('path'); +const inquirer = require('inquirer'); +const sqlite3 = require('sqlite3').verbose(); + +const { BaseTerminologyModule } = require('./tx-import-base'); +const { SnomedSqliteV0Importer } = require('./sqlite-v2/import-snomed-v0'); + +class SnomedSqliteV0Module extends BaseTerminologyModule { + getName() { + return 'snomed-sqlite-v0'; + } + + getDescription() { + return 'SNOMED CT RF2 Snapshot -> SQLite (clean v0 schema)'; + } + + getSupportedFormats() { + return ['rf2', 'directory']; + } + + getDefaultConfig() { + return { + verbose: true, + overwrite: false, + snapshotOnly: true, + skipRefsets: false, + skipClosure: false, + edition: '900000000000207008', + dest: './data/snomed-v0.db' + }; + } + + getEstimatedDuration() { + return '30-180 minutes (depends on edition size and closure)'; + } + + registerCommands(terminologyCommand, globalOptions) { + terminologyCommand + .command('import') + .description('Import SNOMED CT RF2 into SQLite v0 schema') + .option('-s, --source ', 'Source directory containing RF2 files (ideally Snapshot root)') + .option('-d, --dest ', 'Destination SQLite file') + .option('-e, --edition ', 'Edition code (e.g., 900000000000207008 or 731000124108)') + .option('--snomed-version ', 'Version date in YYYYMMDD format') + .option('-u, --uri ', 'Canonical SNOMED URI; overrides edition+version') + .option('--skip-refsets', 'Skip refset/value-set membership import') + .option('--skip-closure', 'Deprecated/ignored: closure is always built') + .option('--include-non-snapshot', 'Include non-Snapshot RF2 files in discovery') + .option('--overwrite', 'Overwrite destination database if it exists') + .option('-y, --yes', 'Skip confirmations') + .action(async (options) => { + await this.handleImportCommand({ ...globalOptions, ...options }); + }); + + terminologyCommand + .command('validate') + .description('Validate source path and discover RF2 file classes') + .option('-s, --source ', 'Source directory') + .option('--include-non-snapshot', 'Include non-Snapshot RF2 files in discovery') + .action(async (options) => { + await this.handleValidateCommand({ ...globalOptions, ...options }); + }); + + terminologyCommand + .command('status') + .description('Show status of a generated SQLite v0 SNOMED database') + .option('-d, --dest ', 'Database file path', './data/snomed-v0.db') + .action(async (options) => { + await this.handleStatusCommand({ ...globalOptions, ...options }); + }); + } + + async handleImportCommand(options) { + try { + const config = options.yes + ? this.buildNonInteractiveConfig(options) + : await this.gatherConfig(options); + + if (!options.yes) { + const confirmed = await this.confirmImport(config); + if (!confirmed) { + this.logInfo('Import cancelled'); + return; + } + } + + this.rememberSuccessfulConfig(config); + await this.runImportWithoutConfigSaving(config); + } catch (error) { + this.logError(`Import command failed: ${error.message}`); + if (options.verbose) { + console.error(error.stack); + } + throw error; + } + } + + async gatherConfig(options) { + const baseConfig = await this.gatherCommonConfig(options); + + const config = { + ...baseConfig, + edition: options.edition || baseConfig.edition || '900000000000207008', + version: options.snomedVersion || options.version || baseConfig.version, + uri: options.uri || baseConfig.uri, + skipRefsets: !!options.skipRefsets, + skipClosure: false, + snapshotOnly: !options.includeNonSnapshot + }; + + if (!config.uri && !config.version) { + const answers = await inquirer.prompt([ + { + type: 'input', + name: 'version', + message: 'SNOMED version (YYYYMMDD):', + validate: validateVersion + } + ]); + config.version = answers.version; + } + + const parsed = parseSnomedUri(config.uri); + if (!config.version && parsed.version) { + config.version = parsed.version; + } + if (!config.edition && parsed.edition) { + config.edition = parsed.edition; + } + + if (!config.uri) { + config.uri = `http://snomed.info/sct/${config.edition}/version/${config.version}`; + } + + if (!options.dest && shouldAutoAssignDest(config.dest) && config.version) { + config.dest = buildDefaultDest(config.edition, config.version); + } + if (options.skipClosure) { + this.logWarning('--skip-closure ignored: closure is always built'); + } + + return config; + } + + buildNonInteractiveConfig(options) { + const parsed = parseSnomedUri(options.uri); + const edition = options.edition || this.getDefaultConfig().edition || parsed.edition; + const version = options.snomedVersion || options.version || parsed.version; + const config = { + ...this.getDefaultConfig(), + ...options, + source: options.source, + dest: options.dest || buildDefaultDest(edition, version), + edition, + version, + uri: options.uri, + skipRefsets: !!options.skipRefsets, + skipClosure: false, + snapshotOnly: !options.includeNonSnapshot, + overwrite: !!options.overwrite, + verbose: !!options.verbose + }; + + if (!config.uri && config.edition && config.version) { + config.uri = `http://snomed.info/sct/${config.edition}/version/${config.version}`; + } + + if (!config.source) { + throw new Error('source is required when using --yes'); + } + if (!config.uri && !config.version) { + throw new Error('Provide --uri or --snomed-version with --edition when using --yes'); + } + if (options.skipClosure) { + this.logWarning('--skip-closure ignored: closure is always built'); + } + + return config; + } + + async confirmImport(config) { + console.log('\nSNOMED SQLite v0 Import Configuration:'); + console.log(` Source: ${config.source}`); + console.log(` Destination: ${config.dest}`); + console.log(` URI: ${config.uri}`); + console.log(` SnapshotOnly: ${config.snapshotOnly ? 'Yes' : 'No'}`); + console.log(` Skip Refsets: ${config.skipRefsets ? 'Yes' : 'No'}`); + console.log(` Skip Closure: ${config.skipClosure ? 'Yes' : 'No'}`); + console.log(` Overwrite: ${config.overwrite ? 'Yes' : 'No'}`); + + const answer = await inquirer.prompt([ + { + type: 'confirm', + name: 'confirmed', + message: 'Proceed with import?', + default: true + } + ]); + + return answer.confirmed; + } + + async runImportWithoutConfigSaving(config) { + try { + const importer = new SnomedSqliteV0Importer(config); + const result = await importer.run(); + this.logSuccess(`SNOMED SQLite v0 import complete: ${result.uri}`); + this.logSuccess(`Concepts: ${result.stats.concepts.toLocaleString()}, Descriptions: ${result.stats.descriptions.toLocaleString()}, Relationships: ${result.stats.relationships.toLocaleString()}`); + } catch (error) { + this.logError(`SNOMED SQLite v0 import failed: ${error.message}`); + if (config.verbose) { + console.error(error.stack); + } + process.exit(1); + } + } + + async handleValidateCommand(options) { + const source = options.source || (await promptForSource()); + + if (!fs.existsSync(source)) { + this.logError(`Source does not exist: ${source}`); + return; + } + + const snapshotOnly = !options.includeNonSnapshot; + const files = SnomedSqliteV0Importer.discoverRf2Files(source, { snapshotOnly }); + + console.log('\nDiscovered RF2 file classes:'); + console.log(` Concepts: ${files.concepts.length}`); + console.log(` Descriptions: ${files.descriptions.length}`); + console.log(` Relationships: ${files.relationships.length}`); + console.log(` Concrete Values: ${files.concreteValues.length}`); + console.log(` Language Refsets: ${files.languageRefsets.length}`); + console.log(` Any Refsets: ${files.refsets.length}`); + + const ok = files.concepts.length > 0 && files.descriptions.length > 0; + if (ok) { + this.logSuccess('Validation passed'); + } else { + this.logError('Validation failed: concepts and descriptions are required'); + } + } + + async handleStatusCommand(options) { + const dbPath = path.resolve(options.dest || './data/snomed-v0.db'); + + if (!fs.existsSync(dbPath)) { + this.logError(`Database not found: ${dbPath}`); + return; + } + + const db = new sqlite3.Database(dbPath, sqlite3.OPEN_READONLY); + + try { + const codeSystem = await getRow(db, 'SELECT cs_id, canonical_uri, edition_code, version, loaded_at FROM code_system ORDER BY cs_id DESC LIMIT 1', []); + if (!codeSystem) { + this.logWarning('No code_system rows found'); + return; + } + + const [concepts, descriptions, relationships, refsets, closure] = await Promise.all([ + getRow(db, 'SELECT COUNT(*) AS n FROM concept WHERE cs_id = ?', [codeSystem.cs_id]), + getRow( + db, + `SELECT COUNT(*) AS n + FROM designation d + JOIN concept c ON c.concept_id = d.concept_id + WHERE c.cs_id = ?`, + [codeSystem.cs_id] + ), + getRow( + db, + `SELECT COUNT(*) AS n + FROM concept_link l + JOIN concept c ON c.concept_id = l.source_concept_id + WHERE c.cs_id = ?`, + [codeSystem.cs_id] + ), + getRow(db, 'SELECT COUNT(*) AS n FROM value_set WHERE cs_id = ?', [codeSystem.cs_id]), + getRow( + db, + `SELECT COUNT(*) AS n + FROM closure cl + JOIN concept c ON c.concept_id = cl.ancestor_id + WHERE c.cs_id = ?`, + [codeSystem.cs_id] + ) + ]); + + const [ftsDisplay, ftsDesignation, ftsLiteral] = await Promise.all([ + getCountIfTableExists(db, 'search_fts_display'), + getCountIfTableExists(db, 'search_fts_designation'), + getCountIfTableExists(db, 'search_fts_literal') + ]); + + console.log('\nSNOMED SQLite v0 Status:'); + console.log(` DB: ${dbPath}`); + console.log(` Canonical URI: ${codeSystem.canonical_uri}`); + console.log(` Edition: ${codeSystem.edition_code || '(none)'}`); + console.log(` Version: ${codeSystem.version || '(none)'}`); + console.log(` Loaded At: ${codeSystem.loaded_at}`); + console.log(` Concepts: ${(concepts?.n || 0).toLocaleString()}`); + console.log(` Descriptions: ${(descriptions?.n || 0).toLocaleString()}`); + console.log(` Relationships: ${(relationships?.n || 0).toLocaleString()}`); + console.log(` Refsets: ${(refsets?.n || 0).toLocaleString()}`); + console.log(` Closure rows: ${(closure?.n || 0).toLocaleString()}`); + console.log(` FTS display: ${(ftsDisplay?.n || 0).toLocaleString()}`); + console.log(` FTS desig.: ${(ftsDesignation?.n || 0).toLocaleString()}`); + console.log(` FTS literal: ${(ftsLiteral?.n || 0).toLocaleString()}`); + + this.logSuccess('Status read complete'); + } finally { + await closeDb(db); + } + } +} + +function validateVersion(input) { + if (!input) return 'Version is required'; + if (!/^\d{8}$/.test(input)) return 'Version must be YYYYMMDD'; + + const year = Number(input.slice(0, 4)); + const month = Number(input.slice(4, 6)); + const day = Number(input.slice(6, 8)); + + if (year < 1900 || year > 2100) return 'Invalid year'; + if (month < 1 || month > 12) return 'Invalid month'; + if (day < 1 || day > 31) return 'Invalid day'; + + return true; +} + +async function promptForSource() { + const answer = await inquirer.prompt([ + { + type: 'input', + name: 'source', + message: 'Source directory:', + validate: (input) => input ? true : 'Source is required' + } + ]); + + return answer.source; +} + +function getRow(db, sql, params) { + return new Promise((resolve, reject) => { + db.get(sql, params, (err, row) => { + if (err) reject(err); + else resolve(row); + }); + }); +} + +async function getCountIfTableExists(db, tableName) { + const exists = await getRow( + db, + `SELECT 1 AS found FROM sqlite_master WHERE type = 'table' AND name = ? LIMIT 1`, + [tableName] + ); + if (!exists) { + return { n: 0 }; + } + return getRow(db, `SELECT COUNT(*) AS n FROM ${tableName}`, []); +} + +function closeDb(db) { + return new Promise((resolve, reject) => { + db.close((err) => { + if (err) reject(err); + else resolve(); + }); + }); +} + +function parseSnomedUri(uri) { + if (!uri || typeof uri !== 'string') { + return { edition: null, version: null }; + } + const m = uri.match(/^https?:\/\/snomed\.info\/sct\/([^/]+)\/version\/(\d{8})$/i); + if (!m) { + return { edition: null, version: null }; + } + return { edition: m[1], version: m[2] }; +} + +function buildDefaultDest(edition, version) { + if (!version) { + return path.resolve('./data/snomed-v0.db'); + } + const label = + edition === '900000000000207008' + ? 'intl' + : edition === '731000124108' + ? 'us' + : String(edition || 'edition'); + return path.resolve(`./data/sct_${label}_${version}.v0.db`); +} + +function shouldAutoAssignDest(dest) { + if (!dest) return true; + const resolved = path.resolve(dest); + return ( + resolved === path.resolve('./data/snomed-v0.db') || + resolved === path.resolve('./data/snomed-sqlite-v0.db') + ); +} + +module.exports = { + SnomedSqliteV0Module +}; diff --git a/tx/importers/sqlite-v2/README.md b/tx/importers/sqlite-v2/README.md new file mode 100644 index 0000000..664ab6a --- /dev/null +++ b/tx/importers/sqlite-v2/README.md @@ -0,0 +1,90 @@ +# SQLite v0 Importers + +This folder contains clean-start terminology import pipelines targeting the shared SQLite v0 schema. + +Naming note: +- `v0i` was used for some earlier local artifacts during indexing/closure experiments. +- Schema version is still SQLite `v0`; there is no separate `v0i` schema. +- Keep one canonical full DB per terminology/version (closure + FTS) and avoid keeping experimental side files in active cache paths. + +Developer docs: +- `docs/SQLITE_RUNTIME_CONFIG_CONTRACT.md` (contract-level key reference) +- `docs/SQLITE_METADATA_DEVELOPER_GUIDE.md` (annotated SNOMED/LOINC/RxNorm examples) + +Metadata policy: +- Importers now emit runtime-driving metadata only (`runtime.*` keys). +- Legacy duplicate keys (`schemaVersion`, `sourceKind`, `display`, etc.) are intentionally not emitted. + +## SNOMED import command + +Use `tx-import`: + +```bash +tx-import snomed-sqlite-v0 import \ + --yes \ + --source /path/to/Snapshot \ + --dest /path/to/sct_intl_20250201.v0.db \ + --edition 900000000000207008 \ + --snomed-version 20250201 \ + --overwrite +``` + +Use `--skip-closure` only for importer bring-up/debug. Production builds should include full closure. +Recursive fallback is available but now opt-in (`runtime.hierarchy.closure.fallbackRecursive=true`); default is fail-closed. + +Importer now also builds broad trigram FTS tables used by runtime text filtering: +- `search_fts_display` +- `search_fts_designation` +- `search_fts_literal` + +Runtime is configured FTS-first with LIKE fallback via `runtime.search` in `cs_config`. + +## RxNorm import command + +Use `tx-import`: + +```bash +tx-import rxnorm-sqlite-v0 import \ + --yes \ + --source /path/to/RxNorm_full_02022026.zip \ + --dest /path/to/rxnorm_02022026.v0.db \ + --rxnorm-version 02022026 \ + --overwrite +``` + +Use `--skip-closure` for faster iteration imports. + +## LOINC import command + +Use `tx-import`: + +```bash +tx-import loinc-sqlite-v0 import \ + --yes \ + --source /path/to/Loinc_2.81.zip \ + --dest /path/to/loinc_2.81.v0.db \ + --loinc-version 2.81 \ + --overwrite +``` + +Use `--skip-closure` for faster iteration imports. + +## Runtime source type + +`Library` now accepts: + +- `sqlite-v0:` (preferred generic source type) +- `snomed-sqlite-v0:` (alias to `sqlite-v0`) +- `loinc-sqlite-v0:` (alias to `sqlite-v0`) +- `rxnorm-sqlite-v0:` (alias to `sqlite-v0`) + +Loader behavior is generic. If specialized factory behavior is needed, metadata tags +(`runtime.behaviorFlags.tags`) are matched against factories registered through +`SqliteRuntimeV0FactoryProvider.registerSpecializedFactory(...)`. + +Use `!` after the type to mark the default for a code system when multiple versions are loaded: + +- `sqlite-v0!:sct_intl_20250201.v0.db` (default) +- `sqlite-v0:sct_us_20250301.v0.db` (additional version) + +Example config: `tx/tx.snomed-v0.yml`. diff --git a/tx/importers/sqlite-v2/import-snomed-v0.js b/tx/importers/sqlite-v2/import-snomed-v0.js new file mode 100644 index 0000000..d892884 --- /dev/null +++ b/tx/importers/sqlite-v2/import-snomed-v0.js @@ -0,0 +1,1167 @@ +'use strict'; + +const fs = require('fs'); +const path = require('path'); +const readline = require('readline'); +const sqlite3 = require('sqlite3').verbose(); + +const BASE_URI = 'http://snomed.info/sct'; + +const IS_A_TYPE_ID = '116680003'; +const FSN_TYPE_ID = '900000000000003001'; +const SYNONYM_TYPE_ID = '900000000000013009'; + +const CHAR_INFERRED = '900000000000011006'; +const CHAR_STATED = '900000000000010007'; +const CHAR_ADDITIONAL = '900000000000227009'; + +const ACCEPTABILITY_PREFERRED = '900000000000548007'; + +const EDGE_SET_INFERRED = 1; +const EDGE_SET_STATED = 2; +const EDGE_SET_ADDITIONAL = 3; + +const MAX_SQL_PARAMS = 900; +const FLUSH_ROW_TARGET = 5000; + +class SnomedSqliteV0Importer { + constructor(config = {}) { + this.config = { + source: config.source, + dest: config.dest, + edition: config.edition || '900000000000207008', + version: config.version, + uri: config.uri, + snapshotOnly: config.snapshotOnly !== false, + skipRefsets: !!config.skipRefsets, + skipClosure: !!config.skipClosure, + verbose: !!config.verbose, + overwrite: !!config.overwrite + }; + + this.db = null; + this.csId = null; + this.auditRunId = null; + + this.preferredDescriptions = new Set(); + this.seenPropertyCodes = new Set(); + this.propertyIdByCode = new Map(); + this.conceptIdByCode = new Map(); + this.nextConceptId = 1; + this.isAPropertyId = null; + + this.stats = { + concepts: 0, + descriptions: 0, + relationships: 0, + concreteValues: 0, + refsets: 0, + refsetMembers: 0, + closureRows: 0, + ftsDisplayRows: 0, + ftsDesignationRows: 0, + ftsLiteralRows: 0 + }; + + const parsed = parseEditionAndVersion(this.config.uri); + if (parsed.edition && !config.edition) { + this.config.edition = parsed.edition; + } + if (parsed.version && !config.version) { + this.config.version = parsed.version; + } + if (!this.config.uri && this.config.edition && this.config.version) { + this.config.uri = `${BASE_URI}/${this.config.edition}/version/${this.config.version}`; + } + } + + static discoverRf2Files(source, { snapshotOnly = true } = {}) { + const files = { + concepts: [], + descriptions: [], + relationships: [], + concreteValues: [], + languageRefsets: [], + refsets: [] + }; + + scanDirectory(source, files, snapshotOnly); + return files; + } + + async run() { + if (!this.config.source || !this.config.dest) { + throw new Error('source and dest are required'); + } + if (!this.config.uri) { + throw new Error('Either uri or (edition + version) is required'); + } + if (!this.config.version) { + throw new Error('Version (YYYYMMDD) is required for v0 imports'); + } + + await this.openDatabase(); + await this.createSchema(); + + try { + await this.startAudit(); + await this.createCodeSystem(); + + const files = SnomedSqliteV0Importer.discoverRf2Files(this.config.source, { + snapshotOnly: this.config.snapshotOnly + }); + + this.log(`Discovered files: concepts=${files.concepts.length}, descriptions=${files.descriptions.length}, relationships=${files.relationships.length}, concrete=${files.concreteValues.length}, languageRefsets=${files.languageRefsets.length}, refsets=${files.refsets.length}`); + + if (files.concepts.length === 0) { + throw new Error('No concept Snapshot files found'); + } + if (files.descriptions.length === 0) { + throw new Error('No description Snapshot files found'); + } + + await this.importLanguagePreferences(files.languageRefsets); + await this.importConcepts(files.concepts); + await this.importDescriptions(files.descriptions); + await this.deriveConceptDisplays(); + await this.importRelationships(files.relationships); + await this.importConcreteValues(files.concreteValues); + + if (!this.config.skipRefsets) { + await this.importRefsets(files.refsets); + } + + await this.buildSearchIndexes(); + + if (!this.config.skipClosure) { + await this.buildClosure(); + } + + await this.writeCsConfig(); + await this.finalizeDatabase(); + await this.completeAudit('success', null); + } catch (error) { + await this.completeAudit('failed', error); + throw error; + } finally { + await this.closeDatabase(); + } + + return { + csId: this.csId, + uri: this.config.uri, + stats: this.stats + }; + } + + async openDatabase() { + const dir = path.dirname(this.config.dest); + fs.mkdirSync(dir, { recursive: true }); + + if (fs.existsSync(this.config.dest)) { + if (!this.config.overwrite) { + throw new Error(`Destination exists: ${this.config.dest} (use --overwrite)`); + } + fs.unlinkSync(this.config.dest); + } + + this.db = await openSqlite(this.config.dest); + + await this.exec('PRAGMA foreign_keys = OFF'); + await this.exec('PRAGMA journal_mode = WAL'); + await this.exec('PRAGMA synchronous = OFF'); + await this.exec('PRAGMA cache_size = -64000'); + await this.exec('PRAGMA temp_store = MEMORY'); + } + + async closeDatabase() { + if (!this.db) return; + await closeSqlite(this.db); + this.db = null; + } + + async createSchema() { + const schemaPath = path.join(__dirname, 'schema-v0.sql'); + const ddl = fs.readFileSync(schemaPath, 'utf8'); + await this.exec(ddl); + } + + async startAudit() { + const result = await this.runSql( + `INSERT INTO load_audit (started_at, source_path, target_db, terminology, edition_code, version, status) + VALUES (CURRENT_TIMESTAMP, ?, ?, 'snomed', ?, ?, 'running')`, + [this.config.source, this.config.dest, this.config.edition || null, this.config.version || null] + ); + this.auditRunId = result.lastID; + } + + async completeAudit(status, error) { + if (!this.auditRunId) return; + + const payload = { + uri: this.config.uri, + stats: this.stats + }; + + if (error) { + payload.error = { + message: error.message, + stack: this.config.verbose ? error.stack : undefined + }; + } + + await this.runSql( + `UPDATE load_audit + SET completed_at = CURRENT_TIMESTAMP, + status = ?, + stats_json = ? + WHERE run_id = ?`, + [status, JSON.stringify(payload), this.auditRunId] + ); + } + + async createCodeSystem() { + const result = await this.runSql( + `INSERT INTO code_system (base_uri, edition_code, version, canonical_uri, name, source_kind) + VALUES (?, ?, ?, ?, ?, ?)`, + [ + BASE_URI, + this.config.edition || null, + this.config.version || null, + this.config.uri, + snomedName(this.config.edition), + 'rf2-snapshot' + ] + ); + + this.csId = result.lastID; + + await this.runSql( + `INSERT OR IGNORE INTO property_def (cs_id, property_code, value_kind, is_hierarchy, display) + VALUES (?, ?, 'concept', 1, 'is-a')`, + [this.csId, IS_A_TYPE_ID] + ); + this.seenPropertyCodes.add(IS_A_TYPE_ID); + const isARow = await this.get( + `SELECT property_id + FROM property_def + WHERE cs_id = ? AND property_code = ?`, + [this.csId, IS_A_TYPE_ID] + ); + if (!isARow) { + throw new Error(`Unable to resolve property_id for ${IS_A_TYPE_ID}`); + } + this.propertyIdByCode.set(IS_A_TYPE_ID, isARow.property_id); + this.isAPropertyId = isARow.property_id; + } + + async importLanguagePreferences(files) { + if (!files || files.length === 0) { + this.log('No language refset files found; preferred flags will be limited'); + return; + } + + this.log(`Importing language preference markers from ${files.length} files...`); + + let count = 0; + for (const file of files) { + for await (const cols of readTsv(file)) { + if (cols.length < 7) continue; + const active = cols[2] === '1'; + const descriptionId = cols[5]; + const acceptabilityId = cols[6]; + + if (!active) continue; + if (acceptabilityId !== ACCEPTABILITY_PREFERRED) continue; + + this.preferredDescriptions.add(descriptionId); + count += 1; + } + } + + this.log(`Captured ${this.preferredDescriptions.size.toLocaleString()} preferred description ids (${count.toLocaleString()} active rows)`); + } + + async importConcepts(files) { + this.log(`Importing concepts from ${files.length} files...`); + + const rows = []; + let imported = 0; + + for (const file of files) { + for await (const cols of readTsv(file)) { + if (cols.length < 5) continue; + + const code = cols[0]; + const active = cols[2] === '1' ? 1 : 0; + const conceptId = this.nextConceptId++; + + rows.push([ + conceptId, + this.csId, + code, + active, + null, + null + ]); + + this.conceptIdByCode.set(code, conceptId); + imported += 1; + + if (rows.length >= FLUSH_ROW_TARGET) { + await this.bulkInsert( + `INSERT INTO concept (concept_id, cs_id, code, active, display, definition)`, + 6, + rows + ); + rows.length = 0; + this.log(` concepts imported: ${imported.toLocaleString()}`); + } + } + } + + if (rows.length > 0) { + await this.bulkInsert( + `INSERT INTO concept (concept_id, cs_id, code, active, display, definition)`, + 6, + rows + ); + } + + this.stats.concepts = imported; + this.log(`Concept import complete: ${imported.toLocaleString()}`); + } + + async importDescriptions(files) { + this.log(`Importing descriptions from ${files.length} files...`); + + const rows = []; + let imported = 0; + + for (const file of files) { + for await (const cols of readTsv(file)) { + if (cols.length < 9) continue; + + const descriptionId = cols[0]; + const active = cols[2] === '1' ? 1 : 0; + const conceptCode = cols[4]; + const languageCode = cols[5] || null; + const typeId = cols[6] || null; + const term = cols[7] || ''; + const conceptId = this.conceptIdByCode.get(conceptCode); + + if (!conceptId) continue; + + const useCode = mapUseCode(typeId); + const preferred = (typeId === FSN_TYPE_ID || this.preferredDescriptions.has(descriptionId)) ? 1 : 0; + + rows.push([ + conceptId, + active, + languageCode, + useCode, + term, + preferred + ]); + + imported += 1; + + if (rows.length >= FLUSH_ROW_TARGET) { + await this.bulkInsert( + `INSERT INTO designation (concept_id, active, language_code, use_code, term, preferred)`, + 6, + rows + ); + rows.length = 0; + this.log(` descriptions imported: ${imported.toLocaleString()}`); + } + } + } + + if (rows.length > 0) { + await this.bulkInsert( + `INSERT INTO designation (concept_id, active, language_code, use_code, term, preferred)`, + 6, + rows + ); + } + + this.stats.descriptions = imported; + this.log(`Description import complete: ${imported.toLocaleString()}`); + } + + async deriveConceptDisplays() { + this.log('Deriving concept display values from designations...'); + + await this.runSql( + `UPDATE concept + SET display = COALESCE( + ( + SELECT d.term + FROM designation d + WHERE d.concept_id = concept.concept_id + AND d.active = 1 + ORDER BY d.designation_id ASC + LIMIT 1 + ), + concept.code + ) + WHERE cs_id = ?`, + [this.csId] + ); + } + + async importRelationships(files) { + this.log(`Importing relationships from ${files.length} files...`); + + const rows = []; + let imported = 0; + + for (const file of files) { + for await (const cols of readTsv(file)) { + if (cols.length < 10) continue; + + const active = cols[2] === '1' ? 1 : 0; + const sourceCode = cols[4]; + const targetCode = cols[5]; + const groupId = parseInt(cols[6], 10) || 0; + const typeId = cols[7] || null; + const characteristicTypeId = cols[8] || null; + const sourceConceptId = this.conceptIdByCode.get(sourceCode); + const targetConceptId = this.conceptIdByCode.get(targetCode); + if (!sourceConceptId || !targetConceptId) continue; + + const propertyId = await this.ensureProperty(typeId, 'concept', typeId === IS_A_TYPE_ID ? 1 : 0); + if (!propertyId) continue; + + rows.push([ + edgeSetIdFromCharacteristic(characteristicTypeId), + sourceConceptId, + propertyId, + targetConceptId, + groupId, + active, + ]); + + imported += 1; + + if (rows.length >= FLUSH_ROW_TARGET) { + await this.bulkInsert( + `INSERT INTO concept_link (edge_set_id, source_concept_id, property_id, target_concept_id, group_id, active)`, + 6, + rows + ); + rows.length = 0; + this.log(` relationships imported: ${imported.toLocaleString()}`); + } + } + } + + if (rows.length > 0) { + await this.bulkInsert( + `INSERT INTO concept_link (edge_set_id, source_concept_id, property_id, target_concept_id, group_id, active)`, + 6, + rows + ); + } + + this.stats.relationships = imported; + this.log(`Relationship import complete: ${imported.toLocaleString()}`); + } + + async importConcreteValues(files) { + if (!files || files.length === 0) { + this.log('No concrete value files found; skipping'); + return; + } + + this.log(`Importing concrete values from ${files.length} files...`); + + const rows = []; + let imported = 0; + + for (const file of files) { + for await (const cols of readTsv(file)) { + if (cols.length < 10) continue; + + const active = cols[2] === '1' ? 1 : 0; + const sourceCode = cols[4]; + const rawValue = cols[5]; + const groupId = parseInt(cols[6], 10) || 0; + const typeId = cols[7] || null; + const characteristicTypeId = cols[8] || null; + const sourceConceptId = this.conceptIdByCode.get(sourceCode); + if (!sourceConceptId) continue; + + const propertyId = await this.ensureProperty(typeId, 'literal', 0); + if (!propertyId) continue; + + const parsed = parseConcreteValue(rawValue); + + rows.push([ + edgeSetIdFromCharacteristic(characteristicTypeId), + sourceConceptId, + propertyId, + groupId, + active, + rawValue, + parsed.valueText, + parsed.valueNum, + parsed.valueBool, + ]); + + imported += 1; + + if (rows.length >= FLUSH_ROW_TARGET) { + await this.bulkInsert( + `INSERT INTO concept_literal (edge_set_id, source_concept_id, property_id, group_id, active, value_raw, value_text, value_num, value_bool)`, + 9, + rows + ); + rows.length = 0; + this.log(` concrete values imported: ${imported.toLocaleString()}`); + } + } + } + + if (rows.length > 0) { + await this.bulkInsert( + `INSERT INTO concept_literal (edge_set_id, source_concept_id, property_id, group_id, active, value_raw, value_text, value_num, value_bool)`, + 9, + rows + ); + } + + this.stats.concreteValues = imported; + this.log(`Concrete value import complete: ${imported.toLocaleString()}`); + } + + async importRefsets(files) { + if (!files || files.length === 0) { + this.log('No refset files found; skipping'); + return; + } + + this.log(`Importing refsets from ${files.length} files...`); + + const memberRows = []; + const seenRefsets = new Map(); + let memberCount = 0; + + for (const file of files) { + for await (const cols of readTsv(file)) { + if (cols.length < 6) continue; + + const active = cols[2] === '1' ? 1 : 0; + if (!active) continue; + + const refsetId = cols[4]; + const componentId = cols[5]; + const conceptId = this.conceptIdByCode.get(componentId); + + if (!refsetId || !componentId) continue; + if (!conceptId) continue; + + const vsUrl = `${BASE_URI}?fhir_vs=refset/${refsetId}`; + + if (!seenRefsets.has(vsUrl)) { + await this.runSql( + `INSERT OR IGNORE INTO value_set (cs_id, url, version, name) + VALUES (?, ?, ?, ?)`, + [this.csId, vsUrl, this.config.version || null, `SNOMED Refset ${refsetId}`] + ); + const row = await this.get( + `SELECT vs_id + FROM value_set + WHERE cs_id = ? AND url = ? AND version = ?`, + [this.csId, vsUrl, this.config.version || null] + ); + if (!row) continue; + seenRefsets.set(vsUrl, row.vs_id); + } + + memberRows.push([ + seenRefsets.get(vsUrl), + conceptId, + 1 + ]); + memberCount += 1; + + if (memberRows.length >= FLUSH_ROW_TARGET) { + await this.bulkInsert( + `INSERT OR IGNORE INTO value_set_member (vs_id, concept_id, active)`, + 3, + memberRows + ); + memberRows.length = 0; + this.log(` refset members imported: ${memberCount.toLocaleString()}`); + } + } + } + + if (memberRows.length > 0) { + await this.bulkInsert( + `INSERT OR IGNORE INTO value_set_member (vs_id, concept_id, active)`, + 3, + memberRows + ); + } + + this.stats.refsets = seenRefsets.size; + this.stats.refsetMembers = memberCount; + this.log(`Refset import complete: ${seenRefsets.size.toLocaleString()} refsets, ${memberCount.toLocaleString()} members`); + } + + async buildClosure() { + this.log('Building transitive closure (is-a, inferred)...'); + if (!this.isAPropertyId) { + throw new Error('Cannot build closure: is-a property_id not resolved'); + } + + await this.exec('BEGIN TRANSACTION'); + try { + await this.exec('DELETE FROM closure'); + + // Temp frontier tables for iterative breadth expansion. + await this.exec(` + CREATE TEMP TABLE IF NOT EXISTS _closure_frontier ( + ancestor_id INTEGER NOT NULL, + descendant_id INTEGER NOT NULL, + depth INTEGER NOT NULL, + PRIMARY KEY (ancestor_id, descendant_id) + ) WITHOUT ROWID; + + CREATE TEMP TABLE IF NOT EXISTS _closure_next ( + ancestor_id INTEGER NOT NULL, + descendant_id INTEGER NOT NULL, + depth INTEGER NOT NULL, + PRIMARY KEY (ancestor_id, descendant_id) + ) WITHOUT ROWID; + + CREATE INDEX IF NOT EXISTS _idx_closure_frontier_desc + ON _closure_frontier(descendant_id, ancestor_id); + `); + + await this.exec('DELETE FROM _closure_frontier'); + await this.exec('DELETE FROM _closure_next'); + + // Self rows (depth 0) go directly into closure. + await this.runSql( + `INSERT OR IGNORE INTO closure (ancestor_id, descendant_id) + SELECT concept_id, concept_id + FROM concept + WHERE cs_id = ?`, + [this.csId] + ); + + // Direct is-a edges (depth 1) populate closure + initial frontier. + await this.runSql( + `INSERT OR IGNORE INTO closure (ancestor_id, descendant_id) + SELECT target_concept_id, source_concept_id + FROM concept_link + WHERE active = 1 + AND property_id = ? + AND edge_set_id = ?`, + [this.isAPropertyId, EDGE_SET_INFERRED] + ); + + await this.runSql( + `INSERT OR IGNORE INTO _closure_frontier (ancestor_id, descendant_id, depth) + SELECT target_concept_id, source_concept_id, 1 + FROM concept_link + WHERE active = 1 + AND property_id = ? + AND edge_set_id = ?`, + [this.isAPropertyId, EDGE_SET_INFERRED] + ); + + let iteration = 0; + let cumulativeNew = 0; + while (true) { + await this.exec('DELETE FROM _closure_next'); + + await this.runSql( + `INSERT OR IGNORE INTO _closure_next (ancestor_id, descendant_id, depth) + SELECT f.ancestor_id, l.source_concept_id, f.depth + 1 + FROM _closure_frontier f + JOIN concept_link l + ON l.property_id = ? + AND l.edge_set_id = ? + AND l.active = 1 + AND l.target_concept_id = f.descendant_id + WHERE NOT EXISTS ( + SELECT 1 + FROM closure c + WHERE c.ancestor_id = f.ancestor_id + AND c.descendant_id = l.source_concept_id + )`, + [this.isAPropertyId, EDGE_SET_INFERRED] + ); + + const nextCountRow = await this.get(`SELECT COUNT(*) AS n FROM _closure_next`); + const nextCount = nextCountRow ? nextCountRow.n : 0; + if (nextCount === 0) { + if (this.config.verbose) { + this.log(` closure iteration ${iteration + 1}: +0 rows`); + } + break; + } + + await this.runSql( + `INSERT OR IGNORE INTO closure (ancestor_id, descendant_id) + SELECT ancestor_id, descendant_id + FROM _closure_next`, + [] + ); + + await this.exec('DELETE FROM _closure_frontier'); + await this.exec( + `INSERT OR IGNORE INTO _closure_frontier (ancestor_id, descendant_id, depth) + SELECT ancestor_id, descendant_id, depth + FROM _closure_next` + ); + + iteration += 1; + cumulativeNew += nextCount; + if (this.config.verbose || iteration % 5 === 0) { + this.log(` closure iteration ${iteration}: +${nextCount.toLocaleString()} rows (cumulative ${cumulativeNew.toLocaleString()})`); + } + } + + await this.exec('DELETE FROM _closure_frontier'); + await this.exec('DELETE FROM _closure_next'); + + await this.exec('COMMIT'); + } catch (error) { + await this.exec('ROLLBACK'); + throw error; + } + + const row = await this.get( + `SELECT COUNT(*) AS n FROM closure`, + [] + ); + + this.stats.closureRows = row ? row.n : 0; + this.log(`Closure complete: ${this.stats.closureRows.toLocaleString()} rows`); + } + + async writeCsConfig() { + const runtimeFilters = { + concept: { operators: ['=', 'is-a', 'descendent-of', 'in'] }, + code: { operators: ['regex'] } + }; + + const runtimeSearch = { + mode: 'fts-broad', + activeOnly: true, + designationActiveOnly: true, + literalActiveOnly: true, + sources: ['display', 'designation', 'literal'], + ftsTables: { + display: 'search_fts_display', + designation: 'search_fts_designation', + literal: 'search_fts_literal' + }, + likeFallback: { enabled: true, caseInsensitive: true } + }; + + const configRows = [ + ['runtime.versioning', JSON.stringify({ algorithm: 'date', partialMatch: true })], + ['runtime.languages', JSON.stringify({ default: 'en' })], + ['runtime.designations', JSON.stringify({ + useMapping: { + fsn: { system: BASE_URI, code: FSN_TYPE_ID, display: 'Fully specified name' }, + synonym: { system: BASE_URI, code: SYNONYM_TYPE_ID, display: 'Synonym (core metadata concept)' } + }, + primaryDisplay: { + source: 'designation', + strategy: 'first-active', + activeOnly: true, + order: 'designation_id_asc' + } + })], + ['runtime.hierarchy', JSON.stringify({ + propertyCode: IS_A_TYPE_ID, + edgeSetId: EDGE_SET_INFERRED, + closure: { enabled: true, fallbackRecursive: false } + })], + ['runtime.filters', JSON.stringify(runtimeFilters)], + ['runtime.implicitValueSets', JSON.stringify({ + all: { queries: ['fhir_vs', 'fhir_vs=all'] }, + isa: { queryPrefix: 'fhir_vs=isa/', filter: { property: 'concept', op: 'is-a', valueFromSuffix: true } }, + refset: { queryPrefix: 'fhir_vs=refset/', filter: { property: 'concept', op: 'in', valueFromSuffix: true } } + })], + ['runtime.status', JSON.stringify({ + inactive: { source: 'concept.active', invert: true }, + deprecated: { source: 'constant', value: false }, + abstract: { source: 'constant', value: false } + })], + ['runtime.search', JSON.stringify(runtimeSearch)], + ['runtime.behaviorFlags', JSON.stringify({ + tags: ['snomed'] + })] + ]; + + for (const [key, value] of configRows) { + await this.runSql( + `INSERT OR REPLACE INTO cs_config (cs_id, key, value) VALUES (?, ?, ?)`, + [this.csId, key, typeof value === 'string' ? value : JSON.stringify(value)] + ); + } + } + + async finalizeDatabase() { + this.log('Finalizing SQLite database...'); + await this.exec('ANALYZE'); + await this.exec('PRAGMA journal_mode = DELETE'); + await this.exec('PRAGMA synchronous = NORMAL'); + await this.exec('VACUUM'); + } + + async buildSearchIndexes() { + this.log('Building broad text search indexes (display/designation/literal)...'); + + await this.exec('BEGIN TRANSACTION'); + try { + await this.exec('DELETE FROM search_fts_display'); + await this.exec('DELETE FROM search_fts_designation'); + await this.exec('DELETE FROM search_fts_literal'); + + const display = await this.runSql( + `INSERT INTO search_fts_display(rowid, term) + SELECT concept_id, trim(display) + FROM concept + WHERE cs_id = ? + AND display IS NOT NULL + AND trim(display) <> ''`, + [this.csId] + ); + + const designation = await this.runSql( + `INSERT INTO search_fts_designation(rowid, term) + SELECT d.designation_id, trim(d.term) + FROM designation d + JOIN concept c ON c.concept_id = d.concept_id + WHERE c.cs_id = ? + AND d.term IS NOT NULL + AND trim(d.term) <> ''`, + [this.csId] + ); + + const literal = await this.runSql( + `INSERT INTO search_fts_literal(rowid, term) + SELECT literal_id, txt + FROM ( + SELECT cl.literal_id AS literal_id, + trim(COALESCE(NULLIF(cl.value_text, ''), NULLIF(cl.value_raw, ''))) AS txt + FROM concept_literal cl + JOIN concept c ON c.concept_id = cl.source_concept_id + WHERE c.cs_id = ? + ) x + WHERE txt IS NOT NULL + AND txt <> ''`, + [this.csId] + ); + + await this.exec(`INSERT INTO search_fts_display(search_fts_display) VALUES ('optimize')`); + await this.exec(`INSERT INTO search_fts_designation(search_fts_designation) VALUES ('optimize')`); + await this.exec(`INSERT INTO search_fts_literal(search_fts_literal) VALUES ('optimize')`); + + await this.exec('COMMIT'); + + this.stats.ftsDisplayRows = display.changes || 0; + this.stats.ftsDesignationRows = designation.changes || 0; + this.stats.ftsLiteralRows = literal.changes || 0; + + this.log( + `Search index complete: display=${this.stats.ftsDisplayRows.toLocaleString()}, ` + + `designation=${this.stats.ftsDesignationRows.toLocaleString()}, ` + + `literal=${this.stats.ftsLiteralRows.toLocaleString()}` + ); + } catch (error) { + await this.exec('ROLLBACK'); + throw error; + } + } + + async ensureProperty(propertyCode, valueKind, isHierarchy) { + if (!propertyCode) return null; + if (this.propertyIdByCode.has(propertyCode)) { + return this.propertyIdByCode.get(propertyCode); + } + + await this.runSql( + `INSERT OR IGNORE INTO property_def (cs_id, property_code, value_kind, is_hierarchy) + VALUES (?, ?, ?, ?)`, + [this.csId, propertyCode, valueKind, isHierarchy] + ); + + const row = await this.get( + `SELECT property_id + FROM property_def + WHERE cs_id = ? AND property_code = ?`, + [this.csId, propertyCode] + ); + if (!row) return null; + + this.propertyIdByCode.set(propertyCode, row.property_id); + this.seenPropertyCodes.add(propertyCode); + return row.property_id; + } + + async bulkInsert(sqlPrefix, columnCount, rows) { + if (!rows.length) return; + + const chunkSize = Math.max(1, Math.floor(MAX_SQL_PARAMS / columnCount)); + await this.exec('BEGIN TRANSACTION'); + + try { + for (let i = 0; i < rows.length; i += chunkSize) { + const chunk = rows.slice(i, i + chunkSize); + const placeholders = chunk.map(() => `(${new Array(columnCount).fill('?').join(',')})`).join(','); + const flat = []; + for (const row of chunk) { + for (const value of row) flat.push(value); + } + + await this.runSql(`${sqlPrefix} VALUES ${placeholders}`, flat); + } + + await this.exec('COMMIT'); + } catch (error) { + await this.exec('ROLLBACK'); + throw error; + } + } + + async runSql(sql, params = []) { + return new Promise((resolve, reject) => { + this.db.run(sql, params, function onRun(err) { + if (err) { + reject(err); + } else { + resolve({ changes: this.changes || 0, lastID: this.lastID }); + } + }); + }); + } + + async get(sql, params = []) { + return new Promise((resolve, reject) => { + this.db.get(sql, params, (err, row) => { + if (err) reject(err); + else resolve(row); + }); + }); + } + + async exec(sql) { + return new Promise((resolve, reject) => { + this.db.exec(sql, (err) => { + if (err) reject(err); + else resolve(); + }); + }); + } + + log(message) { + if (!this.config.verbose) return; + console.log(message); + } +} + +function parseEditionAndVersion(uri) { + if (!uri) return { edition: null, version: null }; + + const match = uri.match(/\/sct\/(\d+)\/version\/(\d{8})/); + if (!match) return { edition: null, version: null }; + + return { + edition: match[1], + version: match[2] + }; +} + +function snomedName(editionCode) { + if (!editionCode) return 'SNOMED CT'; + if (editionCode === '900000000000207008') return 'SNOMED CT International'; + if (editionCode === '731000124108') return 'SNOMED CT US Edition'; + return `SNOMED CT ${editionCode}`; +} + +function edgeSetIdFromCharacteristic(characteristicTypeId) { + if (characteristicTypeId === CHAR_STATED) return EDGE_SET_STATED; + if (characteristicTypeId === CHAR_ADDITIONAL) return EDGE_SET_ADDITIONAL; + if (characteristicTypeId === CHAR_INFERRED) return EDGE_SET_INFERRED; + return EDGE_SET_INFERRED; +} + +function mapUseCode(typeId) { + if (typeId === FSN_TYPE_ID) return 'fsn'; + if (typeId === SYNONYM_TYPE_ID) return 'synonym'; + return typeId || null; +} + +function parseConcreteValue(rawValue) { + if (rawValue === null || rawValue === undefined) { + return { valueText: null, valueNum: null, valueBool: null }; + } + + if (rawValue.startsWith('#')) { + const n = Number(rawValue.slice(1)); + return { + valueText: null, + valueNum: Number.isFinite(n) ? n : null, + valueBool: null + }; + } + + if (rawValue === 'true' || rawValue === 'false') { + return { + valueText: null, + valueNum: null, + valueBool: rawValue === 'true' ? 1 : 0 + }; + } + + if (rawValue.startsWith('"') && rawValue.endsWith('"') && rawValue.length >= 2) { + return { + valueText: rawValue.slice(1, -1), + valueNum: null, + valueBool: null + }; + } + + return { + valueText: rawValue, + valueNum: null, + valueBool: null + }; +} + +function classifyRf2File(filePath, firstLine, files) { + if (!firstLine) return; + + if (firstLine.startsWith('id\teffectiveTime\tactive\tmoduleId\tdefinitionStatusId')) { + files.concepts.push(filePath); + return; + } + + if (firstLine.startsWith('id\teffectiveTime\tactive\tmoduleId\tconceptId\tlanguageCode\ttypeId\tterm\tcaseSignificanceId')) { + files.descriptions.push(filePath); + return; + } + + if (firstLine.startsWith('id\teffectiveTime\tactive\tmoduleId\tsourceId\tdestinationId\trelationshipGroup\ttypeId\tcharacteristicTypeId\tmodifierId')) { + if (filePath.toLowerCase().includes('statedrelationship')) { + return; + } + files.relationships.push(filePath); + return; + } + + if (firstLine.startsWith('id\teffectiveTime\tactive\tmoduleId\tsourceId\tvalue\trelationshipGroup\ttypeId\tcharacteristicTypeId\tmodifierId')) { + files.concreteValues.push(filePath); + return; + } + + if (firstLine.startsWith('id\teffectiveTime\tactive\tmoduleId\trefsetId\treferencedComponentId\tacceptabilityId')) { + files.languageRefsets.push(filePath); + files.refsets.push(filePath); + return; + } + + if (firstLine.startsWith('id\teffectiveTime\tactive\tmoduleId\trefsetId\treferencedComponentId')) { + files.refsets.push(filePath); + } +} + +function scanDirectory(dir, files, snapshotOnly) { + if (!fs.existsSync(dir)) return; + + const entries = fs.readdirSync(dir, { withFileTypes: true }); + for (const entry of entries) { + const fullPath = path.join(dir, entry.name); + + if (entry.isDirectory()) { + if (!entry.name.startsWith('.')) { + scanDirectory(fullPath, files, snapshotOnly); + } + continue; + } + + if (!entry.isFile() || !entry.name.endsWith('.txt')) continue; + + if (snapshotOnly && !fullPath.toLowerCase().includes('snapshot')) { + continue; + } + + const firstLine = readFirstLine(fullPath); + classifyRf2File(fullPath, firstLine, files); + } +} + +function readFirstLine(filePath) { + const fd = fs.openSync(filePath, 'r'); + try { + const buf = Buffer.alloc(1024); + const count = fs.readSync(fd, buf, 0, buf.length, 0); + if (count <= 0) return ''; + + const text = buf.toString('utf8', 0, count); + const index = text.indexOf('\n'); + if (index < 0) return text.trim(); + return text.slice(0, index).replace(/\r$/, ''); + } finally { + fs.closeSync(fd); + } +} + +async function* readTsv(filePath) { + const stream = fs.createReadStream(filePath); + const rl = readline.createInterface({ input: stream, crlfDelay: Infinity }); + + let lineNumber = 0; + for await (const line of rl) { + lineNumber += 1; + if (lineNumber === 1) continue; + if (!line) continue; + yield line.split('\t'); + } +} + +function openSqlite(filePath) { + return new Promise((resolve, reject) => { + const db = new sqlite3.Database(filePath, (err) => { + if (err) reject(err); + else resolve(db); + }); + }); +} + +function closeSqlite(db) { + return new Promise((resolve, reject) => { + db.close((err) => { + if (err) reject(err); + else resolve(); + }); + }); +} + +module.exports = { + SnomedSqliteV0Importer, + constants: { + BASE_URI, + IS_A_TYPE_ID, + FSN_TYPE_ID, + SYNONYM_TYPE_ID, + CHAR_INFERRED, + CHAR_STATED, + CHAR_ADDITIONAL, + ACCEPTABILITY_PREFERRED, + EDGE_SET_INFERRED, + EDGE_SET_STATED, + EDGE_SET_ADDITIONAL + } +}; diff --git a/tx/importers/sqlite-v2/schema-v0.sql b/tx/importers/sqlite-v2/schema-v0.sql new file mode 100644 index 0000000..c50bed6 --- /dev/null +++ b/tx/importers/sqlite-v2/schema-v0.sql @@ -0,0 +1,183 @@ +PRAGMA foreign_keys = OFF; + +CREATE TABLE IF NOT EXISTS code_system ( + cs_id INTEGER PRIMARY KEY AUTOINCREMENT, + base_uri TEXT NOT NULL, + edition_code TEXT, + version TEXT, + canonical_uri TEXT NOT NULL, + name TEXT, + source_kind TEXT, + loaded_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_code_system_base_version + ON code_system(base_uri, version); + +CREATE TABLE IF NOT EXISTS cs_config ( + cs_id INTEGER NOT NULL, + key TEXT NOT NULL, + value TEXT NOT NULL, + PRIMARY KEY (cs_id, key), + FOREIGN KEY (cs_id) REFERENCES code_system(cs_id) +); + +CREATE TABLE IF NOT EXISTS concept ( + concept_id INTEGER PRIMARY KEY, + cs_id INTEGER NOT NULL, + code TEXT NOT NULL, + active INTEGER NOT NULL DEFAULT 1, + display TEXT, + definition TEXT, + FOREIGN KEY (cs_id) REFERENCES code_system(cs_id) +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_concept_cs_code + ON concept(cs_id, code); + +CREATE INDEX IF NOT EXISTS idx_concept_active + ON concept(cs_id, active); + +-- Case-insensitive lookups used by generic property/link filters. +CREATE INDEX IF NOT EXISTS idx_concept_cs_code_nocase + ON concept(cs_id, code COLLATE NOCASE, concept_id); + +CREATE INDEX IF NOT EXISTS idx_concept_cs_display_nocase + ON concept(cs_id, display COLLATE NOCASE, concept_id); + +CREATE TABLE IF NOT EXISTS designation ( + designation_id INTEGER PRIMARY KEY AUTOINCREMENT, + concept_id INTEGER NOT NULL, + active INTEGER NOT NULL DEFAULT 1, + language_code TEXT, + use_code TEXT, + term TEXT NOT NULL, + preferred INTEGER NOT NULL DEFAULT 0, + FOREIGN KEY (concept_id) REFERENCES concept(concept_id) +); + +CREATE INDEX IF NOT EXISTS idx_designation_concept + ON designation(concept_id, active); +CREATE INDEX IF NOT EXISTS idx_designation_concept_pref_term + ON designation(concept_id, preferred DESC, term); + +CREATE TABLE IF NOT EXISTS property_def ( + property_id INTEGER PRIMARY KEY AUTOINCREMENT, + cs_id INTEGER NOT NULL, + property_code TEXT NOT NULL, + value_kind TEXT NOT NULL DEFAULT 'concept', + is_hierarchy INTEGER NOT NULL DEFAULT 0, + display TEXT, + FOREIGN KEY (cs_id) REFERENCES code_system(cs_id) +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_propdef_cs_code + ON property_def(cs_id, property_code); + +CREATE TABLE IF NOT EXISTS concept_link ( + edge_id INTEGER PRIMARY KEY AUTOINCREMENT, + edge_set_id INTEGER NOT NULL DEFAULT 1, + source_concept_id INTEGER NOT NULL, + property_id INTEGER NOT NULL, + target_concept_id INTEGER NOT NULL, + group_id INTEGER NOT NULL DEFAULT 0, + active INTEGER NOT NULL DEFAULT 1, + FOREIGN KEY (source_concept_id) REFERENCES concept(concept_id), + FOREIGN KEY (target_concept_id) REFERENCES concept(concept_id), + FOREIGN KEY (property_id) REFERENCES property_def(property_id) +); + +CREATE INDEX IF NOT EXISTS idx_concept_link_source + ON concept_link(source_concept_id, property_id, edge_set_id, active); + +CREATE INDEX IF NOT EXISTS idx_concept_link_target + ON concept_link(target_concept_id, property_id, edge_set_id, active); + +-- Property-driven link filters perform better with property-first access. +CREATE INDEX IF NOT EXISTS idx_concept_link_prop_active_source + ON concept_link(property_id, edge_set_id, active, source_concept_id, target_concept_id); + +CREATE INDEX IF NOT EXISTS idx_concept_link_prop_active_target + ON concept_link(property_id, edge_set_id, active, target_concept_id, source_concept_id); + +CREATE TABLE IF NOT EXISTS concept_literal ( + literal_id INTEGER PRIMARY KEY AUTOINCREMENT, + edge_set_id INTEGER NOT NULL DEFAULT 1, + source_concept_id INTEGER NOT NULL, + property_id INTEGER NOT NULL, + value_raw TEXT, + value_text TEXT, + value_num REAL, + value_bool INTEGER, + group_id INTEGER NOT NULL DEFAULT 0, + active INTEGER NOT NULL DEFAULT 1, + FOREIGN KEY (source_concept_id) REFERENCES concept(concept_id), + FOREIGN KEY (property_id) REFERENCES property_def(property_id) +); + +CREATE INDEX IF NOT EXISTS idx_concept_literal_source + ON concept_literal(source_concept_id, property_id, edge_set_id, active); + +-- Property/text predicates need value-oriented access paths. +CREATE INDEX IF NOT EXISTS idx_concept_literal_prop_active_text_nocase + ON concept_literal(property_id, active, value_text COLLATE NOCASE, source_concept_id); + +CREATE INDEX IF NOT EXISTS idx_concept_literal_prop_active_raw_nocase + ON concept_literal(property_id, active, value_raw COLLATE NOCASE, source_concept_id); + +-- Broad text search surfaces (rowid-linked, contentless FTS5). +-- These power fast filter text matching across display/designation/literal. +CREATE VIRTUAL TABLE IF NOT EXISTS search_fts_display + USING fts5(term, tokenize='trigram', content=''); + +CREATE VIRTUAL TABLE IF NOT EXISTS search_fts_designation + USING fts5(term, tokenize='trigram', content=''); + +CREATE VIRTUAL TABLE IF NOT EXISTS search_fts_literal + USING fts5(term, tokenize='trigram', content=''); + +CREATE TABLE IF NOT EXISTS closure ( + ancestor_id INTEGER NOT NULL, + descendant_id INTEGER NOT NULL, + PRIMARY KEY (ancestor_id, descendant_id) +) WITHOUT ROWID; + +CREATE TABLE IF NOT EXISTS value_set ( + vs_id INTEGER PRIMARY KEY AUTOINCREMENT, + cs_id INTEGER NOT NULL, + url TEXT NOT NULL, + version TEXT, + name TEXT, + FOREIGN KEY (cs_id) REFERENCES code_system(cs_id) +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_value_set_cs_url_version + ON value_set(cs_id, url, version); + +CREATE TABLE IF NOT EXISTS value_set_member ( + member_id INTEGER PRIMARY KEY AUTOINCREMENT, + vs_id INTEGER NOT NULL, + concept_id INTEGER NOT NULL, + active INTEGER NOT NULL DEFAULT 1, + FOREIGN KEY (vs_id) REFERENCES value_set(vs_id), + FOREIGN KEY (concept_id) REFERENCES concept(concept_id) +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_vsm_unique + ON value_set_member(vs_id, concept_id); + +CREATE INDEX IF NOT EXISTS idx_vsm_vs + ON value_set_member(vs_id); + +CREATE TABLE IF NOT EXISTS load_audit ( + run_id INTEGER PRIMARY KEY AUTOINCREMENT, + started_at TEXT NOT NULL, + completed_at TEXT, + source_path TEXT, + target_db TEXT, + terminology TEXT, + edition_code TEXT, + version TEXT, + status TEXT NOT NULL, + stats_json TEXT +); diff --git a/tx/library.js b/tx/library.js index c1cada9..50335c1 100644 --- a/tx/library.js +++ b/tx/library.js @@ -19,6 +19,8 @@ const {RxNormServicesFactory} = require("./cs/cs-rxnorm"); const {NdcServicesFactory} = require("./cs/cs-ndc"); const {UniiServicesFactory} = require("./cs/cs-unii"); const {SnomedServicesFactory} = require("./cs/cs-snomed"); +const {SqliteRuntimeV0FactoryProvider} = require("./cs/cs-sqlite-runtime-v0"); +require("./cs/cs-sqlite-v0-specializers"); const {CPTServicesFactory} = require("./cs/cs-cpt"); const {OMOPServicesFactory} = require("./cs/cs-omop"); const {PackageValueSetProvider} = require("./vs/vs-package"); @@ -239,6 +241,13 @@ class Library { await this.loadSnomed(details, isDefault, mode); break; + case 'sqlite-v0': + case 'snomed-sqlite-v0': + case 'loinc-sqlite-v0': + case 'rxnorm-sqlite-v0': + await this.loadSqliteV0(details, isDefault, mode); + break; + case 'cpt': await this.loadCpt(details, isDefault, mode); break; @@ -404,6 +413,17 @@ class Library { this.registerProvider(sctFN, sct, isDefault); } + async loadSqliteV0(details, isDefault, mode) { + const sqliteFN = await this.getOrDownloadFile(details); + if (mode === "fetch" || mode === "npm") { + return; + } + const factory = await SqliteRuntimeV0FactoryProvider.createFromMetadata( + this.i18n, sqliteFN, { idPrefix: 'sqlite-v0' } + ); + this.registerProvider(sqliteFN, factory, isDefault); + } + async loadCpt(details, isDefault, mode) { const cptFN = await this.getOrDownloadFile(details); if (mode === "fetch" || mode === "npm") { diff --git a/tx/params.js b/tx/params.js index 2c0528e..62f9837 100644 --- a/tx/params.js +++ b/tx/params.js @@ -557,6 +557,17 @@ class TxParameters { } assign(other) { + this.count = other.count; + this.offset = other.offset; + this.limit = other.limit; + this.filter = other.filter; + this.limitedExpansion = other.limitedExpansion; + this.incompleteOK = other.incompleteOK; + this.abstractOk = other.abstractOk; + this.inferSystem = other.inferSystem; + if (other.supplements) { + this.supplements = new Set(other.supplements); + } if (other.FVersionRules) { this.FVersionRules = [...other.FVersionRules]; } diff --git a/tx/perf-counters.js b/tx/perf-counters.js new file mode 100644 index 0000000..533796f --- /dev/null +++ b/tx/perf-counters.js @@ -0,0 +1,49 @@ +/** + * Lightweight opt-in counters and timers for new code paths. + * Disabled by default; call enable() from test harnesses. + * + * bump(name) — record that a branch was taken + * begin(name) — start a timer, returns a token + * end(token) — stop the timer, accumulate elapsed ms + * snapshot() — { counts: {name: N}, timings: {name: {calls, totalMs}} } + */ + +let enabled = false; +const counts = {}; +const timings = {}; + +function bump(name) { + if (!enabled) return; + counts[name] = (counts[name] || 0) + 1; +} + +function begin(name) { + if (!enabled) return null; + return { name, t0: performance.now() }; +} + +function end(token) { + if (!token) return; + const ms = performance.now() - token.t0; + const entry = timings[token.name] || (timings[token.name] = { calls: 0, totalMs: 0 }); + entry.calls++; + entry.totalMs += ms; +} + +function reset() { + for (const k of Object.keys(counts)) delete counts[k]; + for (const k of Object.keys(timings)) delete timings[k]; +} + +function snapshot() { + const t = {}; + for (const [k, v] of Object.entries(timings)) { + t[k] = { calls: v.calls, totalMs: +v.totalMs.toFixed(2) }; + } + return { counts: { ...counts }, timings: t }; +} + +function enable() { enabled = true; } +function disable() { enabled = false; } + +module.exports = { bump, begin, end, reset, snapshot, enable, disable }; diff --git a/tx/workers/expand.js b/tx/workers/expand.js index b227e45..faf10c3 100644 --- a/tx/workers/expand.js +++ b/tx/workers/expand.js @@ -13,6 +13,7 @@ const {TxParameters} = require("../params"); const {Designations, SearchFilterText} = require("../library/designations"); const {Extensions} = require("../library/extensions"); const {getValuePrimitive, getValueName} = require("../../library/utilities"); +const perfCounters = require('../perf-counters'); const {div} = require("../../library/html"); const {Issue, OperationOutcome} = require("../library/operation-outcome"); const crypto = require('crypto'); @@ -20,9 +21,9 @@ const ValueSet = require("../library/valueset"); const {VersionUtilities} = require("../../library/version-utilities"); // Expansion limits (from Pascal constants) -const UPPER_LIMIT_NO_TEXT = 1000; -const UPPER_LIMIT_TEXT = 1000; -const INTERNAL_LIMIT = 10000; +const UPPER_LIMIT_NO_TEXT = 100000; +const UPPER_LIMIT_TEXT = 100000; +const INTERNAL_LIMIT = 100000; const EXPANSION_DEAD_TIME_SECS = 30; const CACHE_WHEN_DEBUGGING = false; @@ -226,7 +227,17 @@ class ValueSetExpander { } async listDisplaysFromProvider(displays, cs, context) { - await cs.designations(context, displays); + const langs = this.params.workingLanguages?.(); + if (!this.params.includeDesignations && langs && cs.hasAnyDisplays(langs)) { + perfCounters.bump('display.fastPath'); + const d = await cs.display(context); + if (d) { + displays.addDesignation(true, 'active', null, null, d); + } + } else { + perfCounters.bump('display.fullPath'); + await cs.designations(context, displays); + } displays.source = cs; } @@ -755,13 +766,13 @@ class ValueSetExpander { let set = await cs.executeFilters(prep); this.worker.opContext.log('iterate filters'); while (await cs.filterMore(ctxt, set)) { - this.worker.deadCheck('processCodes#4'); const c = await cs.filterConcept(ctxt, set); + this.worker.deadCheck('processCodes#4'); if (await this.passesFilters(cs, c, prep, filters, 0)) { const cds = new Designations(this.worker.i18n.languageDefinitions); await this.listDisplaysFromProvider(cds, cs, c); await this.includeCode(cs, null, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), await cs.deprecated(c), await cs.getCodeStatus(c), - cds, await cs.definition(c), await cs.itemWeight(c), expansion, valueSets, await cs.getExtensions(c), null, await cs.getProperties(c), null, excludeInactive, vsSrc.url); + cds, await cs.definition(c), await cs.itemWeight(c), expansion, valueSets, await cs.getExtensions(c), null, await this._getPropsIfRequested(cs, c), null, excludeInactive, vsSrc.url); } } this.worker.opContext.log('iterate filters done'); @@ -776,7 +787,7 @@ class ValueSetExpander { this.worker.deadCheck('processCodes#3'); cds.clear(); Extensions.checkNoModifiers(cc, 'ValueSetExpander.processCodes', 'set concept reference'); - const cctxt = await cs.locate(cc.code, this.allAltCodes); + let cctxt = await cs.locate(cc.code, this.allAltCodes); if (cctxt && cctxt.context && (!this.params.activeOnly || !await cs.isInactive(cctxt.context)) && await this.passesFilters(cs, cctxt.context, prep, filters, 0)) { await this.listDisplaysFromProvider(cds, cs, cctxt.context); this.listDisplaysFromIncludeConcept(cds, cc, vsSrc); @@ -786,7 +797,7 @@ class ValueSetExpander { ov = await cs.itemWeight(cctxt.context); } let added = await this.includeCode(cs, null, cs.system(), cs.version(), cc.code, await cs.isAbstract(cctxt.context), await cs.isInactive(cctxt.context), await cs.isDeprecated(cctxt.context), await cs.getStatus(cctxt.context), cds, - await cs.definition(cctxt.context), ov, expansion, valueSets, await cs.extensions(cctxt.context), cc.extension, await cs.properties(cctxt.context), null, excludeInactive, vsSrc.url); + await cs.definition(cctxt.context), ov, expansion, valueSets, await cs.extensions(cctxt.context), cc.extension, await this._propsIfRequested(cs, cctxt.context), null, excludeInactive, vsSrc.url); if (added) { this.addToTotal(); } @@ -801,7 +812,7 @@ class ValueSetExpander { const fcl = cset.filter; const prep = await cs.getPrepContext(true); if (!filter.isNull) { - await cs.searchFilter(filter, prep, true); + await cs.searchFilter(prep, filter, true); } if (cs.specialEnumeration()) { @@ -828,8 +839,8 @@ class ValueSetExpander { this.worker.opContext.log('iterate filters'); while (await cs.filterMore(prep, fset[0])) { - this.worker.deadCheck('processCodes#5'); const c = await cs.filterConcept(prep, fset[0]); + this.worker.deadCheck('processCodes#5'); const ok = (!this.params.activeOnly || !await cs.isInactive(c)) && (await this.passesFilters(cs, c, prep, fset, 1)); if (ok) { // count++; @@ -844,7 +855,7 @@ class ValueSetExpander { } let added = await this.includeCode(cs, parent, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), await cs.isDeprecated(c), await cs.getStatus(c), cds, await cs.definition(c), await cs.itemWeight(c), - expansion, null, await cs.extensions(c), null, await cs.properties(c), null, excludeInactive, vsSrc.url); + expansion, null, await cs.extensions(c), null, await this._propsIfRequested(cs, c), null, excludeInactive, vsSrc.url); if (added) { this.addToTotal(); } @@ -880,6 +891,24 @@ class ValueSetExpander { return true; } + async _propsIfRequested(cs, context) { + if (this.params.properties.length) { + perfCounters.bump('props.loaded'); + return await cs.properties(context); + } + perfCounters.bump('props.skipped'); + return null; + } + + async _getPropsIfRequested(cs, context) { + if (this.params.properties.length) { + perfCounters.bump('props.loaded'); + return await cs.getProperties(context); + } + perfCounters.bump('props.skipped'); + return null; + } + async excludeCodes(cset, path, vsSrc, filter, expansion, excludeInactive, notClosed) { this.worker.deadCheck('processCodes#1'); const valueSets = []; @@ -962,11 +991,11 @@ class ValueSetExpander { notClosed.value = true; } const prep = await cs.getPrepContext(true); - const ctxt = await cs.searchFilter(filter, prep, false); + const ctxt = await cs.searchFilter(prep, filter, false); await cs.prepare(prep); while (await cs.filterMore(ctxt)) { - this.worker.deadCheck('processCodes#4'); const c = await cs.filterConcept(ctxt); + this.worker.deadCheck('processCodes#4'); if (await this.passesFilters(cs, c, prep, filters, 0)) { this.excludeCode(cs, await cs.system(), await cs.version(), await cs.code(c), expansion, valueSets, vsSrc.url); } @@ -977,6 +1006,7 @@ class ValueSetExpander { if (cset.concept) { this.worker.opContext.log('iterate concepts'); const cds = new Designations(this.worker.i18n.languageDefinitions); + for (const cc of cset.concept) { this.worker.deadCheck('processCodes#3'); cds.clear(); @@ -998,7 +1028,7 @@ class ValueSetExpander { this.worker.opContext.log('prep filters'); const prep = await cs.getPrepContext(true); if (!filter.isNull) { - await cs.searchFilter(filter, prep, true); + await cs.searchFilter(prep, filter, true); } if (cs.specialEnumeration()) { @@ -1019,8 +1049,8 @@ class ValueSetExpander { } //let count = 0; while (await cs.filterMore(prep, fset[0])) { - this.worker.deadCheck('processCodes#5'); const c = await cs.filterConcept(prep, fset[0]); + this.worker.deadCheck('processCodes#5'); const ok = (!this.params.activeOnly || !await cs.isInactive(c)) && (await this.passesFilters(cs, c, prep, fset, 1)); if (ok) { //count++; @@ -1053,7 +1083,7 @@ class ValueSetExpander { const cds = new Designations(this.worker.i18n.languageDefinitions); await this.listDisplaysFromProvider(cds, cs, context); const t = await this.includeCode(cs, parent, await cs.system(), await cs.version(), context.code, await cs.isAbstract(context), await cs.isInactive(context), await cs.isDeprecated(context), await cs.getStatus(context), cds, await cs.definition(context), - await cs.itemWeight(context), expansion, imports, await cs.extensions(context), null, await cs.properties(context), null, excludeInactive, srcUrl); + await cs.itemWeight(context), expansion, imports, await cs.extensions(context), null, await this._propsIfRequested(cs, context), null, excludeInactive, srcUrl); if (t != null) { result++; } @@ -1122,20 +1152,198 @@ class ValueSetExpander { this.worker.opContext.log('compose #2'); + // Try expandForValueSet: group includes+excludes by system + const excludeInactive = this.excludeInactives(source); + const handledIncludes = await this._tryExpandForValueSet( + source, filter, expansion, excludeInactive, notClosed + ); + + // Determine which systems were fully handled (includes+excludes baked in) + const handledSystems = new Set(); + const includes = source.jsonObj.compose.include || []; + for (const idx of handledIncludes) { + if (includes[idx]?.system) handledSystems.add(includes[idx].system); + } + + // Process excludes for unhandled systems (populates this.excluded safety net) let i = 0; for (const c of source.jsonObj.compose.exclude || []) { this.worker.deadCheck('handleCompose#4'); - await this.excludeCodes(c, "ValueSet.compose.exclude["+i+"]", source, filter, expansion, this.excludeInactives(source), notClosed); + if (!handledSystems.has(c.system)) { + await this.excludeCodes(c, "ValueSet.compose.exclude["+i+"]", source, filter, expansion, this.excludeInactives(source), notClosed); + } + i++; } + // Fall back to per-include processing for anything not handled i = 0; - for (const c of source.jsonObj.compose.include || []) { + for (const c of includes) { this.worker.deadCheck('handleCompose#5'); - await this.includeCodes(c, "ValueSet.compose.include["+i+"]", source, filter, expansion, this.excludeInactives(source), notClosed); + if (!handledIncludes.has(i)) { + await this.includeCodes(c, "ValueSet.compose.include["+i+"]", source, filter, expansion, excludeInactive, notClosed); + } i++; } } + /** + * Group includes/excludes by code system and try expandForValueSet on each. + * Returns a Set of include indices that were successfully handled. + */ + async _tryExpandForValueSet(source, filter, expansion, excludeInactive, notClosed) { + const handled = new Set(); + const includes = source.jsonObj.compose.include || []; + const excludes = source.jsonObj.compose.exclude || []; + + // Group eligible includes by system + const bySystem = new Map(); // system → { cs, indices, includes, excludes } + for (let idx = 0; idx < includes.length; idx++) { + const cset = includes[idx]; + // Only eligible if it has a system and no valueSet references + if (!cset.system || (cset.valueSet && cset.valueSet.length > 0)) continue; + // Must have concept or filter (not bare "whole code system" which has + // special enumeration / iterator logic) + if (!cset.concept && !cset.filter) continue; + + let entry = bySystem.get(cset.system); + if (!entry) { + entry = { indices: [], includes: [], excludes: [] }; + bySystem.set(cset.system, entry); + } + entry.indices.push(idx); + entry.includes.push({ + concepts: cset.concept || null, + filters: (cset.filter || []).map(f => ({ property: f.property, op: f.op, value: f.value })), + }); + } + + if (bySystem.size === 0) return handled; + + // Add matching excludes to each system's group + for (const cset of excludes) { + if (!cset.system) continue; + const entry = bySystem.get(cset.system); + if (!entry) continue; + entry.excludes.push({ + concepts: cset.concept || null, + filters: (cset.filter || []).map(f => ({ property: f.property, op: f.op, value: f.value })), + }); + } + + // Try expandForValueSet for each system + for (const [system, group] of bySystem) { + const cset0 = includes[group.indices[0]]; + const cs = await this.worker.findCodeSystem( + system, cset0.version, this.params, + ['complete', 'fragment'], false, false, true, null, this.requiredSupplements + ); + if (!cs || !cs.expandForValueSet) continue; + + // Paging is only safe when this is the sole system being expanded. + // With multiple systems the global offset/count doesn't map to any single + // system's result set — applying it would skip or lose codes. + // CONTRACT: if offset/count are non-null and the provider returns an iterable, + // the provider MUST have applied them. We zero this.offset below, so the + // provider's SQL is the sole paging authority. If the provider can't handle + // paging, it should return null to fall back to the framework's iterator. + const singleSystem = bySystem.size === 1; + const spec = { + includes: group.includes, + excludes: group.excludes, + activeOnly: !!(this.params.activeOnly || excludeInactive), + searchText: filter.isNull ? null : filter.text, + includeDesignations: !!this.params.includeDesignations, + properties: this.params.properties || [], + offset: singleSystem && this.offset > 0 ? this.offset : null, + count: singleSystem && this.count > 0 ? this.count : null, + }; + + const _t = perfCounters.begin('expandForValueSet'); + let result; + try { + result = await cs.expandForValueSet(spec); + } catch (e) { + perfCounters.end(_t); + throw e; + } + perfCounters.end(_t); + + if (result == null) { + perfCounters.bump('expandForValueSet.fallback'); + continue; + } + + perfCounters.bump('expandForValueSet.handled'); + this.worker.opContext.log('expandForValueSet handled ' + system); + + // Pre-flight: supplements, canonical status, used-codesystem + for (const idx of group.indices) { + this.worker.checkSupplements(cs, includes[idx], this.requiredSupplements, this.usedSupplements); + } + this.checkProviderCanonicalStatus(expansion, cs, this.valueSet); + const sv = this.canonical(await cs.system(), await cs.version()); + this.addParamUri(expansion, 'used-codesystem', sv); + + // Hierarchy is not possible with expandForValueSet + this.canBeHierarchy = false; + this.noTotal(); + + // Iterate results through includeCode + try { + for await (const entry of result) { + this.worker.deadCheck('expandForValueSet#iter'); + const cds = new Designations(this.worker.i18n.languageDefinitions); + if (entry.designations) { + for (const d of entry.designations) { + cds.addDesignation(false, 'active', d.language, d.use, d.value); + } + } + if (entry.display) { + cds.addDesignation(true, 'active', 'en', null, entry.display); + } + await this.includeCode( + cs, null, + entry.system || await cs.system(), + entry.version || await cs.version(), + entry.code, + entry.isAbstract || false, + entry.isInactive || false, + entry.isDeprecated || false, + entry.status || null, + cds, + entry.definition || null, + entry.itemWeight || null, + expansion, null, + entry.extensions || null, null, + entry.properties || null, null, + excludeInactive, + source.url + ); + } + } catch (e) { + if (e.finished) { + // setFinished sentinel — paging limit reached, normal + } else { + throw e; + } + } + + // Only zero the framework offset when we actually passed paging params + // to the provider. Otherwise the framework still needs to apply offset + // during finalization. + if (singleSystem) { + this.offset = 0; + } + + // Mark all includes for this system as handled + for (const idx of group.indices) { + handled.add(idx); + } + } + + return handled; + } + excludeInactives(source) { return source.jsonObj.compose && source.jsonObj.compose.inactive != undefined && !source.jsonObj.compose.inactive; } diff --git a/tx/workers/validate.js b/tx/workers/validate.js index 36ba555..a81652e 100644 --- a/tx/workers/validate.js +++ b/tx/workers/validate.js @@ -768,7 +768,7 @@ class ValueSetChecker { ver.value = cs.version(); contentMode.value = cs.contentMode(); let msg = ''; - excluded = (system === '%%null%%' || cs.system() === system) && await this.checkConceptSet(path, 'not in', cs, cc, code, displays, this.valueSet, msg, inactive, normalForm, vstatus, op, vcc); + excluded = (system === '%%null%%' || cs.system() === system) && await this.checkConceptSet(path, 'not in', cs, cc, code, displays, this.valueSet, msg, inactive, normalForm, vstatus, op, vcc, messages); if (msg) { messages.push(msg); } diff --git a/tx/workers/worker.js b/tx/workers/worker.js index c1c749d..ee34ca3 100644 --- a/tx/workers/worker.js +++ b/tx/workers/worker.js @@ -7,6 +7,7 @@ const {Issue} = require("../library/operation-outcome"); const {Languages} = require("../../library/languages"); const {ConceptMap} = require("../library/conceptmap"); const {Renderer} = require("../library/renderer"); +const perfCounters = require('../perf-counters'); /** * Custom error for terminology setup issues @@ -43,6 +44,7 @@ class TerminologyWorker { this.noCacheThisOne = false; this.params = null; // Will be set by subclasses this.renderer = new Renderer(i18n, languages, provider); + this._providerCache = new Map(); } /** @@ -144,6 +146,17 @@ class TerminologyWorker { if (!noVParams) { version = this.determineVersionBase(url, version, params); } + + // Memoize by resolved url|version|supplements within a single request + const suppKey = statedSupplements ? [...statedSupplements].sort().join(',') : ''; + const kindsKey = Array.isArray(kinds) ? kinds.join(',') : String(kinds); + const cacheKey = `${url}|${version}|${kindsKey}|${suppKey}`; + if (this._providerCache.has(cacheKey)) { + perfCounters.bump('cache.hit'); + return this._providerCache.get(cacheKey); + } + perfCounters.bump('cache.miss'); + let codeSystemResource = null; let provider = null; const supplements = this.loadSupplements(url, version, statedSupplements); @@ -184,6 +197,7 @@ class TerminologyWorker { if (checkVer) { this.checkVersion(url, provider.version(), params, provider.versionAlgorithm(), op); } + this._providerCache.set(cacheKey, provider); } return provider;