|
| 1 | +# Faster ValueSet Expansion via `expandForValueSet` |
| 2 | + |
| 3 | +## The Problem |
| 4 | + |
| 5 | +The `CodeSystemProvider` interface (`tx/cs/cs-api.js`) treats each provider as an |
| 6 | +**iterator + attribute oracle**: given a filter, produce matching codes one at a |
| 7 | +time, then answer per-code questions (`isInactive`, `getStatus`, `designations`, |
| 8 | +etc.). The expansion worker assembles the result by driving this loop. |
| 9 | + |
| 10 | +For SQL-backed providers like RxNorm and LOINC, this creates two performance |
| 11 | +problems: |
| 12 | + |
| 13 | +1. **Per-code overhead.** The worker calls 8–10 methods per code. If contexts aren't |
| 14 | + self-sufficient, each call can trigger a SQL query. For RxNorm, expanding 1100 |
| 15 | + codes originally cost 3301 SQL queries (1 filter + 1100 × 3 per-code status |
| 16 | + lookups). This was fixed independently by making contexts carry all attributes |
| 17 | + at creation time — zero interface changes required. |
| 18 | + |
| 19 | +2. **No paging push-down.** To return page 11 (offset=1000, count=100), the worker |
| 20 | + processes all 1100 codes, then discards the first 1000. It can't tell the |
| 21 | + provider to skip ahead because paging is a ValueSet-level concern (multiple |
| 22 | + includes, excludes, dedup across code systems). A SQL provider could do |
| 23 | + `LIMIT 100 OFFSET 1000` and return in ~1ms instead of ~200ms. |
| 24 | + |
| 25 | +Similarly, excludes and `activeOnly` filtering happen code-by-code in the worker. |
| 26 | +A SQL provider could handle all of these in the WHERE clause. |
| 27 | + |
| 28 | +We briefly considered adding `{ offset, count, activeOnly }` options to the |
| 29 | +existing `executeFilters()` method, but this creates awkward partially-resolved |
| 30 | +states — the provider handles some post-filters but not others, and the worker |
| 31 | +has to know which. Piecemeal push-down doesn't compose well. |
| 32 | + |
| 33 | +## The Interface Change |
| 34 | + |
| 35 | +One new optional method on `CodeSystemProvider`: |
| 36 | + |
| 37 | +```js |
| 38 | +async expandForValueSet(spec) { |
| 39 | + // Returns an AsyncIterable<ExpandedEntry> or null (can't handle → fall back) |
| 40 | + return null; |
| 41 | +} |
| 42 | +``` |
| 43 | + |
| 44 | +Instead of threading individual parameters through the 10-call-per-code loop, |
| 45 | +**give the provider the full hull of includes and excludes that apply to its code |
| 46 | +system** and let it handle everything in one shot. |
| 47 | + |
| 48 | +### Input |
| 49 | + |
| 50 | +The worker groups compose entries by code system and builds: |
| 51 | + |
| 52 | +```js |
| 53 | +{ |
| 54 | + includes: [ // compose.include blocks for this CS |
| 55 | + { concepts: [{code, display?}], // explicit code list (may be null) |
| 56 | + filters: [{property, op, value}] // property filters (may be empty) |
| 57 | + }, ... |
| 58 | + ], |
| 59 | + excludes: [ // compose.exclude blocks for this CS |
| 60 | + { concepts: [{code}], |
| 61 | + filters: [{property, op, value}] |
| 62 | + }, ... |
| 63 | + ], |
| 64 | + activeOnly: boolean, // exclude inactive codes |
| 65 | + searchText: string | null, // expansion 'filter' parameter |
| 66 | + includeDesignations: boolean, // whether designations are needed |
| 67 | + properties: string[], // which properties to include |
| 68 | + languages: Languages, // requested display languages |
| 69 | + |
| 70 | + // Paging — non-null only when safe (single code system); must apply if present |
| 71 | + offset: number | null, |
| 72 | + count: number | null, |
| 73 | +} |
| 74 | +``` |
| 75 | +
|
| 76 | +### Output |
| 77 | +
|
| 78 | +`AsyncIterable<ExpandedEntry>` where each entry is a fully-resolved code: |
| 79 | +
|
| 80 | +```js |
| 81 | +{ |
| 82 | + code: string, |
| 83 | + display: string, |
| 84 | + isAbstract: boolean, |
| 85 | + isInactive: boolean, |
| 86 | + isDeprecated: boolean, |
| 87 | + status: string | null, |
| 88 | + definition: string | null, |
| 89 | + designations: [{language, use, value}], |
| 90 | + properties: [{code, valueType, value}], |
| 91 | + extensions: [...] | null, |
| 92 | +} |
| 93 | +``` |
| 94 | +
|
| 95 | +The worker iterates these entries via `includeCode()`, which still handles dedup |
| 96 | +across code systems, import filtering, expansion limits, and FHIR object construction. |
| 97 | +
|
| 98 | +### Paging contract |
| 99 | +
|
| 100 | +`offset` and `count` are **non-null only when the worker can verify they're |
| 101 | +safe** (currently: single code system in the compose). When provided: |
| 102 | +
|
| 103 | +- The provider **must apply them** (via SQL `LIMIT/OFFSET`). The worker zeros |
| 104 | + its own offset after a successful `expandForValueSet` call, so the provider's |
| 105 | + SQL is the sole paging authority. |
| 106 | +- If the provider can handle filters and excludes but **not** paging, it should |
| 107 | + return `null` to fall back to the framework's iterator path, which applies |
| 108 | + offset/count during finalization. |
| 109 | +
|
| 110 | +When `null` (multi-system compose, or offset/count not requested): |
| 111 | +
|
| 112 | +- The provider returns all matching codes. The framework accumulates them into |
| 113 | + `fullList` and applies paging during finalization. Still faster than baseline |
| 114 | + because the provider handles filters, excludes, and activeOnly in SQL — just |
| 115 | + without the LIMIT/OFFSET shortcut. |
| 116 | +
|
| 117 | +Each expansion request is fully stateless — there is no cross-request memory of |
| 118 | +previous pages. A request for offset=1000 re-derives the full ordered result set |
| 119 | +and skips the first 1000 entries. SQL `LIMIT/OFFSET` lets the provider do this skip in |
| 120 | +the B-tree index instead of iterating through rows. |
| 121 | +
|
| 122 | +### Why the provider gets excludes |
| 123 | +
|
| 124 | +With paging push-down, SQL-side excludes are **15x faster** than JS-side filtering |
| 125 | +(1.1ms vs 15.9ms at offset=10000). The B-tree index handles seek + exclude together. |
| 126 | +The provider needs excludes to make paging accurate. |
| 127 | +
|
| 128 | +### Fallback protocol |
| 129 | +
|
| 130 | +1. Provider returns `null` → worker falls back to the existing iterator-oracle |
| 131 | + pattern, completely unchanged. SNOMED and any provider that doesn't implement |
| 132 | + this method are unaffected. |
| 133 | +2. Provider returns an iterable → worker iterates entries, skips the framework's |
| 134 | + manual `excludeCodes()` and `includeCodes()` paths for that code system. |
| 135 | +3. `this.offset = 0` after expansion — the provider's SQL handled OFFSET; prevents |
| 136 | + the framework's finalization from double-skipping. |
| 137 | +
|
| 138 | +### Why `better-sqlite3` |
| 139 | +
|
| 140 | +The async `sqlite3` package can't support this pattern: `db.each()` can't abort |
| 141 | +early, so there's no way to stop after N rows. `better-sqlite3` provides |
| 142 | +synchronous lazy cursors via `stmt.iterate()` — the provider can break after the |
| 143 | +page is filled. It's also 2x faster for bulk loads. |
| 144 | +
|
| 145 | +The sync API is fine here — SQLite operations are inherently single-threaded and |
| 146 | +in-process. The async wrappers add overhead for no concurrency benefit. |
| 147 | +
|
| 148 | +--- |
| 149 | +
|
| 150 | +## RxNorm Implementation |
| 151 | +
|
| 152 | +### SQL strategies |
| 153 | +
|
| 154 | +The RxNorm provider maps each filter/option to SQL: |
| 155 | +
|
| 156 | +| Filter | SQL | Index used | |
| 157 | +|--------|-----|------------| |
| 158 | +| TTY (e.g., SBD) | `WHERE TTY IN (...)` | `(SAB, TTY, RXCUI)` — covers ORDER BY | |
| 159 | +| STY (semantic type) | `JOIN rxnsty ON rxnsty.RXCUI = rxnconso.RXCUI WHERE TUI = ?` | `X_RXNSTY_2(TUI)` drives, probes `X_RXNCONSO_1(RXCUI)` | |
| 160 | +| Concepts | `WHERE RXCUI IN (...)` with `+SAB = @sab` (unary `+` suppresses SAB index) | `X_RXNCONSO_1(RXCUI)` | |
| 161 | +| Excludes | `AND RXCUI NOT IN (@p1, @p2, ...)` | — | |
| 162 | +| activeOnly | `AND SUPPRESS <> '1'` | — | |
| 163 | +| searchText | `AND UPPER(STR) LIKE @pattern` | — | |
| 164 | +
|
| 165 | +**GROUP BY for JOINs**: When STY joins are present, rxnconso has 1–8 rows per RXCUI |
| 166 | +(different TTY values). `GROUP BY RXCUI` deduplicates at the SQL level so |
| 167 | +LIMIT/OFFSET counts unique codes, not raw rows. |
| 168 | +
|
| 169 | +**Index lesson**: Adding indexes can *hurt* SQLite — a composite `rxnsty(TUI, RXCUI)` |
| 170 | +index caused the planner to switch to a worse strategy. Prefer query shaping (JOIN |
| 171 | +order, unary `+`) over explicit index hints. |
| 172 | +
|
| 173 | +### Results (13 tests) |
| 174 | +
|
| 175 | +``` |
| 176 | +Test | Opt (ms) | Base (ms) | Speedup | Result |
| 177 | +------------------------------|----------|----------|---------|------- |
| 178 | +filter-tty-sbd-10 | 6.8 | 248.7 | 36.5x | ✅ exact |
| 179 | +concept-5 | 2.5 | 3.8 | 1.5x | ✅ exact |
| 180 | +exclude-concepts-3 | 3.2 | 235.2 | 73.8x | ✅ exact |
| 181 | +multi-include-2 | 63.9 | 471.0 | 7.4x | ✅ sets equal (40k) |
| 182 | +activeonly-sbd | 2.2 | 194.9 | 82.3x | ✅ exact |
| 183 | +filter-tty-in-multi | 1.4 | 476.0 | 341.6x | ✅ exact |
| 184 | +filter-sty-t200 | 217.1 | 1521.6 | 7.0x | ✅ exact |
| 185 | +paged-offset-100 | 1.3 | 228.2 | 177.4x | ✅ exact |
| 186 | +text-aspirin | 1.8 | TIMEOUT | ∞ | ✅ opt works |
| 187 | +exclude-filter | 3.9 | 320.5 | 83.2x | ✅ exact |
| 188 | +multi-include-concept+filter | 140.2 | 189.5 | 1.4x | ✅ sets equal (23k) |
| 189 | +combo-active-text-paged | 1.4 | TIMEOUT | ∞ | ✅ opt works |
| 190 | +multi-include-multi-exclude | 99.7 | 673.7 | 6.8x | ✅ sets equal (40k) |
| 191 | +``` |
| 192 | +
|
| 193 | +**All 13 pass.** Median speedup ~37x. Best case 342x (multi-value IN with index). |
| 194 | +
|
| 195 | +--- |
| 196 | +
|
| 197 | +## LOINC Implementation |
| 198 | +
|
| 199 | +### Baseline problem |
| 200 | +
|
| 201 | +LOINC's existing provider loads **all 240k codes into memory** at startup. Filter |
| 202 | +queries run SQL to find matching CodeKeys, then **materialize the entire result |
| 203 | +set** into an array before iterating. For large filters this takes 2–5 seconds: |
| 204 | +STATUS=ACTIVE materializes 163k rows, CLASSTYPE=Lab materializes 73k rows. |
| 205 | +
|
| 206 | +The LOINC database is more normalized than RxNorm: |
| 207 | +- `Codes` (240k) — CodeKey PK, Code, Type (1=Code, 2=Part, 3=AnswerList, 4=Answer), StatusKey |
| 208 | +- `Relationships` (1.2M) — links codes to parts (COMPONENT, CLASS, SYSTEM, SCALE_TYP, etc.) |
| 209 | +- `Properties` (347k) + `PropertyValues` — key-value attributes (CLASSTYPE, ORDER_OBS) |
| 210 | +
|
| 211 | +### SQL strategies |
| 212 | +
|
| 213 | +| Filter | SQL | Notes | |
| 214 | +|--------|-----|-------| |
| 215 | +| Relationship (COMPONENT, CLASS, etc.) | `JOIN Relationships r ON r.TargetKey = (SELECT CodeKey FROM Codes WHERE Code = ?) AND r.RelationshipTypeKey = ? AND r.SourceKey = c.CodeKey` | Naturally scopes to Type=1 codes | |
| 216 | +| STATUS | `WHERE c.StatusKey = ?` | Direct column match | |
| 217 | +| CLASSTYPE | `JOIN Properties p ... JOIN PropertyValues pv ... AND pv.Value = ?` | Value "1" → "Laboratory class" via lookup | |
| 218 | +| LIST (answers-for) | `JOIN Relationships r ON r.SourceKey = (SELECT ...) AND r.RelationshipTypeKey = 40 AND r.TargetKey = c.CodeKey` | Reversed direction — list is source, answers are targets | |
| 219 | +| activeOnly | `WHERE c.StatusKey = 1` | — | |
| 220 | +
|
| 221 | +**Multi-include via UNION**: Each compose include becomes a separate SELECT. Multiple |
| 222 | +includes are `UNION ALL`'d, with the outer query applying `GROUP BY Code` for dedup: |
| 223 | +
|
| 224 | +```sql |
| 225 | +SELECT Code, Description FROM ( |
| 226 | + SELECT c.Code, c.CodeKey, d.Description FROM Codes c |
| 227 | + JOIN Descriptions d ON d.CodeKey = c.CodeKey AND d.DescriptionTypeKey = 1 |
| 228 | + JOIN Relationships r1 ON ... -- include 1 filters |
| 229 | + UNION ALL |
| 230 | + SELECT c.Code, c.CodeKey, d.Description FROM Codes c |
| 231 | + JOIN Descriptions d ON d.CodeKey = c.CodeKey AND d.DescriptionTypeKey = 1 |
| 232 | + JOIN Relationships r2 ON ... -- include 2 filters |
| 233 | +) GROUP BY Code ORDER BY CodeKey LIMIT ? OFFSET ? |
| 234 | +``` |
| 235 | +
|
| 236 | +This avoids AND-semantics where JOINs from different includes would all need to |
| 237 | +match simultaneously. |
| 238 | +
|
| 239 | +**Key decisions:** |
| 240 | +- **ORDER BY CodeKey** (not Code string) matches baseline iteration order |
| 241 | +- **No blanket Type=1 filter** — relationship JOINs scope naturally; a Type=1 |
| 242 | + restriction would break LIST queries (answers are Type=4) |
| 243 | +- **Existing indexes sufficient** — `RelationshipsTarget`, `PropertiesCode1`, |
| 244 | + `CodesCode` cover all patterns without new indexes |
| 245 | +- **Concept-only includes fall back** — `expandForValueSet` returns `null`, lets |
| 246 | + the framework handle via `locate()` (efficient for small lists) |
| 247 | +
|
| 248 | +### Results (14 tests) |
| 249 | +
|
| 250 | +``` |
| 251 | +Test | Opt (ms) | Base (ms) | Speedup | Result |
| 252 | +------------------------------|----------|-----------|---------|--------------------------- |
| 253 | +filter-component-bacteria | 11.3 | 31.4 | 2.7x | ✅ exact |
| 254 | +filter-class-chem | 13.8 | 941.0 | 68.2x | ✅ exact |
| 255 | +filter-scale-qn | 50.8 | 2611.3 | 51.4x | ✅ exact |
| 256 | +filter-system-ser | 21.0 | 680.4 | 32.4x | ✅ exact |
| 257 | +concept-5 | 2.8 | 1.9 | 0.6x | ✅ sets equal (5) |
| 258 | +exclude-concepts | 2.3 | 57.4 | 24.9x | ✅ exact |
| 259 | +activeonly-class | 14.0 | 119.4 | 8.7x | ✅ exact |
| 260 | +filter-list-ll150 | 2.3 | 14.7 | 6.4x | ✅ sets equal (255) |
| 261 | +filter-classtype-lab | 82.2 | 2020.7 | 24.6x | ✅ exact |
| 262 | +paged-class-offset-100 | 14.0 | 102.8 | 7.5x | ✅ exact |
| 263 | +multi-filter-comp-scale | 2.4 | 80.0 | 34.6x | ✅ exact |
| 264 | +filter-status-active | 50.1 | 5324.6 | 106.4x | ✅ exact |
| 265 | +text-glucose | 1.3 | 1.4 | 1.1x | ✅ exact |
| 266 | +multi-include-2-components | 4.2 | 87.6 | 25.0x | ✅ sets equal (743) |
| 267 | +``` |
| 268 | +
|
| 269 | +**All 14 pass.** Median speedup ~25x. Biggest wins: STATUS=ACTIVE 5.3s → 50ms, |
| 270 | +SCALE_TYP=Qn 2.6s → 51ms, CLASSTYPE=Lab 2.0s → 82ms. |
| 271 | +
|
| 272 | +--- |
| 273 | +
|
| 274 | +## Validation |
| 275 | +
|
| 276 | +**Unit tests:** LOINC 37/37 pass, RxNorm 45/45 skip (require raw import data). |
| 277 | +
|
| 278 | +**Replay tests** (18 captured production queries): |
| 279 | +- RxNorm: 3/3 ✅ (validate-code, all 200) |
| 280 | +- LOINC: 4/4 functionally correct (1 exact match, 3 now return 200 where production |
| 281 | + returned 422 — we handle queries production rejected) |
| 282 | +- SNOMED: 5/5 expected failures (not loaded) |
| 283 | +- Other: 6/6 ✅ (batch-validate, multi-system expansions) |
| 284 | +
|
| 285 | +## Summary |
| 286 | +
|
| 287 | +The entire interface change is **one optional method** — `expandForValueSet`. It's |
| 288 | +additive: providers that don't implement it are completely unaffected. The method |
| 289 | +gives SQL-backed providers the information they need (the full compose hull, |
| 290 | +activeOnly, excludes, paging) to push everything into a single query with |
| 291 | +LIMIT/OFFSET. |
| 292 | +
|
| 293 | +| What | RxNorm | LOINC | |
| 294 | +|------|--------|-------| |
| 295 | +| Tests | 13/13 pass | 14/14 pass | |
| 296 | +| Median speedup | ~37x | ~25x | |
| 297 | +| Best speedup | 342x | 106x | |
| 298 | +| Biggest absolute win | text+combo: TIMEOUT → <2ms | STATUS=ACTIVE: 5.3s → 50ms | |
| 299 | +| New indexes needed | None | None | |
| 300 | +| Existing tests broken | None | None | |
0 commit comments