From 99cabe07b292f37769bcba5f848a719994621cba Mon Sep 17 00:00:00 2001 From: Guillermo Date: Fri, 21 Nov 2025 13:16:55 +0100 Subject: [PATCH 1/2] [#sansara/6461] document CodeSystem streaming operations for large terminologies Add documentation for Aidbox-specific $import and $export operations that enable handling of large CodeSystems Updates: - Add Aidbox-Specific Operations section to codesystem.md with $import/$export usage examples - Update capabilities.md to distinguish FHIR standard vs Aidbox-specific operations - Document BOX_WEB_MAX_BODY configuration requirement (1GB recommended) - Include wget examples for both operations Co-Authored-By: Claude --- .../aidbox-terminology-module/capabilities.md | 23 +++--- .../fhir-terminology/codesystem.md | 74 ++++++++++++++++++- 2 files changed, 86 insertions(+), 11 deletions(-) diff --git a/docs/terminology-module/aidbox-terminology-module/capabilities.md b/docs/terminology-module/aidbox-terminology-module/capabilities.md index 971a8c7af..dea74ea68 100644 --- a/docs/terminology-module/aidbox-terminology-module/capabilities.md +++ b/docs/terminology-module/aidbox-terminology-module/capabilities.md @@ -16,16 +16,18 @@ This page provides a comprehensive overview of all FHIR terminology operations a ## Supported Operations -| Resource | Operation | Status | -| -------------- | ---------------- | ------ | -| **CodeSystem** | `$lookup` | ✅ | -| | `$validate-code` | ✅ | -| | `$subsumes` | ❌ | -| | `$find-matches` | ❌ | -| **ValueSet** | `$expand` | ✅ | -| | `$validate-code` | ✅ | -| **ConceptMap** | `$translate` | ✅ | -| | `$closure` | ❌ | +| Resource | Operation | Status | FHIR | +| -------------- | ---------------- | ------ | ---- | +| **CodeSystem** | `$lookup` | ✅ | ✅ | +| | `$validate-code` | ✅ | ✅ | +| | `$subsumes` | ❌ | ✅ | +| | `$find-matches` | ❌ | ✅ | +| | `$import` | ✅ | ❌ | +| | `$export` | ✅ | ❌ | +| **ValueSet** | `$expand` | ✅ | ✅ | +| | `$validate-code` | ✅ | ✅ | +| **ConceptMap** | `$translate` | ✅ | ✅ | +| | `$closure` | ❌ | ✅ | ## Features @@ -58,4 +60,5 @@ This page provides a comprehensive overview of all FHIR terminology operations a | Ad-hoc ValueSets | ✅ | ValueSet as a `Parameter` | 2507 | | R4/R5/R6 format conversion | ✅ | E.g.: `expansion.contains.concept.property` or extensions | 2507 | | Batch validation | ❌ | | | +| Streaming operations | ✅ | Aidbox specific, large terminologies | 2511 | diff --git a/docs/terminology-module/fhir-terminology/codesystem.md b/docs/terminology-module/fhir-terminology/codesystem.md index 1ebd96c42..5693519a9 100644 --- a/docs/terminology-module/fhir-terminology/codesystem.md +++ b/docs/terminology-module/fhir-terminology/codesystem.md @@ -264,7 +264,7 @@ GET /fhir/CodeSystem/$validate-code?url=http://hl7.org/fhir/administrative-gende "resourceType": "Parameters", "parameter": [ { - "name": "result", + "name": "result", "valueBoolean": false }, { @@ -276,3 +276,75 @@ GET /fhir/CodeSystem/$validate-code?url=http://hl7.org/fhir/administrative-gende ``` {% endtab %} {% endtabs %} + +## Aidbox-Specific Operations + +The following operations are Aidbox-specific extensions that are not part of the FHIR specification. These streaming operations are designed to handle large CodeSystems that exceed the default request body size limit. + +The Aidbox Terminology Module is optimized for typical FHIR interactions (create, read, update) with relatively small content. By default, Aidbox supports a maximum request body size of [**20,971,520 bytes**](../../reference/all-settings.md#web.max-body) (approximately 20 MB). For large terminologies like RxNorm (which contains approximately 225,000 concepts), regular CRUD operations become impractical. + +**Consider using the streaming operations when:** +- Your CodeSystem content exceeds the default body size limit +- You need to import/export large terminologies (e.g. RxNorm) + +> **Important:** Regular Aidbox validation does NOT run when using streaming operations. These operations prioritize performance and scalability for large terminologies over comprehensive validation. + +### $import + +Imports a large CodeSystem using HTTP streaming. This operation accepts a complete CodeSystem resource and streams is content efficiently in the Fhir Artifacts Registry. + +{% tabs %} +{% tab title="Request" %} +```bash +POST /fhir/CodeSystem/:id/$import +Content-Type: application/octet-stream + +< /tx/rxnorm.json +``` + +Example using wget: +```bash +wget --debug \ + --method=POST \ + --header='content-type: application/octet-stream' \ + --output-document=/tmp/rxnorm.json \ + --progress=dot:mega \ + --body-file=/tx/rxnorm.json \ + 'http://localhost:8080/fhir/CodeSystem/rxnorm/$import' +``` + +The request body should contain a complete CodeSystem resource in JSON format with all concepts included. +{% endtab %} + +### $export + +Exports a CodeSystem using HTTP streaming, allowing you to retrieve large terminologies without running out of memory. + +{% tabs %} +{% tab title="Request" %} +```bash +GET /fhir/CodeSystem/:id/$export +``` + +Example using wget: +```bash +wget --debug \ + --method=GET \ + --output-document=/tmp/rxnorm.json \ + --progress=dot:mega \ + 'http://localhost:8080/fhir/CodeSystem/rxnorm/$export' +``` +{% endtab %} + +The operation streams the complete CodeSystem resource including all concepts. +{% endtab %} +{% endtabs %} + +### Configuration Requirements + +To use streaming operations with large CodeSystems, you must increase the maximum [request body size](../../reference/all-settings.md#web.max-body): + +**Environment Variable:** +```yaml +BOX_WEB_MAX_BODY=1073741824 +``` \ No newline at end of file From ff1adb822e8067d7649d084ab940e67c83afa2dd Mon Sep 17 00:00:00 2001 From: Guillermo Date: Mon, 24 Nov 2025 11:04:42 +0100 Subject: [PATCH 2/2] [#sansara/6461] changing streaming operations names to include prefix: $codesystem- --- .../aidbox-terminology-module/capabilities.md | 26 +++++++++---------- .../fhir-terminology/codesystem.md | 15 +++++------ 2 files changed, 20 insertions(+), 21 deletions(-) diff --git a/docs/terminology-module/aidbox-terminology-module/capabilities.md b/docs/terminology-module/aidbox-terminology-module/capabilities.md index dea74ea68..9a4b7d3db 100644 --- a/docs/terminology-module/aidbox-terminology-module/capabilities.md +++ b/docs/terminology-module/aidbox-terminology-module/capabilities.md @@ -16,18 +16,18 @@ This page provides a comprehensive overview of all FHIR terminology operations a ## Supported Operations -| Resource | Operation | Status | FHIR | -| -------------- | ---------------- | ------ | ---- | -| **CodeSystem** | `$lookup` | ✅ | ✅ | -| | `$validate-code` | ✅ | ✅ | -| | `$subsumes` | ❌ | ✅ | -| | `$find-matches` | ❌ | ✅ | -| | `$import` | ✅ | ❌ | -| | `$export` | ✅ | ❌ | -| **ValueSet** | `$expand` | ✅ | ✅ | -| | `$validate-code` | ✅ | ✅ | -| **ConceptMap** | `$translate` | ✅ | ✅ | -| | `$closure` | ❌ | ✅ | +| Resource | Operation | Status | FHIR Standard | +| -------------- | -------------------- | ------ | ------------- | +| **CodeSystem** | `$lookup` | ✅ | ✅ | +| | `$validate-code` | ✅ | ✅ | +| | `$subsumes` | ❌ | ✅ | +| | `$find-matches` | ❌ | ✅ | +| | `$codesystem-import` | ✅ | ❌ | +| | `$codesystem-export` | ✅ | ❌ | +| **ValueSet** | `$expand` | ✅ | ✅ | +| | `$validate-code` | ✅ | ✅ | +| **ConceptMap** | `$translate` | ✅ | ✅ | +| | `$closure` | ❌ | ✅ | ## Features @@ -44,7 +44,7 @@ This page provides a comprehensive overview of all FHIR terminology operations a | ValueSet inclusion/exclusion | ✅ | Deep set operations support | 2507 | | Lookup displays, designation, and properties | ✅ | All concept attributes | 2507 | | Text search filter | ✅ | Free-text concept search | 2507 | -| Property filters | ✅ | Property-based filtering: `=`, `in`, `regex`, etc | 2507 | +| Property filters | ✅ | Property-based filtering: `=`, `in`, `regex`, etc | 2507 ****| | Multi-language support | ✅ | Translations via `displaylanguage`, HTTP header, designation, etc | 2507 | | Active/Inactive filtering | ✅ | Via `status`, `inactive`, `notSelectable`, etc | 2507 | | Hierarchy via `parent`, `child` | ✅ | | 2507 | diff --git a/docs/terminology-module/fhir-terminology/codesystem.md b/docs/terminology-module/fhir-terminology/codesystem.md index 5693519a9..1dc5a347c 100644 --- a/docs/terminology-module/fhir-terminology/codesystem.md +++ b/docs/terminology-module/fhir-terminology/codesystem.md @@ -289,14 +289,14 @@ The Aidbox Terminology Module is optimized for typical FHIR interactions (create > **Important:** Regular Aidbox validation does NOT run when using streaming operations. These operations prioritize performance and scalability for large terminologies over comprehensive validation. -### $import +### $codesystem-import -Imports a large CodeSystem using HTTP streaming. This operation accepts a complete CodeSystem resource and streams is content efficiently in the Fhir Artifacts Registry. +Imports a large CodeSystem using HTTP streaming. This operation accepts a complete CodeSystem resource and streams is content efficiently in the Fhir Artifacts Registry(FAR). {% tabs %} {% tab title="Request" %} ```bash -POST /fhir/CodeSystem/:id/$import +POST /fhir/CodeSystem/:id/$codesystem-import Content-Type: application/octet-stream < /tx/rxnorm.json @@ -307,23 +307,22 @@ Example using wget: wget --debug \ --method=POST \ --header='content-type: application/octet-stream' \ - --output-document=/tmp/rxnorm.json \ --progress=dot:mega \ --body-file=/tx/rxnorm.json \ - 'http://localhost:8080/fhir/CodeSystem/rxnorm/$import' + 'http://localhost:8080/fhir/CodeSystem/rxnorm/$codesystem-import' ``` The request body should contain a complete CodeSystem resource in JSON format with all concepts included. {% endtab %} -### $export +### $codesystem-export Exports a CodeSystem using HTTP streaming, allowing you to retrieve large terminologies without running out of memory. {% tabs %} {% tab title="Request" %} ```bash -GET /fhir/CodeSystem/:id/$export +GET /fhir/CodeSystem/:id/$codesystem-export ``` Example using wget: @@ -332,7 +331,7 @@ wget --debug \ --method=GET \ --output-document=/tmp/rxnorm.json \ --progress=dot:mega \ - 'http://localhost:8080/fhir/CodeSystem/rxnorm/$export' + 'http://localhost:8080/fhir/CodeSystem/rxnorm/$codesystem-export' ``` {% endtab %}