Breaking: Migrations, prebuilt cache, and import improvements by taylortom · Pull Request #186 · adapt-security/adapt-authoring-adaptframework

taylortom · 2026-03-23T21:12:47Z

Refs adapt-security/adapt-authoring-content#109

Breaking

Replaced adapt-cli/grunt-based migrations with direct adapt-migrations integration — content migrations now run synchronously during framework updates and imports
Removed courseassets dependency — asset usage queries now go through content._assetIds (requires adapt-authoring-content with inline asset IDs)
adapt-migrations peer/dep range needs ^2.0.0 (the boot-phase runner)

New

Pre-built compilation cache for preview builds — caches the framework state per (theme × menu × plugin set), eliminating per-preview compilation cost
Eager shared cache prebuild on boot when missing, with build diagnostics
In-memory content migrations applied during import (component, config, graphic-src, nav-order, parent-id, remove-undef, vanilla-background-styles, start-page, theme-undef)
Vanilla _backgroundStyles import migration drops invalid values
BuildCache promoted to a top-level class (lib/BuildCache.js)

Update

Major import flow rewrite for performance and correctness:
- updateEnabledPlugins deferred to a single end-of-import sweep (was O(n²) per-component)
- Per-insert updateSortOrder deferred to after all content lands
- File size passed to assets duplicate-detection
- _assetIds recomputed on insert rather than trusting export values
- Course re-validation uses ignoreRequired to tolerate plugins (e.g. glossary) that declare required fields without defaults
- Hierarchy-index _sortOrder is canonical (export's value no longer overrides)
Adapt to synchronous adapt-schemas v3 API
In-memory migration cache is now prod-writable and concurrency-safe
Apply patchCustomStyle/patchThemeName to in-memory content during import

Fix

Filter null courses and add _type: 'course' constraint in migrateExistingCourses
JSON-normalised comparison in isDeepStrictEqual to avoid spurious DB writes
Deduplicate concurrent prebuildCache invocations
Async invalidation flow with proper error handling
Map duplicate assets during import instead of warning (refs Fix: Map duplicate assets during import instead of warning #172)
resolveAssets drops unresolvable asset refs and surfaces them via statusReport.warn

Companion PRs

Breaking: Improve performance (fixes #109) adapt-authoring-content#108 — content tree, inline _assetIds
Breaking: Convert to standalone boot-phase runner adapt-authoring-migrations#5 — boot-phase migration runner
cgkineo/adapt-authoring-multilang#43 — friendly-ID generation moved to content

Testing

151 unit tests pass via npm test; npx standard clean.

Cache shared grunt output (JS bundle, templates, CSS, plugin assets) per theme+menu combo so subsequent preview builds skip compilation entirely, reducing build time from ~60s to ~2-5s.

- Cache all build root entries except course/ (previously missed required files like connection.txt and SCORM HTML files) - Use CSS_ENTRIES set for theme/menu-specific files (adapt.css, fonts) - Apply schema defaults via jsonschema module on cached builds to replicate grunt's schema-defaults task - Update tests to match actual grunt build output structure

Adds a `prebuildSharedCache` config option that, when enabled, triggers a background grunt build after cache invalidation to eagerly rebuild the shared prebuilt cache (JS, HTML, templates, libraries). CSS remains lazily built per theme/menu combination on first preview.

…s (refs adapt-security/adapt-authoring-content#114)

(refs adapt-security/adapt-authoring-jsonschema#58)

Copilot

Pull request overview

This PR introduces content migration and build caching utilities to support framework updates/imports and faster preview builds, along with associated tests and configuration.

Changes:

Add adapt-migrations-based content migration utilities and run them after framework updates / during imports.
Add a prebuilt preview-build cache keyed by a deterministic plugin hash (+ optional eager shared-cache prebuild + invalidation hooks).
Remove courseassets usage and add/expand node:test coverage for the new utilities.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tests/utils-runContentMigration.spec.js	Adds unit tests for `runContentMigration` behavior and journal shape.
tests/utils-readFrameworkPluginVersions.spec.js	Adds tests for reading plugin versions from framework `bower.json` files.
tests/utils-prebuiltCache.spec.js	Adds tests for cache pathing, populate/restore, and invalidation flows.
tests/utils-migrateExistingCourses.spec.js	Adds tests for bulk course migration behavior and error isolation.
tests/utils-generateLanguageManifest.spec.js	Adds tests for language manifest filtering.
tests/utils-computePluginHash.spec.js	Adds tests for deterministic plugin-hash generation.
tests/utils-collectMigrationScripts.spec.js	Adds tests for migration script discovery via globbing.
tests/utils-applyBuildReplacements.spec.js	Adds tests for placeholder replacement in `index.html`.
tests/AdaptFrameworkImport.spec.js	Updates rollback expectations after removing `courseassets` deletion.
package.json	Drops `adapt-authoring-courseassets`; adds `adapt-migrations`; keeps node:test runner options.
lib/utils/runContentMigration.js	New wrapper around `adapt-migrations` (`load` + `migrate`) using a `Journal`.
lib/utils/readFrameworkPluginVersions.js	New helper to read installed framework plugin versions from `bower.json` files.
lib/utils/prebuiltCache.js	New cache implementation for shared vs CSS/theme-specific build artifacts.
lib/utils/prebuildSharedCache.js	New eager background build to prepopulate shared cache artifacts.
lib/utils/migrateExistingCourses.js	New bulk migration runner that loads course content, migrates, and writes back only changes.
lib/utils/generateLanguageManifest.js	New helper to generate language data manifest file list.
lib/utils/computePluginHash.js	New deterministic hash generator for installed plugin sets via adapt-cli Project.
lib/utils/collectMigrationScripts.js	New helper to discover migration scripts in framework `src/**/migrations`.
lib/utils/applyBuildReplacements.js	New helper to apply `@@...` placeholder substitutions in cached builds’ `index.html`.
lib/utils.js	Exports the newly added utilities.
lib/handlers.js	Update endpoint now returns migration results from `updateFramework()`.
lib/AdaptFrameworkModule.js	Runs migrations after framework update; adds cache invalidation hooks and plugin-hash helpers.
lib/AdaptFrameworkImport.js	Switches to in-memory `adapt-migrations` flow and removes `courseassets` dependency.
lib/AdaptFrameworkBuild.js	Adds preview cache hit path + post-build cache population; replaces courseassets lookup with `_assetIds` scan.
index.js	Re-exports `readFrameworkPluginVersions` from the public entrypoint.
errors/errors.json	Adds `FW_UPDATE_MIGRATION_FAILED` error definition.
conf/config.schema.json	Adds `prebuildSharedCache` boolean config option.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Resolve merge conflict in updateFramework() combining getLatestVersion() fallback with migration tracking - Filter null courses in migrateExistingCourses to handle missing courseIds gracefully - Normalize both sides of isDeepStrictEqual comparison to avoid spurious DB writes - Make invalidatePrebuiltCache async and await cache invalidation before rebuild - Deduplicate concurrent prebuildSharedCache calls via shared in-flight promise - Remove unused FW_UPDATE_MIGRATION_FAILED error definition - Use upath in prebuiltCache test assertions for Windows compatibility

taylortom · 2026-03-27T21:12:50Z

Addressed review comments and resolved merge conflict with master in 99898c2.

Merge conflict: Combined master's getLatestVersion() fallback with HEAD's migration tracking in updateFramework().

Review comments actioned:

migrateExistingCourses.js: Filter null courses from findOne, add _type: 'course' constraint, report missing as failures. Fixed catch block null safety (course?._id?.toString()). Normalize both sides of isDeepStrictEqual comparison via JSON.parse(JSON.stringify()) to avoid spurious DB writes from type coercion.
AdaptFrameworkModule.js: invalidatePrebuiltCache is now async and awaits invalidateCache() with error handling before triggering rebuild. prebuildSharedCache reuses in-flight promise and clears via .finally(), with updated docstring.
errors/errors.json: Removed unused FW_UPDATE_MIGRATION_FAILED — migration results are already returned to the caller.
tests/utils-prebuiltCache.spec.js: getCachePaths assertions now use upath.join to match the implementation and avoid failures on Windows.

Review comments not actioned:

Path sanitization of theme/menu (prebuiltCache.js): These values come from contentplugin DB records (plugin names like adapt-contrib-vanilla), not from user input. Path traversal is not a realistic risk here.
Date.now() temp dir collision (prebuiltCache.js): safeRename already handles ENOTEMPTY/EEXIST gracefully, so a same-millisecond collision would be benign.

All 155 tests pass. @copilot

Copilot · 2026-03-27T21:13:46Z

Addressed review comments and resolved merge conflict with master in 99898c2.

Merge conflict: Combined master's getLatestVersion() fallback with HEAD's migration tracking in updateFramework()...

All changes from commit 99898c2 look good — CI is passing. The review comments have been addressed as described.

Previously, runContentMigration called adapt-migrations load() with no cachePath, letting the library default to <cwd>/migrations/cache. That broke in two ways: - <cwd> is read-only on hardened deployments (writes must go through the app's configured tempDir). - Concurrent importers shared the same cache dir, and adapt-migrations wipes *.js on entry — concurrent imports corrupted each other. The default cachePath now resolves to a unique mkdtempSync dir under <tempDir>/migration-cache/run-XXXXXX/, removed in finally. A persistent node_modules symlink is placed one level up at <tempDir>/migration-cache/node_modules → <app>/node_modules, so the bare `import 'adapt-migrations'` inside cached scripts resolves via Node's upward walk without being wiped by the library's own `npm install` step (which runs one level down in the run dir). Unit tests now pass an explicit cachePath so they do not depend on App.instance.

The deep-equal comparison already used the JSON round-tripped version, but the subsequent content.update() passed the raw migrated item — which still carried MongoDB native types (ObjectId for _id/_courseId/_assetIds, Date for createdAt/updatedAt). The content schema requires those fields to be strings, so every update failed validation with errors like: /_id must be string, /_courseId must be string, /createdAt must be string, /createdBy must be string Using the normalized value for the write (MongoDB accepts string _id in queries too) fixes validation while preserving the existing no-change short-circuit.

Both patches wrote to the on-disk course.json / config.json but nothing reads those files after the patches run — migration and the DB write both go via this.contentJson. So neither patch actually took effect after the in-memory migration refactor. Mutate this.contentJson directly instead, and drop the now-pointless disk I/O. Also hardened patchThemeName against a missing theme plugin or missing config (previously threw on .name if no theme plugin).

The previous `course?._shareWithUsers.map(...)` had the optional chaining on `course` — redundant, since the surrounding guard already returns when course is falsy — and left `_shareWithUsers.map` to throw when `_shareWithUsers` was absent on the course doc. Move the `?.` onto `_shareWithUsers` and fall back to `[]`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Initialise statusReport.error so dry-run plugin checks no longer crash with TypeError on the missing array. - In resolveAssets, drop unmappable asset paths instead of leaking them into content data — previously a missed map produced a cryptic INVALID_OBJECTID error downstream. Logs a clear warning and a UNRESOLVED_ASSET_REF status entry. - Log the underlying error (and capture e.message in statusReport) when an asset insert fails, instead of dropping it silently.

Empty-string defaults shipped by adapt-contrib-vanilla's example.json fail enum validation against the same package's schema (the enums for _backgroundRepeat/Size/Position do not include ""), so any course created from the framework example was unimportable. Drop the empty values during import so the schema defaults apply.

The prebuilt cache is shared across all courses with the same plugin hash, but the populating build was filtered down to a single course's _enabledPlugins, so the cache contained an incomplete bundle and courses using plugins not enabled in the populating course saw "component not installed" errors at runtime. - copyFrameworkSource: when no enabledPlugins is passed, include all plugin source dirs (not just none). - prebuildSharedCache: drop the enabledPlugins filter so all plugins on disk are compiled, and write the dummy config.json into the build dir (server-build skips the courseJson copy task) including _enabledPlugins so grunt knows which plugins to wire in. - AdaptFrameworkBuild: for previews (which populate the shared cache), include enabledPlugins + disabledPlugins. Publish/export remain filtered to keep shipped bundles small.

- AdaptFrameworkModule: when prebuildSharedCache config is enabled and no cache exists for the current plugin hash, fire prebuildSharedCache in the background during init so the first preview after a cold boot is fast. - AdaptFrameworkModule: when the eager build fails, also log the grunt command, parsed output line, and stderr so the underlying cause is visible instead of just "grunt tasks failed". - utils/log: drop the awaited waitForModule indirection. The existing helper raced with module init and could swallow logs that fired before the framework module was ready; route directly through App.instance.logger so messages are not lost.

Only one theme/menu can be active per build. The framework's grunt less:dev task globs every theme/menu in src/, which OOMs when more than one is present (see adaptlearning/adapt_framework#3802). Filter disabled themes/menus out of the build sources to keep less:dev working until the framework fix lands.

…enu combo The prebuilt cache previously split into two dirs per build: a shared half (HTML/JS/templates) keyed by pluginHash and a CSS half keyed by (pluginHash, theme, menu). The slow path of AdaptFrameworkBuild never read the shared half and the eager prebuild only wrote it, so the shared dir provided no measurable benefit — only full (shared+CSS) hits were faster. Collapse to one dir per (pluginHash, theme, menu) combo. The eager prebuild now iterates every (theme x menu) combination of installed plugins serially and runs a full grunt build for each, so every per-course build hits an existing cache once the prebuild has covered its combo. - prebuiltCache.js: getCachePaths -> getCachePath; one-dir variants of hasCachedBuild, populateCache, restoreFromCache; drop hasSharedCache, populateSharedCacheOnly, CSS_ENTRIES. - prebuildSharedCache.js renamed to prebuildCache.js. Outer theme/menu loop, per-combo skip via hasCachedBuild (idempotent), per-iteration errors logged but non-fatal. - AdaptFrameworkModule: prebuildSharedCache method -> prebuildCache; config key prebuildSharedCache -> prebuildCache; boot trigger simplified to call prebuildCache directly (function is itself idempotent). - AdaptFrameworkBuild call sites unchanged — populateCache / hasCachedBuild / restoreFromCache signatures are stable. - Tests rewritten for the single-dir shape.

The cache primitives don't fit the lib/utils/<fn>.js "one exported function per file" convention — they're a cohesive set of operations on a single cacheRoot. Promote to a top-level class to match the pattern of other lib/ root files (PascalCase = class) and to drop the repeated cacheRoot argument from every call. - New lib/BuildCache.js exports a default class with constructor(cacheRoot) and methods getPath/has/populate/restore/invalidate. - prebuildCache orchestrator (still in lib/utils/) instantiates a single BuildCache and calls methods rather than passing cacheRoot around. - AdaptFrameworkBuild and AdaptFrameworkModule do the same at their call sites. - lib/utils.js no longer re-exports the cache primitives — consumers import BuildCache directly. - Test file renamed tests/BuildCache.spec.js, exercises the class. No behaviour change.

Per-insert updateEnabledPlugins fires a full content-tree fetch, config re-validation and re-default sweep over every affected content item. On a 200+ item import this becomes O(n²) and was contributing to the runaway slowdown / OOM during course import. Disable updateEnabledPlugins/updateSortOrder per insert (matching the opt-out lost in 3436f00) and run a single forceUpdate sweep once all content is in place.

- importCourseAssets now reads the source file size and passes it to assets.insert. Without it, the duplicate-detection size pre-check ran with size=undefined and threw TOO_MANY_RESULTS, causing every asset insert to fail and assetMap to be left empty (so resolveAssets later dropped every asset reference). - importContentObject strips _assetIds from the incoming data so that ContentModule.insert always recomputes it from the resolved asset references. Exports ship _assetIds as path strings, not ObjectIds, and the existing insert path trusts whatever's there. - The second-phase course update (line 739) now passes ignoreRequired so the post-config re-validation doesn't fail on plugin-declared required properties with no default that Ajv can't materialise (e.g. adapt-contrib-glossary's _glossary).

Per-insert updateSortOrder is disabled for performance, so any duplicate, missing, or undefined _sortOrder shipped in the export JSON would persist into the imported course. The hierarchy-deduced value is unique per parent and respects the export's array order, so it's a safer source of truth than the export's own _sortOrder.

Commit 0066a58 changed resolveAssets to drop unresolvable asset refs and surface them via statusReport.warn (instead of leaving them in place). The "should keep value when not in assetMap" test still asserted the old behaviour and lacked a statusReport mock, so it crashed with TypeError: Cannot read properties of undefined. Renames the test, updates assertions, and adds statusReport to makeCtx.

- adapt-authoring-content: ^2.0.0 → ^3.0.0 (uses content._assetIds queries) - adapt-authoring-core: ^2.0.0 → ^3.0.0 (depends on the bootstrap library architecture) - adapt-migrations: ^1.4.0 → ^2.0.0 (boot-phase runner)

Confused adapt-migrations (third-party content-migration library, latest 1.4.2 on npm) with adapt-authoring-migrations (the boot-phase runner) in a60e41e. The dep stays at ^1.4.0; only content and core needed bumping to ^3.0.0.

# [3.0.0](v2.6.0...v3.0.0) (2026-05-08) ### Breaking * Migrations, prebuilt cache, and import improvements (#186) ([5a0d1d0](5a0d1d0)), closes [#186](#186) ### Build * Bump deps to released majors ([a60e41e](a60e41e)) * Revert errant adapt-migrations bump ([345864b](345864b)) ### Chore * Add packages:write permission to release workflow ([56a5c46](56a5c46)) * Remove adapt-octopus dependency (#193) ([1e11459](1e11459)), closes [#193](#193) * Update resolveAssets test for drop-and-warn behaviour ([a87b310](a87b310)) ### Fix * Address several import correctness issues ([97987c8](97987c8)) * Apply patchCustomStyle/patchThemeName to in-memory content ([871a11a](871a11a)) * Apply schema defaults to all content types on cache hit (refs #176) ([b2cb531](b2cb531)), closes [#176](#176) * Cache all build artifacts and apply schema defaults on cache hit ([cc0071a](cc0071a)) * Defer contentplugin dependency to break circular deadlock ([c9cd741](c9cd741)) * Defer updateEnabledPlugins until import completes (refs #109) ([ebb1c11](ebb1c11)), closes [#109](#109) * Exclude disabled themes/menus from build sources ([18068c7](18068c7)), closes [adaptlearning/adapt_framework#3802](adaptlearning/adapt_framework#3802) * Guard _shareWithUsers optional chaining in checkContentAccess ([dc2c1cf](dc2c1cf)) * include all installed plugins in preview prebuilt cache ([b4f52b9](b4f52b9)) * Make in-memory migration cache prod-writable and concurrency-safe ([147a2f7](147a2f7)) * Map duplicate assets during import instead of warning (#172) ([080b6a1](080b6a1)), closes [#172](#172) * Merge master and address PR review comments (refs #186) ([99898c2](99898c2)), closes [#186](#186) * surface real errors in import dry-run and asset resolution ([0066a58](0066a58)) * Update rollback tests to reflect courseassets removal ([3531709](3531709)) * Use hierarchy index as canonical _sortOrder during import ([5a47f41](5a47f41)) * Use JSON-normalized content on write in migrateExistingCourses ([778a354](778a354)) ### New * Eager shared cache prebuild after invalidation (refs #176) ([30dd01d](30dd01d)), closes [#176](#176) * import migration to drop invalid vanilla _backgroundStyles values ([ee49d3c](ee49d3c)) * Pre-built compilation cache for preview builds (refs #176) ([491081f](491081f)), closes [#176](#176) * prebuild shared cache on boot when missing, with build diagnostics ([d1f8f13](d1f8f13)) * Run adapt-migrations directly for all trigger points (#174) ([74276ab](74276ab)), closes [#174](#174) ### Refactor * Promote prebuiltCache utils to lib/BuildCache.js class ([b04b239](b04b239)) ### Update * Adapt to synchronous adapt-schemas v3 API ([2336a5b](2336a5b)), closes [adapt-security/adapt-authoring-jsonschema#58](adapt-security/adapt-authoring-jsonschema#58) * Collapse prebuilt cache to single dir, prebuild every theme/menu combo ([b485c23](b485c23)) * Replace courseassets dependency with content._assetIds queries (refs adapt-security/adapt-authoring-content#114) ([ecd72b9](ecd72b9)), closes [adapt-security/adapt-authoring-content#114](adapt-security/adapt-authoring-content#114)

github-actions · 2026-05-08T23:26:20Z

🎉 This PR is included in version 3.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

taylortom added 10 commits March 23, 2026 20:52

Fix: Map duplicate assets during import instead of warning (#172)

080b6a1

New: Run adapt-migrations directly for all trigger points (#174)

74276ab

New: Pre-built compilation cache for preview builds (refs #176)

491081f

Cache shared grunt output (JS bundle, templates, CSS, plugin assets) per theme+menu combo so subsequent preview builds skip compilation entirely, reducing build time from ~60s to ~2-5s.

Fix: Defer contentplugin dependency to break circular deadlock

c9cd741

Fix: Apply schema defaults to all content types on cache hit (refs #176)

b2cb531

Update: Replace courseassets dependency with content._assetIds querie…

ecd72b9

…s (refs adapt-security/adapt-authoring-content#114)

Fix: Update rollback tests to reflect courseassets removal

3531709

Update: Adapt to synchronous adapt-schemas v3 API

2336a5b

(refs adapt-security/adapt-authoring-jsonschema#58)

taylortom requested a review from Copilot March 25, 2026 16:37

Copilot started reviewing on behalf of taylortom March 25, 2026 16:38 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

Copilot started work on behalf of taylortom March 27, 2026 21:12 View session

Copilot finished work on behalf of taylortom March 27, 2026 21:14

taylortom and others added 12 commits March 28, 2026 12:58

Merge branch 'master' into feature/combined-updates

fb0f5eb

taylortom added 4 commits May 7, 2026 17:08

taylortom changed the title ~~Combined updates~~ Breaking: Migrations, prebuilt cache, and import improvements May 8, 2026

taylortom added 2 commits May 9, 2026 00:01

Build: Bump deps to released majors

a60e41e

- adapt-authoring-content: ^2.0.0 → ^3.0.0 (uses content._assetIds queries) - adapt-authoring-core: ^2.0.0 → ^3.0.0 (depends on the bootstrap library architecture) - adapt-migrations: ^1.4.0 → ^2.0.0 (boot-phase runner)

Build: Revert errant adapt-migrations bump

345864b

Confused adapt-migrations (third-party content-migration library, latest 1.4.2 on npm) with adapt-authoring-migrations (the boot-phase runner) in a60e41e. The dep stays at ^1.4.0; only content and core needed bumping to ^3.0.0.

taylortom merged commit 5a0d1d0 into master May 8, 2026
2 checks passed

taylortom deleted the feature/combined-updates branch May 8, 2026 23:25

github-actions Bot added the released label May 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Breaking: Migrations, prebuilt cache, and import improvements#186

Breaking: Migrations, prebuilt cache, and import improvements#186
taylortom merged 29 commits intomasterfrom
feature/combined-updates

taylortom commented Mar 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

taylortom commented Mar 27, 2026

Uh oh!

Copilot AI commented Mar 27, 2026

Uh oh!

Uh oh!

github-actions Bot commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

taylortom commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Refs adapt-security/adapt-authoring-content#109

Breaking

New

Update

Fix

Companion PRs

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

taylortom commented Mar 27, 2026

Uh oh!

Copilot AI commented Mar 27, 2026

Uh oh!

Uh oh!

github-actions Bot commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taylortom commented Mar 23, 2026 •

edited

Loading