diff --git a/AGENTS.md b/AGENTS.md index a3204729..c75659dc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,7 +6,15 @@ Diese Datei ist das Codex-Pendant zu `CLAUDE.md`. Wenn sich Workflow, Branch-Reg --- -## 0. Projektziel / Vision +## -1. Arbeitsprinzip / Top Memory + +MasterSelects wird nicht fuer kurzfristige Loesungen optimiert. Da das Projekt AI-powered sehr schnell entwickelt werden kann und aktuell keine externen User blockiert, sind grosse, richtige Architekturentscheidungen ausdruecklich erlaubt und bevorzugt. + +Default ist: langfristig denken, echte Zielarchitektur bauen, keine MVPs, keine Mocks, keine Wegwerf-Prototypen und keine kleinen Zwischenloesungen, wenn die robuste Loesung direkt erreichbar ist. Kurzfristige Hacks nur dann verwenden, wenn der User sie explizit verlangt oder ein harter technischer Blocker keine bessere Umsetzung erlaubt. + +--- + +## 0. Projektziel MasterSelects soll bis Juni 2026 ALLE Media-Dateien unterstuetzen, nicht nur Video, Audio und Bilder, sondern wirklich alles: diff --git a/CLAUDE.md b/CLAUDE.md index 74d0ac87..39502233 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,10 +1,16 @@ # CLAUDE.md +## -1. Arbeitsprinzip / Top Memory + +MasterSelects wird nicht fuer kurzfristige Loesungen optimiert. Da das Projekt AI-powered sehr schnell entwickelt werden kann und aktuell keine externen User blockiert, sind grosse, richtige Architekturentscheidungen ausdruecklich erlaubt und bevorzugt. + +Default ist: langfristig denken, echte Zielarchitektur bauen, keine MVPs, keine Mocks, keine Wegwerf-Prototypen und keine kleinen Zwischenloesungen, wenn die robuste Loesung direkt erreichbar ist. Kurzfristige Hacks nur dann verwenden, wenn der User sie explizit verlangt oder ein harter technischer Blocker keine bessere Umsetzung erlaubt. + Anweisungen für AI-Assistenten (Claude, GPT, etc.) bei der Arbeit an diesem Projekt. --- -## 0. Projektziel / Vision (Deadline: Juni 2026) +## 0. Projektziel (Deadline: Juni 2026) **MasterSelects muss ALLE Media-Dateien unterstützen** — nicht nur Video/Audio/Bild, sondern wirklich ALLES: 3D (OBJ, FBX, glTF), PDF, SVG, CAD (DXF/STEP), Binärdaten, Point Clouds, JSON, CSV, und mehr. diff --git a/docs/Features/Audio.md b/docs/Features/Audio.md index 2826ebcc..7b0ffeda 100644 --- a/docs/Features/Audio.md +++ b/docs/Features/Audio.md @@ -77,12 +77,14 @@ Audio export is handled by `engine/audio`: - `FrameExporter` uses `AudioExportPipeline.exportAudio()` for normal video exports with audio. - `FrameExporter` uses `AudioExportPipeline.exportRawAudio()` for the FFmpeg export path. - `ExportPanel` also exposes standalone audio export through the same pipeline. +- Audio-only WAV export uses `exportRawAudio()` and writes a 16-bit PCM WAV file. - The pipeline applies clip trimming, speed changes, EQ, volume, mixing, and then encoding. - `AudioEncoderWrapper` prefers AAC-LC and falls back to Opus if the browser supports it. - Peak normalization is optional and only happens during export when enabled. Important limitation: -the WebCodecs audio encoder is required for the standalone encoded audio path. +the WebCodecs audio encoder is required for the standalone browser-compressed audio path. +Audio-only WAV export does not require WebCodecs audio encoding. FFmpeg exports can still receive raw audio because they use `exportRawAudio()`. ## Limitations diff --git a/docs/Features/Debugging.md b/docs/Features/Debugging.md index 3ca4dc90..f4abe2c8 100644 --- a/docs/Features/Debugging.md +++ b/docs/Features/Debugging.md @@ -161,6 +161,7 @@ The playback-related AI tools read from the same sources: - `getStatsHistory` - `getLogs` - `getPlaybackTrace` +- `purgePlaybackPath` Those tools surface: - Engine state and readiness @@ -171,6 +172,10 @@ Those tools surface: - Render loop and render dispatcher state - WebCodecs / VF pipeline event windows +`purgePlaybackPath` resets the live playback path at the current playhead without a page reload. It clears VideoSync warmups/seeks, retargets active HTMLVideo/WebCodecs providers, resets GPU-ready state, and can resume playback automatically. The health monitor can invoke the same path when `vf_preview_frame` telemetry shows the playhead target moving while the preview frame remains frozen. + +When playback start has to wait for active HTML video readiness, `TimelineState.playbackWarmup` is set until the readiness gate finishes or is canceled. The main preview renders a small `Preparing playback` overlay only for that pre-start gate, so background VideoSync warmups during normal playback do not look like blocking loading states. + `getStatsHistory` is capped to 1-30 samples, `getLogs` caps the returned buffer to 1-500 entries, and `getPlaybackTrace` caps the inspected time window and event count so the bridge stays responsive. --- diff --git a/docs/Features/Export.md b/docs/Features/Export.md index 71d0f00f..1c472b7d 100644 --- a/docs/Features/Export.md +++ b/docs/Features/Export.md @@ -25,7 +25,7 @@ FCPXML is exposed as a selectable export container for NLE interchange. - `Basic` contains output naming and container selection. The container row is grouped by `Video`, `Image`, `Audio`, and `XML`, and switches output mode by selecting a deliverable directly. - The `Video` group contains codec selection, resolution, frame rate, bitrate/rate controls, animated GIF palette controls, stacked-alpha, and range toggles. - In `Image` mode the same middle group becomes an `Image` panel with format-aware resolution and quality controls, and it can export either the current playhead frame or a numbered image sequence folder. -- The `Audio` group contains sample rate, bitrate, normalization, and audio-only range controls. +- The `Audio` group contains audio-only format selection, sample rate, bitrate for browser-compressed audio, normalization, and audio-only range controls. - Lower in the panel, legacy `Advanced Video`, `Advanced Audio`, and `Range & Summary` sections still exist for raw-value access. - Export settings and preset selection now live in a shared store, so changes inside the Export tab participate in global undo/redo and are restored with the project. @@ -152,10 +152,13 @@ Audio export is handled separately from the video encoder. - Audio is extracted from the selected timeline range. - `AudioExportPipeline` renders the mixed audio. +- Audio-only WAV export writes the mixed `AudioBuffer` as 16-bit PCM WAV. - WebCodecs export can mux the audio chunks into the final file. ### Supported Behavior +- Audio-only export supports uncompressed WAV (`.wav`) without WebCodecs audio encoding. +- The existing browser-compressed audio-only path writes the detected browser codec (`.aac` or `.ogg`). - AAC is used for MP4 when supported. - Opus is used for WebM when supported. - If the browser cannot encode a usable audio format, the export can proceed without audio. diff --git a/docs/Features/Math-Scene-Clips-Plan.md b/docs/Features/Math-Scene-Clips-Plan.md index 6e13a678..5816b4ec 100644 --- a/docs/Features/Math-Scene-Clips-Plan.md +++ b/docs/Features/Math-Scene-Clips-Plan.md @@ -91,7 +91,7 @@ Scope: - Add the dedicated `Math` dock panel. - Add object list, parameter list, animation list, and inspector. -- Add reusable Math Scene media items in the Media panel. +- Expand the reusable Math Scene media items in the Media panel beyond the current default preset item. - Add preset buttons: function, point, tangent, area, label, camera move. - Add timeline badges and better clip display. - Add copy/paste and duplicate behavior for Math Scene objects. diff --git a/docs/Features/Motion-Design.md b/docs/Features/Motion-Design.md index 4116d4f6..9b2de86d 100644 --- a/docs/Features/Motion-Design.md +++ b/docs/Features/Motion-Design.md @@ -14,7 +14,7 @@ The motion design system follows `docs/plans/motion-design-system-plan.md`. It i - `src/services/properties/PropertyRegistry.ts` describes transform, effect, color, mask, vector-animation, and motion properties without owning Zustand state. - `src/stores/timeline/motionClipSlice.ts` can create rectangle/ellipse shape clips, null clips, adjustment clips, update motion definitions, and convert solid clips to motion rectangle clips. - `src/components/panels/properties/MotionShapeTab.tsx` exposes primitive, size, corner radius, fill, and stroke controls for motion shape clips. -- Video track-header context menus can create Motion Rectangle and Motion Ellipse clips at the playhead. +- The Media panel add/context menu can create Motion Rectangle and Motion Ellipse preset items that can be dragged to video tracks. - Solid clip context menus can convert the selected solid to a motion shape while preserving its clip id and timing. - The Motion tab exposes a first Grid Replicator section with enable, count, spacing, and opacity fade controls. - `src/engine/motion/MotionRenderer.ts` renders rectangle and ellipse primitives into transparent `rgba8unorm` textures using analytic WGSL SDFs. diff --git a/docs/Features/Timeline.md b/docs/Features/Timeline.md index 21085e4c..ba351337 100644 --- a/docs/Features/Timeline.md +++ b/docs/Features/Timeline.md @@ -195,7 +195,7 @@ Soloing multiple tracks is supported. Non-solo tracks dim visually when any solo ### Track Header Context Menu -- Video tracks can create Motion Rectangle, Motion Ellipse, or Math Scene clips at the playhead. +- Math Scene and Motion Shape presets are created from the Media panel add/context menu and then dragged to video tracks. - `Duplicate Track` currently creates a new empty track of the same type. - `Delete` is blocked for the last remaining track of that type. - Deleting a populated track shows the affected clip count in the menu label/tooltip. diff --git a/docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md b/docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md new file mode 100644 index 00000000..89048e39 --- /dev/null +++ b/docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md @@ -0,0 +1,752 @@ +# Wasm/WIT + Signal IR + Worker/Capability Runtime Plan + +**Status:** Architektur- und Multi-Agent-Ausfuehrungsplan +**Datum:** 2026-05-15 +**Ziel:** MasterSelects bekommt eine echte Runtime-Schicht, in der jede Datei als typisiertes Signal importiert, analysiert, transformiert, gerendert und exportiert werden kann. Keine Mock-Architektur, kein Wegwerf-MVP. Der erste Integrationsschnitt muss echte Dateien, echte Artefakte und echte Projekt-Persistenz bedienen. + +--- + +## 1. Entscheidung + +Die richtige strategische Richtung ist: + +```text +Signal IR + -> Content-addressed Artifact Store + -> Extension Manifest + Capability Policy + -> Worker Runtime + -> Wasm Component/WIT Runtime + -> Importer/Analyzer/Operator/Renderer Adapter + -> Timeline/NodeGraph/Render Integration +``` + +Pandino oder ein anderer Service-Registry-Mechanismus kann spaeter als Bundle-/Service-Layer oben drauf kommen. Fuer den Kern ist jetzt wichtiger, dass MasterSelects ein stabiles Datenmodell, einen sicheren Ausfuehrungsraum und eine versionierte ABI bekommt. Sonst registrieren wir nur alte Kopplung in neuem Gewand. + +--- + +## 2. Codebase-Befund + +### 2.1 Import ist zentral, aber noch typbegrenzt + +Relevante Dateien: + +- `src/stores/mediaStore/helpers/importPipeline.ts` +- `src/stores/mediaStore/helpers/mediaTypeHelpers.ts` +- `src/stores/mediaStore/slices/fileImportSlice.ts` +- `src/stores/mediaStore/types.ts` +- `src/services/project/types/media.types.ts` + +Befund: + +- `classifyMediaType(file)` kennt Video, Audio, Image, Model, Gaussian Splat und Vector Animation. +- Unbekannte Dateien werden aktuell hart abgelehnt. +- `ProjectMediaFile.type` ist noch eine feste Union. +- Der Importpfad kopiert nach `Raw/`, erzeugt Thumbnails/Proxy/Metadaten und baut sofort ein `MediaFile`. + +Konsequenz: + +- Der Universal-Importer darf den existierenden Pfad nicht ersetzen, bevor er dessen Projekt-/Thumbnail-/Proxy-Verhalten abbildet. +- Neue Importer muessen zuerst als Adapter um den bestehenden Import laufen und dann Schritt fuer Schritt den festen Typknoten aufbrechen. + +### 2.2 NodeGraph hat schon die richtige Sprache, aber nicht die Runtime + +Relevante Dateien: + +- `src/types/nodeGraph.ts` +- `src/stores/timeline/nodeGraphSlice.ts` +- `src/services/nodeGraph/clipGraphProjection.ts` +- `src/services/nodeGraph/aiNodeRuntime.ts` + +Befund: + +- `NodeGraphSignalType` kennt bereits `texture`, `audio`, `geometry`, `curve`, `mask`, `text`, `metadata`, `event`, `time`, `scene`, `timeline`, `render-target`, `number`, `boolean`, `string`. +- `NodeGraphRuntimeKind` kennt bereits `worker` und `wasm`. +- `aiNodeRuntime.ts` fuehrt generierten JS-Code aktuell ueber `new Function(...)` im Main Context aus. +- Textur-Processing ist stark limitiert und Canvas-basiert. + +Konsequenz: + +- Signal IR kann auf den vorhandenen Signaltypen aufbauen. +- Der erste harte Sicherheitsgewinn ist: AI/generated/custom nodes laufen nicht mehr im Main Context, sondern ueber Worker/Capability Runtime. + +### 2.3 Worker/Wasm existiert nur punktuell + +Relevante Dateien/Pfade: + +- `src/workers/transcriptionWorker.ts` +- `src/services/sam2/sam2Worker.ts` +- `src/engine/gaussian/core/splatOrderSortWorker.ts` +- `src/engine/ffmpeg/FFmpegBridge.ts` +- `@playcanvas/splat-transform` WebP/Wasm-Nutzung + +Befund: + +- Es gibt mehrere Worker-/Wasm-Nutzungen, aber keinen gemeinsamen Host, keine gemeinsame Job-Lifecycle-API, keine Capability-Policy und keinen Artifact-Output-Vertrag. +- FFmpeg, SAM2 und Gaussian Sort loesen jeweils lokale Spezialfaelle. + +Konsequenz: + +- Die Runtime muss als eigene Schicht entstehen und existierende Spezial-Worker spaeter adaptieren, nicht sofort umschreiben. + +### 2.4 Projektcache existiert, aber nicht als universeller Artifact Store + +Relevante Dateien: + +- `src/services/project/core/constants.ts` +- `src/services/project/domains/CacheService.ts` +- `src/services/project/domains/RawMediaService.ts` +- `src/services/projectDB.ts` +- `src/stores/mediaStore/helpers/fileHashHelpers.ts` + +Befund: + +- Projektfolder haben `Raw`, `Proxy`, `Analysis`, `Transcripts`, `Cache/thumbnails`, `Cache/splats`, `Cache/waveforms`. +- Hashing ist primaer Import-/Dedup-Hilfe, nicht universelle CAS-Basis. +- Thumbnails, Splats, Waveforms und Proxy Frames sind getrennte Spezialcaches. + +Konsequenz: + +- Fuer Signal IR brauchen wir einen generischen Artifact Store mit Manifesten, nicht nur weitere Spezialordner. +- Bestehende Folder koennen bleiben, aber neue Artefakte sollten unter `Cache/artifacts//...` oder einem aequivalenten Schema landen. + +### 2.5 Policy-Muster existiert + +Relevante Dateien: + +- `src/services/aiTools/policy/types.ts` +- `src/services/aiTools/policy/registry.ts` + +Befund: + +- AI Tools haben bereits `readOnly`, `riskLevel`, `requiresConfirmation`, `sensitiveDataAccess`, `localFileAccess`, `allowedCallers`. + +Konsequenz: + +- Die Extension-/Runtime-Capabilities sollten dasselbe Denkmuster wiederverwenden, aber feiner fuer Dateizugriff, Netzwerk, GPU, Zeit, Random, Projekt-Schreibzugriff, Artifact-Schreibzugriff und Timeline-Mutation. + +### 2.6 Effekte sind Registry-basiert, aber build-time + +Relevante Dateien: + +- `src/effects/index.ts` +- `src/effects/types.ts` +- `src/effects/*` + +Befund: + +- Effekte sind klar registriert, aber ueber statische Imports. +- WGSL-Shader und Uniform-Packing sind gute Kandidaten fuer spaetere Operator-Provider. + +Konsequenz: + +- Effekt-Registry bleibt zunaechst stabil. +- Ein spaeterer `signal-operator` kann Effektdefinitionen spiegeln, statt den Renderpfad sofort zu veraendern. + +--- + +## 3. Zielarchitektur + +### 3.1 Signal IR + +Signal IR ist das gemeinsame Datenmodell fuer alles, was MasterSelects verarbeitet. + +Kernobjekte: + +- `SignalAsset`: importierte Datei oder erzeugtes Asset. +- `SignalRef`: referenzierbarer Output eines Assets, Operators oder NodeGraph. +- `SignalKind`: `texture`, `audio`, `geometry`, `point-cloud`, `mesh`, `scene`, `table`, `document`, `curve`, `mask`, `text`, `metadata`, `event`, `timeline`, `render-target`, `binary`. +- `SignalArtifact`: gespeichertes Ergebnis mit Content Hash, MIME, Byte Range, Codec/Encoding und Provenance. +- `SignalOperator`: pure oder stateful Transformation von Input-Signalen zu Output-Signalen. +- `SignalGraph`: persistierbare Verbindung von Quellen, Operatoren und Outputs. + +Wichtige Designregel: + +- Timeline und Render Engine duerfen nicht sofort direkt auf jedes neue Format zugreifen muessen. +- Jede neue Datei wird zuerst SignalAsset plus mindestens ein SignalRef. Danach entscheiden Adapter, ob daraus Clip, Node, Geometry, Texture, Table oder Document Preview wird. + +### 3.2 Extension ABI + +Extension-Typen: + +- `importer`: erkennt Dateien, extrahiert Metadaten, erzeugt SignalRefs und Artefakte. +- `analyzer`: erzeugt Analyseartefakte, z.B. transcript, waveform, mesh stats, table schema, document outline. +- `operator`: transformiert SignalRefs, z.B. table -> curve, mesh -> point-cloud, document page -> texture. +- `renderer-adapter`: erzeugt renderbare Layer-Inputs fuer WebGPU/Canvas/HTML. +- `exporter`: erzeugt Dateien aus SignalGraph/Timeline/Render Targets. + +ABI-Schichten: + +- TypeScript Host API fuer built-in Provider. +- Worker RPC ABI fuer JS/TS Provider. +- WIT ABI fuer Wasm Components. + +Warum WIT: + +- WIT beschreibt Contracts fuer WebAssembly Components, nicht Verhalten. +- WIT kann Interfaces und Worlds definieren. +- WIT Resources eignen sich fuer Handles, die nicht als riesige Bytes kopiert werden sollen. +- `jco` ist der relevante JS-nahe Toolchain-Kandidat fuer Components im Browser-/Node-Umfeld. + +### 3.3 Worker/Capability Runtime + +Runtime-Aufgaben: + +- Provider isoliert ausfuehren. +- Jobs starten, abbrechen, priorisieren und monitoren. +- Transferables nutzen: `ArrayBuffer`, `ImageBitmap`, spaeter `VideoFrame` wo moeglich. +- Capabilities erzwingen: Dateilesen, Dateischreiben, Projektcache, Netzwerk, Random, Time, GPU, Timeline-Mutation. +- Progress, Logs, Diagnostics und Artifact-Outputs standardisieren. + +Nicht-Ziel: + +- Kein versteckter globaler DI-Container als erstes Fundament. +- Keine Main-Thread-Ausfuehrung von untrusted/generated Plugin-Code. + +### 3.4 Artifact Store + +Der Artifact Store ist die Bruecke zwischen Import, Analyse, Runtime und Projektpersistenz. + +Pflicht: + +- SHA-256 Content Hash fuer echte Artefakte. +- Manifest mit `artifactId`, `hash`, `size`, `mimeType`, `encoding`, `producer`, `sourceRefs`, `createdAt`, `schemaVersion`. +- Speicherziel im Projektordner und IndexedDB-Index. +- Backwards-kompatible Nutzung bestehender `Cache/thumbnails`, `Cache/splats`, `Cache/waveforms`, solange die alten Pfade gebraucht werden. + +Zielpfad-Vorschlag: + +```text +Cache/artifacts/ + sha256/ + ab/ + abcdef.../ + artifact.bin + manifest.json +``` + +--- + +## 4. Parallele Agentenstruktur + +Wir starten mit 6 Implementierungs-Agenten plus 3 Review-/Synthese-Agenten. Die 6 Implementierungs-Agenten arbeiten parallel mit klarer Ownership. Die 3 Review-Agenten pruefen danach Befunde, API-Schnittstellen und Integrationsrisiken. + +### Agent 1: Signal IR + Typmigration + +Ownership: + +- `src/signals/**` +- `src/types/nodeGraph.ts` nur additive Anpassungen +- Tests unter `tests/unit/signals/**` + +Auftrag: + +- `SignalKind`, `SignalAsset`, `SignalRef`, `SignalArtifact`, `SignalGraph`, `SignalOperatorDescriptor` definieren. +- Mapping von bestehendem `MediaFile`, `ProjectMediaFile`, `TimelineSourceType`, `NodeGraphSignalType` auf Signal IR dokumentieren. +- Keine Importpipeline umbauen. +- Keine Render-Hotpaths anfassen. + +Ergebnis: + +- Kompilierende TypeScript-Typen. +- Mapping-Dokument als Kommentar oder `docs/Features/Signal-IR.md`. +- Unit-Tests fuer Schema Guards/normalization. + +### Agent 2: Artifact Store + Projektpersistenz + +Ownership: + +- `src/artifacts/**` +- `src/services/project/domains/ArtifactService.ts` +- additive Anpassungen in `src/services/project/core/constants.ts` +- additive Anpassungen in `src/services/projectDB.ts` +- Tests unter `tests/unit/artifacts/**` + +Auftrag: + +- Generischen Artifact Store bauen. +- SHA-256 Hashing fuer Blob/ArrayBuffer/File implementieren. +- Manifest lesen/schreiben. +- Project folder + IndexedDB Index verbinden. +- Bestehende Cache Services nicht entfernen. + +Ergebnis: + +- `putArtifact`, `getArtifact`, `hasArtifact`, `listArtifactsBySource`, `deleteArtifact`. +- Projektordner-Konstante fuer `Cache/artifacts`. +- Tests mit echten Blob-Artefakten. + +### Agent 3: Extension Registry + Capability Policy + +Ownership: + +- `src/extensions/**` +- `src/runtime/capabilities/**` +- Tests unter `tests/unit/extensions/**` + +Auftrag: + +- Provider Manifest definieren. +- Capability-Modell definieren. +- Registry fuer built-in, worker und wasm Provider bauen. +- Policy-Pruefung analog zu AI Tool Policy, aber fuer Runtime-Jobs. + +Capability-Vorschlag: + +```ts +type RuntimeCapability = + | 'file.read' + | 'file.write' + | 'artifact.read' + | 'artifact.write' + | 'project.read' + | 'project.write' + | 'network.fetch' + | 'time.now' + | 'random' + | 'gpu.compute' + | 'timeline.mutate' + | 'ai.invoke'; +``` + +Ergebnis: + +- Provider koennen registriert und nach File Signature, MIME, Extension, SignalKind und RuntimeKind gesucht werden. +- Unknown Provider/Capability fail-closed. +- Tests fuer erlaubte/verbotene Capabilities. + +### Agent 4: Worker Runtime + +Ownership: + +- `src/runtime/worker/**` +- `src/workers/runtimeHost.worker.ts` +- Tests unter `tests/unit/runtime/worker/**` + +Auftrag: + +- Gemeinsames Worker-Protokoll fuer Runtime-Jobs bauen. +- Job lifecycle: `queued`, `running`, `progress`, `completed`, `failed`, `cancelled`. +- AbortController-Unterstuetzung. +- Transferables sauber behandeln. +- Logs/Diagnostics aus Worker zurueckfuehren. + +Ergebnis: + +- `WorkerRuntimeHost`. +- `RuntimeJobClient`. +- Testbarer Echo-/Hash-/CSV-Worker als echter Worker-Fixture, nicht als Mock. +- Keine Kopplung an MediaStore. + +### Agent 5: Wasm/WIT Host + ABI + +Ownership: + +- `wit/masterselects/**` +- `src/runtime/wasm/**` +- `scripts/wasm/**` +- Tests unter `tests/unit/runtime/wasm/**` + +Auftrag: + +- WIT Packages fuer MasterSelects Provider definieren. +- Host-Facade fuer Wasm Components entwerfen. +- `jco`/Component Model Toolchain als realen Buildpfad pruefen. +- Minimaler echter Wasm-Provider: CSV oder binary metadata importer, der echte Bytes verarbeitet und ein Signal Manifest ausgibt. + +WIT-Startpunkt: + +```wit +package masterselects:runtime@0.1.0; + +interface signals { + enum signal-kind { + texture, + audio, + geometry, + point-cloud, + table, + document, + curve, + mask, + text, + metadata, + binary, + } + + record artifact-ref { + id: string, + hash: string, + mime-type: string, + size: u64, + } + + record signal-ref { + id: string, + kind: signal-kind, + artifact: option, + metadata-json: string, + } +} + +interface importer { + use signals.{signal-ref}; + + record import-request { + file-name: string, + mime-type: string, + bytes: list, + } + + record import-result { + signals: list, + diagnostics-json: string, + } + + can-import: func(file-name: string, mime-type: string, header: list) -> bool; + import-file: func(request: import-request) -> result; +} + +world masterselects-importer { + export importer; +} +``` + +Ergebnis: + +- WIT ist versioniert. +- Wasm-Host kann mindestens einen echten Component-Importer laden oder, falls Browser-Component-Support blockiert, via `jco transpile` als ES-Modul nutzen. +- Dokumentierter Toolchain-Befehl. + +### Agent 6: Universal Import Orchestrator + Kompatibilitaetsadapter + +Ownership: + +- `src/importers/**` +- additive Anpassungen in `src/stores/mediaStore/helpers/importPipeline.ts` +- additive Anpassungen in `src/stores/mediaStore/slices/fileImportSlice.ts` +- Tests unter `tests/unit/importers/**` + +Auftrag: + +- Import-Orchestrator bauen, der zuerst Provider Discovery macht und danach den passenden Importpfad ausfuehrt. +- Bestehende Medienformate weiter ueber den aktuellen Importpfad laufen lassen. +- Neue Signal-Importer fuer CSV und PLY/OBJ oder SVG als echte erste Verticals anschliessen. +- Fallback: unknown file wird als `binary` SignalAsset mit Metadaten importiert, nicht hart verworfen. + +Ergebnis: + +- Bestehende Video/Audio/Image/Model/Gaussian/Vector Imports bleiben kompatibel. +- Mindestens ein bisher unbekanntes Format landet als echtes SignalAsset mit Artifact und Project-Persistenz. +- Import UI/MediaStore bekommt eine Kompatibilitaetsdarstellung, ohne Timeline-Zwang. + +--- + +## 5. Review- und Konsens-Agenten + +### Review Agent A: API/Kontrakt + +Prueft: + +- Passen Signal IR, Artifact Store, Extension Registry, Worker Runtime und WIT zusammen? +- Gibt es doppelte Begriffe oder inkonsistente IDs? +- Sind alte Media-/Timeline-Typen sauber adaptierbar? + +Output: + +- `docs/plans/wasm-signal-review-api.md` + +### Review Agent B: Runtime/Security + +Prueft: + +- Laeuft generated/custom code wirklich ausserhalb des Main Context? +- Sind Capabilities fail-closed? +- Gibt es unkontrollierten File-, Network-, Random-, Time- oder Project-Zugriff? +- Sind Worker-Jobs abbrechbar und diagnosierbar? + +Output: + +- `docs/plans/wasm-signal-review-runtime-security.md` + +### Review Agent C: Integration/Performance + +Prueft: + +- Wird der WebGPU Render-Hotpath geschont? +- Gibt es unnoetige Kopien grosser Dateien? +- Bleiben bestehende Imports, Project Save/Load und Export ungebrochen? +- Sind Tests realistisch und nicht nur Type-Tests? + +Output: + +- `docs/plans/wasm-signal-review-integration-performance.md` + +### Konsens + +Nach den drei Reviews schreibt der Integrator: + +- `docs/plans/wasm-signal-runtime-consensus.md` + +Muss enthalten: + +- gemeinsam akzeptierte API +- offene harte Entscheidungen +- Reihenfolge der Merge-Slices +- Liste der Dateien, die noch nicht parallel angefasst werden duerfen + +--- + +## 6. Abhaengigkeiten und Parallelisierung + +### Wave 1: Contracts und Fundamente + +Parallel startbar: + +- Agent 1 Signal IR +- Agent 2 Artifact Store +- Agent 3 Extension Registry + Capability Policy +- Agent 4 Worker Runtime +- Agent 5 WIT Host/ABI + +Agent 6 kann parallel vorbereiten: + +- aktuelle Importpipeline kapseln +- Tests fuer bestehende Importklassifikation schreiben +- noch keine finale Integration ohne Signal IR + Registry + Artifact Store + +### Wave 2: Realer Vertical Slice + +Startbedingung: + +- `SignalRef`, `SignalArtifact`, Provider Manifest und Artifact Store Interfaces sind stabil genug. + +Parallel: + +- Agent 5 liefert echten Wasm/WIT Importer. +- Agent 4 liefert Worker-Ausfuehrung. +- Agent 6 verbindet Orchestrator mit CSV/SVG/PLY oder binary fallback. +- Agent 1 ergaenzt Mapping fuer Timeline/NodeGraph. +- Agent 2 bindet Artefakte in Projektpersistenz ein. +- Agent 3 erzwingt Capabilities im Orchestrator. + +### Wave 3: NodeGraph und Runtime Migration + +Neue oder Folge-Agenten: + +- NodeGraph-Agent migriert `aiNodeRuntime.ts` auf Worker/Capability Runtime. +- Render-Adapter-Agent baut `SignalRef -> LayerSource` Adapter. +- Persistence-Agent erweitert Project Save/Load um SignalAssets und Artifacts. + +Erst hier werden Hotpaths wie `LayerCollector`, `RenderDispatcher`, `LayerBuilderService` oder Timeline-Clip-Typen groesser angefasst. + +--- + +## 7. Merge-Slices + +### Slice 1: Pure Contracts + +Enthaelt: + +- `src/signals/**` +- `src/extensions/**` +- `src/runtime/capabilities/**` +- `wit/masterselects/**` +- Tests fuer Types/Policy + +Akzeptanz: + +- `npm run test -- tests/unit/signals tests/unit/extensions` +- `npm run build` + +### Slice 2: Artifact Store + +Enthaelt: + +- `src/artifacts/**` +- Project constants +- ProjectDB additive schema/index changes +- Tests mit echten Blob/File Daten + +Akzeptanz: + +- Artefakt schreiben, lesen, erneut deduplizieren. +- Altes Thumbnail/Splat/Waveform Verhalten bleibt unveraendert. + +### Slice 3: Worker Runtime + +Enthaelt: + +- Worker Host +- Job Client +- Runtime Worker Fixture +- Cancellation/Progress/Diagnostics Tests + +Akzeptanz: + +- Echter Worker-Test verarbeitet echte Bytes. +- Job kann abgebrochen werden. +- Transferables werden genutzt. + +### Slice 4: Wasm/WIT Toolchain + +Enthaelt: + +- WIT package +- Build Script +- Beispiel-Component +- Host Adapter + +Akzeptanz: + +- Ein echter Wasm/WIT Importer verarbeitet eine Fixture-Datei. +- Toolchain ist lokal reproduzierbar dokumentiert. +- Wenn Browser-Loading nicht direkt geht, ist `jco transpile` als ES-Modul-Fallback bewiesen. + +### Slice 5: Universal Importer + +Enthaelt: + +- Import Orchestrator +- Adapter zum bestehenden `processImport` +- Binary/CSV/SVG/PLY erster Signal Import +- MediaStore-Kompatibilitaet + +Akzeptanz: + +- Bestehende Medienimports laufen weiter. +- Eine bisher unbekannte Datei wird nicht abgelehnt, sondern als SignalAsset importiert. +- Project Save/Load verliert die Signal-Referenz nicht. + +### Slice 6: NodeGraph Runtime Migration + +Enthaelt: + +- `aiNodeRuntime.ts` weg von `new Function` im Main Context. +- Worker/Capability Runtime fuer generated/custom nodes. +- SignalRef-basierte Inputs/Outputs. + +Akzeptanz: + +- Custom/AI Node erzeugt reales Ergebnis ueber Worker. +- Main Thread fuehrt keinen generated code direkt aus. +- Bestehende Node Workspace UI bleibt bedienbar. + +--- + +## 8. Harte Architekturregeln + +1. Keine neuen Plugin-/Generated-Code-Pfade im Main Context. +2. Keine neue Dateiart darf nur als UI-Mock auftauchen. Sie muss ein echtes SignalAsset und mindestens ein reales Artifact oder nachvollziehbare Metadata erzeugen. +3. Bestehende Imports fuer Video/Audio/Image/Model/Gaussian/Vector duerfen nicht regressieren. +4. Hotpaths (`RenderDispatcher`, `LayerCollector`, `WebGPUEngine`) werden erst nach Contract- und Adapter-Schicht angefasst. +5. Unknown file ist kein Fehlerfall mehr. Mindestens `binary` SignalAsset ist Pflicht. +6. Jede Runtime-Ausfuehrung hat Job-ID, Logs, Progress, Result, Diagnostics und Cancellation. +7. Alle Capabilities fail-closed. +8. Project Save/Load muss SignalAssets, Artifacts und Provider-Versionen versioniert persistieren. +9. Artefakte sind content-addressed. Pfadnamen oder Media IDs allein reichen nicht. +10. Wasm/WIT ABI wird versioniert und darf nicht implizit aus TypeScript-Typen generiert werden, ohne die WIT-Dateien zu reviewen. + +--- + +## 9. Erste echte Verticals + +Diese Verticals sind sinnvoll, weil sie verschiedene Signal-Klassen erzwingen: + +### Vertical A: CSV -> table -> curve/texture preview + +Warum: + +- Kleine Dateien, schnelle Tests, hoher Nutzen fuer "jede Datei wird Signal". +- Erzwingt `table`, `metadata`, optional `curve`. + +Akzeptanz: + +- CSV importiert als `SignalAsset`. +- Header/Rows/Column Types werden analysiert. +- Preview kann mindestens eine Tabellen-/Texturansicht erzeugen. + +### Vertical B: PLY/OBJ -> geometry/point-cloud + +Warum: + +- Nahe an bestehender Model/Gaussian-Welt. +- Erzwingt Geometry/Point Cloud Artefakte. + +Akzeptanz: + +- PLY oder OBJ wird importiert, auch wenn es nicht ueber den alten Modelpfad laeuft. +- Stats werden als Metadata Artifact persistiert. +- Spaeterer Render-Adapter kann daraus LayerSource bauen. + +### Vertical C: SVG -> document/vector/texture + +Warum: + +- Passt zum Ziel "Dokumente/SVG". +- Erzwingt Document/Vector/Texture-Bruecke. + +Akzeptanz: + +- SVG wird als `document` oder `vector` SignalAsset importiert. +- Raster Preview wird als Artifact erzeugt. +- Original bleibt als binary/document Artifact erhalten. + +--- + +## 10. Agent-Prompts + +### Prompt fuer Agent 1 + +```text +Du bist Agent 1 fuer MasterSelects Wasm/WIT + Signal IR. Lies AGENTS.md und den Plan docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md. Deine Ownership ist src/signals/**, additive Aenderungen in src/types/nodeGraph.ts und tests/unit/signals/**. Baue echte TypeScript-Typen und Guards fuer SignalAsset, SignalRef, SignalArtifact, SignalGraph und SignalOperatorDescriptor. Schreibe deine Befunde und geaenderten Dateien am Ende auf. Fasse dich nicht mit Importpipeline oder Render-Hotpaths an. +``` + +### Prompt fuer Agent 2 + +```text +Du bist Agent 2 fuer MasterSelects Artifact Store. Lies AGENTS.md und den Plan docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md. Deine Ownership ist src/artifacts/**, src/services/project/domains/ArtifactService.ts, additive Aenderungen in src/services/project/core/constants.ts und src/services/projectDB.ts sowie tests/unit/artifacts/**. Implementiere echten content-addressed Artifact Store mit SHA-256, Manifesten und Projektfolder/IndexedDB-Anbindung. Schreibe Befunde und geaenderte Dateien am Ende auf. Bestehende Cache-Services nicht entfernen. +``` + +### Prompt fuer Agent 3 + +```text +Du bist Agent 3 fuer MasterSelects Extension Registry und Capability Policy. Lies AGENTS.md und den Plan docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md. Deine Ownership ist src/extensions/**, src/runtime/capabilities/** und tests/unit/extensions/**. Definiere Provider Manifest, RuntimeCapability, Registry und fail-closed Policy-Pruefung fuer built-in, worker und wasm Provider. Schreibe Befunde und geaenderte Dateien am Ende auf. +``` + +### Prompt fuer Agent 4 + +```text +Du bist Agent 4 fuer MasterSelects Worker Runtime. Lies AGENTS.md und den Plan docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md. Deine Ownership ist src/runtime/worker/**, src/workers/runtimeHost.worker.ts und tests/unit/runtime/worker/**. Baue echten WorkerRuntimeHost mit Job Lifecycle, Progress, Diagnostics, Cancellation und Transferables. Nutze echte Worker-Fixtures statt Mocks. Schreibe Befunde und geaenderte Dateien am Ende auf. +``` + +### Prompt fuer Agent 5 + +```text +Du bist Agent 5 fuer MasterSelects Wasm/WIT Runtime. Lies AGENTS.md und den Plan docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md. Deine Ownership ist wit/masterselects/**, src/runtime/wasm/**, scripts/wasm/** und tests/unit/runtime/wasm/**. Definiere die WIT ABI, pruefe jco/component-model Toolchain und liefere einen echten Wasm/WIT Importer fuer eine Fixture-Datei. Schreibe Befunde, Toolchain-Befehle und geaenderte Dateien am Ende auf. +``` + +### Prompt fuer Agent 6 + +```text +Du bist Agent 6 fuer MasterSelects Universal Import Orchestrator. Lies AGENTS.md und den Plan docs/plans/wasm-wit-signal-ir-worker-runtime-plan.md. Deine Ownership ist src/importers/**, additive Aenderungen in src/stores/mediaStore/helpers/importPipeline.ts und src/stores/mediaStore/slices/fileImportSlice.ts sowie tests/unit/importers/**. Kapsle den bestehenden Importpfad, schliesse Provider Discovery an und sorge dafuer, dass unknown files als binary SignalAsset importiert werden. Bestehende Medienimports muessen weiter funktionieren. Schreibe Befunde und geaenderte Dateien am Ende auf. +``` + +--- + +## 11. Quellen und technische Anker + +- WebAssembly Component Model WIT: https://component-model.bytecodealliance.org/design/wit.html +- WIT Spezifikation im Component Model Repository: https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md +- Bytecode Alliance jco: https://github.com/bytecodealliance/jco +- ComponentizeJS: https://github.com/bytecodealliance/ComponentizeJS +- WASI Interfaces: https://wasi.dev/interfaces + +--- + +## 12. Naechster praktischer Schritt + +1. Plan in sechs Agenten parallel ausfuehren lassen. +2. Jeder Agent schreibt Befunde und geaenderte Dateien. +3. Drei Review-Agenten pruefen API, Runtime/Security und Integration/Performance. +4. Integrator schreibt Konsens und merged zuerst nur Slice 1. +5. Danach echte Verticals mergen, beginnend mit Artifact Store + Worker Runtime + CSV/binary Import. diff --git a/docs/research/pandino-scr-evaluation-starter.md b/docs/research/pandino-scr-evaluation-starter.md new file mode 100644 index 00000000..1a59743a --- /dev/null +++ b/docs/research/pandino-scr-evaluation-starter.md @@ -0,0 +1,148 @@ +# Pandino-SCR Adoption — Evaluation Starter + +> **Status:** Starting point only. This document is intentionally neutral. It is meant to give an AI agent (or human reader) a fair entry point before they read the actual codebase and form their own opinion. +> +> **Origin:** Proposed in [GitHub Issue #133](https://github.com/Sportinger/MasterSelects/issues/133). The proposer suggests adopting Pandino (an OSGi-style Service Component Registry for JavaScript) as the foundational architecture for MasterSelects. +> +> **What this document is NOT:** A decision, a recommendation, a plan, or a final analysis. It does not advocate for or against adoption. +> +> **How to use this document:** Read it, then read the actual code paths referenced below, then form an independent judgment. The arguments listed are deliberately balanced (equal count) so that no side is favored by the framing. + +--- + +## 1. Core ideas behind Pandino-SCR + +Pandino is a JavaScript/TypeScript port of the OSGi Service Component Runtime model. The key concepts: + +- **Bundles** — units of code with a lifecycle (install, start, stop, uninstall). +- **Services** — interface-typed objects published into a central registry by bundles. +- **Component decorators** — `@Component`, `@Reference`, `@Activate`, `@Deactivate` declare what a class provides and what it consumes. +- **Service Component Registry (SCR)** — scans decorated classes, resolves dependencies topologically, instantiates and wires them. +- **Dynamic binding** — services can appear and disappear at runtime; dependent components re-bind or deactivate. +- **Fragments** — non-standalone bundles that extend a host bundle's content (resources, types, configuration). + +The mental model is **inversion of control via discoverable, lifecycle-managed services**, instead of explicit imports and manual wiring. + +Reference: https://github.com/BlackBeltTechnology/pandino + +--- + +## 2. Current state of the MasterSelects codebase + +### 2.1 Existing registry patterns + +The codebase already uses typed registry patterns in six locations: + +| Registry | File | Pattern | +|----------|------|---------| +| Effects | `src/effects/index.ts` | `Map`, populated by explicit imports + `registerEffects()` | +| Properties | `src/services/properties/PropertyRegistry.ts` | Class with `register()`, `registerResolver()`, `registerProvider()` methods | +| Media Runtime | `src/services/mediaRuntime/registry.ts` | Class managing `BasicMediaSourceRuntime` instances per descriptor | +| AI Tool Policy | `src/services/aiTools/policy/registry.ts` | `Map` | +| AI Tool Handlers | `src/services/aiTools/handlers/index.ts` | Multiple `Record` dispatch maps | +| Node Graph descriptors | `src/types/nodeGraph.ts` | Type-only definitions backing a Zustand slice | + +### 2.2 Registration timing + +All six registries are populated at **module-load time**. None of them support runtime add/remove of definitions. Instance lifecycles (e.g., per-clip media runtimes) are dynamic, but the set of registered *types* is fixed at build time. + +### 2.3 Build configuration + +`tsconfig.app.json` currently sets: +- `erasableSyntaxOnly: true` +- `verbatimModuleSyntax: true` +- `strict: true` +- No `experimentalDecorators` +- No `emitDecoratorMetadata` + +Pandino-SCR uses legacy TypeScript decorators with metadata emission. Adopting it requires changing the first three of these settings. + +### 2.4 Module-discovery mechanism + +The codebase does **not** currently use `import.meta.glob` (Vite's native pattern for build-time module discovery). All registry entries are reached through explicit `import` statements. + +### 2.5 Runtime architecture + +Several characteristics are relevant to any DI/lifecycle discussion: + +- **60fps WebGPU render loop** orchestrated from `src/engine/render/RenderLoop.ts` and `RenderDispatcher.ts`. +- **HMR-survival singletons** for the WebGPU engine, FFmpeg bridge, and SAM2 service (pattern documented in `CLAUDE.md` §4). +- **WebGPU device-loss handling** — surfaces and pipelines can become invalid at runtime and need reinitialization. +- **Monitoring infrastructure** — `src/services/monitoring/` contains playback health, frame-phase, and pipeline-event monitors used for debugging. + +### 2.6 Coupling observations + +- `src/stores/timeline/index.ts` combines ~23 feature slices into a single Zustand store. +- This store is imported by many feature modules (LayerBuilder, EffectsPipeline, PropertyRegistry consumers, NodeGraph runtime, export pipeline, AI tool handlers). +- Effects, panels, and services are organized into clear folders, but their dependency graph routes through the central timeline store. + +### 2.7 Cost of adding a new entity (today, baseline) + +- **New effect**: create effect folder + shader, add export to category `index.ts`, automatic pickup. ~2–3 file edits. +- **New AI tool**: handler function + handler map entry + tool definition + policy entry. ~4 file edits across `src/services/aiTools/`. +- **New media source runtime**: implement runtime class + register via descriptor. ~2 file edits. + +### 2.8 Plugin/extension architecture + +There is currently **no third-party plugin loading mechanism**. No dynamic `import()` chains for user-supplied code. The CLAUDE.md §0 vision describes a future where MasterSelects supports all media types via extensible runtimes; the mechanism for that is not yet built. + +--- + +## 3. Arguments **for** adopting Pandino-SCR + +1. **Self-registering components** — a class with `@Component` registers itself; no central registry file needs to be edited per addition. This reduces merge-conflict surface when multiple contributors (or AI agents) add features in parallel. + +2. **Built-in lifecycle hooks** — `@Activate`, `@Deactivate`, `@Modified` give a standardized place for setup/teardown logic. The HMR-singleton pattern currently spread across services could be expressed uniformly. + +3. **Dynamic service binding** — services can be added or removed at runtime, with dependents notified automatically. This is the foundation pattern for plugin/extension ecosystems. + +4. **Service ranking and filter expressions** — when multiple implementations exist (e.g., several video decoders), SCR provides a declarative way to choose between them, instead of ad-hoc selection logic. + +5. **Fragment Pattern** — allows extending an existing bundle without modifying it (adding resources, properties, configuration), which is a mature solution for theme/locale/plugin contributions. + +6. **Established methodology with long track record** — OSGi/SCR concepts have been refined for ~25 years in the JVM world (Eclipse, Equinox, Apache Felix). The semantics are well-understood and documented. + +7. **Declarative dependency graph** — `@Reference` annotations make a component's dependencies visible at the declaration site, rather than scattered across constructor calls and imports. + +--- + +## 4. Arguments **against** adopting Pandino-SCR + +1. **Build-config incompatibility** — `erasableSyntaxOnly: true` and `verbatimModuleSyntax: true` would have to be removed to support `emitDecoratorMetadata`. These were deliberate choices for fast/strict builds; reversing them affects the whole project, not only the DI-adopting parts. + +2. **Implicit wiring reduces grep-ability** — `EFFECT_REGISTRY.set("blur", ...)` is a string-searchable line; `@Component` plus container resolution is not. Stack traces gain framework frames between caller and callee. + +3. **Runtime overhead** — `reflect-metadata` adds bundle weight (~50KB) and decorator evaluation runs at startup. Magnitude depends on component count; needs measurement, but is non-zero. + +4. **Vite-HMR integration is unproven for this stack** — Pandino targets OSGi-style hosts. Behavior under Vite's module replacement (especially for WebGPU device-loss recovery and HMR-singleton survival) is not documented and would need verification. + +5. **Niche in the JavaScript ecosystem** — unlike Angular DI or NestJS, OSGi-style SCR has a small JS user base. Issues hit during adoption may have to be solved without community precedent. + +6. **Learning curve for contributors** — anyone touching the project (human or AI agent) would need to understand the SCR lifecycle model (ServiceFactory, ServiceRanking, References cardinality, dynamic vs. static policy) on top of the existing stack. + +7. **Solves problems the codebase does not yet have** — runtime plugin loading, multi-vendor extension marketplaces, and dynamic service rebinding are SCR's primary value, but MasterSelects does not currently ship in a context where those are required. Adopting the runtime now is buying capability ahead of demand. + +--- + +## 5. Open questions worth investigating + +These are unanswered facts that should inform the decision but are not settled by this document: + +- Does Pandino survive Vite-HMR cycles for long-lived services without re-initializing GPU resources? +- What is the bundle-size delta after adding `reflect-metadata` and a working SCR setup? +- How does decorator-based registration interact with tree-shaking in a production build? +- Can existing monitoring services (`playbackHealthMonitor`, etc.) be expressed as `@Component`s without performance regression in the 60fps render path? +- Is there a hybrid approach (e.g., Pandino for the plugin-facing surface, explicit registries for the render core) that captures most upside with less risk? +- What does a 1-effect or 1-media-importer prototype look like in practice, and how does it compare to the current per-feature edit cost? + +--- + +## 6. Suggested next step for an evaluating agent + +1. Read the six registry files listed in §2.1 directly. +2. Trace the dependency graph from `src/stores/timeline/index.ts` and `src/services/layerBuilder/LayerBuilderService.ts` to feel the actual coupling shape. +3. Read `CLAUDE.md` §4 (HMR Singleton Pattern) and §5 (Debugging) to understand the runtime constraints. +4. Skim the Pandino README and Fragment Pattern doc linked above. +5. Decide whether the arguments in §3 or §4 weigh more for *this* codebase, or whether one of the open questions in §5 needs to be answered before a position is possible. + +This document is a starting point, not a verdict. The agent's own reading of the code is expected to refine, contradict, or extend the points above. diff --git a/package-lock.json b/package-lock.json index cde82a09..3f009099 100644 --- a/package-lock.json +++ b/package-lock.json @@ -41,6 +41,7 @@ "eslint": "^9.39.1", "eslint-plugin-react-hooks": "^7.0.1", "eslint-plugin-react-refresh": "^0.4.24", + "fast-check": "^4.8.0", "globals": "^16.5.0", "jsdom": "^28.0.0", "typescript": "~5.9.3", @@ -4138,6 +4139,29 @@ "node": ">=12.0.0" } }, + "node_modules/fast-check": { + "version": "4.8.0", + "resolved": "https://registry.npmjs.org/fast-check/-/fast-check-4.8.0.tgz", + "integrity": "sha512-GOJ158CUMnN6cSahsv4+ExARvIDuzzinFjkp0E9WtiBa5zcVeLozVkWaE4IzFcc+Y48Wp1EDlUZsXRyAztQcSg==", + "dev": true, + "funding": [ + { + "type": "individual", + "url": "https://github.com/sponsors/dubzzz" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fast-check" + } + ], + "license": "MIT", + "dependencies": { + "pure-rand": "^8.0.0" + }, + "engines": { + "node": ">=12.17.0" + } + }, "node_modules/fast-deep-equal": { "version": "3.1.3", "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", @@ -5482,6 +5506,23 @@ "node": ">=6" } }, + "node_modules/pure-rand": { + "version": "8.4.0", + "resolved": "https://registry.npmjs.org/pure-rand/-/pure-rand-8.4.0.tgz", + "integrity": "sha512-IoM8YF/jY0hiugFo/wOWqfmarlE6J0wc6fDK1PhftMk7MGhVZl88sZimmqBBFomLOCSmcCCpsfj7wXASCpvK9A==", + "dev": true, + "funding": [ + { + "type": "individual", + "url": "https://github.com/sponsors/dubzzz" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fast-check" + } + ], + "license": "MIT" + }, "node_modules/rc": { "version": "1.2.8", "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", diff --git a/package.json b/package.json index ca6ce86b..5e8c897a 100644 --- a/package.json +++ b/package.json @@ -57,6 +57,7 @@ "eslint": "^9.39.1", "eslint-plugin-react-hooks": "^7.0.1", "eslint-plugin-react-refresh": "^0.4.24", + "fast-check": "^4.8.0", "globals": "^16.5.0", "jsdom": "^28.0.0", "typescript": "~5.9.3", diff --git a/src/changelog-data.json b/src/changelog-data.json index 01157567..cbb54256 100644 --- a/src/changelog-data.json +++ b/src/changelog-data.json @@ -1,4 +1,36 @@ [ + { + "date": "2026-05-19", + "type": "new", + "title": "WAV Export and Runtime Updates", + "description": "Audio export now includes WAV output support alongside runtime updates that keep export behavior aligned with the current media pipeline.", + "section": "Export / Runtime", + "commits": ["69db7f5a"] + }, + { + "date": "2026-05-19", + "type": "improve", + "title": "Export Preview and Fast Decode Reliability", + "description": "Export previews, timeline locking, and fast export frame lookup now coordinate more reliably across parallel decode and locked-track scenarios.", + "section": "Export / Timeline", + "commits": ["d7f74a59", "291a303b"] + }, + { + "date": "2026-05-19", + "type": "improve", + "title": "Playback Recovery and Board Organization", + "description": "Playback recovery, warmup feedback, media board organization, and track lock UI have been tightened for clearer editing feedback.", + "section": "Playback / Project Board", + "commits": ["aa6f6587", "f6f51528"] + }, + { + "date": "2026-05-19", + "type": "improve", + "title": "Property-Based Editor Invariants", + "description": "The test suite now stress-tests timeline ranges, clip slicing, keyframes, transforms, easing, speed integration, stop markers, camera lenses, export ranges, and file type helpers with randomized property tests.", + "section": "Testing / Reliability", + "commits": ["a3961981"] + }, { "date": "2026-05-12", "type": "new", diff --git a/src/components/export/ExportPanel.tsx b/src/components/export/ExportPanel.tsx index 54ae6e03..0feec0c8 100644 --- a/src/components/export/ExportPanel.tsx +++ b/src/components/export/ExportPanel.tsx @@ -9,7 +9,7 @@ import { projectFileService } from '../../services/projectFileService'; const log = Logger.create('ExportPanel'); import { FrameExporter, downloadBlob } from '../../engine/export'; import type { VideoCodec, ContainerFormat } from '../../engine/export'; -import { AudioExportPipeline } from '../../engine/audio'; +import { AudioExportPipeline, encodeAudioBufferToWavBlob } from '../../engine/audio'; import { useShallow } from 'zustand/react/shallow'; import { useTimelineStore } from '../../stores/timeline'; import { useMediaStore } from '../../stores/mediaStore'; @@ -241,7 +241,7 @@ export function ExportPanel() { gifAlphaThreshold, setGifAlphaThreshold, isFFmpegLoading, isFFmpegReady, ffmpegLoadError, stackedAlpha, setStackedAlpha, - includeAudio, setIncludeAudio, audioSampleRate, setAudioSampleRate, + includeAudio, setIncludeAudio, audioOnlyFormat, setAudioOnlyFormat, audioSampleRate, setAudioSampleRate, audioBitrate, setAudioBitrate, normalizeAudio, setNormalizeAudio, videoEnabled, setVideoEnabled, visualMode, setVisualMode, @@ -368,7 +368,9 @@ export function ExportPanel() { // Handle cancel const handleCancel = useCallback(() => { - if (visualMode === 'gif' || visualMode === 'image') { + if (!videoEnabled) { + ffmpegAudioPipelineRef.current?.cancel(); + } else if (visualMode === 'gif' || visualMode === 'image') { ffmpegFrameRendererRef.current?.cancel(); const ffmpeg = getFFmpegBridge(); ffmpeg.cancel(); @@ -387,7 +389,7 @@ export function ExportPanel() { setExportPhase('idle'); // End export progress in timeline endExport(); - }, [exporter, encoder, endExport, setExporter, setExportPhase, setIsExporting, visualMode]); + }, [exporter, encoder, endExport, setExporter, setExportPhase, setIsExporting, videoEnabled, visualMode]); // Handle browser-side GIF export. This is not a WebCodecs codec, but it shares // the browser render path so GIF is available without loading FFmpeg. @@ -848,8 +850,42 @@ export function ExportPanel() { bitrate: audioBitrate, normalize: normalizeAudio, }); + ffmpegAudioPipelineRef.current = audioPipeline; + let timelineExportStarted = false; try { + startExport(startTime, endTime); + timelineExportStarted = true; + + if (audioOnlyFormat === 'wav') { + const audioBuffer = await audioPipeline.exportRawAudio( + startTime, + endTime, + (audioProgress) => { + setProgress({ + phase: 'audio', + currentFrame: 0, + totalFrames: 0, + percent: audioProgress.percent, + estimatedTimeRemaining: 0, + currentTime: endTime, + audioPhase: audioProgress.phase, + audioPercent: audioProgress.percent, + }); + setExportProgress(audioProgress.percent, endTime); + } + ); + + if (audioBuffer && audioBuffer.length > 0) { + const audioBlob = encodeAudioBufferToWavBlob(audioBuffer); + downloadBlob(audioBlob, `${filename}.wav`); + return; + } + + setError('No audio clips found in the selected range'); + return; + } + const audioResult = await audioPipeline.exportAudio( startTime, endTime, @@ -864,6 +900,7 @@ export function ExportPanel() { audioPhase: audioProgress.phase, audioPercent: audioProgress.percent, }); + setExportProgress(audioProgress.percent, endTime); } ); @@ -887,9 +924,27 @@ export function ExportPanel() { log.error('Audio export failed', e); setError(e instanceof Error ? e.message : 'Audio export failed'); } finally { + ffmpegAudioPipelineRef.current = null; setIsExporting(false); + if (timelineExportStarted) { + endExport(); + } } - }, [getCurrentExportRange, filename, isExporting, audioSampleRate, audioBitrate, normalizeAudio, setError, setIsExporting, setProgress]); + }, [ + audioBitrate, + audioOnlyFormat, + audioSampleRate, + endExport, + filename, + getCurrentExportRange, + isExporting, + normalizeAudio, + setError, + setExportProgress, + setIsExporting, + setProgress, + startExport, + ]); // Handle FCPXML export const handleExportFCPXML = useCallback(() => { @@ -1168,7 +1223,9 @@ export function ExportPanel() { let estimatedBitrate: number; if (!videoEnabled) { - estimatedBitrate = audioBitrate; + estimatedBitrate = audioOnlyFormat === 'wav' + ? audioSampleRate * 2 * 16 + : audioBitrate; } else if (encoder === 'webcodecs' || encoder === 'htmlvideo') { estimatedBitrate = bitrate; } else if (ffmpegRateControl === 'crf') { @@ -1262,14 +1319,19 @@ export function ExportPanel() { : 'AAC'; const effectiveIncludeAudio = (isVideoMode || isXmlMode) && includeAudio && !isGifMode; const selectedImageFormat = IMAGE_FORMATS.find(({ id }) => id === imageFormat) ?? IMAGE_FORMATS[0]; - const audioExtension = audioCodec === 'opus' ? 'ogg' : 'aac'; - const audioCodecLabel = audioCodec?.toUpperCase() ?? 'AAC'; + const browserAudioExtension = audioCodec === 'opus' ? 'ogg' : 'aac'; + const browserAudioCodecLabel = audioCodec?.toUpperCase() ?? 'AAC'; + const audioOnlyExtension = audioOnlyFormat === 'wav' ? 'wav' : browserAudioExtension; + const audioOnlyCodecLabel = audioOnlyFormat === 'wav' ? 'WAV PCM' : browserAudioCodecLabel; + const browserAudioUnavailable = isWebCodecsEncoder && !isAudioSupported && !(isAudioOnlyMode && audioOnlyFormat === 'wav'); const currentAudioCodecLabel = isVideoMode && encoder === 'ffmpeg' ? ffmpegAudioCodecLabel - : audioCodecLabel; + : isAudioOnlyMode + ? audioOnlyCodecLabel + : browserAudioCodecLabel; const outputHeight = stackedAlpha && isVideoMode && !isGifMode ? actualHeight * 2 : actualHeight; const frameCount = isImageSequenceMode ? imageSequenceFrameCount : isVideoMode ? Math.ceil((endTime - startTime) * actualFps) : 1; - const displayExtension = isXmlMode ? 'fcpxml' : isAudioOnlyMode ? audioExtension : isImageMode ? (isImageSequenceMode ? imageSequenceOutputLabel.toLowerCase() : imageFormat) : isGifMode ? 'gif' : currentContainerId; + const displayExtension = isXmlMode ? 'fcpxml' : isAudioOnlyMode ? audioOnlyExtension : isImageMode ? (isImageSequenceMode ? imageSequenceOutputLabel.toLowerCase() : imageFormat) : isGifMode ? 'gif' : currentContainerId; const displayOutputName = isImageSequenceMode ? imageSequenceOutputName : `${filename || 'export'}.${displayExtension}`; const displayContainerLabel = isImageSequenceMode ? imageSequenceOutputLabel : `.${displayExtension}`; const estimatedSizeLabel = isXmlMode ? 'Metadata only' : isImageMode ? (isImageSequenceMode ? `${imageSequenceFrameCount} frames ${imageSequenceOutputLabel}` : 'Current frame') : (!videoEnabled && !includeAudio && !isGifMode) ? '-' : estimatedSize(); @@ -1291,7 +1353,7 @@ export function ExportPanel() { isExporting || (isImageSequenceMode && endTime <= startTime) || (!isImageMode && !isXmlMode && endTime <= startTime) || - isAudioOnlyMode && (!includeAudio || !isAudioSupported) || + isAudioOnlyMode && (!includeAudio || (audioOnlyFormat === 'browser' && !isAudioSupported)) || (isVideoMode && encoder === 'ffmpeg' && isFFmpegLoading); const primaryExportLabel = 'Export'; const usesBrowserProgress = isImageSequenceMode || encoder === 'webcodecs' || encoder === 'htmlvideo'; @@ -1299,7 +1361,7 @@ export function ExportPanel() { ? [ { label: currentAudioCodecLabel, target: 'audio-format' as const }, { label: `${audioSampleRate / 1000} kHz`, target: 'audio-format' as const }, - { label: `${Math.round(audioBitrate / 1000)} kbps`, target: 'audio-quality' as const }, + { label: isAudioOnlyMode && audioOnlyFormat === 'wav' ? '16-bit PCM' : `${Math.round(audioBitrate / 1000)} kbps`, target: 'audio-quality' as const }, { label: normalizeAudio ? 'Normalized' : 'Unprocessed', target: 'audio-processing' as const }, ] : []; @@ -1815,15 +1877,28 @@ export function ExportPanel() {
+
@@ -1921,7 +1996,7 @@ export function ExportPanel() { ) : isAudioOnlyMode ? (
- Audio-only export uses the detected browser codec. + Audio-only export writes the selected audio file format.
) : null} @@ -2529,7 +2604,7 @@ export function ExportPanel() { type="button" className={`export-toggle${!isGifMode && (isXmlMode ? includeAudio : !isImageMode && includeAudio) ? ' is-active' : ''}`} onClick={() => setIncludeAudio(!includeAudio)} - disabled={isImageMode || isGifMode || (!isXmlMode && isWebCodecsEncoder && !isAudioSupported)} + disabled={isImageMode || isGifMode || (!isXmlMode && browserAudioUnavailable)} > {!isGifMode && (isXmlMode ? includeAudio : !isImageMode && includeAudio) ? 'On' : 'Off'} @@ -2555,9 +2630,29 @@ export function ExportPanel() { {currentAudioCodecLabel}
- - {currentAudioCodecLabel}{videoEnabled && encoder === 'ffmpeg' ? ' auto' : ''} - + {isAudioOnlyMode ? ( + <> + + + + ) : ( + + {currentAudioCodecLabel}{videoEnabled && encoder === 'ffmpeg' ? ' auto' : ''} + + )} {audioSampleRatePresets.map((preset) => ( - ))} + {isAudioOnlyMode && audioOnlyFormat === 'wav' ? ( + 16-bit PCM + ) : ( + audioBitratePresets.map((preset) => ( + + )) + )}
@@ -2614,7 +2713,7 @@ export function ExportPanel() { )} - {isWebCodecsEncoder && !isAudioSupported && ( + {browserAudioUnavailable && (
Browser audio encoding is not available here. Video export still works.
@@ -2956,7 +3055,7 @@ export function ExportPanel() { type="checkbox" checked={includeAudio} onChange={(e) => setIncludeAudio(e.target.checked)} - disabled={isGifMode || ((encoder === 'webcodecs' || encoder === 'htmlvideo') && !isAudioSupported)} + disabled={isGifMode || browserAudioUnavailable} /> Include Audio @@ -2965,7 +3064,7 @@ export function ExportPanel() { GIF is silent )} - {(encoder === 'webcodecs' || encoder === 'htmlvideo') && !isAudioSupported && ( + {browserAudioUnavailable && ( Not supported @@ -2987,25 +3086,33 @@ export function ExportPanel() {
- + {isAudioOnlyMode && audioOnlyFormat === 'wav' ? ( + + 16-bit PCM + + ) : ( + + )}
- {encoder === 'ffmpeg' && ( + {(encoder === 'ffmpeg' || isAudioOnlyMode) && (
- {ffmpegContainer === 'mov' ? 'AAC' : - ffmpegContainer === 'mkv' ? 'FLAC' : - ffmpegContainer === 'avi' ? 'PCM' : - ffmpegContainer === 'mxf' ? 'PCM' : 'AAC'} (auto) + {isAudioOnlyMode + ? currentAudioCodecLabel + : `${ffmpegContainer === 'mov' ? 'AAC' : + ffmpegContainer === 'mkv' ? 'FLAC' : + ffmpegContainer === 'avi' ? 'PCM' : + ffmpegContainer === 'mxf' ? 'PCM' : 'AAC'} (auto)`}
)} diff --git a/src/components/export/useExportState.ts b/src/components/export/useExportState.ts index 28b967da..585a9b6e 100644 --- a/src/components/export/useExportState.ts +++ b/src/components/export/useExportState.ts @@ -25,6 +25,7 @@ import { type ExportEncoderType, type ExportImageFormat, type ExportImageMode, + type ExportAudioFormat, type ExportSpecialContainer, type ExportVisualMode, } from '../../stores/exportStore'; @@ -74,6 +75,7 @@ export function useExportState(_composition: Composition | undefined) { gifAlphaThreshold, stackedAlpha, includeAudio, + audioOnlyFormat, audioSampleRate, audioBitrate, normalizeAudio, @@ -116,8 +118,11 @@ export function useExportState(_composition: Composition | undefined) { log.info(`Audio codec detected: ${result.codec.toUpperCase()}`); } else { setIsAudioSupported(false); - setSettings({ includeAudio: false }); - log.warn('No audio encoding supported in this browser'); + const currentSettings = useExportStore.getState().settings; + if (currentSettings.videoEnabled || currentSettings.audioOnlyFormat === 'browser') { + setSettings({ includeAudio: false }); + } + log.warn('No browser audio encoding supported in this browser'); } }); }, [setSettings]); @@ -334,6 +339,8 @@ export function useExportState(_composition: Composition | undefined) { setStackedAlpha: (value: boolean) => setSettings({ stackedAlpha: value }), includeAudio, setIncludeAudio: (value: boolean) => setSettings({ includeAudio: value }), + audioOnlyFormat, + setAudioOnlyFormat: (value: ExportAudioFormat) => setSettings({ audioOnlyFormat: value }), audioSampleRate, setAudioSampleRate: (value: 44100 | 48000) => setSettings({ audioSampleRate: value }), audioBitrate, diff --git a/src/components/panels/MediaPanel.tsx b/src/components/panels/MediaPanel.tsx index f3ac9b69..9eb8a754 100644 --- a/src/components/panels/MediaPanel.tsx +++ b/src/components/panels/MediaPanel.tsx @@ -76,7 +76,14 @@ import { isProxyFrameCountComplete } from '../../stores/mediaStore/helpers/proxy const log = Logger.create('MediaPanel'); import { useMediaStore } from '../../stores/mediaStore'; -import type { MediaFile, Composition, ProjectItem, SolidItem, CameraItem } from '../../stores/mediaStore'; +import type { + CameraItem, + Composition, + MediaFile, + MotionShapeItem, + ProjectItem, + SolidItem, +} from '../../stores/mediaStore'; import { useTimelineStore } from '../../stores/timeline'; import { useDockStore } from '../../stores/dockStore'; import { useContextMenuPosition } from '../../hooks/useContextMenuPosition'; @@ -384,10 +391,12 @@ export function MediaPanel() { const compositions = useMediaStore(state => state.compositions); const folders = useMediaStore(state => state.folders); const textItems = useMediaStore(state => state.textItems); - const solidItems = useMediaStore(state => state.solidItems); - const meshItems = useMediaStore(state => state.meshItems); - const cameraItems = useMediaStore(state => state.cameraItems); - const splatEffectorItems = useMediaStore(state => state.splatEffectorItems); + const solidItems = useMediaStore(state => state.solidItems ?? []); + const meshItems = useMediaStore(state => state.meshItems ?? []); + const cameraItems = useMediaStore(state => state.cameraItems ?? []); + const splatEffectorItems = useMediaStore(state => state.splatEffectorItems ?? []); + const mathSceneItems = useMediaStore(state => state.mathSceneItems ?? []); + const motionShapeItems = useMediaStore(state => state.motionShapeItems ?? []); const selectedIds = useMediaStore(state => state.selectedIds); const expandedFolderIds = useMediaStore(state => state.expandedFolderIds); const fileSystemSupported = useMediaStore(state => state.fileSystemSupported); @@ -423,6 +432,7 @@ export function MediaPanel() { removeTextItem, createSolidItem, getOrCreateSolidFolder, + removeSolidItem, updateSolidItem, createMeshItem, getOrCreateMeshFolder, @@ -433,6 +443,12 @@ export function MediaPanel() { createSplatEffectorItem, getOrCreateSplatEffectorFolder, removeSplatEffectorItem, + createMathSceneItem, + getOrCreateMathSceneFolder, + removeMathSceneItem, + createMotionShapeItem, + getOrCreateMotionShapeFolder, + removeMotionShapeItem, setLabelColor, importGaussianSplat, } = useMediaStore.getState(); @@ -1319,12 +1335,15 @@ export function MediaPanel() { else if (compositions.find(c => c.id === id)) removeComposition(id); else if (folders.find(f => f.id === id)) removeFolder(id); else if (textItems.find(t => t.id === id)) removeTextItem(id); + else if (solidItems.find(s => s.id === id)) removeSolidItem(id); else if (meshItems.find(m => m.id === id)) removeMeshItem(id); else if (cameraItems.find(c => c.id === id)) removeCameraItem(id); else if (splatEffectorItems.find(e => e.id === id)) removeSplatEffectorItem(id); + else if (mathSceneItems.find(m => m.id === id)) removeMathSceneItem(id); + else if (motionShapeItems.find(m => m.id === id)) removeMotionShapeItem(id); }); closeContextMenu(); - }, [selectedIds, files, compositions, folders, textItems, meshItems, cameraItems, splatEffectorItems, removeFile, removeComposition, removeFolder, removeTextItem, removeMeshItem, removeCameraItem, removeSplatEffectorItem, closeContextMenu]); + }, [selectedIds, files, compositions, folders, textItems, solidItems, meshItems, cameraItems, splatEffectorItems, mathSceneItems, motionShapeItems, removeFile, removeComposition, removeFolder, removeTextItem, removeSolidItem, removeMeshItem, removeCameraItem, removeSplatEffectorItem, removeMathSceneItem, removeMotionShapeItem, closeContextMenu]); // Get the active parent folder (icons view: current open folder, classic/board view: selected folder or null) const getActiveParentId = useCallback((): string | null => { @@ -1389,6 +1408,18 @@ export function MediaPanel() { closeContextMenu(); }, [createSplatEffectorItem, getOrCreateSplatEffectorFolder, closeContextMenu]); + const handleNewMathScene = useCallback(() => { + const mathFolderId = getOrCreateMathSceneFolder(); + createMathSceneItem(undefined, mathFolderId); + closeContextMenu(); + }, [createMathSceneItem, getOrCreateMathSceneFolder, closeContextMenu]); + + const handleNewMotionShape = useCallback((primitive: import('../../types/motionDesign').ShapePrimitive) => { + const motionFolderId = getOrCreateMotionShapeFolder(); + createMotionShapeItem(primitive, undefined, motionFolderId); + closeContextMenu(); + }, [createMotionShapeItem, getOrCreateMotionShapeFolder, closeContextMenu]); + // Import Gaussian Avatar (.zip) — opens file picker, imports with forced gaussian-avatar type const handleImportGaussianSplat = useCallback(() => { const input = document.createElement('input'); @@ -1572,6 +1603,42 @@ export function MediaPanel() { return; } + if (item.type === 'math-scene') { + setExternalDragPayload({ + kind: 'math-scene', + id: item.id, + duration: item.duration, + hasAudio: false, + isAudio: false, + isVideo: true, + }); + e.dataTransfer.setData('application/x-math-scene-item-id', item.id); + e.dataTransfer.effectAllowed = 'copyMove'; + if (e.currentTarget instanceof HTMLElement) { + e.dataTransfer.setDragImage(e.currentTarget, 10, 10); + } + return; + } + + if (item.type === 'motion-shape') { + const motionShape = item as MotionShapeItem; + setExternalDragPayload({ + kind: 'motion-shape', + id: item.id, + duration: motionShape.duration, + hasAudio: false, + isAudio: false, + isVideo: true, + primitive: motionShape.primitive, + }); + e.dataTransfer.setData('application/x-motion-shape-item-id', item.id); + e.dataTransfer.effectAllowed = 'copyMove'; + if (e.currentTarget instanceof HTMLElement) { + e.dataTransfer.setDragImage(e.currentTarget, 10, 10); + } + return; + } + // Handle media file drag const mediaFile = item as MediaFile; if (mediaFile.isImporting || mediaNeedsRelink(mediaFile)) { @@ -2048,6 +2115,7 @@ export function MediaPanel() { if (mf.fps) parts.push(`${mf.fps} fps`); if (mf.fileSize) parts.push(formatFileSize(mf.fileSize)); if (mf.bitrate) parts.push(formatBitrate(mf.bitrate)); + if (!mf.duration && 'duration' in item && item.duration) parts.push(formatDuration(item.duration)); } return parts.join('\n'); @@ -2067,7 +2135,7 @@ export function MediaPanel() { const importProgress = getItemImportProgress(item); // Duration badge: videos + compositions - const duration = mediaFile?.duration || comp?.duration; + const duration = mediaFile?.duration || comp?.duration || ('duration' in item ? item.duration : undefined); // Folder item count const folderCount = isFolder ? getItemsForParent(item.id).length : 0; @@ -2126,7 +2194,9 @@ export function MediaPanel() { ...meshItems, ...cameraItems, ...splatEffectorItems, - ]), [files, compositions, folders, textItems, solidItems, meshItems, cameraItems, splatEffectorItems]); + ...mathSceneItems, + ...motionShapeItems, + ]), [files, compositions, folders, textItems, solidItems, meshItems, cameraItems, splatEffectorItems, mathSceneItems, motionShapeItems]); const projectListItems = useMemo(() => ([ ...folders, @@ -2136,8 +2206,10 @@ export function MediaPanel() { ...meshItems, ...cameraItems, ...splatEffectorItems, + ...mathSceneItems, + ...motionShapeItems, ...files, - ]), [folders, compositions, textItems, solidItems, meshItems, cameraItems, splatEffectorItems, files]); + ]), [folders, compositions, textItems, solidItems, meshItems, cameraItems, splatEffectorItems, mathSceneItems, motionShapeItems, files]); const allProjectItemsById = useMemo(() => new Map(allProjectItems.map((item) => [item.id, item])), [allProjectItems]); const totalItems = allProjectItems.length; @@ -3051,6 +3123,29 @@ export function MediaPanel() { }; } + if (item.type === 'math-scene') { + return { + kind: 'math-scene', + id: item.id, + duration: item.duration, + hasAudio: false, + isAudio: false, + isVideo: true, + }; + } + + if (item.type === 'motion-shape') { + return { + kind: 'motion-shape', + id: item.id, + duration: item.duration, + hasAudio: false, + isAudio: false, + isVideo: true, + primitive: item.primitive, + }; + } + if (isImportedMediaFileItem(item) && item.file && !item.isImporting) { const isAudioOnly = item.file.type.startsWith('audio/') || @@ -3896,6 +3991,24 @@ export function MediaPanel() { 3D Effector +
+
{ handleNewMathScene(); setAddDropdownOpen(false); }}> + + Math Scene +
+
+ + Motion Shape + ▶ +
+
{ handleNewMotionShape('rectangle'); setAddDropdownOpen(false); }}> + Rectangle +
+
{ handleNewMotionShape('ellipse'); setAddDropdownOpen(false); }}> + Ellipse +
+
+
Mesh @@ -4095,7 +4208,9 @@ export function MediaPanel() { solidItems.find(s => s.id === contextMenu.itemId) || meshItems.find(m => m.id === contextMenu.itemId) || cameraItems.find(c => c.id === contextMenu.itemId) || - splatEffectorItems.find(e => e.id === contextMenu.itemId) + splatEffectorItems.find(e => e.id === contextMenu.itemId) || + mathSceneItems.find(m => m.id === contextMenu.itemId) || + motionShapeItems.find(m => m.id === contextMenu.itemId) : null; const isVideoFile = selectedItem && 'type' in selectedItem && selectedItem.type === 'video'; const isComposition = selectedItem && 'type' in selectedItem && selectedItem.type === 'composition'; @@ -4153,6 +4268,24 @@ export function MediaPanel() { 3D Effector
+
+
{ handleNewMathScene(); closeContextMenu(); }}> + + Math Scene +
+
+ + Motion Shape + ▶ +
+
{ handleNewMotionShape('rectangle'); closeContextMenu(); }}> + Rectangle +
+
{ handleNewMotionShape('ellipse'); closeContextMenu(); }}> + Ellipse +
+
+
Mesh diff --git a/src/components/panels/media/FileTypeIcon.tsx b/src/components/panels/media/FileTypeIcon.tsx index 1a4ef62b..2bf48fe0 100644 --- a/src/components/panels/media/FileTypeIcon.tsx +++ b/src/components/panels/media/FileTypeIcon.tsx @@ -118,6 +118,22 @@ export const FileTypeIcon = memo(({ type, large }: FileTypeIconProps) => { ); + case 'math-scene': + return ( + + + + + + ); + case 'motion-shape': + return ( + + + + + + ); case 'gaussian-avatar': return ( @@ -318,6 +334,25 @@ const LargeIcon = memo(({ type, style }: { type?: string; style: React.CSSProper ); + case 'math-scene': + return ( + + + + + + + ); + case 'motion-shape': + return ( + + + + + + + + ); case 'gaussian-avatar': return ( diff --git a/src/components/panels/media/board/layout.ts b/src/components/panels/media/board/layout.ts index 601e3d13..66abb57b 100644 --- a/src/components/panels/media/board/layout.ts +++ b/src/components/panels/media/board/layout.ts @@ -97,6 +97,8 @@ export function getMediaBoardTypeLabel(item: MediaBoardItem): string { } if (item.type === 'splat-effector') return 'Effector'; if (item.type === 'solid') return 'Solid'; + if (item.type === 'math-scene') return 'Math Scene'; + if (item.type === 'motion-shape') return 'Motion Shape'; if (item.type === 'model') return 'Model'; return item.type.charAt(0).toUpperCase() + item.type.slice(1); } @@ -118,7 +120,7 @@ export function getMediaBoardItemAspectRatio(item: MediaBoardItem): number { return clampMediaBoardNumber(width / height, MEDIA_BOARD_NODE_ASPECT_MIN, MEDIA_BOARD_NODE_ASPECT_MAX); } - if (item.type === 'camera' || item.type === 'model' || item.type === 'splat-effector') { + if (item.type === 'camera' || item.type === 'model' || item.type === 'splat-effector' || item.type === 'motion-shape') { return 1; } diff --git a/src/components/panels/media/board/overviewCanvas.ts b/src/components/panels/media/board/overviewCanvas.ts index 26d5f33a..46c231c8 100644 --- a/src/components/panels/media/board/overviewCanvas.ts +++ b/src/components/panels/media/board/overviewCanvas.ts @@ -53,6 +53,8 @@ function getMediaBoardOverviewFill(item: MediaBoardItem): string { if (item.type === 'composition') return 'rgba(100, 58, 138, 0.82)'; if (item.type === 'text') return 'rgba(46, 59, 78, 0.92)'; if (item.type === 'camera') return 'rgba(139, 108, 45, 0.86)'; + if (item.type === 'math-scene') return 'rgba(49, 72, 108, 0.9)'; + if (item.type === 'motion-shape') return 'rgba(89, 56, 76, 0.9)'; if (item.type === 'image') return 'rgba(38, 64, 84, 0.88)'; if (item.type === 'video') return 'rgba(45, 43, 64, 0.9)'; if (item.type === 'audio') return 'rgba(52, 71, 55, 0.88)'; diff --git a/src/components/panels/media/itemTypeGuards.ts b/src/components/panels/media/itemTypeGuards.ts index c2b8ced8..36a03b7f 100644 --- a/src/components/panels/media/itemTypeGuards.ts +++ b/src/components/panels/media/itemTypeGuards.ts @@ -10,7 +10,9 @@ export function isImportedMediaFileItem(item: ProjectItem): item is MediaFile { item.type === 'text' || item.type === 'solid' || item.type === 'camera' || - item.type === 'splat-effector' + item.type === 'splat-effector' || + item.type === 'math-scene' || + item.type === 'motion-shape' ) { return false; } diff --git a/src/components/preview/Preview.css b/src/components/preview/Preview.css index b8ace2a7..1c487604 100644 --- a/src/components/preview/Preview.css +++ b/src/components/preview/Preview.css @@ -212,6 +212,17 @@ display: block; } +.preview-export-frame { + position: absolute; + max-width: 100%; + max-height: 100%; + object-fit: contain; + background: var(--bg-canvas); + display: block; + pointer-events: none; + z-index: 6; +} + .preview-container { position: relative; overflow: hidden; @@ -458,6 +469,79 @@ color: var(--text-on-accent); } +.preview-playback-waiter-overlay { + position: absolute; + inset: 0; + z-index: 23; + display: flex; + align-items: center; + justify-content: center; + padding: 16px; + pointer-events: none; + background: rgba(0, 0, 0, 0.14); +} + +.preview-playback-waiter { + display: inline-flex; + align-items: center; + gap: 10px; + max-width: 100%; + min-height: 44px; + padding: 9px 12px; + color: var(--text-primary); + background: rgba(10, 12, 16, 0.82); + border: 1px solid rgba(255, 255, 255, 0.14); + border-radius: 6px; + box-shadow: 0 10px 32px rgba(0, 0, 0, 0.32); + backdrop-filter: blur(8px); +} + +.preview-playback-waiter-spinner { + flex: 0 0 auto; + width: 18px; + height: 18px; + border: 2px solid rgba(255, 255, 255, 0.18); + border-top-color: var(--accent); + border-radius: 50%; + animation: preview-playback-waiter-spin 0.78s linear infinite; +} + +.preview-playback-waiter-copy { + display: flex; + flex-direction: column; + gap: 2px; + min-width: 0; +} + +.preview-playback-waiter-title { + overflow: hidden; + color: var(--text-primary); + font-size: 12px; + font-weight: 600; + line-height: 1.25; + text-overflow: ellipsis; + white-space: nowrap; +} + +.preview-playback-waiter-detail { + color: var(--text-secondary); + font-size: 11px; + line-height: 1.2; + white-space: nowrap; +} + +@keyframes preview-playback-waiter-spin { + to { + transform: rotate(360deg); + } +} + +@media (prefers-reduced-motion: reduce) { + .preview-playback-waiter-spinner { + animation-duration: 1.8s; + } +} + .preview-splat-progress-overlay { position: absolute; left: 50%; diff --git a/src/components/preview/Preview.tsx b/src/components/preview/Preview.tsx index 53fc40d0..e00c5be4 100644 --- a/src/components/preview/Preview.tsx +++ b/src/components/preview/Preview.tsx @@ -492,7 +492,30 @@ export function Preview({ panelId, source, showTransparencyGrid }: PreviewProps) const setSceneGizmoClipIdOverride = useEngineStore((s) => s.setSceneGizmoClipIdOverride); const activeSplatLoadProgress = useEngineStore(selectActiveGaussianSplatLoadProgress); const setSceneNavFpsMoveSpeed = useEngineStore((s) => s.setSceneNavFpsMoveSpeed); - const { clips, selectedClipIds, primarySelectedClipId, selectClip, updateClipTransform, updateTextProperties, updateTextBoundsVertex, updateTextBoundsVertices, setPropertyValue, getInterpolatedTextBounds, maskEditMode, maskPanelActive, layers, selectedLayerId, selectLayer, updateLayer, tracks, isPlaying, playheadPosition } = useTimelineStore(useShallow(s => ({ + const { + clips, + selectedClipIds, + primarySelectedClipId, + selectClip, + updateClipTransform, + updateTextProperties, + updateTextBoundsVertex, + updateTextBoundsVertices, + setPropertyValue, + getInterpolatedTextBounds, + maskEditMode, + maskPanelActive, + layers, + selectedLayerId, + selectLayer, + updateLayer, + tracks, + isPlaying, + playheadPosition, + playbackWarmup, + isExporting, + exportPreviewFrame, + } = useTimelineStore(useShallow(s => ({ clips: s.clips, selectedClipIds: s.selectedClipIds, primarySelectedClipId: s.primarySelectedClipId, @@ -512,6 +535,9 @@ export function Preview({ panelId, source, showTransparencyGrid }: PreviewProps) tracks: s.tracks, isPlaying: s.isPlaying, playheadPosition: s.playheadPosition, + playbackWarmup: s.playbackWarmup, + isExporting: s.isExporting, + exportPreviewFrame: s.exportPreviewFrame, }))); const { compositions, activeCompositionId } = useMediaStore(useShallow(s => ({ compositions: s.compositions, @@ -535,6 +561,7 @@ export function Preview({ panelId, source, showTransparencyGrid }: PreviewProps) const containerRef = useRef(null); const canvasWrapperRef = useRef(null); const canvasRef = useRef(null); + const exportPreviewCanvasRef = useRef(null); const overlayRef = useRef(null); const [canvasSize, setCanvasSize] = useState({ width: 1920, height: 1080 }); const [containerSize, setContainerSize] = useState({ width: 0, height: 0 }); @@ -1954,6 +1981,41 @@ export function Preview({ panelId, source, showTransparencyGrid }: PreviewProps) return () => resizeObserver.disconnect(); }, [effectiveResolution.width, effectiveResolution.height]); + const exportPreviewDisplaySize = useMemo(() => { + if (!exportPreviewFrame || containerSize.width <= 0 || containerSize.height <= 0) { + return canvasSize; + } + + const frameAspect = exportPreviewFrame.width / Math.max(1, exportPreviewFrame.height); + const containerAspect = containerSize.width / Math.max(1, containerSize.height); + if (containerAspect > frameAspect) { + const height = containerSize.height; + return { width: Math.floor(height * frameAspect), height: Math.floor(height) }; + } + + const width = containerSize.width; + return { width: Math.floor(width), height: Math.floor(width / frameAspect) }; + }, [ + canvasSize, + containerSize.height, + containerSize.width, + exportPreviewFrame, + ]); + + useEffect(() => { + const canvas = exportPreviewCanvasRef.current; + if (!canvas || !isExporting || !exportPreviewFrame) return; + + if (canvas.width !== exportPreviewFrame.width) canvas.width = exportPreviewFrame.width; + if (canvas.height !== exportPreviewFrame.height) canvas.height = exportPreviewFrame.height; + + const ctx = canvas.getContext('2d', { alpha: false }); + if (!ctx) return; + + ctx.clearRect(0, 0, canvas.width, canvas.height); + ctx.drawImage(exportPreviewFrame, 0, 0); + }, [exportPreviewFrame, isExporting]); + const zoomEditCameraOrthoView = useCallback((e: PreviewWheelEvent): boolean => { if (!editCameraOrthoViewActive || !activeEditCameraOrthoFrame || !editCameraOrthoMode) return false; if (!isCanvasInteractionTarget(e.target)) return false; @@ -2399,6 +2461,16 @@ export function Preview({ panelId, source, showTransparencyGrid }: PreviewProps) const editCameraGizmoTransform = editCameraModeActive && activeCameraClipAtPlayhead ? resolveCameraClipTransformAtPlayhead(activeCameraClipAtPlayhead) : null; + const showPlaybackWaiter = Boolean( + isEngineReady && + !sourceMonitorActive && + playbackWarmup + ); + const playbackWaiterVideoCount = playbackWarmup?.pendingVideoCount ?? 0; + const playbackWaiterLabel = 'Preparing playback'; + const playbackWaiterDetail = playbackWaiterVideoCount > 0 + ? `${playbackWaiterVideoCount} video${playbackWaiterVideoCount === 1 ? '' : 's'}` + : ''; return (
+ {isExporting && exportPreviewFrame && ( + + )} {isEditableSource && maskPanelActive && maskEditMode !== 'none' && ( + {showPlaybackWaiter && ( +
+
+ +
+ )} + {textPreviewEditorEnabled && selectedClip?.textProperties && selectedTextLayer && ( { + if (isExporting) { + e.preventDefault(); + e.dataTransfer.dropEffect = 'none'; + return; + } if (trackMap.get(trackId)?.locked) { e.preventDefault(); e.dataTransfer.dropEffect = 'none'; @@ -495,9 +508,14 @@ export function Timeline() { } else { handleTrackDragOver(e, trackId); } - }, [trackMap, isTransitionDrag, handleTransitionDragOver, handleTrackDragOver, scrollX, pixelToTime]); + }, [isExporting, trackMap, isTransitionDrag, handleTransitionDragOver, handleTrackDragOver, scrollX, pixelToTime]); const handleCombinedDrop = useCallback((e: React.DragEvent, trackId: string) => { + if (isExporting) { + e.preventDefault(); + e.dataTransfer.dropEffect = 'none'; + return; + } if (trackMap.get(trackId)?.locked) { e.preventDefault(); e.dataTransfer.dropEffect = 'none'; @@ -513,7 +531,7 @@ export function Timeline() { } else { handleTrackDrop(e, trackId); } - }, [trackMap, isTransitionDrag, handleTransitionDrop, handleTrackDrop, scrollX, pixelToTime]); + }, [isExporting, trackMap, isTransitionDrag, handleTransitionDrop, handleTrackDrop, scrollX, pixelToTime]); const handleCombinedDragLeave = useCallback((e: React.DragEvent) => { handleTransitionDragLeave(); @@ -1302,7 +1320,7 @@ export function Timeline() { (timelineRef as React.MutableRefObject).current = el; (trackLanesRef as React.MutableRefObject).current = el; }} - className={`timeline-tracks ${clipDrag ? 'dragging-clip' : ''} ${marquee ? 'marquee-selecting' : ''}`} + className={`timeline-tracks ${clipDrag ? 'dragging-clip' : ''} ${marquee ? 'marquee-selecting' : ''} ${isExporting ? 'export-locked' : ''}`} data-ai-id="timeline-tracks" onMouseDown={handleMarqueeMouseDown} onDragOver={(e) => e.preventDefault()} diff --git a/src/components/timeline/TimelineInteractions.css b/src/components/timeline/TimelineInteractions.css index b6c2ecc4..37a1d346 100644 --- a/src/components/timeline/TimelineInteractions.css +++ b/src/components/timeline/TimelineInteractions.css @@ -278,6 +278,15 @@ cursor: crosshair; } +.timeline-tracks.export-locked, +.timeline-tracks.export-locked .timeline-clip, +.timeline-tracks.export-locked .trim-handle, +.timeline-tracks.export-locked .fade-handle, +.timeline-tracks.export-locked .in-out-marker, +.timeline-tracks.export-locked .marker-flag { + cursor: not-allowed !important; +} + .marquee-selection { position: absolute; background: rgba(59, 130, 246, 0.15); diff --git a/src/components/timeline/TimelineKeyframes.css b/src/components/timeline/TimelineKeyframes.css index 0955af75..08981b9b 100644 --- a/src/components/timeline/TimelineKeyframes.css +++ b/src/components/timeline/TimelineKeyframes.css @@ -324,7 +324,7 @@ align-items: center; justify-content: flex-start; padding: 0 8px; - padding-right: 30px; + padding-right: 44px; margin: 0; box-sizing: border-box; position: relative; diff --git a/src/components/timeline/TimelineTracks.css b/src/components/timeline/TimelineTracks.css index 43797955..3278d5fd 100644 --- a/src/components/timeline/TimelineTracks.css +++ b/src/components/timeline/TimelineTracks.css @@ -38,6 +38,29 @@ .track-header.locked { border-left: 2px solid var(--warning); + filter: saturate(0.78) brightness(0.86); +} + +.track-header.locked::after { + content: ""; + position: absolute; + inset: 0; + z-index: 1; + pointer-events: none; + background: + repeating-linear-gradient( + 135deg, + rgba(255, 255, 255, 0.08) 0, + rgba(255, 255, 255, 0.08) 1px, + transparent 1px, + transparent 7px + ), + rgba(0, 0, 0, 0.14); +} + +.track-header.locked > * { + position: relative; + z-index: 2; } /* Green separator line between video and audio track sections */ @@ -123,15 +146,31 @@ } .track-controls { - display: flex; - flex-direction: row; + display: grid; + grid-template-columns: repeat(2, 16px); + grid-auto-rows: 15px; align-items: center; + justify-items: center; gap: 2px; margin-left: auto; position: absolute; right: 4px; top: 50%; transform: translateY(-50%); + z-index: 3; +} + +.track-controls .btn-icon { + display: inline-flex; + align-items: center; + justify-content: center; + width: 16px; + height: 15px; + min-width: 16px; + padding: 0; + border-radius: 3px; + font-size: 9px; + line-height: 1; } /* Track pick whip for layer parenting */ @@ -222,6 +261,24 @@ .track-lane.locked { background-color: color-mix(in srgb, var(--bg-secondary) 88%, var(--warning) 12%); + filter: saturate(0.52) brightness(0.74); +} + +.track-lane.locked::after { + content: ""; + position: absolute; + inset: 0; + z-index: 28; + pointer-events: none; + background: + repeating-linear-gradient( + 135deg, + rgba(255, 255, 255, 0.13) 0, + rgba(255, 255, 255, 0.13) 1px, + transparent 1px, + transparent 9px + ), + rgba(0, 0, 0, 0.22); } .composition-exit-clips-overlay { diff --git a/src/components/timeline/TrackContextMenu.tsx b/src/components/timeline/TrackContextMenu.tsx index b00c3b00..1bd2484d 100644 --- a/src/components/timeline/TrackContextMenu.tsx +++ b/src/components/timeline/TrackContextMenu.tsx @@ -59,29 +59,6 @@ export function TrackContextMenu({ menu, onClose }: TrackContextMenuProps) { onClose(); }; - const handleAddMathScene = () => { - if (menu.trackType !== 'video') return; - const { playheadPosition, addMathSceneClip, selectClip } = useTimelineStore.getState(); - const clipId = addMathSceneClip(menu.trackId, playheadPosition); - if (clipId) { - selectClip(clipId); - } - onClose(); - }; - - const handleAddMotionShape = (primitive: 'rectangle' | 'ellipse') => { - if (menu.trackType !== 'video') return; - const { playheadPosition, addMotionShapeClip, selectClip } = useTimelineStore.getState(); - const clipId = addMotionShapeClip(menu.trackId, playheadPosition, { - primitive, - name: primitive === 'ellipse' ? 'Motion Ellipse' : 'Motion Rectangle', - }); - if (clipId) { - selectClip(clipId); - } - onClose(); - }; - const handleDeleteTrack = () => { useTimelineStore.getState().removeTrack(menu.trackId); onClose(); @@ -111,20 +88,6 @@ export function TrackContextMenu({ menu, onClose }: TrackContextMenuProps) {
+ Add Audio Track
- {menu.trackType === 'video' && ( - <> -
-
- + Add Math Scene -
-
handleAddMotionShape('rectangle')}> - + Add Motion Rectangle -
-
handleAddMotionShape('ellipse')}> - + Add Motion Ellipse -
- - )}
Duplicate Track diff --git a/src/components/timeline/hooks/useClipDrag.ts b/src/components/timeline/hooks/useClipDrag.ts index 135d78c5..1bc22a03 100644 --- a/src/components/timeline/hooks/useClipDrag.ts +++ b/src/components/timeline/hooks/useClipDrag.ts @@ -20,6 +20,7 @@ interface UseClipDragProps { selectedClipIds: Set; scrollX: number; snappingEnabled: boolean; + isExporting: boolean; // Actions selectClip: (clipId: string | null, addToSelection?: boolean, setPrimaryOnly?: boolean) => void; @@ -48,6 +49,7 @@ export function useClipDrag({ selectedClipIds, scrollX, snappingEnabled, + isExporting, selectClip, moveClip, openCompositionTab, @@ -80,6 +82,7 @@ export function useClipDrag({ if (e.button !== 0) return; e.stopPropagation(); e.preventDefault(); + if (isExporting) return; // Use ref for current clipMap to avoid stale closure const currentClipMap = clipMapRef.current; @@ -474,7 +477,7 @@ export function useClipDrag({ document.addEventListener('mousemove', handleMouseMove); document.addEventListener('mouseup', handleMouseUp); }, - [trackLanesRef, timelineRef, clipMap, tracks, scrollX, snappingEnabled, pixelToTime, selectClip, getSnappedPosition, getPositionWithResistance, moveClip] + [trackLanesRef, timelineRef, clipMap, tracks, scrollX, snappingEnabled, isExporting, pixelToTime, selectClip, getSnappedPosition, getPositionWithResistance, moveClip] ); // Handle double-click on clip - open composition if it's a nested comp diff --git a/src/components/timeline/hooks/useClipFade.ts b/src/components/timeline/hooks/useClipFade.ts index a4ab60d8..25f0b44b 100644 --- a/src/components/timeline/hooks/useClipFade.ts +++ b/src/components/timeline/hooks/useClipFade.ts @@ -22,6 +22,7 @@ interface UseClipFadeProps { // Clip and track data clipMap: Map; tracks: TimelineTrack[]; + isExporting: boolean; // Keyframe actions addKeyframe: (clipId: string, property: AnimatableProperty, value: number, time?: number, easing?: EasingType) => void; @@ -47,6 +48,7 @@ interface UseClipFadeReturn { export function useClipFade({ clipMap, tracks, + isExporting, addKeyframe, removeKeyframe, moveKeyframe, @@ -177,6 +179,7 @@ export function useClipFade({ (e: React.MouseEvent, clipId: string, edge: 'left' | 'right') => { e.stopPropagation(); e.preventDefault(); + if (isExporting) return; const clip = clipMap.get(clipId); if (!clip) return; @@ -337,7 +340,7 @@ export function useClipFade({ document.addEventListener('mousemove', handleMouseMove); document.addEventListener('mouseup', handleMouseUp); }, - [clipMap, tracks, getFadeInDuration, getFadeOutDuration, getClipKeyframes, pixelToTime, addKeyframe, moveKeyframe, removeKeyframe, isAudioClip, ensureAudioVolumeEffect] + [clipMap, tracks, isExporting, getFadeInDuration, getFadeOutDuration, getClipKeyframes, pixelToTime, addKeyframe, moveKeyframe, removeKeyframe, isAudioClip, ensureAudioVolumeEffect] ); return { diff --git a/src/components/timeline/hooks/useClipTrim.ts b/src/components/timeline/hooks/useClipTrim.ts index 5e10d83e..be5fb7e4 100644 --- a/src/components/timeline/hooks/useClipTrim.ts +++ b/src/components/timeline/hooks/useClipTrim.ts @@ -10,6 +10,7 @@ interface UseClipTrimProps { // Clip data clipMap: Map; tracks: TimelineTrack[]; + isExporting: boolean; // Actions selectClip: (clipId: string | null, addToSelection?: boolean) => void; @@ -34,6 +35,7 @@ function canLoopExtendVectorClip(clip: TimelineClip): boolean { export function useClipTrim({ clipMap, tracks, + isExporting, selectClip, trimClip, moveClip, @@ -50,6 +52,7 @@ export function useClipTrim({ (e: React.MouseEvent, clipId: string, edge: 'left' | 'right') => { e.stopPropagation(); e.preventDefault(); + if (isExporting) return; const clip = clipMap.get(clipId); if (!clip) return; @@ -187,7 +190,7 @@ export function useClipTrim({ document.addEventListener('mousemove', handleMouseMove); document.addEventListener('mouseup', handleMouseUp); }, - [clipMap, tracks, pixelToTime, selectClip, trimClip, moveClip] + [clipMap, tracks, isExporting, pixelToTime, selectClip, trimClip, moveClip] ); return { diff --git a/src/components/timeline/hooks/useExternalDrop.ts b/src/components/timeline/hooks/useExternalDrop.ts index 01585e9c..385dd3c5 100644 --- a/src/components/timeline/hooks/useExternalDrop.ts +++ b/src/components/timeline/hooks/useExternalDrop.ts @@ -29,6 +29,7 @@ import { import type { ExternalDragState } from '../types'; import type { TimelineTrack, TimelineClip } from '../../../types'; import type { Composition, MediaFile } from '../../../stores/mediaStore'; +import type { ShapePrimitive } from '../../../types/motionDesign'; import { NativeHelperClient } from '../../../services/nativeHelper/NativeHelperClient'; import { Logger } from '../../../services/logger'; @@ -153,6 +154,7 @@ interface UseExternalDropProps { scrollX: number; tracks: TimelineTrack[]; clips: TimelineClip[]; + isExporting: boolean; pixelToTime: (pixel: number) => number; addTrack: (type: 'video' | 'audio') => string | undefined; addClip: (trackId: string, file: File, startTime: number, duration?: number, mediaFileId?: string, mediaTypeOverride?: string) => void; @@ -162,6 +164,8 @@ interface UseExternalDropProps { addMeshClip: (trackId: string, startTime: number, meshType: import('../../../stores/mediaStore/types').MeshPrimitiveType, duration?: number, skipMediaItem?: boolean) => string | null; addCameraClip: (trackId: string, startTime: number, duration?: number, skipMediaItem?: boolean) => string | null; addSplatEffectorClip: (trackId: string, startTime: number, duration?: number, skipMediaItem?: boolean) => string | null; + addMathSceneClip: (trackId: string, startTime: number, duration?: number, skipMediaItem?: boolean) => string | null; + addMotionShapeClip: (trackId: string, startTime: number, options?: { primitive?: ShapePrimitive; duration?: number; name?: string }) => string | null; } interface UseExternalDropReturn { @@ -195,6 +199,10 @@ function getPayloadMimeTypes(payload: ExternalDragPayload): string[] { return ['application/x-camera-item-id']; case 'splat-effector': return ['application/x-splat-effector-item-id']; + case 'math-scene': + return ['application/x-math-scene-item-id']; + case 'motion-shape': + return ['application/x-motion-shape-item-id']; case 'media-file': return payload.isAudio ? ['application/x-media-file-id', 'application/x-media-is-audio'] @@ -209,6 +217,8 @@ function getPayloadMimeData(payload: ExternalDragPayload, mimeType: string): str if (mimeType === 'application/x-mesh-item-id' && payload.kind === 'mesh') return payload.id; if (mimeType === 'application/x-camera-item-id' && payload.kind === 'camera') return payload.id; if (mimeType === 'application/x-splat-effector-item-id' && payload.kind === 'splat-effector') return payload.id; + if (mimeType === 'application/x-math-scene-item-id' && payload.kind === 'math-scene') return payload.id; + if (mimeType === 'application/x-motion-shape-item-id' && payload.kind === 'motion-shape') return payload.id; if (mimeType === 'application/x-media-file-id' && payload.kind === 'media-file') return payload.id; if (mimeType === 'application/x-media-is-audio' && payload.kind === 'media-file' && payload.isAudio) return 'true'; return ''; @@ -293,6 +303,7 @@ export function useExternalDrop({ scrollX, tracks, clips, + isExporting, pixelToTime, addTrack, addClip, @@ -302,6 +313,8 @@ export function useExternalDrop({ addMeshClip, addCameraClip, addSplatEffectorClip, + addMathSceneClip, + addMotionShapeClip, }: UseExternalDropProps): UseExternalDropReturn { const [externalDrag, setExternalDrag] = useState(null); const dragCounterRef = useRef(0); @@ -318,6 +331,16 @@ export function useExternalDrop({ setExternalDrag(null); }, [resetVideoNewTrackGesture]); + const rejectDropDuringExport = useCallback((e: React.DragEvent) => { + if (!isExporting) return false; + e.preventDefault(); + e.stopPropagation(); + e.dataTransfer.dropEffect = 'none'; + dragCounterRef.current = 0; + clearExternalDragState(); + return true; + }, [clearExternalDragState, isExporting]); + const updateVideoNewTrackGesture = useCallback((clientY: number, isAudio: boolean) => { const rect = timelineRef.current?.getBoundingClientRect(); if (!rect) { @@ -580,6 +603,46 @@ export function useExternalDrop({ }; } + if (e.dataTransfer.types.includes('application/x-math-scene-item-id')) { + if (dragPayload?.kind === 'math-scene') { + return { + duration: dragPayload.duration ?? 5, + hasAudio: false, + isAudio: false, + isVideo: true, + }; + } + + const mathSceneItemId = e.dataTransfer.getData('application/x-math-scene-item-id'); + const mathSceneItem = mediaStore.mathSceneItems.find((item) => item.id === mathSceneItemId); + return { + duration: mathSceneItem?.duration ?? 5, + hasAudio: false, + isAudio: false, + isVideo: true, + }; + } + + if (e.dataTransfer.types.includes('application/x-motion-shape-item-id')) { + if (dragPayload?.kind === 'motion-shape') { + return { + duration: dragPayload.duration ?? 5, + hasAudio: false, + isAudio: false, + isVideo: true, + }; + } + + const motionShapeItemId = e.dataTransfer.getData('application/x-motion-shape-item-id'); + const motionShapeItem = mediaStore.motionShapeItems.find((item) => item.id === motionShapeItemId); + return { + duration: motionShapeItem?.duration ?? 5, + hasAudio: false, + isAudio: false, + isVideo: true, + }; + } + if (e.dataTransfer.types.includes('application/x-media-file-id')) { if (dragPayload?.kind === 'media-file') { if (dragPayload.file && dragPayload.isVideo && dragPayload.duration === undefined) { @@ -685,6 +748,7 @@ export function useExternalDrop({ // Handle external file drag enter on track const handleTrackDragEnter = useCallback( (e: React.DragEvent, trackId: string) => { + if (rejectDropDuringExport(e)) return; e.preventDefault(); dragCounterRef.current++; @@ -987,6 +1051,80 @@ export function useExternalDrop({ return; } + if (e.dataTransfer.types.includes('application/x-math-scene-item-id')) { + const mathSceneItemId = dragPayload?.kind === 'math-scene' + ? dragPayload.id + : e.dataTransfer.getData('application/x-math-scene-item-id'); + const mathSceneItem = mathSceneItemId + ? mediaStore.mathSceneItems.find((item) => item.id === mathSceneItemId) + : null; + const duration = dragPayload?.kind === 'math-scene' + ? dragPayload.duration ?? 5 + : mathSceneItem?.duration ?? 5; + + if (isAudioTrack) { + setExternalDrag(applyVideoNewTrackOffer({ + trackId: '', + startTime, + x: e.clientX, + y: e.clientY, + duration, + hasAudio: false, + isVideo: true, + isAudio: false, + })); + return; + } + setExternalDrag(buildTrackPreviewState({ + trackId, + desiredStartTime: startTime, + x: e.clientX, + y: e.clientY, + duration, + hasAudio: false, + isVideo: true, + isAudio: false, + })); + return; + } + + if (e.dataTransfer.types.includes('application/x-motion-shape-item-id')) { + const motionShapeItemId = dragPayload?.kind === 'motion-shape' + ? dragPayload.id + : e.dataTransfer.getData('application/x-motion-shape-item-id'); + const motionShapeItem = motionShapeItemId + ? mediaStore.motionShapeItems.find((item) => item.id === motionShapeItemId) + : null; + const duration = dragPayload?.kind === 'motion-shape' + ? dragPayload.duration ?? 5 + : motionShapeItem?.duration ?? 5; + + if (isAudioTrack) { + setExternalDrag(applyVideoNewTrackOffer({ + trackId: '', + startTime, + x: e.clientX, + y: e.clientY, + duration, + hasAudio: false, + isVideo: true, + isAudio: false, + })); + return; + } + setExternalDrag(buildTrackPreviewState({ + trackId, + desiredStartTime: startTime, + x: e.clientX, + y: e.clientY, + duration, + hasAudio: false, + isVideo: true, + isAudio: false, + })); + return; + } + if (e.dataTransfer.types.includes('Files')) { let dur: number | undefined; let hasAudio: boolean | undefined; @@ -1050,12 +1188,13 @@ export function useExternalDrop({ })); } }, - [tracks, getDesiredStartTime, buildTrackPreviewState, requestVideoDragMetadata, updateResolvedDragMetadata, applyVideoNewTrackOffer] + [tracks, rejectDropDuringExport, getDesiredStartTime, buildTrackPreviewState, requestVideoDragMetadata, updateResolvedDragMetadata, applyVideoNewTrackOffer] ); // Handle external file drag over track const handleTrackDragOver = useCallback( (e: React.DragEvent, trackId: string) => { + if (rejectDropDuringExport(e)) return; e.preventDefault(); e.dataTransfer.dropEffect = 'copy'; @@ -1063,10 +1202,25 @@ export function useExternalDrop({ const isMediaPanelDrag = e.dataTransfer.types.includes('application/x-media-file-id'); const isTextDrag = e.dataTransfer.types.includes('application/x-text-item-id'); const isSolidDrag = e.dataTransfer.types.includes('application/x-solid-item-id'); + const isMeshDrag = e.dataTransfer.types.includes('application/x-mesh-item-id'); const isCameraDrag = e.dataTransfer.types.includes('application/x-camera-item-id'); + const isSplatEffectorDrag = e.dataTransfer.types.includes('application/x-splat-effector-item-id'); + const isMathSceneDrag = e.dataTransfer.types.includes('application/x-math-scene-item-id'); + const isMotionShapeDrag = e.dataTransfer.types.includes('application/x-motion-shape-item-id'); const isFileDrag = e.dataTransfer.types.includes('Files'); - if (isCompDrag || isMediaPanelDrag || isTextDrag || isSolidDrag || isCameraDrag || isFileDrag) { + if ( + isCompDrag || + isMediaPanelDrag || + isTextDrag || + isSolidDrag || + isMeshDrag || + isCameraDrag || + isSplatEffectorDrag || + isMathSceneDrag || + isMotionShapeDrag || + isFileDrag + ) { const desiredStartTime = getDesiredStartTime(e.clientX); const preview = resolveImmediateDragPreview(e); @@ -1094,8 +1248,11 @@ export function useExternalDrop({ return; } - // Text, solid, camera, and composition items can only go on video tracks - if ((isTextDrag || isSolidDrag || isCameraDrag || isCompDrag) && isAudioTrack) { + // Generated visual items and compositions can only go on video tracks + if ( + (isTextDrag || isSolidDrag || isMeshDrag || isCameraDrag || isSplatEffectorDrag || isMathSceneDrag || isMotionShapeDrag || isCompDrag) && + isAudioTrack + ) { e.dataTransfer.dropEffect = 'none'; setExternalDrag((prev) => prev ? { ...prev, @@ -1174,7 +1331,7 @@ export function useExternalDrop({ */ } }, - [tracks, getDesiredStartTime, buildTrackPreviewState, resolveImmediateDragPreview, updateVideoNewTrackGesture] + [tracks, rejectDropDuringExport, getDesiredStartTime, buildTrackPreviewState, resolveImmediateDragPreview, updateVideoNewTrackGesture] ); // Handle external file drag leave @@ -1199,6 +1356,7 @@ export function useExternalDrop({ // Handle drag over "new track" drop zone const handleNewTrackDragOver = useCallback( (e: React.DragEvent, trackType: 'video' | 'audio') => { + if (rejectDropDuringExport(e)) return; e.preventDefault(); e.stopPropagation(); @@ -1251,12 +1409,13 @@ export function useExternalDrop({ })); } }, - [timelineRef, getDesiredStartTime, resolveImmediateDragPreview, updateVideoNewTrackGesture] + [timelineRef, rejectDropDuringExport, getDesiredStartTime, resolveImmediateDragPreview, updateVideoNewTrackGesture] ); // Handle drop on "new track" zone - creates new track and adds clip const handleNewTrackDrop = useCallback( async (e: React.DragEvent, trackType: 'video' | 'audio') => { + if (rejectDropDuringExport(e)) return; e.preventDefault(); e.stopPropagation(); @@ -1378,6 +1537,30 @@ export function useExternalDrop({ } } + const mathSceneItemId = e.dataTransfer.getData('application/x-math-scene-item-id'); + if (mathSceneItemId) { + const mediaStore = useMediaStore.getState(); + const mathSceneItem = mediaStore.mathSceneItems.find((item) => item.id === mathSceneItemId); + if (mathSceneItem) { + addMathSceneClip(newTrackId, startTime, mathSceneItem.duration, true); + return; + } + } + + const motionShapeItemId = e.dataTransfer.getData('application/x-motion-shape-item-id'); + if (motionShapeItemId) { + const mediaStore = useMediaStore.getState(); + const motionShapeItem = mediaStore.motionShapeItems.find((item) => item.id === motionShapeItemId); + if (motionShapeItem) { + addMotionShapeClip(newTrackId, startTime, { + primitive: motionShapeItem.primitive, + duration: motionShapeItem.duration, + name: motionShapeItem.name, + }); + return; + } + } + // Handle media panel drag if (mediaFileId) { const mediaStore = useMediaStore.getState(); @@ -1445,12 +1628,13 @@ export function useExternalDrop({ } } }, - [scrollX, pixelToTime, addTrack, addCompClip, addClip, addTextClip, addSolidClip, addMeshClip, addCameraClip, addSplatEffectorClip, externalDrag, timelineRef, clearExternalDragState, updateVideoNewTrackGesture] + [scrollX, pixelToTime, addTrack, addCompClip, addClip, addTextClip, addSolidClip, addMeshClip, addCameraClip, addSplatEffectorClip, addMathSceneClip, addMotionShapeClip, externalDrag, timelineRef, clearExternalDragState, updateVideoNewTrackGesture, rejectDropDuringExport] ); // Handle external file drop on track const handleTrackDrop = useCallback( async (e: React.DragEvent, trackId: string) => { + if (rejectDropDuringExport(e)) return; e.preventDefault(); const desiredStartTime = getDesiredStartTime(e.clientX); @@ -1532,6 +1716,30 @@ export function useExternalDrop({ } } + const mathSceneItemId = e.dataTransfer.getData('application/x-math-scene-item-id'); + if (mathSceneItemId) { + const mediaStore = useMediaStore.getState(); + const mathSceneItem = mediaStore.mathSceneItems.find((item) => item.id === mathSceneItemId); + if (mathSceneItem && isVideoTrack) { + addMathSceneClip(trackId, resolveDropStartTime(mathSceneItem.duration), mathSceneItem.duration, true); + return; + } + } + + const motionShapeItemId = e.dataTransfer.getData('application/x-motion-shape-item-id'); + if (motionShapeItemId) { + const mediaStore = useMediaStore.getState(); + const motionShapeItem = mediaStore.motionShapeItems.find((item) => item.id === motionShapeItemId); + if (motionShapeItem && isVideoTrack) { + addMotionShapeClip(trackId, resolveDropStartTime(motionShapeItem.duration), { + primitive: motionShapeItem.primitive, + duration: motionShapeItem.duration, + name: motionShapeItem.name, + }); + return; + } + } + const mediaFileId = e.dataTransfer.getData('application/x-media-file-id'); if (mediaFileId) { const mediaStore = useMediaStore.getState(); @@ -1630,7 +1838,7 @@ export function useExternalDrop({ } } }, - [addCompClip, addClip, addTextClip, addSolidClip, addMeshClip, addCameraClip, addSplatEffectorClip, externalDrag, tracks, getDesiredStartTime, resolveTrackStartTime, clearExternalDragState] + [addCompClip, addClip, addTextClip, addSolidClip, addMeshClip, addCameraClip, addSplatEffectorClip, addMathSceneClip, addMotionShapeClip, externalDrag, tracks, rejectDropDuringExport, getDesiredStartTime, resolveTrackStartTime, clearExternalDragState] ); useEffect(() => { diff --git a/src/components/timeline/hooks/usePlayheadDrag.ts b/src/components/timeline/hooks/usePlayheadDrag.ts index 3abff215..147df830 100644 --- a/src/components/timeline/hooks/usePlayheadDrag.ts +++ b/src/components/timeline/hooks/usePlayheadDrag.ts @@ -16,6 +16,7 @@ interface UsePlayheadDragProps { outPoint: number | null; isRamPreviewing: boolean; isPlaying: boolean; + isExporting: boolean; // Actions setPlayheadPosition: (time: number) => void; @@ -42,6 +43,7 @@ export function usePlayheadDrag({ outPoint, isRamPreviewing, isPlaying, + isExporting, setPlayheadPosition, setDraggingPlayhead, setInPoint, @@ -59,6 +61,7 @@ export function usePlayheadDrag({ if (e.button !== 0) return; e.stopPropagation(); e.preventDefault(); + if (isExporting) return; // Pause playback when user clicks on ruler (like Premiere/DaVinci) if (isPlaying) { @@ -82,6 +85,7 @@ export function usePlayheadDrag({ }, [ isPlaying, + isExporting, pause, isRamPreviewing, cancelRamPreview, @@ -98,6 +102,10 @@ export function usePlayheadDrag({ const handlePlayheadMouseDown = useCallback( (e: React.MouseEvent) => { e.stopPropagation(); + if (isExporting) { + e.preventDefault(); + return; + } // Pause playback when user drags playhead (like Premiere/DaVinci) if (isPlaying) { @@ -113,7 +121,7 @@ export function usePlayheadDrag({ setDraggingPlayhead(true); }, - [isPlaying, pause, isRamPreviewing, cancelRamPreview, setDraggingPlayhead] + [isPlaying, isExporting, pause, isRamPreviewing, cancelRamPreview, setDraggingPlayhead] ); // Handle In/Out marker drag @@ -121,6 +129,7 @@ export function usePlayheadDrag({ (e: React.MouseEvent, type: 'in' | 'out') => { e.stopPropagation(); e.preventDefault(); + if (isExporting) return; if (isPlaying) { pause(); @@ -135,7 +144,7 @@ export function usePlayheadDrag({ originalTime, }); }, - [isPlaying, pause, inPoint, outPoint] + [isPlaying, isExporting, pause, inPoint, outPoint] ); // Handle marker dragging diff --git a/src/components/timeline/utils/externalDragSession.ts b/src/components/timeline/utils/externalDragSession.ts index 6b1d20db..112cc64e 100644 --- a/src/components/timeline/utils/externalDragSession.ts +++ b/src/components/timeline/utils/externalDragSession.ts @@ -1,5 +1,14 @@ export interface ExternalDragPayload { - kind: 'media-file' | 'composition' | 'text' | 'solid' | 'mesh' | 'camera' | 'splat-effector'; + kind: + | 'media-file' + | 'composition' + | 'text' + | 'solid' + | 'mesh' + | 'camera' + | 'splat-effector' + | 'math-scene' + | 'motion-shape'; id: string; duration?: number; hasAudio?: boolean; @@ -7,6 +16,7 @@ export interface ExternalDragPayload { isVideo: boolean; file?: File; meshType?: import('../../../stores/mediaStore/types').MeshPrimitiveType; + primitive?: import('../../../types/motionDesign').ShapePrimitive; } export const EXTERNAL_DRAG_BRIDGE_EVENT = 'masterselects:external-drag-bridge'; diff --git a/src/engine/ParallelDecodeManager.ts b/src/engine/ParallelDecodeManager.ts index b92d7788..47e794ca 100644 --- a/src/engine/ParallelDecodeManager.ts +++ b/src/engine/ParallelDecodeManager.ts @@ -41,6 +41,44 @@ interface Sample { timescale: number; } +export interface SamplePresentationTiming { + cts: number; + timescale: number; +} + +export function getPresentationOffsetSeconds(samples: readonly SamplePresentationTiming[]): number { + let firstPresentationTime = Infinity; + + for (const sample of samples) { + if (!Number.isFinite(sample.cts) || !Number.isFinite(sample.timescale) || sample.timescale <= 0) { + continue; + } + + firstPresentationTime = Math.min(firstPresentationTime, sample.cts / sample.timescale); + } + + return Number.isFinite(firstPresentationTime) ? firstPresentationTime : 0; +} + +export function getNormalizedSampleSourceTime( + sample: SamplePresentationTiming, + presentationOffsetSeconds: number +): number { + if (!Number.isFinite(sample.cts) || !Number.isFinite(sample.timescale) || sample.timescale <= 0) { + return 0; + } + + const sourceTime = (sample.cts / sample.timescale) - presentationOffsetSeconds; + return Number.isFinite(sourceTime) ? Math.max(0, sourceTime) : 0; +} + +export function getNormalizedSampleTimestampMicroseconds( + sample: SamplePresentationTiming, + presentationOffsetSeconds: number +): number { + return Math.round(getNormalizedSampleSourceTime(sample, presentationOffsetSeconds) * 1_000_000); +} + interface MP4VideoTrack { id: number; codec: string; @@ -101,8 +139,8 @@ interface ClipInfo { interface DecodedFrame { frame: VideoFrame; - sourceTime: number; // Time in source video (seconds) - timestamp: number; // Original timestamp from VideoFrame (microseconds) + sourceTime: number; // Normalized time in source video (seconds) + timestamp: number; // Normalized timestamp from VideoFrame (microseconds) } interface ClipDecoder { @@ -113,6 +151,7 @@ interface ClipDecoder { sampleIndex: number; videoTrack: MP4VideoTrack; codecConfig: VideoDecoderConfig; + presentationOffsetSeconds: number; frameBuffer: Map; // timestamp (μs) -> decoded frame sortedTimestamps: number[]; // Sorted list for O(log n) lookup oldestTimestamp: number; // Track bounds for quick rejection @@ -124,6 +163,10 @@ interface ClipDecoder { needsKeyframe: boolean; // True after flush - must start from keyframe } +export interface ParallelDecodeFrameLookupOptions { + toleranceMultiplier?: number; +} + // Buffer settings - tuned for speed like After Effects const BUFFER_AHEAD_FRAMES = 60; // Pre-decode this many frames ahead (1 second at 60fps) const BACKGROUND_DECODE_MARGIN_FRAMES = 180; // Refill earlier during fast export so decode output can catch up @@ -133,6 +176,7 @@ const SEEK_BATCH_MULTIPLIER = 5; // Multiplier for batch size after seeks (5x = const FAR_TARGET_SEEK_THRESHOLD_FRAMES = 30; const UPCOMING_CLIP_PREFETCH_SECONDS = 2.0; const MAX_PREWARM_CLIP_STARTS = 32; +const SLOW_BLOCKING_PREFETCH_WARN_MS = 250; export class ParallelDecodeManager { private clipDecoders: Map = new Map(); @@ -206,6 +250,11 @@ export class ParallelDecodeManager { throw e; } + const presentationOffsetSeconds = getPresentationOffsetSeconds(parseResult.samples); + if (Math.abs(presentationOffsetSeconds) > 0.0005) { + log.info(`"${clipInfo.clipName}": normalizing MP4 presentation offset ${presentationOffsetSeconds.toFixed(3)}s so source starts at 0.000s`); + } + const clipDecoder: ClipDecoder = { clipId: clipInfo.clipId, clipName: clipInfo.clipName, @@ -214,6 +263,7 @@ export class ParallelDecodeManager { sampleIndex: 0, videoTrack: parseResult.videoTrack, codecConfig, + presentationOffsetSeconds, frameBuffer: new Map(), sortedTimestamps: [], oldestTimestamp: Infinity, @@ -424,8 +474,8 @@ export class ParallelDecodeManager { return false; } - if (targetTimestamp < clipDecoder.oldestTimestamp - this.frameTolerance) { - return true; + if (targetTimestamp < clipDecoder.oldestTimestamp - tolerance) { + return false; } if (targetTimestamp > clipDecoder.newestTimestamp + tolerance) { @@ -615,9 +665,14 @@ export class ParallelDecodeManager { const needsDecodingBack = !frameInBuffer && clipDecoder.sampleIndex > targetSampleIndex + 30; // Too far ahead const needsDecoding = needsDecodingAhead || needsDecodingBack; const isBehindTarget = clipDecoder.sampleIndex <= targetSampleIndex; // Are we behind the current target? + const targetAheadFrames = targetSampleIndex - clipDecoder.sampleIndex; + const targetBehindFrames = clipDecoder.sampleIndex - targetSampleIndex; const shouldSeekDirectlyToTarget = !frameInBuffer && - Math.abs(targetSampleIndex - clipDecoder.sampleIndex) > FAR_TARGET_SEEK_THRESHOLD_FRAMES; + ( + targetBehindFrames > FAR_TARGET_SEEK_THRESHOLD_FRAMES || + (clipDecoder.frameBuffer.size === 0 && targetAheadFrames > FAR_TARGET_SEEK_THRESHOLD_FRAMES) + ); if (needsDecoding && !clipDecoder.isDecoding) { log.debug(`"${clipInfo.clipName}": Triggering decode - samples=${clipDecoder.samples.length}, targetIdx=${targetSampleIndex}, currentIdx=${clipDecoder.sampleIndex}, decodeTarget=${decodeTarget}, frameInBuffer=${frameInBuffer}, isBehindTarget=${isBehindTarget}, needsBackSeek=${needsDecodingBack}, directSeek=${shouldSeekDirectlyToTarget}`); @@ -634,7 +689,28 @@ export class ParallelDecodeManager { targetSampleIndex ); if (prefetchTarget.shouldBlock) { + const directDecodeStart = performance.now(); await decodePromise; + const directDecodeMs = performance.now() - directDecodeStart; + if (directDecodeMs >= SLOW_BLOCKING_PREFETCH_WARN_MS) { + log.warn(`${clipDecoder.clipName}: slow direct prefetch decode`, { + timelineTime: Number(prefetchTarget.timelineTime.toFixed(3)), + sourceTime: Number(sourceTime.toFixed(3)), + targetSampleIndex, + decodeTarget, + sampleIndex: clipDecoder.sampleIndex, + directDecodeMs: Number(directDecodeMs.toFixed(1)), + directSeek: shouldSeekDirectlyToTarget, + decodeQueueSize: clipDecoder.decoder.decodeQueueSize, + bufferSize: clipDecoder.frameBuffer.size, + bufferedStart: Number.isFinite(clipDecoder.oldestTimestamp) + ? Number((clipDecoder.oldestTimestamp / 1_000_000).toFixed(3)) + : null, + bufferedEnd: Number.isFinite(clipDecoder.newestTimestamp) + ? Number((clipDecoder.newestTimestamp / 1_000_000).toFixed(3)) + : null, + }); + } } else { void decodePromise; } @@ -660,8 +736,11 @@ export class ParallelDecodeManager { const sourceTime = this.timelineToSourceTime(clipInfo, timelineTime); const targetTimestamp = sourceTime * 1_000_000; const targetSampleIndex = this.findSampleIndexForTime(clipDecoder, sourceTime); + const blockingStart = performance.now(); + let attemptsUsed = 0; for (let attempt = 0; attempt < 10; attempt++) { + attemptsUsed = attempt + 1; // Wait for pending decode to complete if (clipDecoder.pendingDecode) { await clipDecoder.pendingDecode; @@ -700,6 +779,26 @@ export class ParallelDecodeManager { // Final check - strict export should fail instead of using a nearby frame. const finalTolerance = this.frameTolerance * 3; // 3x tolerance for final check const finalCheck = this.hasUsableBufferedFrame(clipDecoder, targetTimestamp, finalTolerance); + const blockingMs = performance.now() - blockingStart; + if (blockingMs >= SLOW_BLOCKING_PREFETCH_WARN_MS) { + log.warn(`${clipDecoder.clipName}: slow blocking prefetch`, { + timelineTime: Number(timelineTime.toFixed(3)), + sourceTime: Number(sourceTime.toFixed(3)), + targetSampleIndex, + sampleIndex: clipDecoder.sampleIndex, + attempts: attemptsUsed, + blockingMs: Number(blockingMs.toFixed(1)), + decodeQueueSize: clipDecoder.decoder.decodeQueueSize, + bufferSize: clipDecoder.frameBuffer.size, + bufferedStart: Number.isFinite(clipDecoder.oldestTimestamp) + ? Number((clipDecoder.oldestTimestamp / 1_000_000).toFixed(3)) + : null, + bufferedEnd: Number.isFinite(clipDecoder.newestTimestamp) + ? Number((clipDecoder.newestTimestamp / 1_000_000).toFixed(3)) + : null, + finalCheck, + }); + } if (!finalCheck) { const availableFrames = Array.from(clipDecoder.frameBuffer.values()) .map(f => (f.timestamp / 1_000_000).toFixed(3)) @@ -820,18 +919,28 @@ export class ParallelDecodeManager { // Otherwise if we're past the target, framesToDecode will be negative and we'll return early if (needsSeek) { // Need to seek - find nearest keyframe before the ACTUAL target we need - const seekTarget = seekTargetSampleIndex ?? targetSampleIndex; + const seekTarget = Math.max( + 0, + Math.min(seekTargetSampleIndex ?? targetSampleIndex, clipDecoder.samples.length - 1) + ); // Find keyframe candidates by CTS (display time), not decode order. // Due to B-frame reordering, a keyframe earlier in decode order // can have a LATER CTS than the target, causing wrong frames to be decoded. - const targetCTS = clipDecoder.samples[seekTarget].cts; + const targetSourceTime = getNormalizedSampleSourceTime( + clipDecoder.samples[seekTarget], + clipDecoder.presentationOffsetSeconds + ); const keyframeCandidates: number[] = []; for (let i = 0; i < clipDecoder.samples.length; i++) { if (clipDecoder.samples[i].is_sync) { - if (clipDecoder.samples[i].cts <= targetCTS) { + const sampleSourceTime = getNormalizedSampleSourceTime( + clipDecoder.samples[i], + clipDecoder.presentationOffsetSeconds + ); + if (sampleSourceTime <= targetSourceTime) { keyframeCandidates.push(i); } else { - break; // Keyframe CTS values increase monotonically + break; // Keyframe presentation times increase monotonically } } } @@ -846,14 +955,17 @@ export class ParallelDecodeManager { for (let k = keyframeCandidates.length - 1; k >= keyframeCandidates.length - maxAttempts; k--) { const candidateIndex = keyframeCandidates[k]; const candidateSample = clipDecoder.samples[candidateIndex]; - const candidateCTS = (candidateSample.cts / clipDecoder.videoTrack.timescale).toFixed(3); + const candidateSourceTime = getNormalizedSampleSourceTime( + candidateSample, + clipDecoder.presentationOffsetSeconds + ).toFixed(3); clipDecoder.decoder.reset(); clipDecoder.decoder.configure(exportConfig); const chunk = new EncodedVideoChunk({ type: 'key', - timestamp: (candidateSample.cts * 1_000_000) / candidateSample.timescale, + timestamp: getNormalizedSampleTimestampMicroseconds(candidateSample, clipDecoder.presentationOffsetSeconds), duration: (candidateSample.duration * 1_000_000) / candidateSample.timescale, data: candidateSample.data, }); @@ -861,10 +973,10 @@ export class ParallelDecodeManager { try { clipDecoder.decoder.decode(chunk); clipDecoder.sampleIndex = candidateIndex + 1; // Already decoded this one - log.debug(`${clipDecoder.clipName}: Seek keyframe accepted at sample ${candidateIndex} (CTS=${candidateCTS}s, targetCTS=${(targetCTS / clipDecoder.videoTrack.timescale).toFixed(3)}s, bufferTarget=${targetSampleIndex})`); + log.debug(`${clipDecoder.clipName}: Seek keyframe accepted at sample ${candidateIndex} (source=${candidateSourceTime}s, targetSource=${targetSourceTime.toFixed(3)}s, bufferTarget=${targetSampleIndex})`); break; } catch (e) { - log.debug(`${clipDecoder.clipName}: Seek keyframe REJECTED at sample ${candidateIndex} (CTS=${candidateCTS}s) - not a real IDR, trying earlier`); + log.debug(`${clipDecoder.clipName}: Seek keyframe REJECTED at sample ${candidateIndex} (source=${candidateSourceTime}s) - not a real IDR, trying earlier`); if (k === keyframeCandidates.length - maxAttempts) { // Last attempt failed - reset and start from first sample clipDecoder.decoder.reset(); @@ -947,7 +1059,7 @@ export class ParallelDecodeManager { const chunk = new EncodedVideoChunk({ type: sample.is_sync ? 'key' : 'delta', - timestamp: (sample.cts * 1_000_000) / sample.timescale, + timestamp: getNormalizedSampleTimestampMicroseconds(sample, clipDecoder.presentationOffsetSeconds), duration: (sample.duration * 1_000_000) / sample.timescale, data: sample.data, }); @@ -1009,7 +1121,6 @@ export class ParallelDecodeManager { * due to B-frame reordering. Binary search doesn't work here. */ private findSampleIndexForTime(clipDecoder: ClipDecoder, sourceTime: number): number { - const targetTime = sourceTime * clipDecoder.videoTrack.timescale; const samples = clipDecoder.samples; if (samples.length === 0) return 0; @@ -1019,7 +1130,11 @@ export class ParallelDecodeManager { let closestDiff = Infinity; for (let i = 0; i < samples.length; i++) { - const diff = Math.abs(samples[i].cts - targetTime); + const sampleSourceTime = getNormalizedSampleSourceTime( + samples[i], + clipDecoder.presentationOffsetSeconds + ); + const diff = Math.abs(sampleSourceTime - sourceTime); if (diff < closestDiff) { closestDiff = diff; targetIndex = i; @@ -1034,11 +1149,16 @@ export class ParallelDecodeManager { * Returns null if frame isn't ready (shouldn't happen if prefetch was called) * Optimized: O(log n) binary search instead of O(n) linear scan */ - getFrameForClip(clipId: string, timelineTime: number): VideoFrame | null { + getFrameForClip( + clipId: string, + timelineTime: number, + options: ParallelDecodeFrameLookupOptions = {} + ): VideoFrame | null { const clipDecoder = this.clipDecoders.get(clipId); if (!clipDecoder) return null; const clipInfo = clipDecoder.clipInfo; + const lookupTolerance = this.frameTolerance * Math.max(1, options.toleranceMultiplier ?? 1); // Check if time is within clip range (handles nested clips too) if (!this.isTimeInClipRange(clipInfo, timelineTime)) { @@ -1056,7 +1176,7 @@ export class ParallelDecodeManager { return null; } - const useLastFrame = targetTimestamp > clipDecoder.newestTimestamp + this.frameTolerance; + const useLastFrame = targetTimestamp > clipDecoder.newestTimestamp + lookupTolerance; if (useLastFrame) { const lastTimestamp = clipDecoder.sortedTimestamps[clipDecoder.sortedTimestamps.length - 1]; const lastFrame = clipDecoder.frameBuffer.get(lastTimestamp); @@ -1064,7 +1184,7 @@ export class ParallelDecodeManager { return null; } - const useFirstFrame = targetTimestamp < clipDecoder.oldestTimestamp - this.frameTolerance; + const useFirstFrame = targetTimestamp < clipDecoder.oldestTimestamp - lookupTolerance; if (useFirstFrame) { const firstTimestamp = clipDecoder.sortedTimestamps[0]; const firstFrame = clipDecoder.frameBuffer.get(firstTimestamp); @@ -1102,8 +1222,8 @@ export class ParallelDecodeManager { const decodedFrame = clipDecoder.frameBuffer.get(frameTimestamp); if (decodedFrame) { - if (frameDiff >= this.frameTolerance) { - log.warn(`${clipDecoder.clipName}: nearest frame at ${(frameTimestamp/1_000_000).toFixed(3)}s is outside tolerance for target ${(targetTimestamp/1_000_000).toFixed(3)}s (diff=${(frameDiff/1000).toFixed(1)}ms, tolerance=${(this.frameTolerance/1000).toFixed(1)}ms)`); + if (frameDiff >= lookupTolerance) { + log.warn(`${clipDecoder.clipName}: nearest frame at ${(frameTimestamp/1_000_000).toFixed(3)}s is outside tolerance for target ${(targetTimestamp/1_000_000).toFixed(3)}s (diff=${(frameDiff/1000).toFixed(1)}ms, tolerance=${(lookupTolerance/1000).toFixed(1)}ms)`); return null; } return decodedFrame.frame; diff --git a/src/engine/WebCodecsExportMode.ts b/src/engine/WebCodecsExportMode.ts index 3b3e4ccb..4c3ee916 100644 --- a/src/engine/WebCodecsExportMode.ts +++ b/src/engine/WebCodecsExportMode.ts @@ -28,6 +28,7 @@ export class WebCodecsExportMode { private static readonly INITIAL_LOOKAHEAD_SAMPLES = 90; private static readonly DECODE_LOOKAHEAD_SAMPLES = 60; private static readonly KEEP_FRAMES_BEHIND = 24; + private static readonly WARM_AHEAD_THRESHOLD_SAMPLES = 30; private player: ExportModePlayer; @@ -37,6 +38,8 @@ export class WebCodecsExportMode { private exportFramesCts: number[] = []; // Sorted CTS values for index-based lookup private exportCurrentIndex = 0; private decodeCursorIndex = 0; + private presentationOffsetUs = 0; + private pendingWarmBuffer: Promise | null = null; constructor(player: ExportModePlayer) { this.player = player; @@ -57,10 +60,26 @@ export class WebCodecsExportMode { return (1_000_000 / Math.max(this.player.getFrameRate(), 1)) * multiplier; } + private updatePresentationOffset(samples: readonly Sample[]): void { + let firstPresentationUs = Infinity; + for (const sample of samples) { + if (!Number.isFinite(sample.cts) || !Number.isFinite(sample.timescale) || sample.timescale <= 0) { + continue; + } + firstPresentationUs = Math.min(firstPresentationUs, (sample.cts * 1_000_000) / sample.timescale); + } + + this.presentationOffsetUs = Number.isFinite(firstPresentationUs) ? firstPresentationUs : 0; + } + private getSampleTimestampUs(sample: Sample): number { return (sample.cts * 1_000_000) / sample.timescale; } + private getNormalizedSampleTimestampUs(sample: Sample): number { + return Math.max(0, this.getSampleTimestampUs(sample) - this.presentationOffsetUs); + } + private findClosestSampleIndex(targetTimeSeconds: number): number { const timescale = this.player.getVideoTrackTimescale(); const samples = this.player.getSamples(); @@ -68,12 +87,12 @@ export class WebCodecsExportMode { return 0; } - const targetTimeInTimescale = targetTimeSeconds * timescale; + const targetUs = targetTimeSeconds * 1_000_000; let targetSampleIndex = 0; let closestDiff = Infinity; for (let i = 0; i < samples.length; i++) { - const diff = Math.abs(samples[i].cts - targetTimeInTimescale); + const diff = Math.abs(this.getNormalizedSampleTimestampUs(samples[i]) - targetUs); if (diff < closestDiff) { closestDiff = diff; targetSampleIndex = i; @@ -181,7 +200,7 @@ export class WebCodecsExportMode { const sample = samples[i]; const chunk = new EncodedVideoChunk({ type: sample.is_sync ? 'key' : 'delta', - timestamp: this.getSampleTimestampUs(sample), + timestamp: this.getNormalizedSampleTimestampUs(sample), duration: (sample.duration * 1_000_000) / sample.timescale, data: sample.data, }); @@ -234,10 +253,24 @@ export class WebCodecsExportMode { await this.decodeSampleWindow( this.decodeCursorIndex, endIndexExclusive, - this.getSampleTimestampUs(targetSample) + this.getNormalizedSampleTimestampUs(targetSample) ); } + private scheduleWarmBufferAroundSample(targetSampleIndex: number): void { + if (this.pendingWarmBuffer) { + return; + } + + this.pendingWarmBuffer = this.warmBufferAroundSample(targetSampleIndex) + .catch((error) => { + log.warn('Background export decode warmup failed', error); + }) + .finally(() => { + this.pendingWarmBuffer = null; + }); + } + private async restartFromKeyframe(targetSampleIndex: number): Promise { const samples = this.player.getSamples(); const targetSample = samples[targetSampleIndex]; @@ -266,7 +299,7 @@ export class WebCodecsExportMode { await this.decodeSampleWindow( keyframeIndex, endIndexExclusive, - this.getSampleTimestampUs(targetSample) + this.getNormalizedSampleTimestampUs(targetSample) ); } @@ -343,11 +376,12 @@ export class WebCodecsExportMode { this.isActive = true; const allSamples = this.player.getSamples(); - const targetTimeInTimescale = startTimeSeconds * timescale; + this.updatePresentationOffset(allSamples); + const targetUs = startTimeSeconds * 1_000_000; let startSampleIndex = 0; let closestDiff = Infinity; for (let i = 0; i < allSamples.length; i++) { - const diff = Math.abs(allSamples[i].cts - targetTimeInTimescale); + const diff = Math.abs(this.getNormalizedSampleTimestampUs(allSamples[i]) - targetUs); if (diff < closestDiff) { closestDiff = diff; startSampleIndex = i; @@ -376,12 +410,12 @@ export class WebCodecsExportMode { await this.decodeSampleWindow( keyframeIndex, decodeEnd, - this.getSampleTimestampUs(startSample) + this.getNormalizedSampleTimestampUs(startSample) ); endDecode(); const startFrameIndex = this.findBufferedFrameIndex( - this.getSampleTimestampUs(startSample), + this.getNormalizedSampleTimestampUs(startSample), this.getFrameToleranceUs(3) ); @@ -390,7 +424,7 @@ export class WebCodecsExportMode { this.player.setCurrentFrame(this.exportFrameBuffer.get(startCts) || null); this.exportCurrentIndex = startFrameIndex; } else if (this.exportFramesCts.length > 0) { - const fallbackIndex = this.findClosestFrameIndex(this.getSampleTimestampUs(startSample)); + const fallbackIndex = this.findClosestFrameIndex(this.getNormalizedSampleTimestampUs(startSample)); const fallbackCts = this.exportFramesCts[Math.max(0, fallbackIndex)]; this.player.setCurrentFrame(this.exportFrameBuffer.get(fallbackCts) || null); this.exportCurrentIndex = Math.max(0, fallbackIndex); @@ -511,11 +545,14 @@ export class WebCodecsExportMode { this.exportCurrentIndex = bestIndex; const framesRemaining = this.exportFramesCts.length - bestIndex; - if (framesRemaining < 30 && this.decodeCursorIndex < this.player.getSamples().length) { + if ( + framesRemaining < WebCodecsExportMode.WARM_AHEAD_THRESHOLD_SAMPLES && + this.decodeCursorIndex < this.player.getSamples().length + ) { log.debug( `Decoding ahead: ${framesRemaining} frames remaining, sampleIndex=${this.decodeCursorIndex}/${this.player.getSamples().length}` ); - await this.warmBufferAroundSample(targetSampleIndex); + this.scheduleWarmBufferAroundSample(targetSampleIndex); } this.cleanupOldFrames(bestIndex - WebCodecsExportMode.KEEP_FRAMES_BEHIND); @@ -534,6 +571,18 @@ export class WebCodecsExportMode { `Frame not in buffer: target=${targetCts.toFixed(0)}, range=[${minCtsInBuffer.toFixed(0)}-${maxCtsInBuffer.toFixed(0)}], bufferSize=${this.exportFramesCts.length}` ); + if (this.pendingWarmBuffer) { + await this.pendingWarmBuffer; + bestIndex = this.findBufferedFrameIndex(targetCts, this.getFrameToleranceUs(3)); + if (bestIndex >= 0 && bestIndex < this.exportFramesCts.length) { + const cts = this.exportFramesCts[bestIndex]; + this.player.setCurrentFrame(this.exportFrameBuffer.get(cts) || null); + this.exportCurrentIndex = bestIndex; + this.cleanupOldFrames(bestIndex - WebCodecsExportMode.KEEP_FRAMES_BEHIND); + return; + } + } + if ( targetCts < minCtsInBuffer || targetSampleIndex < this.decodeCursorIndex - WebCodecsExportMode.KEEP_FRAMES_BEHIND @@ -628,6 +677,7 @@ export class WebCodecsExportMode { this.exportFramesCts = []; this.exportCurrentIndex = 0; this.decodeCursorIndex = 0; + this.pendingWarmBuffer = null; log.info('Export mode ended'); } @@ -642,6 +692,7 @@ export class WebCodecsExportMode { this.exportFramesCts = []; this.exportCurrentIndex = 0; this.decodeCursorIndex = 0; + this.pendingWarmBuffer = null; this.isActive = false; } } diff --git a/src/engine/audio/AudioFileEncoder.ts b/src/engine/audio/AudioFileEncoder.ts new file mode 100644 index 00000000..c14718cd --- /dev/null +++ b/src/engine/audio/AudioFileEncoder.ts @@ -0,0 +1,104 @@ +export type AudioOnlyExportFormat = 'wav' | 'browser'; + +export type WavBitDepth = 16; + +export interface AudioBufferLike { + sampleRate: number; + numberOfChannels: number; + length: number; + getChannelData(channel: number): Float32Array; +} + +export interface WavEncodeOptions { + bitDepth?: WavBitDepth; +} + +const WAV_HEADER_BYTES = 44; +const WAV_FORMAT_PCM = 1; + +function writeAscii(view: DataView, offset: number, value: string): void { + for (let i = 0; i < value.length; i++) { + view.setUint8(offset + i, value.charCodeAt(i)); + } +} + +function floatToPcm16(sample: number): number { + const clamped = Math.max(-1, Math.min(1, Number.isFinite(sample) ? sample : 0)); + return clamped < 0 + ? Math.round(clamped * 0x8000) + : Math.round(clamped * 0x7fff); +} + +export function estimateWavByteSize(buffer: AudioBufferLike, options?: WavEncodeOptions): number { + const bitDepth = options?.bitDepth ?? 16; + const bytesPerSample = bitDepth / 8; + return WAV_HEADER_BYTES + buffer.length * buffer.numberOfChannels * bytesPerSample; +} + +export function encodeAudioBufferToWavBytes(buffer: AudioBufferLike, options?: WavEncodeOptions): Uint8Array { + const bitDepth = options?.bitDepth ?? 16; + if (bitDepth !== 16) { + throw new Error(`Unsupported WAV bit depth: ${bitDepth}`); + } + + const channelCount = Math.floor(buffer.numberOfChannels); + if (!Number.isFinite(buffer.sampleRate) || buffer.sampleRate <= 0) { + throw new Error('Cannot encode WAV with an invalid sample rate'); + } + if (!Number.isFinite(channelCount) || channelCount < 1) { + throw new Error('Cannot encode WAV without audio channels'); + } + if (!Number.isFinite(buffer.length) || buffer.length < 0) { + throw new Error('Cannot encode WAV with an invalid sample count'); + } + + const sampleCount = Math.floor(buffer.length); + const bytesPerSample = bitDepth / 8; + const blockAlign = channelCount * bytesPerSample; + const dataSize = sampleCount * blockAlign; + const fileSize = WAV_HEADER_BYTES + dataSize; + const riffChunkSize = fileSize - 8; + + if (riffChunkSize > 0xffffffff) { + throw new Error('WAV export is limited to 4 GB RIFF files'); + } + + const bytes = new Uint8Array(fileSize); + const view = new DataView(bytes.buffer); + + writeAscii(view, 0, 'RIFF'); + view.setUint32(4, riffChunkSize, true); + writeAscii(view, 8, 'WAVE'); + + writeAscii(view, 12, 'fmt '); + view.setUint32(16, 16, true); + view.setUint16(20, WAV_FORMAT_PCM, true); + view.setUint16(22, channelCount, true); + view.setUint32(24, Math.round(buffer.sampleRate), true); + view.setUint32(28, Math.round(buffer.sampleRate) * blockAlign, true); + view.setUint16(32, blockAlign, true); + view.setUint16(34, bitDepth, true); + + writeAscii(view, 36, 'data'); + view.setUint32(40, dataSize, true); + + const channelData = Array.from({ length: channelCount }, (_, channel) => buffer.getChannelData(channel)); + let offset = WAV_HEADER_BYTES; + + for (let sampleIndex = 0; sampleIndex < sampleCount; sampleIndex++) { + for (let channel = 0; channel < channelCount; channel++) { + const pcm = floatToPcm16(channelData[channel]?.[sampleIndex] ?? 0); + view.setInt16(offset, pcm, true); + offset += bytesPerSample; + } + } + + return bytes; +} + +export function encodeAudioBufferToWavBlob(buffer: AudioBufferLike, options?: WavEncodeOptions): Blob { + const bytes = encodeAudioBufferToWavBytes(buffer, options); + const arrayBuffer = new ArrayBuffer(bytes.byteLength); + new Uint8Array(arrayBuffer).set(bytes); + return new Blob([arrayBuffer], { type: 'audio/wav' }); +} diff --git a/src/engine/audio/index.ts b/src/engine/audio/index.ts index 64c0c1b4..0acb9e85 100644 --- a/src/engine/audio/index.ts +++ b/src/engine/audio/index.ts @@ -13,6 +13,8 @@ export { AudioExtractor, audioExtractor } from './AudioExtractor'; export { AudioEncoderWrapper, getRecommendedAudioBitrate, AUDIO_CODEC_INFO } from './AudioEncoder'; export type { AudioEncoderSettings, EncodedAudioResult, AudioEncoderProgressCallback, AudioCodec } from './AudioEncoder'; +export { encodeAudioBufferToWavBlob, encodeAudioBufferToWavBytes, estimateWavByteSize } from './AudioFileEncoder'; +export type { AudioBufferLike, AudioOnlyExportFormat, WavBitDepth, WavEncodeOptions } from './AudioFileEncoder'; export { AudioMixer, audioMixer } from './AudioMixer'; export type { AudioTrackData, MixerSettings, MixProgress, MixProgressCallback } from './AudioMixer'; export { TimeStretchProcessor, timeStretchProcessor } from './TimeStretchProcessor'; diff --git a/src/engine/export/ClipPreparation.ts b/src/engine/export/ClipPreparation.ts index 951f8dcb..cab0f47c 100644 --- a/src/engine/export/ClipPreparation.ts +++ b/src/engine/export/ClipPreparation.ts @@ -497,45 +497,7 @@ async function initializeFastMode( ): Promise { const { WebCodecsPlayer } = await import('../WebCodecsPlayer'); const fileDataCache: ClipFileDataCache = new Map(); - - // Separate composition clips from regular video clips - const regularVideoClips: TimelineClip[] = []; - const nestedVideoClips: Array<{ clip: TimelineClip; parentClip: TimelineClip }> = []; - - for (const clip of videoClips) { - if (clip.source?.type !== 'video') continue; - - if (clip.isComposition) { - clipStates.set(clip.id, { - clipId: clip.id, - webCodecsPlayer: null, - lastSampleIndex: 0, - isSequential: false, - }); - log.debug(`Clip ${clip.name}: Composition with nested clips`); - - // Collect nested video clips - if (clip.nestedClips) { - for (const nestedClip of clip.nestedClips) { - if (nestedClip.source?.type === 'video' && nestedClip.source.videoElement) { - nestedVideoClips.push({ clip: nestedClip, parentClip: clip }); - } - } - } - } else { - regularVideoClips.push(clip); - } - } - - // Use parallel decoding if we have 2+ total video clips - const totalVideoClips = regularVideoClips.length + nestedVideoClips.length; - if (totalVideoClips >= 2) { - log.info(`Using PARALLEL decoding for ${regularVideoClips.length} regular + ${nestedVideoClips.length} nested = ${totalVideoClips} video clips`); - return initializeParallelDecoding(regularVideoClips, mediaFiles, startTime, endTime, nestedVideoClips, clipStates, fps, endPrepare, fileDataCache); - } - - // Single clip: use sequential approach - for (const clip of regularVideoClips) { + const initializeSequentialClip = async (clip: TimelineClip): Promise => { const mediaFileId = getClipMediaFileId(clip); const mediaFile = mediaFileId ? mediaFiles.find(f => f.id === mediaFileId) : null; @@ -591,6 +553,64 @@ async function initializeFastMode( }); log.debug(`Clip ${clip.name}: FAST mode enabled (${exportPlayer.width}x${exportPlayer.height})`); + }; + + // Separate composition clips from regular video clips + const regularVideoClips: TimelineClip[] = []; + const nestedVideoClips: Array<{ clip: TimelineClip; parentClip: TimelineClip }> = []; + + for (const clip of videoClips) { + if (clip.source?.type !== 'video') continue; + + if (clip.isComposition) { + clipStates.set(clip.id, { + clipId: clip.id, + webCodecsPlayer: null, + lastSampleIndex: 0, + isSequential: false, + }); + log.debug(`Clip ${clip.name}: Composition with nested clips`); + + // Collect nested video clips + if (clip.nestedClips) { + for (const nestedClip of clip.nestedClips) { + if (nestedClip.source?.type === 'video' && nestedClip.source.videoElement) { + nestedVideoClips.push({ clip: nestedClip, parentClip: clip }); + } + } + } + } else { + regularVideoClips.push(clip); + } + } + + // Use parallel decoding if we have 2+ total video clips + const totalVideoClips = regularVideoClips.length + nestedVideoClips.length; + if (totalVideoClips >= 2) { + if (nestedVideoClips.length === 0) { + log.info(`Using multi-clip sequential WebCodecs export for ${regularVideoClips.length} regular video clips`); + for (const clip of regularVideoClips) { + await initializeSequentialClip(clip); + } + + log.info(`All ${regularVideoClips.length} clips using FAST WebCodecs sequential decoding`); + endPrepare(); + + return { + clipStates, + parallelDecoder: null, + useParallelDecode: false, + exportMode: 'fast', + }; + } + + log.info(`Using PARALLEL decoding for ${regularVideoClips.length} regular + ${nestedVideoClips.length} nested = ${totalVideoClips} video clips`); + return initializeParallelDecoding(regularVideoClips, mediaFiles, startTime, endTime, nestedVideoClips, clipStates, fps, endPrepare, fileDataCache); + } + + // Single clip: use sequential approach + for (const clip of regularVideoClips) { + await initializeSequentialClip(clip); } log.info(`All ${videoClips.length} clips using FAST WebCodecs sequential decoding`); @@ -617,6 +637,7 @@ async function initializeParallelDecoding( ): Promise { const parallelDecoder = new ParallelDecodeManager(); + try { // Load all clip file data in parallel const endLoadAll = log.time('loadAllClipFileData'); const loadPromises: Promise[] = clips.map(async (clip) => { @@ -773,6 +794,10 @@ async function initializeParallelDecoding( useParallelDecode: true, exportMode: 'fast', }; + } catch (e) { + parallelDecoder.cleanup(); + throw e; + } } /** diff --git a/src/engine/export/ExportLayerBuilder.ts b/src/engine/export/ExportLayerBuilder.ts index f3247dd1..a59d2bfb 100644 --- a/src/engine/export/ExportLayerBuilder.ts +++ b/src/engine/export/ExportLayerBuilder.ts @@ -30,6 +30,7 @@ import { textRenderer } from '../../services/textRenderer'; let cachedVideoTracks: TimelineTrack[] | null = null; let cachedAnyVideoSolo = false; const MAX_EXPORT_NESTING_DEPTH = 4; +const FAST_EXPORT_FRAME_LOOKUP_TOLERANCE_MULTIPLIER = 3; export function initializeLayerBuilder(tracks: TimelineTrack[]): void { cachedVideoTracks = tracks.filter(t => t.type === 'video'); @@ -449,7 +450,9 @@ function buildVideoLayer( throw new Error(`FAST export failed: parallel decoder is not initialized for clip "${clip.name}".`); } if (parallelDecoder.hasClip(clip.id)) { - const videoFrame = parallelDecoder.getFrameForClip(clip.id, time); + const videoFrame = parallelDecoder.getFrameForClip(clip.id, time, { + toleranceMultiplier: FAST_EXPORT_FRAME_LOOKUP_TOLERANCE_MULTIPLIER, + }); if (videoFrame) { return { ...baseLayerProps, @@ -607,7 +610,9 @@ function buildNestedLayerForExport( throw new Error(`FAST export failed: parallel decoder is not initialized for nested clip "${nestedClip.name}".`); } if (parallelDecoder.hasClip(nestedClip.id)) { - const videoFrame = parallelDecoder.getFrameForClip(nestedClip.id, mainTimelineTime); + const videoFrame = parallelDecoder.getFrameForClip(nestedClip.id, mainTimelineTime, { + toleranceMultiplier: FAST_EXPORT_FRAME_LOOKUP_TOLERANCE_MULTIPLIER, + }); if (videoFrame) { return { ...baseLayer, diff --git a/src/engine/export/FrameExporter.ts b/src/engine/export/FrameExporter.ts index 53e463ad..0aaa7c8c 100644 --- a/src/engine/export/FrameExporter.ts +++ b/src/engine/export/FrameExporter.ts @@ -33,6 +33,8 @@ import { } from './codecHelpers'; export class FrameExporter { + private static readonly PREVIEW_FRAME_INTERVAL_MS = 0; + private settings: FullExportSettings; private encoder: VideoEncoderWrapper | null = null; private audioPipeline: AudioExportPipeline | null = null; @@ -42,6 +44,7 @@ export class FrameExporter { private exportMode: ExportMode; private parallelDecoder: ParallelDecodeManager | null = null; private useParallelDecode = false; + private lastPreviewFramePublishMs = Number.NEGATIVE_INFINITY; constructor(settings: FullExportSettings) { this.settings = settings; @@ -315,6 +318,7 @@ export class FrameExporter { log.error(`Failed to create VideoFrame at frame ${frame}`); continue; } + this.publishExportPreviewFrame(videoFrame, time); const encodeStart = performance.now(); await this.encoder.encodeVideoFrame(videoFrame, frame, keyframeInterval); encodeMs = performance.now() - encodeStart; @@ -331,6 +335,7 @@ export class FrameExporter { log.error(`Failed to read pixels at frame ${frame}`); continue; } + this.publishExportPreviewPixels(pixels, width, height, time); const encodeStart = performance.now(); await this.encoder.encodeFrame(pixels, frame, keyframeInterval); encodeMs = performance.now() - encodeStart; @@ -448,6 +453,67 @@ export class FrameExporter { cleanupExportMode(this.clipStates, this.parallelDecoder); } + private shouldPublishExportPreviewFrame(): boolean { + const now = performance.now(); + if (now - this.lastPreviewFramePublishMs < FrameExporter.PREVIEW_FRAME_INTERVAL_MS) { + return false; + } + this.lastPreviewFramePublishMs = now; + return true; + } + + private publishBitmapWhenStillExporting( + bitmapPromise: Promise, + currentTime: number, + cleanup?: () => void, + ): void { + void bitmapPromise + .then((bitmap) => { + const timeline = useTimelineStore.getState(); + if (!timeline.isExporting) { + bitmap.close(); + return; + } + timeline.setExportPreviewFrame(bitmap, currentTime); + }) + .catch((error) => { + log.warn('Could not publish export preview frame', error); + }) + .finally(() => { + cleanup?.(); + }); + } + + private publishExportPreviewFrame(videoFrame: VideoFrame, currentTime: number): void { + if (!this.shouldPublishExportPreviewFrame()) return; + + let previewFrame: VideoFrame; + try { + previewFrame = videoFrame.clone(); + } catch (error) { + log.warn('Could not clone export frame for preview', error); + return; + } + + this.publishBitmapWhenStillExporting( + createImageBitmap(previewFrame), + currentTime, + () => previewFrame.close(), + ); + } + + private publishExportPreviewPixels( + pixels: Uint8ClampedArray, + width: number, + height: number, + currentTime: number, + ): void { + if (!this.shouldPublishExportPreviewFrame()) return; + + const imageData = new ImageData(new Uint8ClampedArray(pixels), width, height); + this.publishBitmapWhenStillExporting(createImageBitmap(imageData), currentTime); + } + private cleanup(originalDimensions: { width: number; height: number }): void { cleanupExportMode(this.clipStates, this.parallelDecoder); cleanupLayerBuilder(); diff --git a/src/engine/export/VideoEncoderWrapper.ts b/src/engine/export/VideoEncoderWrapper.ts index 9deaf330..e7dbf3ae 100644 --- a/src/engine/export/VideoEncoderWrapper.ts +++ b/src/engine/export/VideoEncoderWrapper.ts @@ -53,39 +53,48 @@ export class VideoEncoderWrapper { bitrate: this.settings.bitrate, framerate: this.settings.fps, }; - try { - const requestedSupport = await VideoEncoder.isConfigSupported({ + const buildEncoderConfigCandidates = (bitrateMode: VideoEncoderBitrateMode): VideoEncoderConfig[] => [ + { ...supportCheckConfig, - bitrateMode: requestedBitrateMode, - }); - - this.effectiveBitrateMode = requestedBitrateMode; - - if (!requestedSupport.supported) { - if (requestedBitrateMode !== 'constant') { - log.error(`Codec not supported: ${codecString}`); - return false; - } - - log.warn('Constant bitrate support check failed for this encoder config, will try configure() and fall back to variable if needed'); + latencyMode: 'realtime', + hardwareAcceleration: 'prefer-hardware', + bitrateMode, + contentHint: 'motion', + }, + { + ...supportCheckConfig, + latencyMode: 'realtime', + hardwareAcceleration: 'no-preference', + bitrateMode, + contentHint: 'motion', + }, + { + ...supportCheckConfig, + latencyMode: 'quality', + hardwareAcceleration: 'prefer-hardware', + bitrateMode, + contentHint: 'motion', + }, + { + ...supportCheckConfig, + latencyMode: 'quality', + hardwareAcceleration: 'no-preference', + bitrateMode, + contentHint: 'motion', + }, + ]; + const bitrateModesToTry: VideoEncoderBitrateMode[] = requestedBitrateMode === 'constant' + ? ['constant', 'variable'] + : ['variable']; + const supportedEncoderConfigs: VideoEncoderConfig[] = []; - const fallbackSupport = await VideoEncoder.isConfigSupported({ - ...supportCheckConfig, - bitrateMode: 'variable', - }); - if (!fallbackSupport.supported) { - this.effectiveBitrateMode = 'variable'; - log.error(`Codec not supported: ${codecString}`); - return false; - } - } else if (requestedBitrateMode === 'constant') { - const fallbackSupport = await VideoEncoder.isConfigSupported({ - ...supportCheckConfig, - bitrateMode: 'variable', - }); - if (!fallbackSupport.supported) { - log.error(`Codec not supported: ${codecString}`); - return false; + try { + for (const bitrateMode of bitrateModesToTry) { + for (const config of buildEncoderConfigCandidates(bitrateMode)) { + const support = await VideoEncoder.isConfigSupported(config); + if (support.supported) { + supportedEncoderConfigs.push(config); + } } } } catch (e) { @@ -93,6 +102,11 @@ export class VideoEncoderWrapper { return false; } + if (supportedEncoderConfigs.length === 0) { + log.error(`Codec not supported: ${codecString}`); + return false; + } + // Create muxer (MediaBunny adapter) this.createMuxer(); @@ -110,27 +124,32 @@ export class VideoEncoderWrapper { }, }); - const buildEncoderConfig = (bitrateMode: VideoEncoderBitrateMode): VideoEncoderConfig => ({ - ...supportCheckConfig, - latencyMode: 'quality', - bitrateMode, - }); + let selectedEncoderConfig: VideoEncoderConfig | null = null; - try { - this.encoder.configure(buildEncoderConfig(requestedBitrateMode)); - this.effectiveBitrateMode = requestedBitrateMode; - } catch (error) { - if (requestedBitrateMode !== 'constant') { - throw error; + for (const config of supportedEncoderConfigs) { + try { + this.encoder.configure(config); + selectedEncoderConfig = config; + this.effectiveBitrateMode = config.bitrateMode ?? 'variable'; + break; + } catch (error) { + log.warn( + `Encoder configure failed for ${config.latencyMode ?? 'default'} / ${config.hardwareAcceleration ?? 'default'} / ${config.bitrateMode ?? 'default'}, trying next config`, + error + ); } + } + + if (!selectedEncoderConfig) { + throw new Error(`Failed to configure encoder for codec ${codecString}`); + } - log.warn('Constant bitrate configure() failed, falling back to variable bitrate', error); - this.encoder.configure(buildEncoderConfig('variable')); - this.effectiveBitrateMode = 'variable'; + if (requestedBitrateMode !== this.effectiveBitrateMode) { + log.warn(`Requested ${requestedBitrateMode} bitrate mode is not supported for this encoder config; using ${this.effectiveBitrateMode}`); } log.info( - `Initialized: ${this.settings.width}x${this.settings.height} @ ${this.settings.fps}fps (${this.effectiveVideoCodec.toUpperCase()}, ${(this.settings.bitrate / 1_000_000).toFixed(1)} Mbps, ${this.effectiveBitrateMode})` + `Initialized: ${this.settings.width}x${this.settings.height} @ ${this.settings.fps}fps (${this.effectiveVideoCodec.toUpperCase()}, ${(this.settings.bitrate / 1_000_000).toFixed(1)} Mbps, ${this.effectiveBitrateMode}, ${selectedEncoderConfig.latencyMode ?? 'default'} latency, ${selectedEncoderConfig.hardwareAcceleration ?? 'default'} hw)` ); return true; } diff --git a/src/hooks/useGlobalHistory.ts b/src/hooks/useGlobalHistory.ts index 5ae416ba..6f29e2ea 100644 --- a/src/hooks/useGlobalHistory.ts +++ b/src/hooks/useGlobalHistory.ts @@ -205,6 +205,8 @@ export function useGlobalHistory() { folders: state.folders, textItems: state.textItems, solidItems: state.solidItems, + mathSceneItems: state.mathSceneItems, + motionShapeItems: state.motionShapeItems, }), (curr, prev) => { if (useHistoryStore.getState().isApplying) return; @@ -219,6 +221,10 @@ export function useGlobalHistory() { debouncedCapture('Modify text items'); } else if (curr.solidItems !== prev.solidItems) { debouncedCapture('Modify solid items'); + } else if (curr.mathSceneItems !== prev.mathSceneItems) { + debouncedCapture('Modify math scene items'); + } else if (curr.motionShapeItems !== prev.motionShapeItems) { + debouncedCapture('Modify motion shape items'); } }, { equalityFn: shallowEqual, fireImmediately: false } diff --git a/src/services/aiTools/definitions/stats.ts b/src/services/aiTools/definitions/stats.ts index 455fd487..fdea00c8 100644 --- a/src/services/aiTools/definitions/stats.ts +++ b/src/services/aiTools/definitions/stats.ts @@ -60,4 +60,30 @@ export const statsToolDefinitions: ToolDefinition[] = [ }, }, }, + { + type: 'function', + function: { + name: 'purgePlaybackPath', + description: 'Reset the live playback path at the current playhead without reloading the app. Clears VideoSync warmups/seeks, retargets active video/WebCodecs providers, resets GPU-ready state, and optionally resumes playback.', + parameters: { + type: 'object', + properties: { + mode: { + type: 'string', + enum: ['targeted', 'full'], + description: 'targeted resets active playback clips; full also clears broader preview caches. Defaults to targeted.', + }, + resumePlayback: { + type: 'boolean', + description: 'Whether to resume playback after the purge. Defaults to the pre-purge playing state.', + }, + reason: { + type: 'string', + description: 'Short diagnostic reason included in logs and tool result.', + }, + }, + required: [], + }, + }, + }, ]; diff --git a/src/services/aiTools/handlers/index.ts b/src/services/aiTools/handlers/index.ts index 8beeb652..cd72bd6d 100644 --- a/src/services/aiTools/handlers/index.ts +++ b/src/services/aiTools/handlers/index.ts @@ -121,6 +121,7 @@ import { handleGetLogs, handleGetPlaybackTrace, handleGetStatsHistory, + handlePurgePlaybackPath, } from './stats'; import { handleDebugExport } from './export'; import { @@ -231,6 +232,7 @@ const selfContainedHandlers: Record, call getStatsHistory: handleGetStatsHistory, getLogs: handleGetLogs, getPlaybackTrace: handleGetPlaybackTrace, + purgePlaybackPath: handlePurgePlaybackPath, debugExport: handleDebugExport, getNodeWorkspaceDebugState: handleGetNodeWorkspaceDebugState, sendAINodePrompt: handleSendAINodePrompt, diff --git a/src/services/aiTools/handlers/stats.ts b/src/services/aiTools/handlers/stats.ts index 6008f74e..b2b3d442 100644 --- a/src/services/aiTools/handlers/stats.ts +++ b/src/services/aiTools/handlers/stats.ts @@ -312,3 +312,23 @@ export async function handleGetPlaybackTrace( }, }; } + +export async function handlePurgePlaybackPath( + args: Record +): Promise { + const mode = args.mode === 'full' ? 'full' : 'targeted'; + const result = playbackHealthMonitor.purgePlaybackPath({ + reason: typeof args.reason === 'string' && args.reason.trim() + ? args.reason.trim() + : 'ai-tool', + mode, + resumePlayback: typeof args.resumePlayback === 'boolean' + ? args.resumePlayback + : undefined, + }); + + return { + success: true, + data: result, + }; +} diff --git a/src/services/aiTools/policy/registry.ts b/src/services/aiTools/policy/registry.ts index 666c11e1..7f70a3df 100644 --- a/src/services/aiTools/policy/registry.ts +++ b/src/services/aiTools/policy/registry.ts @@ -112,6 +112,12 @@ const TOOL_POLICY_MAP = new Map([ ['getStatsHistory', bridgeTelemetry()], ['getLogs', bridgeTelemetry()], ['getPlaybackTrace', bridgeTelemetry()], + ['purgePlaybackPath', { + ...bridgeTelemetry(), + readOnly: false, + riskLevel: 'low', + allowedCallers: ['chat', 'devBridge', 'console', 'internal'], + }], ['reloadApp', bridgeTelemetry()], ['debugExport', { ...bridgeTelemetry(), diff --git a/src/services/cloudApi.ts b/src/services/cloudApi.ts index 5a4078aa..2357c81a 100644 --- a/src/services/cloudApi.ts +++ b/src/services/cloudApi.ts @@ -357,6 +357,7 @@ interface ApiRequestInit extends RequestInit { } const DEFAULT_JSON_REQUEST_TIMEOUT_MS = 10_000; +const AI_CHAT_REQUEST_TIMEOUT_MS = 90_000; function createRequestController(signal?: AbortSignal | null, timeoutMs?: number): { cleanup: () => void; @@ -535,6 +536,7 @@ export const cloudApi = { return requestJson('/api/ai/chat', { body: JSON.stringify(body), method: 'POST', + timeoutMs: AI_CHAT_REQUEST_TIMEOUT_MS, }); }, stream(body: CloudAiChatRequest): Promise { @@ -571,6 +573,7 @@ export const cloudApi = { return requestJson('/api/ai/chat', { body: JSON.stringify(body), method: 'POST', + timeoutMs: AI_CHAT_REQUEST_TIMEOUT_MS, }); }, videoCreate(body: { diff --git a/src/services/layerBuilder/VideoSyncManager.ts b/src/services/layerBuilder/VideoSyncManager.ts index 1203a583..326252a7 100644 --- a/src/services/layerBuilder/VideoSyncManager.ts +++ b/src/services/layerBuilder/VideoSyncManager.ts @@ -154,10 +154,13 @@ export class VideoSyncManager { this.lastWarmupRetargetAt = {}; this.lastPausedJumpPreloadPosition = Number.NaN; this.lastPausedJumpPreloadActiveKey = ''; + this.warmingUpVideos = new WeakSet(); + this.warmupRetryCooldown = new WeakMap(); this.warmupAttemptIds = new WeakMap(); this.warmupWatchdogs = new WeakMap(); this.warmupClipIds = new WeakMap(); this.warmupTargetTimes = new WeakMap(); + this.gpuWarmedUp = new WeakSet(); this.nextWarmupAttemptId = 1; // Clear debounce timers for (const id of Object.values(this.preciseSeekTimers)) clearTimeout(id); diff --git a/src/services/playbackDebugStats.ts b/src/services/playbackDebugStats.ts index bdcdeb3e..84e4990f 100644 --- a/src/services/playbackDebugStats.ts +++ b/src/services/playbackDebugStats.ts @@ -562,10 +562,19 @@ function derivePlaybackStatus(stats: Omit): Playba const coldPlayback = stats.coldVideos > 0 && hasLivePlaybackDemand; const healthIssuesDuringPlayback = stats.healthAnomalies > 0 && hasLivePlaybackDemand; const missingReadyFramesDuringPlayback = noReadyFrames && hasLivePlaybackDemand; + const hasPreviewMotionDemand = stats.previewFrames > 0 && stats.stalePreviewWhileTargetMoved > 0; + const previewFreezeDuringPlayback = + stats.previewFreezeEvents > 0 && + stats.stalePreviewWhileTargetMoved > 0 && + (hasLivePlaybackDemand || hasPreviewMotionDemand); + const severePreviewFreeze = + previewFreezeDuringPlayback && + (stats.longestPreviewFreezeMs >= 650 || stats.stalePreviewWhileTargetMoved >= 12); if ( stats.stalls > 0 || severeCadence || + severePreviewFreeze || healthIssuesDuringPlayback || stats.readyStateDrops > 0 || coldPlayback || @@ -578,6 +587,7 @@ function derivePlaybackStatus(stats: Omit): Playba if ( degradedCadence || stats.queuePressureEvents > 30 || + previewFreezeDuringPlayback || stats.seeks >= 3 || (stats.decoderResets ?? 0) >= 3 || (stats.maxPendingSeekMs ?? 0) >= 80 || diff --git a/src/services/playbackHealthMonitor.ts b/src/services/playbackHealthMonitor.ts index f4202c02..6384cfd4 100644 --- a/src/services/playbackHealthMonitor.ts +++ b/src/services/playbackHealthMonitor.ts @@ -14,6 +14,22 @@ import { Logger } from './logger'; import { engine } from '../engine/WebGPUEngine'; import { createFrameContext, getClipTimeInfo, layerBuilder } from './layerBuilder'; import { useTimelineStore } from '../stores/timeline'; +import type { TimelineClip } from '../types'; +import { + buildPlaybackDebugStats, + type PlaybackHealthAnomaly, + type PlaybackHealthVideoState, +} from './playbackDebugStats'; +import { + canUseSharedPreviewRuntimeSession, + getPreviewRuntimeSource, + getRuntimeFrameProvider, + getScrubRuntimeSource, + updateRuntimePlaybackTime, +} from './mediaRuntime/runtimePlayback'; +import type { RuntimeFrameProvider } from './mediaRuntime/types'; +import { vfPipelineMonitor } from './vfPipelineMonitor'; +import { wcPipelineMonitor } from './wcPipelineMonitor'; const log = Logger.create('PlaybackHealth'); @@ -27,7 +43,8 @@ type AnomalyType = | 'READYSTATE_DROP' | 'GPU_SURFACE_COLD' | 'RENDER_STALL' - | 'HIGH_DROP_RATE'; + | 'HIGH_DROP_RATE' + | 'PREVIEW_FREEZE'; interface AnomalyEvent { type: AnomalyType; @@ -42,6 +59,28 @@ interface VideoTimeTracker { staleCount: number; } +type PlaybackPurgeMode = 'targeted' | 'full'; + +interface PlaybackPurgeOptions { + reason?: string; + mode?: PlaybackPurgeMode; + resumePlayback?: boolean; +} + +interface PlaybackPurgeResult { + reason: string; + mode: PlaybackPurgeMode; + playheadPosition: number; + wasPlaying: boolean; + resumeScheduled: boolean; + clips: Array<{ + clipId: string; + targetTime: number; + hadVideoElement: boolean; + webCodecsProvidersReset: number; + }>; +} + // --- Constants --- const POLL_INTERVAL = 500; @@ -55,6 +94,11 @@ const HIGH_DROP_THRESHOLD = 10; const CLIP_ESCALATION_WINDOW_MS = 12000; const CLIP_ESCALATION_THRESHOLD = 3; const CLIP_ESCALATION_COOLDOWN_MS = 15000; +const PREVIEW_FREEZE_WINDOW_MS = 1500; +const PREVIEW_FREEZE_RECOVERY_MS = 650; +const PREVIEW_FREEZE_STALE_FRAMES = 12; +const PLAYBACK_PURGE_COOLDOWN_MS = 8000; +const PLAYBACK_PURGE_RESUME_DELAY_MS = 80; // --- Service --- @@ -80,8 +124,10 @@ export class PlaybackHealthMonitor { GPU_SURFACE_COLD: 0, RENDER_STALL: 0, HIGH_DROP_RATE: 0, + PREVIEW_FREEZE: 0, }; private lastAnomalyTime: Partial> = {}; + private lastPlaybackPurgeAt = 0; private shouldMonitorHtmlVideoHealth( clip: { @@ -272,6 +318,8 @@ export class PlaybackHealthMonitor { this.recordAnomaly('HIGH_DROP_RATE', undefined, `${stats.drops.lastSecond} drops/sec`); } + this.maybeRecoverPreviewFreeze(now, isPlaying, stats.decoder); + // Cleanup stale tracker entries for clips no longer in timeline const currentClipIdSet = new Set(clips.map((c) => c.id)); const htmlHealthClipIdSet = new Set(htmlHealthVideoClips.map((c) => c.id)); @@ -291,6 +339,51 @@ export class PlaybackHealthMonitor { // --- Anomaly recording with cooldown --- + private maybeRecoverPreviewFreeze( + now: number, + isPlaying: boolean, + decoder: ReturnType['decoder'] + ): void { + if (!isPlaying) return; + if (typeof document !== 'undefined' && document.hidden) return; + if (now - this.lastPlaybackPurgeAt < PLAYBACK_PURGE_COOLDOWN_MS) return; + + const healthVideos = this.videos() as PlaybackHealthVideoState[]; + if (healthVideos.length === 0) return; + + const playback = buildPlaybackDebugStats({ + decoder, + now, + windowMs: PREVIEW_FREEZE_WINDOW_MS, + wcTimeline: wcPipelineMonitor.timeline(PREVIEW_FREEZE_WINDOW_MS), + vfTimeline: vfPipelineMonitor.timeline(PREVIEW_FREEZE_WINDOW_MS), + healthVideos, + healthAnomalies: this.anomalyLog as PlaybackHealthAnomaly[], + }); + + const freezeLongEnough = playback.longestPreviewFreezeMs >= PREVIEW_FREEZE_RECOVERY_MS; + const staleEnough = playback.stalePreviewWhileTargetMoved >= PREVIEW_FREEZE_STALE_FRAMES; + if (!freezeLongEnough || !staleEnough) { + return; + } + + const detail = [ + `preview frozen for ${Math.round(playback.longestPreviewFreezeMs)}ms`, + `staleMovingFrames=${playback.stalePreviewWhileTargetMoved}`, + `path=${playback.lastPreviewFreezePath ?? 'unknown'}`, + `pipeline=${playback.pipeline}`, + ].join(', '); + + if (this.recordAnomaly('PREVIEW_FREEZE', playback.lastPreviewFreezeClipId, detail)) { + this.lastPlaybackPurgeAt = now; + this.purgePlaybackPath({ + reason: 'auto-preview-freeze', + mode: 'targeted', + resumePlayback: true, + }); + } + } + private recordAnomaly(type: AnomalyType, clipId?: string, detail?: string): boolean { const now = performance.now(); const lastTime = this.lastAnomalyTime[type]; @@ -405,6 +498,162 @@ export class PlaybackHealthMonitor { vsm.recoverClipPlaybackState(clipId, video, targetTime, { resumePlayback }); } + private safePurgeSeekTime(video: HTMLVideoElement, targetTime: number): number { + const duration = video.duration; + if (!Number.isFinite(duration) || duration <= 0) { + return Math.max(0, targetTime); + } + return Math.max(0, Math.min(targetTime, duration - 0.001)); + } + + private resetRuntimeProvider( + provider: RuntimeFrameProvider | null | undefined, + targetTime: number + ): boolean { + if (!provider?.isFullMode?.()) { + return false; + } + + try { + provider.pause(); + provider.seek(targetTime); + provider.advanceToTime?.(targetTime); + return true; + } catch (error) { + log.warn('Failed to reset WebCodecs provider during playback purge', error); + return false; + } + } + + private resetWebCodecsProvidersForClip( + ctx: ReturnType, + clip: TimelineClip, + targetTime: number + ): number { + const source = clip.source; + if (!source) { + return 0; + } + + const providers = new Set(); + const allowShared = canUseSharedPreviewRuntimeSession(clip, ctx.clipsAtTime); + const previewSource = getPreviewRuntimeSource(source, clip.trackId, allowShared); + const scrubSource = getScrubRuntimeSource(source, clip.trackId, allowShared); + + updateRuntimePlaybackTime(previewSource, targetTime); + updateRuntimePlaybackTime(scrubSource, targetTime); + + const previewProvider = getRuntimeFrameProvider(previewSource); + const scrubProvider = getRuntimeFrameProvider(scrubSource); + if (previewProvider) providers.add(previewProvider); + if (scrubProvider) providers.add(scrubProvider); + if (source.webCodecsPlayer) providers.add(source.webCodecsPlayer); + + let resetCount = 0; + for (const provider of providers) { + if (this.resetRuntimeProvider(provider, targetTime)) { + resetCount++; + } + } + return resetCount; + } + + purgePlaybackPath(options: PlaybackPurgeOptions = {}): PlaybackPurgeResult { + const reason = options.reason ?? 'manual'; + const mode = options.mode ?? 'targeted'; + const state = useTimelineStore.getState(); + const wasPlaying = state.isPlaying; + const previousSpeed = state.playbackSpeed; + const playheadPosition = state.playheadPosition; + const resumePlayback = options.resumePlayback ?? wasPlaying; + + state.setDraggingPlayhead(false); + if (wasPlaying) { + state.pause(); + useTimelineStore.getState().setPlaybackSpeed(previousSpeed); + } + + const ctx = createFrameContext(); + const vsm = layerBuilder.getVideoSyncManager(); + const lc = engine.getLayerCollector(); + const clipsAtPlayhead = ctx.clips.filter( + (clip) => + (clip.source?.videoElement || clip.source?.webCodecsPlayer) && + playheadPosition >= clip.startTime && + playheadPosition < clip.startTime + clip.duration + ); + + this.videoTimeTracker.clear(); + this.warmupStartTimes = new WeakMap(); + this.seekStartTimes.clear(); + this.clipEscalationEvents.clear(); + this.clipEscalationCooldowns.clear(); + vsm.reset(); + engine.clearVideoCache(); + if (mode === 'full') { + engine.clearScrubbingCache(); + engine.clearCompositeCache(); + } + + const purgedClips: PlaybackPurgeResult['clips'] = []; + for (const clip of clipsAtPlayhead) { + const targetTime = getClipTimeInfo(ctx, clip).clipTime; + const video = clip.source?.videoElement; + const webCodecsProvidersReset = this.resetWebCodecsProvidersForClip(ctx, clip, targetTime); + + if (video) { + const safeTargetTime = this.safePurgeSeekTime(video, targetTime); + try { + video.pause(); + video.muted = true; + if ((video.src || video.currentSrc) && Math.abs(video.currentTime - safeTargetTime) > 0.01) { + video.currentTime = safeTargetTime; + } + } catch (error) { + log.warn('Failed to retarget video element during playback purge', error); + } + + lc?.resetVideoGpuReady(video); + const videoSrc = video.currentSrc || video.src; + if (videoSrc) { + engine.clearScrubbingCache(videoSrc); + } + vsm.resetClipRecoveryState(clip.id, video); + vsm.recoverClipPlaybackState(clip.id, video, safeTargetTime, { resumePlayback: false }); + } + + purgedClips.push({ + clipId: clip.id, + targetTime, + hadVideoElement: !!video, + webCodecsProvidersReset, + }); + } + + engine.requestNewFrameRender(); + log.warn(`[PLAYBACK_PURGE] reason=${reason} mode=${mode} clips=${purgedClips.length}`); + + if (resumePlayback) { + window.setTimeout(() => { + const liveState = useTimelineStore.getState(); + liveState.setPlaybackSpeed(previousSpeed); + void liveState.play().catch((error) => { + log.warn('Failed to resume playback after purge', error); + }); + engine.requestNewFrameRender(); + }, PLAYBACK_PURGE_RESUME_DELAY_MS); + } + + return { + reason, + mode, + playheadPosition, + wasPlaying, + resumeScheduled: resumePlayback, + clips: purgedClips, + }; + } + softReset(): void { const { clips, playheadPosition } = useTimelineStore.getState(); const vsm = layerBuilder.getVideoSyncManager(); @@ -498,6 +747,7 @@ export class PlaybackHealthMonitor { this.anomalyCounts[key] = 0; } this.lastAnomalyTime = {}; + this.lastPlaybackPurgeAt = 0; log.info('Health monitor reset'); } @@ -588,10 +838,12 @@ export class PlaybackHealthMonitor { videos: () => ReturnType; recover: { softReset: () => void; + purgePlaybackPath: (mode?: PlaybackPurgeMode) => PlaybackPurgeResult; forceDecodeAll: () => void; clearWarmups: () => void; clearOrphans: () => void; }; + purgePlaybackPath: (mode?: PlaybackPurgeMode) => PlaybackPurgeResult; reset: () => void; }; }).__PLAYBACK_HEALTH__ = { @@ -600,10 +852,12 @@ export class PlaybackHealthMonitor { videos: () => this.videos(), recover: { softReset: () => this.softReset(), + purgePlaybackPath: (mode?: PlaybackPurgeMode) => this.purgePlaybackPath({ reason: 'console', mode }), forceDecodeAll: () => this.forceDecodeAll(), clearWarmups: () => this.clearWarmups(), clearOrphans: () => this.clearOrphans(), }, + purgePlaybackPath: (mode?: PlaybackPurgeMode) => this.purgePlaybackPath({ reason: 'console', mode }), reset: () => this.reset(), }; } diff --git a/src/services/project/core/autosaveRecovery.ts b/src/services/project/core/autosaveRecovery.ts index 1185260c..e8ddea36 100644 --- a/src/services/project/core/autosaveRecovery.ts +++ b/src/services/project/core/autosaveRecovery.ts @@ -9,7 +9,9 @@ function generatedItemCount(project: ProjectFile): number { + (project.solidItems?.length ?? 0) + (project.meshItems?.length ?? 0) + (project.cameraItems?.length ?? 0) - + (project.splatEffectorItems?.length ?? 0); + + (project.splatEffectorItems?.length ?? 0) + + (project.mathSceneItems?.length ?? 0) + + (project.motionShapeItems?.length ?? 0); } export function hasMeaningfulContent(project: ProjectFile): boolean { diff --git a/src/services/project/projectLoad.ts b/src/services/project/projectLoad.ts index 14d6fb5f..7af72a1b 100644 --- a/src/services/project/projectLoad.ts +++ b/src/services/project/projectLoad.ts @@ -997,6 +997,8 @@ export async function loadProjectToStores(): Promise { const meshItems = normalizeItemFolderParents(projectData.meshItems || [], validFolderIds, 'mesh items'); const cameraItems = normalizeItemFolderParents(projectData.cameraItems || [], validFolderIds, 'camera items'); const splatEffectorItems = normalizeItemFolderParents(projectData.splatEffectorItems || [], validFolderIds, 'splat effector items'); + const mathSceneItems = normalizeItemFolderParents(projectData.mathSceneItems || [], validFolderIds, 'math scene items'); + const motionShapeItems = normalizeItemFolderParents(projectData.motionShapeItems || [], validFolderIds, 'motion shape items'); // Update media store useMediaStore.setState({ @@ -1019,6 +1021,8 @@ export async function loadProjectToStores(): Promise { meshItems, cameraItems, splatEffectorItems, + mathSceneItems, + motionShapeItems, activeCompositionId: projectData.activeCompositionId, openCompositionIds: projectData.openCompositionIds || [], expandedFolderIds: projectData.expandedFolderIds || [], diff --git a/src/services/project/projectSave.ts b/src/services/project/projectSave.ts index df2f65fc..821efc5e 100644 --- a/src/services/project/projectSave.ts +++ b/src/services/project/projectSave.ts @@ -536,6 +536,8 @@ export async function syncStoresToProject(): Promise { projectData.meshItems = freshState.meshItems; projectData.cameraItems = freshState.cameraItems; projectData.splatEffectorItems = freshState.splatEffectorItems; + projectData.mathSceneItems = freshState.mathSceneItems; + projectData.motionShapeItems = freshState.motionShapeItems; const flashBoardState = useFlashBoardStore.getState(); const hasBoardsToPersist = flashBoardState.boards.some((board) => board.nodes.length > 0); diff --git a/src/services/project/types/project.types.ts b/src/services/project/types/project.types.ts index 7673c32f..1f1410e5 100644 --- a/src/services/project/types/project.types.ts +++ b/src/services/project/types/project.types.ts @@ -8,7 +8,9 @@ import type { ProjectFlashBoardState } from '../../../stores/flashboardStore/typ import type { ExportStoreData } from '../../../stores/exportStore'; import type { CameraItem, + MathSceneItem, MeshItem, + MotionShapeItem, SolidItem, SplatEffectorItem, TextItem, @@ -145,4 +147,6 @@ export interface ProjectFile { meshItems?: MeshItem[]; cameraItems?: CameraItem[]; splatEffectorItems?: SplatEffectorItem[]; + mathSceneItems?: MathSceneItem[]; + motionShapeItems?: MotionShapeItem[]; } diff --git a/src/services/projectDB.ts b/src/services/projectDB.ts index fdc757b1..29d87373 100644 --- a/src/services/projectDB.ts +++ b/src/services/projectDB.ts @@ -114,6 +114,8 @@ export interface StoredProject { meshItems?: unknown[]; cameraItems?: unknown[]; splatEffectorItems?: unknown[]; + mathSceneItems?: unknown[]; + motionShapeItems?: unknown[]; }; } diff --git a/src/stores/exportStore.ts b/src/stores/exportStore.ts index 226d6815..a7786ddc 100644 --- a/src/stores/exportStore.ts +++ b/src/stores/exportStore.ts @@ -1,6 +1,7 @@ import { create } from 'zustand'; import { subscribeWithSelector } from 'zustand/middleware'; import type { ContainerFormat, VideoCodec } from '../engine/export'; +import type { AudioOnlyExportFormat } from '../engine/audio/AudioFileEncoder'; import type { DnxhrProfile, FFmpegContainer, @@ -14,6 +15,7 @@ export type ExportVisualMode = 'video' | 'image' | 'gif'; export type ExportImageFormat = 'png' | 'jpg' | 'webp' | 'bmp'; export type ExportImageMode = 'frame' | 'sequence'; export type ExportSpecialContainer = 'none' | 'xml'; +export type ExportAudioFormat = AudioOnlyExportFormat; export interface ExportSettings { encoder: ExportEncoderType; @@ -47,6 +49,7 @@ export interface ExportSettings { gifAlphaThreshold: number; stackedAlpha: boolean; includeAudio: boolean; + audioOnlyFormat: ExportAudioFormat; audioSampleRate: 44100 | 48000; audioBitrate: number; normalizeAudio: boolean; @@ -100,6 +103,7 @@ const IMAGE_FORMATS: ExportImageFormat[] = ['png', 'jpg', 'webp', 'bmp']; const IMAGE_EXPORT_MODES: ExportImageMode[] = ['frame', 'sequence']; const VISUAL_MODES: ExportVisualMode[] = ['video', 'image', 'gif']; const SPECIAL_CONTAINERS: ExportSpecialContainer[] = ['none', 'xml']; +const AUDIO_FORMATS: ExportAudioFormat[] = ['wav', 'browser']; const GIF_DITHERS: GifDither[] = ['sierra2_4a', 'floyd_steinberg', 'bayer', 'none']; const GIF_LOOPS: GifLoopMode[] = ['forever', 'once']; const GIF_PALETTE_MODES: GifPaletteMode[] = ['global', 'per-frame']; @@ -141,6 +145,7 @@ export function createDefaultExportSettings(): ExportSettings { gifAlphaThreshold: 128, stackedAlpha: false, includeAudio: true, + audioOnlyFormat: 'wav', audioSampleRate: 48000, audioBitrate: 256_000, normalizeAudio: false, @@ -234,6 +239,7 @@ function sanitizeSettings(input?: Partial | null): ExportSetting gifAlphaThreshold: Math.round(pickNumber(input.gifAlphaThreshold, defaults.gifAlphaThreshold, { min: 0, max: 255 })), stackedAlpha: typeof input.stackedAlpha === 'boolean' ? input.stackedAlpha : defaults.stackedAlpha, includeAudio: isGifOutput ? false : typeof input.includeAudio === 'boolean' ? input.includeAudio : defaults.includeAudio, + audioOnlyFormat: pickEnumValue(input.audioOnlyFormat, AUDIO_FORMATS, defaults.audioOnlyFormat), audioSampleRate: input.audioSampleRate === 44100 || input.audioSampleRate === 48000 ? input.audioSampleRate : defaults.audioSampleRate, diff --git a/src/stores/historyStore.ts b/src/stores/historyStore.ts index 1cbaca47..36c2e7f3 100644 --- a/src/stores/historyStore.ts +++ b/src/stores/historyStore.ts @@ -6,7 +6,15 @@ import { subscribeWithSelector } from 'zustand/middleware'; import { Logger } from '../services/logger'; import { flashBoardMediaBridge } from '../services/flashboard/FlashBoardMediaBridge'; import type { TimelineClip, TimelineTrack, Layer, Keyframe } from '../types'; -import type { MediaFile, Composition, MediaFolder, TextItem, SolidItem } from './mediaStore/types'; +import type { + Composition, + MathSceneItem, + MediaFile, + MediaFolder, + MotionShapeItem, + SolidItem, + TextItem, +} from './mediaStore/types'; import type { TimelineMarker } from './timeline/types'; import type { DockLayout } from '../types/dock'; import type { @@ -46,6 +54,8 @@ interface StateSnapshot { expandedFolderIds: string[]; textItems: TextItem[]; solidItems: SolidItem[]; + mathSceneItems: MathSceneItem[]; + motionShapeItems: MotionShapeItem[]; }; // Dock layout state @@ -119,6 +129,8 @@ interface MediaStoreState { expandedFolderIds: string[]; textItems: TextItem[]; solidItems: SolidItem[]; + mathSceneItems: MathSceneItem[]; + motionShapeItems: MotionShapeItem[]; } interface DockStoreSnapshot { @@ -275,6 +287,8 @@ function createSnapshot(label: string): StateSnapshot { expandedFolderIds: [...(media?.expandedFolderIds || [])], textItems: deepClone(media?.textItems || []), solidItems: deepClone(media?.solidItems || []), + mathSceneItems: deepClone(media?.mathSceneItems || []), + motionShapeItems: deepClone(media?.motionShapeItems || []), }, dock: { layout: deepClone(dock?.layout ?? null), @@ -345,6 +359,8 @@ function applySnapshot(snapshot: StateSnapshot) { expandedFolderIds: [...snapshot.media.expandedFolderIds], textItems: deepClone(snapshot.media.textItems || []), solidItems: deepClone(snapshot.media.solidItems || []), + mathSceneItems: deepClone(snapshot.media.mathSceneItems || []), + motionShapeItems: deepClone(snapshot.media.motionShapeItems || []), }); } diff --git a/src/stores/mediaStore/index.ts b/src/stores/mediaStore/index.ts index 6d69d735..9dc0f104 100644 --- a/src/stores/mediaStore/index.ts +++ b/src/stores/mediaStore/index.ts @@ -33,6 +33,8 @@ export type { MeshItem, CameraItem, SplatEffectorItem, + MathSceneItem, + MotionShapeItem, MeshPrimitiveType, SceneCameraSettings, SlotClipEndBehavior, @@ -53,6 +55,13 @@ const MESH_ITEM_LABELS: Record = { text3d: '3D Text', }; +const MOTION_SHAPE_LABELS: Record = { + rectangle: 'Motion Rectangle', + ellipse: 'Motion Ellipse', + polygon: 'Motion Polygon', + star: 'Motion Star', +}; + // Combined store type with all actions type MediaStoreState = MediaState & FileImportActions & @@ -84,6 +93,12 @@ type MediaStoreState = MediaState & getOrCreateSplatEffectorFolder: () => string; createSplatEffectorItem: (name?: string, parentId?: string | null) => string; removeSplatEffectorItem: (id: string) => void; + getOrCreateMathSceneFolder: () => string; + createMathSceneItem: (name?: string, parentId?: string | null) => string; + removeMathSceneItem: (id: string) => void; + getOrCreateMotionShapeFolder: () => string; + createMotionShapeItem: (primitive: import('../../types/motionDesign').ShapePrimitive, name?: string, parentId?: string | null) => string; + removeMotionShapeItem: (id: string) => void; }; export const useMediaStore = create()( @@ -97,6 +112,8 @@ export const useMediaStore = create()( meshItems: [], cameraItems: [], splatEffectorItems: [], + mathSceneItems: [], + motionShapeItems: [], activeCompositionId: 'comp-1', openCompositionIds: ['comp-1'], slotAssignments: {}, @@ -128,7 +145,18 @@ export const useMediaStore = create()( // Getters getItemsByFolder: (folderId: string | null) => { - const { files, compositions, folders, textItems, solidItems, meshItems, cameraItems, splatEffectorItems } = get(); + const { + files, + compositions, + folders, + textItems, + solidItems, + meshItems, + cameraItems, + splatEffectorItems, + mathSceneItems, + motionShapeItems, + } = get(); return [ ...folders.filter((f) => f.parentId === folderId), ...compositions.filter((c) => c.parentId === folderId), @@ -137,12 +165,25 @@ export const useMediaStore = create()( ...meshItems.filter((m) => m.parentId === folderId), ...cameraItems.filter((c) => c.parentId === folderId), ...splatEffectorItems.filter((e) => e.parentId === folderId), + ...mathSceneItems.filter((m) => m.parentId === folderId), + ...motionShapeItems.filter((m) => m.parentId === folderId), ...files.filter((f) => f.parentId === folderId), ]; }, getItemById: (id: string) => { - const { files, compositions, folders, textItems, solidItems, meshItems, cameraItems, splatEffectorItems } = get(); + const { + files, + compositions, + folders, + textItems, + solidItems, + meshItems, + cameraItems, + splatEffectorItems, + mathSceneItems, + motionShapeItems, + } = get(); return ( files.find((f) => f.id === id) || compositions.find((c) => c.id === id) || @@ -151,7 +192,9 @@ export const useMediaStore = create()( solidItems.find((s) => s.id === id) || meshItems.find((m) => m.id === id) || cameraItems.find((c) => c.id === id) || - splatEffectorItems.find((e) => e.id === id) + splatEffectorItems.find((e) => e.id === id) || + mathSceneItems.find((m) => m.id === id) || + motionShapeItems.find((m) => m.id === id) ); }, @@ -357,6 +400,66 @@ export const useMediaStore = create()( set({ splatEffectorItems: get().splatEffectorItems.filter(e => e.id !== id) }); }, + getOrCreateMathSceneFolder: () => { + const { folders, createFolder } = get(); + const existingFolder = folders.find((f) => f.name === 'Math Scenes' && f.parentId === null); + if (existingFolder) { + return existingFolder.id; + } + const newFolder = createFolder('Math Scenes', null); + return newFolder.id; + }, + + createMathSceneItem: (name?: string, parentId?: string | null) => { + const { mathSceneItems } = get(); + const id = `math-scene-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; + const newMathScene: import('./types').MathSceneItem = { + id, + name: name || `Math Scene ${mathSceneItems.length + 1}`, + type: 'math-scene', + parentId: parentId !== undefined ? parentId : null, + createdAt: Date.now(), + duration: 5, + }; + set({ mathSceneItems: [...mathSceneItems, newMathScene] }); + return id; + }, + + removeMathSceneItem: (id: string) => { + set({ mathSceneItems: get().mathSceneItems.filter((item) => item.id !== id) }); + }, + + getOrCreateMotionShapeFolder: () => { + const { folders, createFolder } = get(); + const existingFolder = folders.find((f) => f.name === 'Motion Shapes' && f.parentId === null); + if (existingFolder) { + return existingFolder.id; + } + const newFolder = createFolder('Motion Shapes', null); + return newFolder.id; + }, + + createMotionShapeItem: (primitive: import('../../types/motionDesign').ShapePrimitive, name?: string, parentId?: string | null) => { + const { motionShapeItems } = get(); + const id = `motion-shape-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; + const label = MOTION_SHAPE_LABELS[primitive] || 'Motion Shape'; + const newMotionShape: import('./types').MotionShapeItem = { + id, + name: name || `${label} ${motionShapeItems.filter((item) => item.primitive === primitive).length + 1}`, + type: 'motion-shape', + parentId: parentId !== undefined ? parentId : null, + createdAt: Date.now(), + primitive, + duration: 5, + }; + set({ motionShapeItems: [...motionShapeItems, newMotionShape] }); + return id; + }, + + removeMotionShapeItem: (id: string) => { + set({ motionShapeItems: get().motionShapeItems.filter((item) => item.id !== id) }); + }, + // Merge all slices ...createFileImportSlice(set, get), ...createFileManageSlice(set, get), diff --git a/src/stores/mediaStore/init.ts b/src/stores/mediaStore/init.ts index 3ceb758b..e162ad48 100644 --- a/src/stores/mediaStore/init.ts +++ b/src/stores/mediaStore/init.ts @@ -265,6 +265,24 @@ function setupItemPersistence(): void { } ); + useMediaStore.subscribe( + (state: MediaState) => state.mathSceneItems, + (mathSceneItems: MediaState['mathSceneItems']) => { + try { + localStorage.setItem('ms-mathSceneItems', JSON.stringify(mathSceneItems)); + } catch { /* quota exceeded or unavailable */ } + } + ); + + useMediaStore.subscribe( + (state: MediaState) => state.motionShapeItems, + (motionShapeItems: MediaState['motionShapeItems']) => { + try { + localStorage.setItem('ms-motionShapeItems', JSON.stringify(motionShapeItems)); + } catch { /* quota exceeded or unavailable */ } + } + ); + log.info('Item persistence setup complete'); } diff --git a/src/stores/mediaStore/slices/projectSlice.ts b/src/stores/mediaStore/slices/projectSlice.ts index b445d14a..18a760db 100644 --- a/src/stores/mediaStore/slices/projectSlice.ts +++ b/src/stores/mediaStore/slices/projectSlice.ts @@ -1,6 +1,17 @@ // Project persistence slice - save, load, init -import type { Composition, MediaFile, MediaFolder, TextItem, SolidItem, CameraItem, MediaSliceCreator, ProxyStatus } from '../types'; +import type { + CameraItem, + Composition, + MathSceneItem, + MediaFile, + MediaFolder, + MediaSliceCreator, + MotionShapeItem, + ProxyStatus, + SolidItem, + TextItem, +} from '../types'; import { DEFAULT_COMPOSITION } from '../constants'; import { generateId } from '../helpers/importPipeline'; import { getExpectedProxyFps, getProxyProgressFromFrameIndices, isProxyFrameIndexSetComplete } from '../helpers/proxyCompleteness'; @@ -180,6 +191,8 @@ export const createProjectSlice: MediaSliceCreator = (set, get) let restoredMeshItems: import('../types').MeshItem[] = []; let restoredCameraItems: CameraItem[] = []; let restoredSplatEffectorItems: import('../types').SplatEffectorItem[] = []; + let restoredMathSceneItems: MathSceneItem[] = []; + let restoredMotionShapeItems: MotionShapeItem[] = []; try { const storedText = localStorage.getItem('ms-textItems'); if (storedText) restoredTextItems = JSON.parse(storedText); @@ -200,6 +213,14 @@ export const createProjectSlice: MediaSliceCreator = (set, get) const storedSplatEffectors = localStorage.getItem('ms-splatEffectorItems'); if (storedSplatEffectors) restoredSplatEffectorItems = JSON.parse(storedSplatEffectors); } catch { /* ignore parse errors */ } + try { + const storedMathScenes = localStorage.getItem('ms-mathSceneItems'); + if (storedMathScenes) restoredMathSceneItems = JSON.parse(storedMathScenes); + } catch { /* ignore parse errors */ } + try { + const storedMotionShapes = localStorage.getItem('ms-motionShapeItems'); + if (storedMotionShapes) restoredMotionShapeItems = JSON.parse(storedMotionShapes); + } catch { /* ignore parse errors */ } set({ files: updatedFiles, @@ -209,6 +230,8 @@ export const createProjectSlice: MediaSliceCreator = (set, get) ...(restoredMeshItems.length > 0 && { meshItems: restoredMeshItems }), ...(restoredCameraItems.length > 0 && { cameraItems: restoredCameraItems }), ...(restoredSplatEffectorItems.length > 0 && { splatEffectorItems: restoredSplatEffectorItems }), + ...(restoredMathSceneItems.length > 0 && { mathSceneItems: restoredMathSceneItems }), + ...(restoredMotionShapeItems.length > 0 && { motionShapeItems: restoredMotionShapeItems }), }); log.info(`Restored ${storedFiles.length} files from IndexedDB`); } catch (e) { @@ -250,6 +273,8 @@ export const createProjectSlice: MediaSliceCreator = (set, get) meshItems: state.meshItems, cameraItems: state.cameraItems, splatEffectorItems: state.splatEffectorItems, + mathSceneItems: state.mathSceneItems, + motionShapeItems: state.motionShapeItems, }, }; @@ -318,6 +343,8 @@ export const createProjectSlice: MediaSliceCreator = (set, get) meshItems: (project.data.meshItems as import('../types').MeshItem[]) || [], cameraItems: (project.data.cameraItems as CameraItem[]) || [], splatEffectorItems: (project.data.splatEffectorItems as import('../types').SplatEffectorItem[]) || [], + mathSceneItems: (project.data.mathSceneItems as MathSceneItem[]) || [], + motionShapeItems: (project.data.motionShapeItems as MotionShapeItem[]) || [], activeCompositionId: null, openCompositionIds: (project.data.openCompositionIds as string[]) || [], expandedFolderIds: project.data.expandedFolderIds, @@ -376,6 +403,8 @@ export const createProjectSlice: MediaSliceCreator = (set, get) meshItems: [], cameraItems: [], splatEffectorItems: [], + mathSceneItems: [], + motionShapeItems: [], activeCompositionId: newCompId, openCompositionIds: [newCompId], selectedIds: [], @@ -399,6 +428,8 @@ export const createProjectSlice: MediaSliceCreator = (set, get) localStorage.removeItem('ms-meshItems'); localStorage.removeItem('ms-cameraItems'); localStorage.removeItem('ms-splatEffectorItems'); + localStorage.removeItem('ms-mathSceneItems'); + localStorage.removeItem('ms-motionShapeItems'); // Load empty timeline timelineStore.loadState(undefined); diff --git a/src/stores/mediaStore/slices/selectionSlice.ts b/src/stores/mediaStore/slices/selectionSlice.ts index aaa2bd34..df72c898 100644 --- a/src/stores/mediaStore/slices/selectionSlice.ts +++ b/src/stores/mediaStore/slices/selectionSlice.ts @@ -38,6 +38,12 @@ export const createSelectionSlice: MediaSliceCreator = (set) = splatEffectorItems: (state.splatEffectorItems || []).map((effector) => itemIds.includes(effector.id) ? { ...effector, parentId: folderId } : effector ), + mathSceneItems: (state.mathSceneItems || []).map((item) => + itemIds.includes(item.id) ? { ...item, parentId: folderId } : item + ), + motionShapeItems: (state.motionShapeItems || []).map((item) => + itemIds.includes(item.id) ? { ...item, parentId: folderId } : item + ), })); }, @@ -89,6 +95,12 @@ export const createSelectionSlice: MediaSliceCreator = (set) = splatEffectorItems: (state.splatEffectorItems || []).map((effector) => itemIds.includes(effector.id) ? { ...effector, labelColor: color } : effector ), + mathSceneItems: (state.mathSceneItems || []).map((item) => + itemIds.includes(item.id) ? { ...item, labelColor: color } : item + ), + motionShapeItems: (state.motionShapeItems || []).map((item) => + itemIds.includes(item.id) ? { ...item, labelColor: color } : item + ), })); }, }); diff --git a/src/stores/mediaStore/types.ts b/src/stores/mediaStore/types.ts index 98eb4a61..37c0d9f0 100644 --- a/src/stores/mediaStore/types.ts +++ b/src/stores/mediaStore/types.ts @@ -10,6 +10,7 @@ import type { } from '../../types'; import type { SplatEffectorSettings } from '../../types/splatEffector'; import type { VectorAnimationMetadata, VectorAnimationProvider } from '../../types/vectorAnimation'; +import type { ShapePrimitive } from '../../types/motionDesign'; // Media item types export type ImportedMediaType = @@ -26,6 +27,8 @@ export type MediaType = | 'composition' | 'text' | 'solid' + | 'math-scene' + | 'motion-shape' | 'camera' | 'splat-effector'; @@ -192,6 +195,17 @@ export interface SplatEffectorItem extends MediaItem { splatEffectorSettings: SplatEffectorSettings; } +export interface MathSceneItem extends MediaItem { + type: 'math-scene'; + duration: number; +} + +export interface MotionShapeItem extends MediaItem { + type: 'motion-shape'; + primitive: ShapePrimitive; + duration: number; +} + // 3D camera configuration for compositions export interface CompositionCamera { enabled: boolean; @@ -258,7 +272,17 @@ export interface ProjectLoadProgress { } // Union type for all items -export type ProjectItem = MediaFile | Composition | MediaFolder | TextItem | SolidItem | MeshItem | CameraItem | SplatEffectorItem; +export type ProjectItem = + | MediaFile + | Composition + | MediaFolder + | TextItem + | SolidItem + | MeshItem + | CameraItem + | SplatEffectorItem + | MathSceneItem + | MotionShapeItem; // Slice creator type for mediaStore export type MediaSliceCreator = ( @@ -277,6 +301,8 @@ export interface MediaState { meshItems: MeshItem[]; cameraItems: CameraItem[]; splatEffectorItems: SplatEffectorItem[]; + mathSceneItems: MathSceneItem[]; + motionShapeItems: MotionShapeItem[]; // Active composition activeCompositionId: string | null; diff --git a/src/stores/timeline/exportEditLock.ts b/src/stores/timeline/exportEditLock.ts new file mode 100644 index 00000000..2d5c7f62 --- /dev/null +++ b/src/stores/timeline/exportEditLock.ts @@ -0,0 +1,218 @@ +import type { TimelineStore } from './types'; +import { Logger } from '../../services/logger'; + +const log = Logger.create('TimelineEditLock'); + +const EXPORT_LOCKED_ACTION_NAMES = new Set([ + 'addTrack', + 'removeTrack', + 'renameTrack', + 'setTrackMuted', + 'setTrackVisible', + 'setTrackSolo', + 'setTrackLocked', + 'setTrackHeight', + 'scaleTracksOfType', + 'setTrackParent', + + 'addClip', + 'addCompClip', + 'updateClip', + 'removeClip', + 'moveClip', + 'trimClip', + 'splitClip', + 'splitClipAtPlayhead', + 'updateClipTransform', + 'toggleClipReverse', + 'setClipParent', + 'setClipPreservesPitch', + + 'addTextClip', + 'updateTextProperties', + 'updateTextBounds', + 'updateTextBoundsVertex', + 'updateTextBoundsVertices', + 'addSolidClip', + 'updateSolidColor', + 'addMathSceneClip', + 'updateMathScene', + 'addMathObject', + 'updateMathObject', + 'removeMathObject', + 'updateMathParameter', + 'addMotionShapeClip', + 'addMotionNullClip', + 'addMotionAdjustmentClip', + 'convertSolidToMotionShape', + 'updateMotionLayer', + 'addMeshClip', + 'updateText3DProperties', + 'addCameraClip', + 'addSplatEffectorClip', + + 'addClipEffect', + 'removeClipEffect', + 'updateClipEffect', + 'setClipEffectEnabled', + 'reorderClipEffect', + + 'ensureColorCorrection', + 'updateColorCorrection', + 'setColorCorrectionEnabled', + 'setColorViewMode', + 'setColorWorkspaceViewport', + 'selectColorNode', + 'addColorNode', + 'removeColorNode', + 'moveColorNode', + 'connectColorNodes', + 'removeColorEdge', + 'updateColorNodeParam', + 'setColorNodeEnabled', + 'renameColorNode', + 'resetColorNode', + 'resetColorCorrection', + 'duplicateColorVersion', + 'deleteColorVersion', + 'setActiveColorVersion', + + 'createLinkedGroup', + 'unlinkGroup', + 'addPendingDownloadClip', + 'completeDownload', + 'setDownloadError', + + 'addKeyframe', + 'addMaskPathKeyframe', + 'addTextBoundsPathKeyframe', + 'removeKeyframe', + 'updateKeyframe', + 'moveKeyframe', + 'moveKeyframes', + 'toggleKeyframeRecording', + 'setPropertyValue', + 'recordMaskPathKeyframe', + 'disableMaskPathKeyframes', + 'recordTextBoundsPathKeyframe', + 'disableTextBoundsPathKeyframes', + 'updateBezierHandle', + + 'setMaskEditMode', + 'setMaskPanelActive', + 'setMaskDrawStart', + 'setActiveMask', + 'selectVertex', + 'selectVertices', + 'deselectAllVertices', + 'addMask', + 'removeMask', + 'updateMask', + 'reorderMasks', + 'addVertex', + 'removeVertex', + 'updateVertex', + 'updateVertices', + 'setVertexHandleMode', + 'closeMask', + 'addRectangleMask', + 'addEllipseMask', + + 'addMarker', + 'removeMarker', + 'updateMarker', + 'moveMarker', + 'clearMarkers', + + 'applyTransition', + 'removeTransition', + 'updateTransitionDuration', + + 'addClipAICustomNode', + 'updateClipAICustomNode', + 'removeClipNodeGraphNode', + 'showClipNodeGraphBuiltIn', + 'connectClipNodeGraphPorts', + 'disconnectClipNodeGraphEdge', + 'moveClipNodeGraphNode', + + 'pasteClips', + 'pasteKeyframes', + 'pasteClipEffects', + 'pasteClipColor', + + 'setPlayheadPosition', + 'setDraggingPlayhead', + 'play', + 'playForward', + 'playReverse', + 'setInPoint', + 'setOutPoint', + 'clearInOut', + 'setInPointAtPlayhead', + 'setOutPointAtPlayhead', + 'setLoopPlayback', + 'toggleLoopPlayback', + 'setPlaybackSpeed', + 'setDuration', + 'setToolMode', + 'toggleCutTool', + + 'loadState', + 'clearTimeline', +]); + +const ASYNC_NULL_ACTION_NAMES = new Set(['addTextClip']); +const ASYNC_VOID_ACTION_NAMES = new Set(['addClip', 'addCompClip', 'completeDownload', 'loadState']); +const STRING_FALLBACK_ACTION_NAMES = new Set([ + 'addTrack', + 'addClipEffect', + 'addColorNode', + 'duplicateColorVersion', + 'addPendingDownloadClip', + 'addMarker', + 'addMask', + 'addVertex', +]); +const NULL_FALLBACK_ACTION_NAMES = new Set([ + 'addSolidClip', + 'addMathSceneClip', + 'addMotionShapeClip', + 'addMotionNullClip', + 'addMotionAdjustmentClip', + 'convertSolidToMotionShape', + 'addMeshClip', + 'addCameraClip', + 'addSplatEffectorClip', + 'addClipAICustomNode', +]); + +function getLockedReturnValue(actionName: string): unknown { + if (ASYNC_NULL_ACTION_NAMES.has(actionName)) return Promise.resolve(null); + if (ASYNC_VOID_ACTION_NAMES.has(actionName)) return Promise.resolve(); + if (STRING_FALLBACK_ACTION_NAMES.has(actionName)) return ''; + if (NULL_FALLBACK_ACTION_NAMES.has(actionName)) return null; + return undefined; +} + +export function lockTimelineEditActions( + actions: T, + get: () => Pick, +): T { + const wrapped = { ...(actions as unknown as Record) }; + + for (const actionName of EXPORT_LOCKED_ACTION_NAMES) { + const action = wrapped[actionName]; + if (typeof action !== 'function') continue; + + wrapped[actionName] = (...args: unknown[]) => { + if (get().isExporting) { + log.warn('Blocked timeline edit during export', { action: actionName }); + return getLockedReturnValue(actionName); + } + return (action as (...nextArgs: unknown[]) => unknown)(...args); + }; + } + + return wrapped as T; +} diff --git a/src/stores/timeline/index.ts b/src/stores/timeline/index.ts index eacca746..c45c62d5 100644 --- a/src/stores/timeline/index.ts +++ b/src/stores/timeline/index.ts @@ -33,9 +33,19 @@ import { createAIActionFeedbackSlice } from './aiActionFeedbackSlice'; import { createPositioningUtils } from './positioningUtils'; import { createSerializationUtils } from './serializationUtils'; import { Logger } from '../../services/logger'; +import { lockTimelineEditActions } from './exportEditLock'; const log = Logger.create('Timeline'); +function closeExportPreviewFrame(frame: ImageBitmap | null): void { + if (!frame) return; + try { + frame.close(); + } catch { + // ImageBitmap.close() is best-effort cleanup; ignore if the browser already released it. + } +} + // Re-export types for convenience export type { TimelineStore, TimelineClip, Keyframe } from './types'; export { DEFAULT_TRANSFORM, DEFAULT_TRACKS, SNAP_THRESHOLD_SECONDS } from './constants'; @@ -143,6 +153,7 @@ export const useTimelineStore = create()( snappingEnabled: true, isPlaying: false, isDraggingPlayhead: false, + playbackWarmup: null, selectedClipIds: new Set(), primarySelectedClipId: null, @@ -177,6 +188,8 @@ export const useTimelineStore = create()( exportProgress: null as number | null, exportCurrentTime: null as number | null, exportRange: null as { start: number; end: number } | null, + exportPreviewFrame: null as ImageBitmap | null, + exportPreviewFrameTime: null as number | null, // Performance toggles (enabled by default) thumbnailsEnabled: true, @@ -256,16 +269,38 @@ export const useTimelineStore = create()( setExportProgress: (progress: number | null, currentTime: number | null) => { set({ exportProgress: progress, exportCurrentTime: currentTime }); }, + setExportPreviewFrame: (frame: ImageBitmap | null, currentTime: number | null) => { + const previousFrame = get().exportPreviewFrame; + if (previousFrame && previousFrame !== frame) { + closeExportPreviewFrame(previousFrame); + } + set({ exportPreviewFrame: frame, exportPreviewFrameTime: currentTime }); + }, startExport: (start: number, end: number) => { - set({ isExporting: true, exportProgress: 0, exportCurrentTime: start, exportRange: { start, end } }); + closeExportPreviewFrame(get().exportPreviewFrame); + set({ + isExporting: true, + exportProgress: 0, + exportCurrentTime: start, + exportRange: { start, end }, + exportPreviewFrame: null, + exportPreviewFrameTime: null, + }); }, endExport: () => { - set({ isExporting: false, exportProgress: null, exportCurrentTime: null, exportRange: null }); + closeExportPreviewFrame(get().exportPreviewFrame); + set({ + isExporting: false, + exportProgress: null, + exportCurrentTime: null, + exportRange: null, + exportPreviewFrame: null, + exportPreviewFrameTime: null, + }); }, }; - return { - ...initialState, + const actions = lockTimelineEditActions({ ...trackActions, ...clipActions, ...textClipActions, @@ -293,6 +328,11 @@ export const useTimelineStore = create()( ...clipboardActions, ...aiActionFeedbackActions, ...utils, + }, get); + + return { + ...initialState, + ...actions, }; }) ); diff --git a/src/stores/timeline/playbackSlice.ts b/src/stores/timeline/playbackSlice.ts index 8866e9f8..a170ec80 100644 --- a/src/stores/timeline/playbackSlice.ts +++ b/src/stores/timeline/playbackSlice.ts @@ -8,6 +8,14 @@ import { getRuntimeFrameProvider } from '../../services/mediaRuntime/runtimePlay import { playheadState, sanitizePlayheadPosition } from '../../services/layerBuilder/PlayheadState'; import { resolvePlaybackStartPosition } from './playbackRange'; +function createPlaybackWarmupRequestId(): string { + return `playback-warmup-${Date.now()}-${Math.random().toString(36).slice(2)}`; +} + +function getWarmupTimestamp(): number { + return typeof performance !== 'undefined' ? performance.now() : Date.now(); +} + // Playback actions only (RAM preview and proxy cache in separate slices) export const createPlaybackSlice: SliceCreator = (set, get) => ({ // Playback actions @@ -35,6 +43,8 @@ export const createPlaybackSlice: SliceCreator = (set, get) => play: async () => { const { clips, inPoint, outPoint, duration, playbackSpeed } = get(); + set({ playbackWarmup: null }); + const playheadPosition = sanitizePlayheadPosition( get().playheadPosition, sanitizePlayheadPosition(playheadState.position, 0) @@ -107,7 +117,20 @@ export const createPlaybackSlice: SliceCreator = (set, get) => ...nestedVideos ])); - if (videosToCheck.length > 0) { + const videosNeedingWarmup = videosToCheck.filter((video) => video.readyState < 3); + + if (videosNeedingWarmup.length > 0) { + const warmupRequestId = createPlaybackWarmupRequestId(); + set({ + playbackWarmup: { + requestId: warmupRequestId, + startedAt: getWarmupTimestamp(), + targetTime: playbackStartPosition, + pendingVideoCount: videosNeedingWarmup.length, + totalVideoCount: videosToCheck.length, + }, + }); + // Wait for all videos to be ready (readyState >= 3 means HAVE_FUTURE_DATA) const waitForReady = async (video: HTMLVideoElement): Promise => { if (video.readyState >= 3) return; @@ -141,22 +164,26 @@ export const createPlaybackSlice: SliceCreator = (set, get) => // Wait for all videos in parallel with a timeout await Promise.race([ - Promise.all(videosToCheck.map(waitForReady)), + Promise.all(videosNeedingWarmup.map(waitForReady)), new Promise(resolve => setTimeout(resolve, 1000)) // Max 1 second wait ]); + + if (get().playbackWarmup?.requestId !== warmupRequestId) { + return; + } } - set({ isPlaying: true }); + set({ playbackWarmup: null, isPlaying: true }); }, pause: () => { // Reset playback speed to normal when pausing // So that Space (play/pause toggle) plays forward again - set({ isPlaying: false, playbackSpeed: 1 }); + set({ isPlaying: false, playbackSpeed: 1, playbackWarmup: null }); }, stop: () => { - set({ isPlaying: false, playheadPosition: 0 }); + set({ isPlaying: false, playheadPosition: 0, playbackWarmup: null }); }, // View actions diff --git a/src/stores/timeline/selectors.ts b/src/stores/timeline/selectors.ts index 6cf5af39..e308264c 100644 --- a/src/stores/timeline/selectors.ts +++ b/src/stores/timeline/selectors.ts @@ -42,6 +42,7 @@ export const selectIsRamPreviewing = (state: TimelineStore) => state.isRamPrevie export const selectIsExporting = (state: TimelineStore) => state.isExporting; export const selectExportProgress = (state: TimelineStore) => state.exportProgress; export const selectExportRange = (state: TimelineStore) => state.exportRange; +export const selectExportPreviewFrame = (state: TimelineStore) => state.exportPreviewFrame; export const selectIsProxyCaching = (state: TimelineStore) => state.isProxyCaching; export const selectProxyCacheProgress = (state: TimelineStore) => state.proxyCacheProgress; @@ -101,6 +102,8 @@ export const selectPreviewExportState = (state: TimelineStore) => ({ isExporting: state.isExporting, exportProgress: state.exportProgress, exportRange: state.exportRange, + exportPreviewFrame: state.exportPreviewFrame, + exportPreviewFrameTime: state.exportPreviewFrameTime, }); // Keyframe state (changes during keyframe edits) diff --git a/src/stores/timeline/types.ts b/src/stores/timeline/types.ts index 42a9be15..4caf7ca8 100644 --- a/src/stores/timeline/types.ts +++ b/src/stores/timeline/types.ts @@ -95,6 +95,14 @@ export interface AIMovingClip { startedAt: number; } +export interface PlaybackWarmupState { + requestId: string; + startedAt: number; + targetTime: number; + pendingVideoCount: number; + totalVideoCount: number; +} + // Timeline marker type export interface TimelineMarker { id: string; @@ -117,6 +125,7 @@ export interface TimelineState { snappingEnabled: boolean; isPlaying: boolean; isDraggingPlayhead: boolean; + playbackWarmup: PlaybackWarmupState | null; selectedClipIds: Set; primarySelectedClipId: string | null; // The clip the user actually clicked (for Properties panel) @@ -151,6 +160,8 @@ export interface TimelineState { exportProgress: number | null; // 0-100 percentage exportCurrentTime: number | null; // Current time being rendered exportRange: { start: number; end: number } | null; + exportPreviewFrame: ImageBitmap | null; // Latest frame captured from the export pipeline + exportPreviewFrameTime: number | null; // Performance toggles thumbnailsEnabled: boolean; @@ -412,6 +423,7 @@ export interface ProxyCacheActions { // Export progress actions interface export interface ExportActions { setExportProgress: (progress: number | null, currentTime: number | null) => void; + setExportPreviewFrame: (frame: ImageBitmap | null, currentTime: number | null) => void; startExport: (start: number, end: number) => void; endExport: () => void; } diff --git a/src/version.ts b/src/version.ts index bd8ca349..d50ed4be 100644 --- a/src/version.ts +++ b/src/version.ts @@ -1,6 +1,6 @@ // App version // Format: MAJOR.MINOR.PATCH -export const APP_VERSION = '1.8.0'; +export const APP_VERSION = '1.8.1'; export interface ChangelogNotice { type: 'info' | 'warning' | 'success' | 'danger'; @@ -36,15 +36,15 @@ export const FEATURED_VIDEO: { // Build/Platform notice shown at top of changelog (set to null to hide) export const BUILD_NOTICE: ChangelogNotice | null = { type: 'success', - title: 'AI Node Workspace', - message: 'Custom AI nodes can now be authored, chatted with, connected in clip node graphs, expose keyframeable parameters, and run in preview/export.', + title: 'Export and Timeline Reliability', + message: 'WAV export, fast export diagnostics, timeline locking, playback recovery, and property-based editor invariants are now tightened for production.', animated: true, }; export const WIP_NOTICE: ChangelogNotice | null = { type: 'info', - title: 'Live node authoring', - message: 'The node workspace now supports manual graph editing, node debugging tools, text layout context, and live render invalidation while paused.', + title: 'Release hardening', + message: 'The editor now has broader randomized coverage for ranges, slices, transforms, keyframes, easing, speed mapping, and file type helpers.', animated: true, }; diff --git a/tests/helpers/storeFactory.ts b/tests/helpers/storeFactory.ts index b512d9f5..2d508115 100644 --- a/tests/helpers/storeFactory.ts +++ b/tests/helpers/storeFactory.ts @@ -27,6 +27,7 @@ import { createDownloadClipSlice } from '../../src/stores/timeline/downloadClipS import { createNodeGraphSlice } from '../../src/stores/timeline/nodeGraphSlice'; import { createPositioningUtils } from '../../src/stores/timeline/positioningUtils'; import { resolvePlaybackStartPosition } from '../../src/stores/timeline/playbackRange'; +import { lockTimelineEditActions } from '../../src/stores/timeline/exportEditLock'; // Minimal initial state sufficient for testing slices function getInitialState(): Partial { @@ -84,6 +85,13 @@ function getInitialState(): Partial { // Proxy cache state isProxyCaching: false, proxyCacheProgress: null, + // Export state + isExporting: false, + exportProgress: null, + exportCurrentTime: null, + exportRange: null, + exportPreviewFrame: null, + exportPreviewFrameTime: null, // Stub functions that slices might call on other slices invalidateCache: () => {}, }; @@ -257,10 +265,13 @@ export function createTestTimelineStore(overrides?: Partial) { // Note: updateClipTransform and updateClipEffect are now provided by clipSlice const stubActions = { updateDuration: () => {}, + setExportProgress: (progress: number | null, currentTime: number | null) => set({ exportProgress: progress, exportCurrentTime: currentTime }), + setExportPreviewFrame: (frame: ImageBitmap | null, currentTime: number | null) => set({ exportPreviewFrame: frame, exportPreviewFrameTime: currentTime }), + startExport: (start: number, end: number) => set({ isExporting: true, exportProgress: 0, exportCurrentTime: start, exportRange: { start, end } }), + endExport: () => set({ isExporting: false, exportProgress: null, exportCurrentTime: null, exportRange: null, exportPreviewFrame: null, exportPreviewFrameTime: null }), }; - return { - ...getInitialState(), + const actions = lockTimelineEditActions({ ...selectionActions, ...trackActions, ...keyframeActions, @@ -278,6 +289,11 @@ export function createTestTimelineStore(overrides?: Partial) { ...positioningUtils, ...playbackActions, ...stubActions, + }, get); + + return { + ...getInitialState(), + ...actions, ...overrides, } as TimelineStore; }); diff --git a/tests/property/cameraLens.property.test.ts b/tests/property/cameraLens.property.test.ts new file mode 100644 index 00000000..eb4927bf --- /dev/null +++ b/tests/property/cameraLens.property.test.ts @@ -0,0 +1,75 @@ +import fc from 'fast-check'; +import { describe, expect, it } from 'vitest'; +import { + MAX_CAMERA_FOV_DEGREES, + MIN_CAMERA_FOV_DEGREES, + clampCameraFov, + fovToFullFrameFocalLengthMm, + fullFrameFocalLengthMmToFov, +} from '../../src/utils/cameraLens'; + +const fcOptions = { + numRuns: 200, + seed: 20260518, +}; + +const finiteNumber = fc.double({ + min: -1_000_000, + max: 1_000_000, + noDefaultInfinity: true, + noNaN: true, +}); + +describe('camera lens properties', () => { + it('clamps finite FOV values into the supported camera range', () => { + fc.assert( + fc.property(finiteNumber, (value) => { + const clamped = clampCameraFov(value); + + expect(Number.isFinite(clamped)).toBe(true); + expect(clamped).toBeGreaterThanOrEqual(MIN_CAMERA_FOV_DEGREES); + expect(clamped).toBeLessThanOrEqual(MAX_CAMERA_FOV_DEGREES); + }), + fcOptions, + ); + }); + + it('round-trips supported FOV values through full-frame focal length', () => { + fc.assert( + fc.property( + fc.double({ + min: MIN_CAMERA_FOV_DEGREES, + max: MAX_CAMERA_FOV_DEGREES, + noDefaultInfinity: true, + noNaN: true, + }), + (fov) => { + const focalLength = fovToFullFrameFocalLengthMm(fov); + const restoredFov = fullFrameFocalLengthMmToFov(focalLength); + + expect(focalLength).toBeGreaterThan(0); + expect(restoredFov).toBeCloseTo(fov, 8); + }, + ), + fcOptions, + ); + }); + + it('maps longer focal lengths to narrower or equal FOVs', () => { + fc.assert( + fc.property( + fc.double({ min: 1, max: 1_000, noDefaultInfinity: true, noNaN: true }), + fc.double({ min: 1, max: 1_000, noDefaultInfinity: true, noNaN: true }), + (a, b) => { + const shorter = Math.min(a, b); + const longer = Math.max(a, b); + + expect(fullFrameFocalLengthMmToFov(longer)).toBeLessThanOrEqual( + fullFrameFocalLengthMmToFov(shorter), + ); + }, + ), + fcOptions, + ); + }); +}); diff --git a/tests/property/clipSlice.property.test.ts b/tests/property/clipSlice.property.test.ts new file mode 100644 index 00000000..983342f0 --- /dev/null +++ b/tests/property/clipSlice.property.test.ts @@ -0,0 +1,295 @@ +import fc from 'fast-check'; +import { describe, expect, it, vi } from 'vitest'; +import { createMockClip, resetIdCounter } from '../helpers/mockData'; +import { createTestTimelineStore } from '../helpers/storeFactory'; + +const propertyConfig = { numRuns: 100, seed: 20260519 }; + +const toSeconds = (ticks: number) => ticks / 10; + +const clipTiming = fc.record({ + startTicks: fc.integer({ min: 0, max: 600 }), + durationTicks: fc.integer({ min: 2, max: 600 }), + inTicks: fc.integer({ min: 0, max: 1_000 }), +}); + +const validTrim = fc + .record({ + inTicks: fc.integer({ min: 0, max: 1_000 }), + durationTicks: fc.integer({ min: 1, max: 600 }), + }) + .map(({ inTicks, durationTicks }) => ({ + inPoint: toSeconds(inTicks), + outPoint: toSeconds(inTicks + durationTicks), + })); + +const moveCase = fc.record({ + startTicks: fc.integer({ min: 0, max: 600 }), + durationTicks: fc.integer({ min: 1, max: 600 }), + requestedStartTicks: fc.integer({ min: -600, max: 600 }), +}); + +function expectBidirectionalLinks(clips: Array<{ id: string; linkedClipId?: string }>) { + for (const clip of clips) { + expect(clip.linkedClipId).toBeDefined(); + const linked = clips.find((candidate) => candidate.id === clip.linkedClipId); + expect(linked).toBeDefined(); + expect(linked!.linkedClipId).toBe(clip.id); + } +} + +describe('clipSlice property invariants', () => { + it('splitClip preserves local clip duration and source continuity for valid interior splits', () => { + fc.assert( + fc.property( + clipTiming, + fc.integer({ min: 1, max: 599 }), + ({ startTicks, durationTicks, inTicks }, splitOffsetTicks) => { + fc.pre(splitOffsetTicks < durationTicks); + resetIdCounter(); + + const startTime = toSeconds(startTicks); + const duration = toSeconds(durationTicks); + const inPoint = toSeconds(inTicks); + const outPoint = inPoint + duration; + const splitTime = startTime + toSeconds(splitOffsetTicks); + const clip = createMockClip({ + id: 'clip-1', + trackId: 'video-1', + startTime, + duration, + inPoint, + outPoint, + }); + const store = createTestTimelineStore({ clips: [clip] }); + + store.getState().splitClip('clip-1', splitTime); + + const clips = [...store.getState().clips].sort((a, b) => a.startTime - b.startTime); + expect(clips).toHaveLength(2); + + const [first, second] = clips; + expect(first.startTime).toBeCloseTo(startTime, 10); + expect(second.startTime).toBeCloseTo(splitTime, 10); + expect(first.duration + second.duration).toBeCloseTo(duration, 10); + expect(first.startTime + first.duration).toBeCloseTo(second.startTime, 10); + + expect(first.inPoint).toBeCloseTo(inPoint, 10); + expect(first.outPoint).toBeCloseTo(second.inPoint, 10); + expect(second.outPoint).toBeCloseTo(outPoint, 10); + expect((first.outPoint - first.inPoint) + (second.outPoint - second.inPoint)).toBeCloseTo(outPoint - inPoint, 10); + }, + ), + propertyConfig, + ); + }); + + it('splitClip at an edge or outside the clip does not add or remove clips', () => { + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined); + try { + fc.assert( + fc.property( + clipTiming, + fc.constantFrom<'start' | 'end' | 'before' | 'after'>('start', 'end', 'before', 'after'), + ({ startTicks, durationTicks, inTicks }, splitKind) => { + resetIdCounter(); + + const startTime = toSeconds(startTicks); + const duration = toSeconds(durationTicks); + const inPoint = toSeconds(inTicks); + const clip = createMockClip({ + id: 'clip-1', + trackId: 'video-1', + startTime, + duration, + inPoint, + outPoint: inPoint + duration, + }); + const splitTime = { + start: startTime, + end: startTime + duration, + before: Math.max(0, startTime - 0.1), + after: startTime + duration + 0.1, + }[splitKind]; + const store = createTestTimelineStore({ clips: [clip] }); + + store.getState().splitClip('clip-1', splitTime); + + expect(store.getState().clips).toHaveLength(1); + expect(store.getState().clips[0].id).toBe('clip-1'); + }, + ), + propertyConfig, + ); + } finally { + warnSpy.mockRestore(); + } + }); + + it('trimClip sets duration to outPoint minus inPoint for valid trims', () => { + fc.assert( + fc.property(validTrim, ({ inPoint, outPoint }) => { + resetIdCounter(); + + const clip = createMockClip({ + id: 'clip-1', + trackId: 'video-1', + startTime: 3, + duration: 10, + inPoint: 0, + outPoint: 10, + }); + const store = createTestTimelineStore({ clips: [clip] }); + + store.getState().trimClip('clip-1', inPoint, outPoint); + + const trimmed = store.getState().clips.find((candidate) => candidate.id === 'clip-1')!; + expect(trimmed.inPoint).toBe(inPoint); + expect(trimmed.outPoint).toBe(outPoint); + expect(trimmed.duration).toBeCloseTo(outPoint - inPoint, 10); + expect(trimmed.duration).toBeGreaterThanOrEqual(0); + }), + propertyConfig, + ); + }); + + it('moveClip clamps startTime to zero or later while preserving unrelated clip fields', () => { + fc.assert( + fc.property(moveCase, ({ startTicks, durationTicks, requestedStartTicks }) => { + resetIdCounter(); + + const clip = createMockClip({ + id: 'clip-1', + trackId: 'video-1', + name: 'Stable fields', + startTime: toSeconds(startTicks), + duration: toSeconds(durationTicks), + inPoint: 1, + outPoint: 1 + toSeconds(durationTicks), + linkedGroupId: 'group-keep', + parentClipId: 'parent-keep', + source: { type: 'video', naturalDuration: 100 }, + }); + const before = { ...clip, transform: clip.transform, effects: clip.effects }; + const store = createTestTimelineStore({ clips: [clip], snappingEnabled: false }); + + store.getState().moveClip('clip-1', toSeconds(requestedStartTicks)); + + const moved = store.getState().clips.find((candidate) => candidate.id === 'clip-1')!; + expect(moved.startTime).toBeGreaterThanOrEqual(0); + expect(moved.id).toBe(before.id); + expect(moved.trackId).toBe(before.trackId); + expect(moved.name).toBe(before.name); + expect(moved.duration).toBe(before.duration); + expect(moved.inPoint).toBe(before.inPoint); + expect(moved.outPoint).toBe(before.outPoint); + expect(moved.linkedGroupId).toBe(before.linkedGroupId); + expect(moved.parentClipId).toBe(before.parentClipId); + expect(moved.source).toEqual(before.source); + expect(moved.transform).toEqual(before.transform); + expect(moved.effects).toEqual(before.effects); + }), + propertyConfig, + ); + }); + + it('moveClip keeps linked clips time-synced when moving the primary clip', () => { + fc.assert( + fc.property(moveCase, ({ startTicks, durationTicks, requestedStartTicks }) => { + resetIdCounter(); + + const startTime = toSeconds(startTicks); + const duration = toSeconds(durationTicks); + const videoClip = createMockClip({ + id: 'clip-v', + trackId: 'video-1', + startTime, + duration, + inPoint: 0, + outPoint: duration, + source: { type: 'video', naturalDuration: duration }, + linkedClipId: 'clip-a', + }); + const audioClip = createMockClip({ + id: 'clip-a', + trackId: 'audio-1', + startTime, + duration, + inPoint: 0, + outPoint: duration, + source: { type: 'audio', naturalDuration: duration }, + linkedClipId: 'clip-v', + }); + const store = createTestTimelineStore({ clips: [videoClip, audioClip], snappingEnabled: false }); + + store.getState().moveClip('clip-v', toSeconds(requestedStartTicks)); + + const movedVideo = store.getState().clips.find((clip) => clip.id === 'clip-v')!; + const movedAudio = store.getState().clips.find((clip) => clip.id === 'clip-a')!; + expect(movedVideo.startTime).toBeGreaterThanOrEqual(0); + expect(movedAudio.startTime).toBeGreaterThanOrEqual(0); + expect(movedAudio.startTime).toBeCloseTo(movedVideo.startTime, 10); + expectBidirectionalLinks([movedVideo, movedAudio]); + }), + propertyConfig, + ); + }); + + it('splitClip keeps linked split pairs bidirectionally linked and duration-aligned', () => { + fc.assert( + fc.property( + clipTiming, + fc.integer({ min: 1, max: 599 }), + ({ startTicks, durationTicks, inTicks }, splitOffsetTicks) => { + fc.pre(splitOffsetTicks < durationTicks); + resetIdCounter(); + + const startTime = toSeconds(startTicks); + const duration = toSeconds(durationTicks); + const inPoint = toSeconds(inTicks); + const splitTime = startTime + toSeconds(splitOffsetTicks); + const videoClip = createMockClip({ + id: 'clip-v', + trackId: 'video-1', + startTime, + duration, + inPoint, + outPoint: inPoint + duration, + source: { type: 'video', naturalDuration: duration }, + linkedClipId: 'clip-a', + }); + const audioClip = createMockClip({ + id: 'clip-a', + trackId: 'audio-1', + startTime, + duration, + inPoint, + outPoint: inPoint + duration, + source: { type: 'audio', naturalDuration: duration }, + linkedClipId: 'clip-v', + }); + const store = createTestTimelineStore({ clips: [videoClip, audioClip] }); + + store.getState().splitClip('clip-v', splitTime); + + const clips = store.getState().clips; + const videoParts = clips.filter((clip) => clip.trackId === 'video-1').sort((a, b) => a.startTime - b.startTime); + const audioParts = clips.filter((clip) => clip.trackId === 'audio-1').sort((a, b) => a.startTime - b.startTime); + + expect(clips).toHaveLength(4); + expect(videoParts).toHaveLength(2); + expect(audioParts).toHaveLength(2); + expectBidirectionalLinks(clips); + + for (let index = 0; index < 2; index += 1) { + expect(videoParts[index].startTime).toBeCloseTo(audioParts[index].startTime, 10); + expect(videoParts[index].duration).toBeCloseTo(audioParts[index].duration, 10); + expect(videoParts[index].linkedClipId).toBe(audioParts[index].id); + expect(audioParts[index].linkedClipId).toBe(videoParts[index].id); + } + }, + ), + propertyConfig, + ); + }); +}); diff --git a/tests/property/easing.property.test.ts b/tests/property/easing.property.test.ts new file mode 100644 index 00000000..e9d5451d --- /dev/null +++ b/tests/property/easing.property.test.ts @@ -0,0 +1,208 @@ +import { describe, expect, it } from 'vitest'; +import fc from 'fast-check'; +import type { EasingType } from '../../src/types'; +import { normalizeEasingType } from '../../src/utils/easing'; +import { easingFunctions, PRESET_BEZIER } from '../../src/utils/keyframeInterpolation'; + +const knownEasings = ['linear', 'ease-in', 'ease-out', 'ease-in-out', 'bezier'] as const satisfies readonly EasingType[]; +const presetEasings = ['linear', 'ease-in', 'ease-out', 'ease-in-out'] as const satisfies readonly Exclude[]; +const aliasCases = [ + ['linear', 'linear'], + [' linear ', 'linear'], + ['LINEAR', 'linear'], + ['ease-in', 'ease-in'], + ['easeIn', 'ease-in'], + ['ease_in', 'ease-in'], + ['ease in', 'ease-in'], + ['EaseInElastic', 'ease-in'], + ['ease-out', 'ease-out'], + ['easeOut', 'ease-out'], + ['ease_out', 'ease-out'], + ['ease out', 'ease-out'], + ['EaseOutElastic', 'ease-out'], + ['ease-in-out', 'ease-in-out'], + ['easeInOut', 'ease-in-out'], + ['ease_in_out', 'ease-in-out'], + ['ease in out', 'ease-in-out'], + ['EaseInOutElastic', 'ease-in-out'], + ['bezier', 'bezier'], + [' BEZIER ', 'bezier'], +] as const satisfies readonly (readonly [string, EasingType])[]; +const aliasCompacts = new Set([ + 'linear', + 'easein', + 'easeout', + 'easeinout', + 'easeinelastic', + 'easeoutelastic', + 'easeinoutelastic', + 'bezier', +]); + +const assertOptions = { seed: 0x5e1ec7, numRuns: 100 }; + +function compactEasing(value: string): string { + return value.trim().toLowerCase().replace(/[\s_-]+/g, ''); +} + +function isKnownEasing(value: string): value is EasingType { + return knownEasings.includes(value as EasingType); +} + +describe('normalizeEasingType properties', () => { + it('normalizes documented aliases independently of fallback', () => { + fc.assert( + fc.property( + fc.constantFrom(...aliasCases), + fc.constantFrom(...knownEasings), + ([alias, expected], fallback) => { + expect(normalizeEasingType(alias, fallback)).toBe(expected); + }, + ), + assertOptions, + ); + }); + + it('keeps known easing values unchanged regardless of fallback', () => { + fc.assert( + fc.property( + fc.constantFrom(...knownEasings), + fc.constantFrom(...knownEasings), + (easing, fallback) => { + expect(normalizeEasingType(easing, fallback)).toBe(easing); + }, + ), + assertOptions, + ); + }); + + it('falls back to the requested default for invalid non-empty strings', () => { + fc.assert( + fc.property( + fc.string({ minLength: 1, maxLength: 32 }).filter((value) => { + const compact = compactEasing(value); + return compact.length > 0 && !aliasCompacts.has(compact); + }), + fc.constantFrom(...knownEasings), + (invalidEasing, fallback) => { + expect(normalizeEasingType(invalidEasing, fallback)).toBe(fallback); + }, + ), + assertOptions, + ); + }); + + it('falls back to the requested default for null, undefined, and empty input', () => { + fc.assert( + fc.property( + fc.constantFrom(null, undefined, ''), + fc.constantFrom(...knownEasings), + (emptyEasing, fallback) => { + expect(normalizeEasingType(emptyEasing, fallback)).toBe(fallback); + }, + ), + assertOptions, + ); + }); + + it('always returns a known easing value when fallback is known', () => { + fc.assert( + fc.property( + fc.option(fc.string({ maxLength: 64 }), { nil: undefined }), + fc.constantFrom(...knownEasings), + (easing, fallback) => { + expect(isKnownEasing(normalizeEasingType(easing, fallback))).toBe(true); + }, + ), + assertOptions, + ); + }); + + it('defaults invalid values to linear when fallback is omitted', () => { + fc.assert( + fc.property( + fc.string({ minLength: 1, maxLength: 32 }).filter((value) => { + const compact = compactEasing(value); + return compact.length > 0 && !aliasCompacts.has(compact); + }), + (invalidEasing) => { + expect(normalizeEasingType(invalidEasing)).toBe('linear'); + }, + ), + assertOptions, + ); + }); +}); + +describe('preset easing properties', () => { + it('anchors every preset function at the normalized progress endpoints', () => { + fc.assert( + fc.property(fc.constantFrom(...presetEasings), (easing) => { + expect(easingFunctions[easing](0)).toBe(0); + expect(easingFunctions[easing](1)).toBe(1); + }), + assertOptions, + ); + }); + + it('is deterministic for normalized preset progress values', () => { + fc.assert( + fc.property( + fc.constantFrom(...presetEasings), + fc.double({ min: 0, max: 1, noNaN: true }), + (easing, progress) => { + const first = easingFunctions[easing](progress); + const second = easingFunctions[easing](progress); + expect(second).toBe(first); + }, + ), + assertOptions, + ); + }); + + it('maps normalized preset progress values to finite values in the unit interval', () => { + fc.assert( + fc.property( + fc.constantFrom(...presetEasings), + fc.double({ min: 0, max: 1, noNaN: true }), + (easing, progress) => { + const eased = easingFunctions[easing](progress); + expect(Number.isFinite(eased)).toBe(true); + expect(eased).toBeGreaterThanOrEqual(0); + expect(eased).toBeLessThanOrEqual(1); + }, + ), + assertOptions, + ); + }); + + it('is monotonic over the normalized progress interval for current presets', () => { + fc.assert( + fc.property( + fc.constantFrom(...presetEasings), + fc.tuple( + fc.double({ min: 0, max: 1, noNaN: true }), + fc.double({ min: 0, max: 1, noNaN: true }), + ), + (easing, [a, b]) => { + const earlier = Math.min(a, b); + const later = Math.max(a, b); + expect(easingFunctions[easing](earlier)).toBeLessThanOrEqual(easingFunctions[easing](later)); + }, + ), + assertOptions, + ); + }); + + it('keeps preset bezier handles finite for conversion consumers', () => { + fc.assert( + fc.property(fc.constantFrom(...presetEasings), (easing) => { + expect(PRESET_BEZIER[easing].p1).toHaveLength(2); + expect(PRESET_BEZIER[easing].p2).toHaveLength(2); + const coordinates = [...PRESET_BEZIER[easing].p1, ...PRESET_BEZIER[easing].p2]; + expect(coordinates.every(Number.isFinite)).toBe(true); + }), + assertOptions, + ); + }); +}); diff --git a/tests/property/exportRange.property.test.ts b/tests/property/exportRange.property.test.ts new file mode 100644 index 00000000..71a6c888 --- /dev/null +++ b/tests/property/exportRange.property.test.ts @@ -0,0 +1,104 @@ +import fc from 'fast-check'; +import { describe, expect, it } from 'vitest'; +import { resolveExportRange } from '../../src/components/export/exportRange'; + +const fcOptions = { + numRuns: 200, + seed: 20260518, +}; + +const finiteTime = fc.double({ + min: -1_000_000, + max: 1_000_000, + noDefaultInfinity: true, + noNaN: true, +}); + +const numericTime = fc.oneof( + finiteTime, + fc.constant(Number.NaN), + fc.constant(Number.POSITIVE_INFINITY), + fc.constant(Number.NEGATIVE_INFINITY), +); + +const nullableNumericTime = fc.oneof(numericTime, fc.constant(null)); + +function safeDuration(duration: number): number { + return Math.max(0, Number.isFinite(duration) ? duration : 0); +} + +function sanitizeTime(value: number | null, fallback: number): number { + return typeof value === 'number' && Number.isFinite(value) ? value : fallback; +} + +describe('resolveExportRange properties', () => { + it('always resolves to a finite monotonic range inside the safe duration', () => { + fc.assert( + fc.property( + numericTime, + nullableNumericTime, + nullableNumericTime, + fc.boolean(), + (duration, inPoint, outPoint, useInOut) => { + const resolved = resolveExportRange({ duration, inPoint, outPoint }, useInOut); + const durationLimit = safeDuration(duration); + + expect(Number.isFinite(resolved.startTime)).toBe(true); + expect(Number.isFinite(resolved.endTime)).toBe(true); + expect(resolved.startTime).toBeGreaterThanOrEqual(0); + expect(resolved.endTime).toBeGreaterThanOrEqual(resolved.startTime); + expect(resolved.endTime).toBeLessThanOrEqual(durationLimit); + expect(resolved.endTime - resolved.startTime).toBeGreaterThanOrEqual(0); + }, + ), + fcOptions, + ); + }); + + it('ignores all marker values when In/Out export is disabled', () => { + fc.assert( + fc.property(numericTime, nullableNumericTime, nullableNumericTime, (duration, inPoint, outPoint) => { + expect(resolveExportRange({ duration, inPoint, outPoint }, false)).toEqual({ + startTime: 0, + endTime: safeDuration(duration), + }); + }), + fcOptions, + ); + }); + + it('defaults null markers to the full safe duration when In/Out export is enabled', () => { + fc.assert( + fc.property(numericTime, (duration) => { + expect(resolveExportRange({ duration, inPoint: null, outPoint: null }, true)).toEqual({ + startTime: 0, + endTime: safeDuration(duration), + }); + }), + fcOptions, + ); + }); + + it('collapses reversed ranges to the clamped start when the requested end is before it', () => { + fc.assert( + fc.property( + fc.double({ min: 0.001, max: 1_000_000, noDefaultInfinity: true, noNaN: true }), + finiteTime, + finiteTime, + (duration, inPoint, outPoint) => { + const durationLimit = safeDuration(duration); + const requestedStart = sanitizeTime(inPoint, 0); + const clampedStart = Math.max(0, Math.min(requestedStart, durationLimit)); + + fc.pre(outPoint < clampedStart); + + expect(resolveExportRange({ duration, inPoint, outPoint }, true)).toEqual({ + startTime: clampedStart, + endTime: clampedStart, + }); + }, + ), + fcOptions, + ); + }); +}); diff --git a/tests/property/fileTypeHelpers.property.test.ts b/tests/property/fileTypeHelpers.property.test.ts new file mode 100644 index 00000000..dedf9773 --- /dev/null +++ b/tests/property/fileTypeHelpers.property.test.ts @@ -0,0 +1,157 @@ +import fc from 'fast-check'; +import { describe, expect, it } from 'vitest'; +import { + isAudioFile, + isGaussianSplatFile, + isMediaFile, + isModelFile, + isVideoFile, +} from '../../src/components/timeline/utils/fileTypeHelpers'; + +const fcOptions = { + numRuns: 200, + seed: 20260518, +}; + +const representativeMediaExtensions = ['mp4', 'wav', 'png', 'glb', 'ply', 'lottie'] as const; +const representativeModelExtensions = ['obj', 'glb'] as const; +const representativeGaussianSplatExtensions = ['ply', 'splat'] as const; + +const knownVideoExtension = 'mp4'; +const knownAudioExtension = 'wav'; +const unknownExtension = fc.array(fc.constantFrom( + '0', '1', '2', '3', '4', '5', '6', '7', + '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', +), { minLength: 8, maxLength: 16 }) + .map((chars) => chars.join('')) + .map((suffix) => `masterselects-property-unknown-${suffix}`); + +const fileStem = fc.string({ minLength: 1, maxLength: 24 }).filter((value) => ( + !value.includes('.') && !value.includes('/') && !value.includes('\\') +)); + +function randomlyCaseExtension(ext: string): fc.Arbitrary { + return fc.array(fc.boolean(), { minLength: ext.length, maxLength: ext.length }) + .map((upperFlags) => ext + .split('') + .map((char, index) => (upperFlags[index] ? char.toUpperCase() : char.toLowerCase())) + .join('')); +} + +function fileWithExtension(name: string, ext: string, type = ''): File { + return new File([''], `${name}.${ext}`, { type }); +} + +describe('timeline file type helper properties', () => { + it('detects representative public media extensions case-insensitively', () => { + fc.assert( + fc.property( + fileStem, + fc.constantFrom(...representativeMediaExtensions).chain((ext) => randomlyCaseExtension(ext)), + (name, ext) => { + expect(isMediaFile(fileWithExtension(name, ext))).toBe(true); + }, + ), + fcOptions, + ); + }); + + it('uses MIME prefixes as media evidence even for unknown extensions', () => { + fc.assert( + fc.property( + fileStem, + fc.constantFrom('video/custom', 'audio/custom', 'image/custom'), + unknownExtension, + (name, type, ext) => { + const file = fileWithExtension(name, ext, type); + + expect(isMediaFile(file)).toBe(true); + expect(isVideoFile(file)).toBe(type.startsWith('video/')); + expect(isAudioFile(file)).toBe(type.startsWith('audio/')); + }, + ), + fcOptions, + ); + }); + + it('keeps representative model and gaussian splat extension classifiers scoped to their public behavior', () => { + fc.assert( + fc.property( + fileStem, + fc.constantFrom( + ...representativeModelExtensions, + ...representativeGaussianSplatExtensions, + ).chain((ext) => randomlyCaseExtension(ext)), + (name, ext) => { + const file = fileWithExtension(name, ext); + const normalizedExt = ext.toLowerCase(); + + expect(isModelFile(file)).toBe( + representativeModelExtensions.includes(normalizedExt as typeof representativeModelExtensions[number]), + ); + expect(isGaussianSplatFile(file)).toBe( + representativeGaussianSplatExtensions.includes( + normalizedExt as typeof representativeGaussianSplatExtensions[number], + ), + ); + }, + ), + fcOptions, + ); + }); + + it('does not classify current unknown sentinel extensions without media MIME evidence', () => { + fc.assert( + fc.property(fileStem, unknownExtension, (name, ext) => { + const file = fileWithExtension(name, ext); + + expect(isMediaFile(file)).toBe(false); + expect(isVideoFile(file)).toBe(false); + expect(isAudioFile(file)).toBe(false); + expect(isModelFile(file)).toBe(false); + expect(isGaussianSplatFile(file)).toBe(false); + }), + fcOptions, + ); + }); + + it('uses the final extension segment for multi-dot names', () => { + fc.assert( + fc.property(fileStem, unknownExtension, (name, ext) => { + const hiddenVideoExtension = new File([''], `${name}.${knownVideoExtension}.${ext}`); + const hiddenAudioExtension = new File([''], `${name}.${knownAudioExtension}.${ext}`); + const finalVideoExtension = new File([''], `${name}.${ext}.${knownVideoExtension}`); + const finalAudioExtension = new File([''], `${name}.${ext}.${knownAudioExtension}`); + + expect(isMediaFile(hiddenVideoExtension)).toBe(false); + expect(isVideoFile(hiddenVideoExtension)).toBe(false); + expect(isMediaFile(hiddenAudioExtension)).toBe(false); + expect(isAudioFile(hiddenAudioExtension)).toBe(false); + + expect(isMediaFile(finalVideoExtension)).toBe(true); + expect(isVideoFile(finalVideoExtension)).toBe(true); + expect(isMediaFile(finalAudioExtension)).toBe(true); + expect(isAudioFile(finalAudioExtension)).toBe(true); + }), + fcOptions, + ); + }); + + it('treats MIME and extension evidence additively when they conflict', () => { + fc.assert( + fc.property(fileStem, (name) => { + const audioNamedVideo = fileWithExtension(name, knownAudioExtension, 'video/custom'); + const videoNamedAudio = fileWithExtension(name, knownVideoExtension, 'audio/custom'); + + expect(isMediaFile(audioNamedVideo)).toBe(true); + expect(isVideoFile(audioNamedVideo)).toBe(true); + expect(isAudioFile(audioNamedVideo)).toBe(true); + + expect(isMediaFile(videoNamedAudio)).toBe(true); + expect(isVideoFile(videoNamedAudio)).toBe(true); + expect(isAudioFile(videoNamedAudio)).toBe(true); + }), + fcOptions, + ); + }); +}); diff --git a/tests/property/keyframeInterpolation.property.test.ts b/tests/property/keyframeInterpolation.property.test.ts new file mode 100644 index 00000000..e82713fe --- /dev/null +++ b/tests/property/keyframeInterpolation.property.test.ts @@ -0,0 +1,311 @@ +import { describe, expect, it } from 'vitest'; +import * as fc from 'fast-check'; +import type { AnimatableProperty, ClipTransform, Keyframe } from '../../src/types'; +import { + getValueFromTransform, + getShortestAngleDeltaDegrees, + interpolateKeyframes, + setValueInTransform, +} from '../../src/utils/keyframeInterpolation'; + +const propertyOptions = { numRuns: 100, seed: 0x4b465001 }; +const finiteValue = fc.double({ min: -1_000_000, max: 1_000_000, noNaN: true, noDefaultInfinity: true }); +const timeValue = fc.double({ min: -10_000, max: 10_000, noNaN: true, noDefaultInfinity: true }); +const interpolationTime = fc.double({ min: 0, max: 1, noNaN: true, noDefaultInfinity: true }); +const angleValue = fc.double({ min: -10_000, max: 10_000, noNaN: true, noDefaultInfinity: true }); +const supportedTransformProperties = [ + 'opacity', + 'position.x', + 'position.y', + 'position.z', + 'scale.all', + 'scale.x', + 'scale.y', + 'scale.z', + 'rotation.x', + 'rotation.y', + 'rotation.z', +] as const satisfies readonly AnimatableProperty[]; +const supportedTransformProperty = fc.constantFrom(...supportedTransformProperties); +const transformValue = fc.double({ min: -10_000, max: 10_000, noNaN: true, noDefaultInfinity: true }); +const transformArbitrary = fc.record({ + opacity: transformValue, + blendMode: fc.constant('normal' as const), + position: fc.record({ + x: transformValue, + y: transformValue, + z: transformValue, + }), + scale: fc.record({ + all: transformValue, + x: transformValue, + y: transformValue, + z: transformValue, + }), + rotation: fc.record({ + x: transformValue, + y: transformValue, + z: transformValue, + }), +}) satisfies fc.Arbitrary; + +function keyframe( + property: AnimatableProperty, + time: number, + value: number, + id = `kf-${property}-${time}`, +): Keyframe { + return { + id, + clipId: 'clip-1', + property, + time, + value, + easing: 'linear', + }; +} + +function cloneTransform(transform: ClipTransform): ClipTransform { + return { + ...transform, + position: { ...transform.position }, + scale: { ...transform.scale }, + rotation: { ...transform.rotation }, + }; +} + +function expectOnlyPropertyChanged( + before: ClipTransform, + after: ClipTransform, + property: AnimatableProperty, + value: number, +): void { + const expected = cloneTransform(before); + + switch (property) { + case 'opacity': + expected.opacity = value; + break; + case 'position.x': + expected.position.x = value; + break; + case 'position.y': + expected.position.y = value; + break; + case 'position.z': + expected.position.z = value; + break; + case 'scale.all': + expected.scale.all = value; + break; + case 'scale.x': + expected.scale.x = value; + break; + case 'scale.y': + expected.scale.y = value; + break; + case 'scale.z': + expected.scale.z = value; + break; + case 'rotation.x': + expected.rotation.x = value; + break; + case 'rotation.y': + expected.rotation.y = value; + break; + case 'rotation.z': + expected.rotation.z = value; + break; + default: + throw new Error(`Unexpected transform property: ${property}`); + } + + expect(after).toEqual(expected); +} + +describe('keyframeInterpolation properties', () => { + it('returns the default value when no keyframes target the requested property', () => { + fc.assert( + fc.property( + fc.array(finiteValue, { maxLength: 20 }), + timeValue, + finiteValue, + (values, time, defaultValue) => { + const otherPropertyKeyframes = values.map((value, index) => + keyframe('scale.x', index, value), + ); + + expect(interpolateKeyframes(otherPropertyKeyframes, 'opacity', time, defaultValue)).toBe(defaultValue); + }, + ), + propertyOptions, + ); + }); + + it('returns the keyed value when exactly one keyframe targets the property', () => { + fc.assert( + fc.property( + finiteValue, + timeValue, + timeValue, + finiteValue, + (value, keyTime, sampleTime, unrelatedValue) => { + const keyframes = [ + keyframe('opacity', keyTime, value), + keyframe('scale.x', keyTime, unrelatedValue), + ]; + + expect(interpolateKeyframes(keyframes, 'opacity', sampleTime, 0)).toBe(value); + }, + ), + propertyOptions, + ); + }); + + it('treats unsorted keyframes the same as sorted keyframes', () => { + fc.assert( + fc.property( + fc.uniqueArray( + fc.record({ + time: fc.integer({ min: -1_000, max: 1_000 }), + value: finiteValue, + }), + { minLength: 2, maxLength: 20, selector: ({ time }) => time }, + ), + timeValue, + (points, sampleTime) => { + const sorted = points + .toSorted((a, b) => a.time - b.time) + .map((point, index) => keyframe('position.x', point.time, point.value, `sorted-${index}`)); + const unsorted = sorted.toReversed(); + + expect(interpolateKeyframes(unsorted, 'position.x', sampleTime, 0)).toBeCloseTo( + interpolateKeyframes(sorted, 'position.x', sampleTime, 0), + ); + }, + ), + propertyOptions, + ); + }); + + it('keeps linear interpolation inside the endpoint value range for times inside a segment', () => { + fc.assert( + fc.property( + fc.integer({ min: -1_000, max: 999 }), + fc.integer({ min: 1, max: 1_000 }), + finiteValue, + finiteValue, + interpolationTime, + (startTime, duration, startValue, endValue, t) => { + const endTime = startTime + duration; + const sampleTime = startTime + duration * t; + const result = interpolateKeyframes( + [ + keyframe('scale.y', startTime, startValue, 'start'), + keyframe('scale.y', endTime, endValue, 'end'), + ], + 'scale.y', + sampleTime, + 1, + ); + const min = Math.min(startValue, endValue); + const max = Math.max(startValue, endValue); + + expect(result).toBeGreaterThanOrEqual(min - 1e-9); + expect(result).toBeLessThanOrEqual(max + 1e-9); + }, + ), + propertyOptions, + ); + }); + + it('always resolves shortest angle deltas within [-180, 180]', () => { + fc.assert( + fc.property(finiteValue, finiteValue, (from, to) => { + const delta = getShortestAngleDeltaDegrees(from, to); + + expect(delta).toBeGreaterThanOrEqual(-180); + expect(delta).toBeLessThanOrEqual(180); + }), + propertyOptions, + ); + }); + + it('round-trips supported transform properties through get/set lenses', () => { + fc.assert( + fc.property( + transformArbitrary, + supportedTransformProperty, + transformValue, + (transform, property, value) => { + const updated = setValueInTransform(transform, property, value); + + expect(getValueFromTransform(updated, property)).toBe(value); + }, + ), + propertyOptions, + ); + }); + + it('sets supported transform properties without mutating the input transform', () => { + fc.assert( + fc.property( + transformArbitrary, + supportedTransformProperty, + transformValue, + (transform, property, value) => { + const before = cloneTransform(transform); + + setValueInTransform(transform, property, value); + + expect(transform).toEqual(before); + }, + ), + propertyOptions, + ); + }); + + it('sets one supported transform property while leaving unrelated fields unchanged', () => { + fc.assert( + fc.property( + transformArbitrary, + supportedTransformProperty, + transformValue, + (transform, property, value) => { + const before = cloneTransform(transform); + const updated = setValueInTransform(transform, property, value); + + expectOnlyPropertyChanged(before, updated, property, value); + }, + ), + propertyOptions, + ); + }); + + it('keeps shortest-angle rotation interpolation within 180 degrees of the segment start', () => { + fc.assert( + fc.property( + angleValue, + angleValue, + interpolationTime, + (startValue, endValue, t) => { + const result = interpolateKeyframes( + [ + { + ...keyframe('rotation.y', 0, startValue, 'start'), + rotationInterpolation: 'shortest', + }, + keyframe('rotation.y', 1, endValue, 'end'), + ], + 'rotation.y', + t, + 0, + ); + + expect(Math.abs(result - startValue)).toBeLessThanOrEqual(180 + 1e-9); + }, + ), + propertyOptions, + ); + }); +}); diff --git a/tests/property/keyframeSlice.property.test.ts b/tests/property/keyframeSlice.property.test.ts new file mode 100644 index 00000000..bd1c72eb --- /dev/null +++ b/tests/property/keyframeSlice.property.test.ts @@ -0,0 +1,205 @@ +import { describe, expect, it } from 'vitest'; +import * as fc from 'fast-check'; +import type { AnimatableProperty, ClipTransform } from '../../src/types'; +import { createTestTimelineStore } from '../helpers/storeFactory'; +import { createMockClip } from '../helpers/mockData'; + +const propertyOptions = { numRuns: 100, seed: 0x4b465002 }; +const durationValue = fc.double({ min: 0.001, max: 10_000, noNaN: true, noDefaultInfinity: true }); +const timeValue = fc.double({ min: -20_000, max: 20_000, noNaN: true, noDefaultInfinity: true }); +const finiteValue = fc.double({ min: -1_000_000, max: 1_000_000, noNaN: true, noDefaultInfinity: true }); +const transformProperties = [ + 'opacity', + 'position.x', + 'position.y', + 'position.z', + 'scale.x', + 'scale.y', + 'rotation.x', + 'rotation.y', + 'rotation.z', +] as const satisfies readonly AnimatableProperty[]; +const transformProperty = fc.constantFrom(...transformProperties); + +function clampTime(time: number, duration: number): number { + return Math.max(0, Math.min(time, duration)); +} + +function createStoreWithClip(duration: number, playheadPosition = 0) { + const clip = createMockClip({ + id: 'clip-1', + trackId: 'video-1', + startTime: 0, + duration, + outPoint: duration, + }); + + return createTestTimelineStore({ clips: [clip], playheadPosition }); +} + +function getTransformValue(transform: ClipTransform, property: AnimatableProperty): number { + switch (property) { + case 'opacity': + return transform.opacity; + case 'position.x': + return transform.position.x; + case 'position.y': + return transform.position.y; + case 'position.z': + return transform.position.z; + case 'scale.x': + return transform.scale.x; + case 'scale.y': + return transform.scale.y; + case 'rotation.x': + return transform.rotation.x; + case 'rotation.y': + return transform.rotation.y; + case 'rotation.z': + return transform.rotation.z; + default: + throw new Error(`Unsupported test property: ${property}`); + } +} + +function expectFiniteKeyframeValues(values: number[]): void { + for (const value of values) { + expect(Number.isFinite(value)).toBe(true); + } +} + +describe('keyframeSlice properties', () => { + it('addKeyframe clamps generated times into the owning clip duration', () => { + fc.assert( + fc.property(durationValue, timeValue, finiteValue, transformProperty, (duration, time, value, property) => { + const store = createStoreWithClip(duration); + + store.getState().addKeyframe('clip-1', property, value, time); + + const keyframes = store.getState().clipKeyframes.get('clip-1') ?? []; + expect(keyframes).toHaveLength(1); + expect(keyframes[0].time).toBeCloseTo(clampTime(time, duration)); + expect(keyframes[0].time).toBeGreaterThanOrEqual(0); + expect(keyframes[0].time).toBeLessThanOrEqual(duration); + }), + propertyOptions, + ); + }); + + it('moveKeyframe clamps generated target times into the owning clip duration', () => { + fc.assert( + fc.property(durationValue, timeValue, timeValue, finiteValue, transformProperty, (duration, initialTime, targetTime, value, property) => { + const store = createStoreWithClip(duration); + store.getState().addKeyframe('clip-1', property, value, initialTime); + const keyframeId = store.getState().clipKeyframes.get('clip-1')![0].id; + + store.getState().moveKeyframe(keyframeId, targetTime); + + const moved = store.getState().clipKeyframes.get('clip-1')!.find(keyframe => keyframe.id === keyframeId); + expect(moved?.time).toBeCloseTo(clampTime(targetTime, duration)); + expect(moved?.time).toBeGreaterThanOrEqual(0); + expect(moved?.time).toBeLessThanOrEqual(duration); + }), + propertyOptions, + ); + }); + + it('moveKeyframes keeps grouped moves within duration and leaves unselected keyframes untouched', () => { + fc.assert( + fc.property( + durationValue, + timeValue, + timeValue, + timeValue, + finiteValue, + finiteValue, + finiteValue, + (duration, firstTime, secondTime, targetTime, firstValue, secondValue, untouchedValue) => { + const store = createStoreWithClip(duration); + store.getState().addKeyframe('clip-1', 'opacity', firstValue, firstTime); + store.getState().addKeyframe('clip-1', 'scale.x', secondValue, secondTime); + store.getState().addKeyframe('clip-1', 'rotation.z', untouchedValue, duration / 2); + + const keyframesBefore = store.getState().clipKeyframes.get('clip-1')!; + const selectedIds = keyframesBefore + .filter(keyframe => keyframe.property === 'opacity' || keyframe.property === 'scale.x') + .map(keyframe => keyframe.id); + const untouched = keyframesBefore.find(keyframe => keyframe.property === 'rotation.z')!; + + store.getState().moveKeyframes(selectedIds, targetTime); + + const keyframesAfter = store.getState().clipKeyframes.get('clip-1')!; + const clampedTarget = clampTime(targetTime, duration); + const selectedAfter = keyframesAfter.filter(keyframe => selectedIds.includes(keyframe.id)); + expect(selectedAfter).toHaveLength(2); + for (const keyframe of selectedAfter) { + expect(keyframe.time).toBeCloseTo(clampedTarget); + expect(keyframe.time).toBeGreaterThanOrEqual(0); + expect(keyframe.time).toBeLessThanOrEqual(duration); + } + + const untouchedAfter = keyframesAfter.find(keyframe => keyframe.id === untouched.id)!; + expect(untouchedAfter.time).toBe(untouched.time); + expect(keyframesAfter.every(keyframe => keyframe.time >= 0 && keyframe.time <= duration)).toBe(true); + }, + ), + propertyOptions, + ); + }); + + it('setPropertyValue while recording stores finite bounded values as keyframes', () => { + fc.assert( + fc.property(durationValue, timeValue, finiteValue, transformProperty, (duration, playheadPosition, value, property) => { + const store = createStoreWithClip(duration, playheadPosition); + store.getState().toggleKeyframeRecording('clip-1', property); + + store.getState().setPropertyValue('clip-1', property, value); + + const keyframes = store.getState().clipKeyframes.get('clip-1') ?? []; + expect(keyframes).toHaveLength(1); + expect(keyframes[0].property).toBe(property); + expect(keyframes[0].value).toBe(value); + expect(keyframes[0].time).toBeCloseTo(clampTime(playheadPosition, duration)); + expectFiniteKeyframeValues(keyframes.map(keyframe => keyframe.value)); + }), + propertyOptions, + ); + }); + + it('setPropertyValue with existing keyframes updates or appends finite bounded values', () => { + fc.assert( + fc.property(durationValue, timeValue, finiteValue, finiteValue, transformProperty, (duration, playheadPosition, initialValue, nextValue, property) => { + const store = createStoreWithClip(duration, playheadPosition); + store.getState().addKeyframe('clip-1', property, initialValue, 0); + + store.getState().setPropertyValue('clip-1', property, nextValue); + + const keyframes = (store.getState().clipKeyframes.get('clip-1') ?? []) + .filter(keyframe => keyframe.property === property); + expect(keyframes.length).toBeGreaterThanOrEqual(1); + expect(keyframes.some(keyframe => keyframe.value === nextValue)).toBe(true); + expect(keyframes.every(keyframe => keyframe.time >= 0 && keyframe.time <= duration)).toBe(true); + expectFiniteKeyframeValues(keyframes.map(keyframe => keyframe.value)); + }), + propertyOptions, + ); + }); + + it('disablePropertyKeyframes removes the property keyframes and preserves the provided current transform value', () => { + fc.assert( + fc.property(durationValue, timeValue, finiteValue, finiteValue, transformProperty, (duration, time, keyedValue, currentValue, property) => { + const store = createStoreWithClip(duration); + store.getState().addKeyframe('clip-1', property, keyedValue, time); + + store.getState().disablePropertyKeyframes('clip-1', property, currentValue); + + const remaining = store.getState().clipKeyframes.get('clip-1') ?? []; + expect(remaining.some(keyframe => keyframe.property === property)).toBe(false); + const clip = store.getState().clips.find(candidate => candidate.id === 'clip-1')!; + expect(getTransformValue(clip.transform, property)).toBe(currentValue); + expect(store.getState().isRecording('clip-1', property)).toBe(false); + }), + propertyOptions, + ); + }); +}); diff --git a/tests/property/playbackRange.property.test.ts b/tests/property/playbackRange.property.test.ts new file mode 100644 index 00000000..76d960dc --- /dev/null +++ b/tests/property/playbackRange.property.test.ts @@ -0,0 +1,93 @@ +import fc from 'fast-check'; +import { describe, expect, it } from 'vitest'; +import { resolvePlaybackStartPosition } from '../../src/stores/timeline/playbackRange'; + +const fcOptions = { + numRuns: 200, + seed: 20260518, +}; + +const finiteTime = fc.double({ + min: -10_000, + max: 10_000, + noDefaultInfinity: true, + noNaN: true, +}); + +const nullableFiniteTime = fc.oneof(finiteTime, fc.constant(null)); + +function safeDuration(duration: number): number { + return Math.max(0, Number.isFinite(duration) ? duration : 0); +} + +function sanitize(value: number, fallback = 0): number { + return Number.isFinite(value) ? value : fallback; +} + +function resolvedRange(inPoint: number | null, outPoint: number | null, duration: number): { start: number; end: number } { + const limit = safeDuration(duration); + const start = Math.max(0, Math.min(inPoint ?? 0, limit)); + const end = Math.max(start, Math.min(outPoint ?? limit, limit)); + return { start, end }; +} + +describe('playback range properties', () => { + it('always resolves playback start to a finite time inside the safe duration', () => { + fc.assert( + fc.property(finiteTime, nullableFiniteTime, nullableFiniteTime, finiteTime, finiteTime, ( + playheadPosition, + inPoint, + outPoint, + duration, + playbackSpeed, + ) => { + const resolved = resolvePlaybackStartPosition(playheadPosition, inPoint, outPoint, duration, playbackSpeed); + + expect(Number.isFinite(resolved)).toBe(true); + expect(resolved).toBeGreaterThanOrEqual(0); + expect(resolved).toBeLessThanOrEqual(safeDuration(duration)); + }), + fcOptions, + ); + }); + + it('returns the clamped playhead when no in/out range is active', () => { + fc.assert( + fc.property(finiteTime, finiteTime, finiteTime, (playheadPosition, duration, playbackSpeed) => { + const limit = safeDuration(duration); + const expected = Math.max(0, Math.min(sanitize(playheadPosition), limit)); + + expect(resolvePlaybackStartPosition(playheadPosition, null, null, duration, playbackSpeed)).toBe(expected); + }), + fcOptions, + ); + }); + + it('starts forward playback at range start when the playhead is outside the active range', () => { + fc.assert( + fc.property(finiteTime, nullableFiniteTime, nullableFiniteTime, finiteTime, (duration, inPoint, outPoint, playhead) => { + const range = resolvedRange(inPoint, outPoint, duration); + const clampedPlayhead = Math.max(0, Math.min(sanitize(playhead, range.start), safeDuration(duration))); + fc.pre(inPoint !== null || outPoint !== null); + fc.pre(clampedPlayhead < range.start || clampedPlayhead >= range.end); + + expect(resolvePlaybackStartPosition(playhead, inPoint, outPoint, duration, 1)).toBe(range.start); + }), + fcOptions, + ); + }); + + it('starts reverse playback at range end when the playhead is outside the active range', () => { + fc.assert( + fc.property(finiteTime, nullableFiniteTime, nullableFiniteTime, finiteTime, (duration, inPoint, outPoint, playhead) => { + const range = resolvedRange(inPoint, outPoint, duration); + const clampedPlayhead = Math.max(0, Math.min(sanitize(playhead, range.start), safeDuration(duration))); + fc.pre(inPoint !== null || outPoint !== null); + fc.pre(clampedPlayhead <= range.start || clampedPlayhead > range.end); + + expect(resolvePlaybackStartPosition(playhead, inPoint, outPoint, duration, -1)).toBe(range.end); + }), + fcOptions, + ); + }); +}); diff --git a/tests/property/speedIntegration.property.test.ts b/tests/property/speedIntegration.property.test.ts new file mode 100644 index 00000000..e9b1b8c0 --- /dev/null +++ b/tests/property/speedIntegration.property.test.ts @@ -0,0 +1,227 @@ +import { describe, expect, it } from 'vitest'; +import fc from 'fast-check'; +import type { AnimatableProperty, EasingType, Keyframe } from '../../src/types'; +import { + calculateSourceTime, + calculateTimelineDuration, + calculateTotalSourceTime, + getMaxSpeed, + hasReverseSpeed, +} from '../../src/utils/speedIntegration'; + +const RUN_OPTIONS = { numRuns: 100, seed: 0x5eed02 }; +const EPSILON = 1e-9; + +const finiteSeconds = fc.double({ + min: 0, + max: 120, + noNaN: true, + noDefaultInfinity: true, +}); + +const finiteSpeed = fc.double({ + min: -16, + max: 16, + noNaN: true, + noDefaultInfinity: true, +}); + +const positiveSpeed = fc.double({ + min: 0.25, + max: 16, + noNaN: true, + noDefaultInfinity: true, +}); + +const sourceDuration = fc.double({ + min: 0.01, + max: 30, + noNaN: true, + noDefaultInfinity: true, +}); + +const nonSpeedProperty = fc.constantFrom( + 'opacity', + 'position.x', + 'position.y', + 'scale.x', + 'rotation.z', + 'camera.fov', +); + +const easing = fc.constantFrom( + 'linear', + 'ease-in', + 'ease-out', + 'ease-in-out', +); + +function keyframe( + index: number, + property: AnimatableProperty, + time: number, + value: number, + easingValue: EasingType = 'linear', +): Keyframe { + return { + id: `kf-${index}`, + clipId: 'clip-1', + time, + property, + value, + easing: easingValue, + }; +} + +const nonSpeedKeyframes = fc.array( + fc.record({ + property: nonSpeedProperty, + time: finiteSeconds, + value: finiteSpeed, + easing, + }), + { maxLength: 12 }, +).map(items => items.map((item, index) => keyframe(index, item.property, item.time, item.value, item.easing))); + +const mixedKeyframes = fc.array( + fc.record({ + property: fc.oneof(fc.constant('speed'), nonSpeedProperty), + time: finiteSeconds, + value: finiteSpeed, + easing, + }), + { maxLength: 16 }, +).map(items => items.map((item, index) => keyframe(index, item.property, item.time, item.value, item.easing))); + +const positiveSpeedKeyframes = fc.uniqueArray( + fc.record({ + timeTick: fc.integer({ min: 0, max: 400 }), + valueTick: fc.integer({ min: 250, max: 16000 }), + }), + { minLength: 2, maxLength: 8, selector: item => item.timeTick }, +).map(items => + items.map((item, index) => + keyframe(index, 'speed', item.timeTick / 10, item.valueTick / 1000, 'linear') + ) +); + +describe('speedIntegration properties', () => { + it('uses clip local time multiplied by default speed when no speed keyframes exist', () => { + fc.assert( + fc.property(nonSpeedKeyframes, finiteSeconds, finiteSpeed, (keyframes, clipLocalTime, defaultSpeed) => { + expect(calculateSourceTime(keyframes, clipLocalTime, defaultSpeed)).toBe(clipLocalTime * defaultSpeed); + expect(calculateTotalSourceTime(keyframes, clipLocalTime, defaultSpeed)).toBe(clipLocalTime * defaultSpeed); + }), + RUN_OPTIONS, + ); + }); + + it('uses clip local time multiplied by the single speed keyframe value when the speed keyframe is at time 0', () => { + fc.assert( + fc.property( + nonSpeedKeyframes, + finiteSpeed, + finiteSpeed, + finiteSeconds, + (otherKeyframes, speedValue, defaultSpeed, clipLocalTime) => { + const speedKeyframe = keyframe(otherKeyframes.length, 'speed', 0, speedValue); + const keyframes = [...otherKeyframes, speedKeyframe]; + + expect(calculateSourceTime(keyframes, clipLocalTime, defaultSpeed)).toBe(clipLocalTime * speedValue); + expect(calculateTotalSourceTime(keyframes, clipLocalTime, defaultSpeed)).toBe(clipLocalTime * speedValue); + }, + ), + RUN_OPTIONS, + ); + }); + + it('calculates equivalent source time for sorted and unsorted speed keyframes', () => { + fc.assert( + fc.property(positiveSpeedKeyframes, positiveSpeed, finiteSeconds, (keyframes, defaultSpeed, clipLocalTime) => { + const sortedKeyframes = keyframes.toSorted((a, b) => a.time - b.time); + const unsortedKeyframes = [...sortedKeyframes].reverse(); + + expect(calculateSourceTime(unsortedKeyframes, clipLocalTime, defaultSpeed)).toBeCloseTo( + calculateSourceTime(sortedKeyframes, clipLocalTime, defaultSpeed), + 9, + ); + expect(calculateTotalSourceTime(unsortedKeyframes, clipLocalTime, defaultSpeed)).toBeCloseTo( + calculateTotalSourceTime(sortedKeyframes, clipLocalTime, defaultSpeed), + 9, + ); + }), + RUN_OPTIONS, + ); + }); + + it('ignores non-speed keyframes when calculating source time', () => { + fc.assert( + fc.property(nonSpeedKeyframes, finiteSeconds, finiteSpeed, (keyframes, clipLocalTime, defaultSpeed) => { + expect(calculateSourceTime(keyframes, clipLocalTime, defaultSpeed)).toBe( + calculateSourceTime([], clipLocalTime, defaultSpeed), + ); + }), + RUN_OPTIONS, + ); + }); + + it('keeps source time nondecreasing for positive default and positive speed keyframes', () => { + fc.assert( + fc.property( + positiveSpeedKeyframes, + positiveSpeed, + finiteSeconds, + finiteSeconds, + (keyframes, defaultSpeed, timeA, timeB) => { + const earlier = Math.min(timeA, timeB); + const later = Math.max(timeA, timeB); + const sourceEarlier = calculateSourceTime(keyframes, earlier, defaultSpeed); + const sourceLater = calculateSourceTime(keyframes, later, defaultSpeed); + + expect(sourceLater + EPSILON).toBeGreaterThanOrEqual(sourceEarlier); + }, + ), + RUN_OPTIONS, + ); + }); + + it('round-trips positive source durations through calculateTimelineDuration', () => { + fc.assert( + fc.property(positiveSpeedKeyframes, sourceDuration, positiveSpeed, (keyframes, duration, defaultSpeed) => { + const timelineDuration = calculateTimelineDuration(keyframes, duration, defaultSpeed); + const recoveredSourceDuration = calculateSourceTime(keyframes, timelineDuration, defaultSpeed); + + expect(recoveredSourceDuration).toBeCloseTo(duration, 2); + }), + RUN_OPTIONS, + ); + }); + + it('reports a max speed at least as large as the absolute default and every speed keyframe value', () => { + fc.assert( + fc.property(mixedKeyframes, finiteSpeed, (keyframes, defaultSpeed) => { + const maxSpeed = getMaxSpeed(keyframes, defaultSpeed); + + expect(maxSpeed).toBeGreaterThanOrEqual(Math.abs(defaultSpeed)); + for (const speedKeyframe of keyframes.filter(kf => kf.property === 'speed')) { + expect(maxSpeed).toBeGreaterThanOrEqual(Math.abs(speedKeyframe.value)); + } + }), + RUN_OPTIONS, + ); + }); + + it('reports reverse speed from the default only when no speed keyframes exist, otherwise from negative speed keyframes', () => { + fc.assert( + fc.property(mixedKeyframes, finiteSpeed, (keyframes, defaultSpeed) => { + const speedKeyframes = keyframes.filter(kf => kf.property === 'speed'); + const expected = speedKeyframes.length === 0 + ? defaultSpeed < 0 + : speedKeyframes.some(kf => kf.value < 0); + + expect(hasReverseSpeed(keyframes, defaultSpeed)).toBe(expected); + }), + RUN_OPTIONS, + ); + }); +}); diff --git a/tests/property/stopMarkers.property.test.ts b/tests/property/stopMarkers.property.test.ts new file mode 100644 index 00000000..5cbb4b30 --- /dev/null +++ b/tests/property/stopMarkers.property.test.ts @@ -0,0 +1,155 @@ +import fc from 'fast-check'; +import { describe, expect, it } from 'vitest'; +import { findStopMarkerInPlaybackRange } from '../../src/services/timeline/stopMarkers'; +import type { TimelineMarker } from '../../src/stores/timeline/types'; + +const STOP_MARKER_EPSILON = 1 / 240; + +const fcOptions = { + numRuns: 200, + seed: 20260518, +}; + +const finiteTime = fc.double({ + min: -10_000, + max: 10_000, + noDefaultInfinity: true, + noNaN: true, +}); + +const positiveMovement = fc.double({ + min: STOP_MARKER_EPSILON * 2, + max: 500, + noDefaultInfinity: true, + noNaN: true, +}); + +const tinyMovement = fc.double({ + min: -STOP_MARKER_EPSILON, + max: STOP_MARKER_EPSILON, + noDefaultInfinity: true, + noNaN: true, +}); + +const nonFiniteTime = fc.constantFrom( + Number.NaN, + Number.POSITIVE_INFINITY, + Number.NEGATIVE_INFINITY, +); + +function createMarker(index: number, time: number, stopPlayback?: boolean): TimelineMarker { + return { + id: `marker-${index}`, + time, + label: `Marker ${index}`, + color: '#ff3366', + stopPlayback, + }; +} + +const markerInput = fc.record({ + time: finiteTime, + stopPlayback: fc.option(fc.boolean(), { nil: undefined }), +}); + +const markerArray = fc.array(markerInput, { minLength: 0, maxLength: 40 }).map((inputs) => ( + inputs + .map((input, index) => createMarker(index, input.time, input.stopPlayback)) + .toSorted((a, b) => a.time - b.time) +)); + +const nonStopMarkerArray = fc.array(finiteTime, { minLength: 0, maxLength: 40 }).map((times) => ( + times + .map((time, index) => createMarker(index, time, index % 2 === 0 ? false : undefined)) + .toSorted((a, b) => a.time - b.time) +)); + +describe('stop marker playback range properties', () => { + it('forward playback picks the earliest crossed stop marker in source bounds', () => { + fc.assert( + fc.property(markerArray, finiteTime, positiveMovement, (markers, fromTime, movement) => { + const toTime = fromTime + movement; + const expected = markers.find((marker) => ( + marker.stopPlayback === true + && marker.time > fromTime + STOP_MARKER_EPSILON + && marker.time <= toTime + STOP_MARKER_EPSILON + )) ?? null; + + const actual = findStopMarkerInPlaybackRange(markers, fromTime, toTime); + + expect(actual).toBe(expected); + if (actual) { + expect(markers.every((marker) => ( + marker === actual + || marker.stopPlayback !== true + || marker.time <= fromTime + STOP_MARKER_EPSILON + || marker.time > actual.time + ))).toBe(true); + } + }), + fcOptions, + ); + }); + + it('reverse playback picks the nearest crossed stop marker in source bounds', () => { + fc.assert( + fc.property(markerArray, finiteTime, positiveMovement, (markers, toTime, movement) => { + const fromTime = toTime + movement; + const expected = markers.findLast((marker) => ( + marker.stopPlayback === true + && marker.time < fromTime - STOP_MARKER_EPSILON + && marker.time >= toTime - STOP_MARKER_EPSILON + )) ?? null; + + const actual = findStopMarkerInPlaybackRange(markers, fromTime, toTime); + + expect(actual).toBe(expected); + if (actual) { + expect(markers.every((marker) => ( + marker === actual + || marker.stopPlayback !== true + || marker.time >= fromTime - STOP_MARKER_EPSILON + || marker.time < actual.time + ))).toBe(true); + } + }), + fcOptions, + ); + }); + + it('ignores markers that are not explicit stopPlayback markers', () => { + fc.assert( + fc.property(nonStopMarkerArray, finiteTime, finiteTime, (markers, fromTime, toTime) => { + expect(findStopMarkerInPlaybackRange(markers, fromTime, toTime)).toBeNull(); + }), + fcOptions, + ); + }); + + it('returns null for movement at or below epsilon', () => { + fc.assert( + fc.property(markerArray, finiteTime, tinyMovement, (markers, fromTime, movement) => { + const toTime = fromTime + movement; + + expect(findStopMarkerInPlaybackRange(markers, fromTime, toTime)).toBeNull(); + }), + fcOptions, + ); + }); + + it('returns null for non-finite range endpoints', () => { + fc.assert( + fc.property(markerArray, nonFiniteTime, finiteTime, (markers, fromTime, toTime) => { + expect(findStopMarkerInPlaybackRange(markers, fromTime, toTime)).toBeNull(); + }), + fcOptions, + ); + + fc.assert( + fc.property(markerArray, finiteTime, nonFiniteTime, (markers, fromTime, toTime) => { + expect(findStopMarkerInPlaybackRange(markers, fromTime, toTime)).toBeNull(); + }), + fcOptions, + ); + }); +}); diff --git a/tests/property/transformComposition.property.test.ts b/tests/property/transformComposition.property.test.ts new file mode 100644 index 00000000..5f306756 --- /dev/null +++ b/tests/property/transformComposition.property.test.ts @@ -0,0 +1,307 @@ +import { describe, expect, it } from 'vitest'; +import fc from 'fast-check'; +import { composeTransforms, wouldCreateCycle } from '../../src/utils/transformComposition'; +import type { BlendMode, ClipTransform } from '../../src/types'; + +const RUN_OPTIONS = { numRuns: 100, seed: 20260518 }; +const EPSILON = 1e-8; + +const blendModes: BlendMode[] = [ + 'normal', + 'multiply', + 'screen', + 'overlay', + 'difference', + 'alpha-add', +]; + +const finiteNumber = fc.double({ + min: -1_000, + max: 1_000, + noNaN: true, + noDefaultInfinity: true, +}); + +const rotationNumber = fc.double({ + min: -720, + max: 720, + noNaN: true, + noDefaultInfinity: true, +}); + +const scaleNumber = fc.double({ + min: -10, + max: 10, + noNaN: true, + noDefaultInfinity: true, +}); + +const optionalScaleNumber = fc.option(scaleNumber, { nil: undefined }); + +const transformArbitrary: fc.Arbitrary = fc.record({ + opacity: fc.double({ min: 0, max: 1, noNaN: true, noDefaultInfinity: true }), + blendMode: fc.constantFrom(...blendModes), + position: fc.record({ + x: finiteNumber, + y: finiteNumber, + z: finiteNumber, + }), + scale: fc.record({ + all: optionalScaleNumber, + x: scaleNumber, + y: scaleNumber, + z: optionalScaleNumber, + }), + rotation: fc.record({ + x: rotationNumber, + y: rotationNumber, + z: rotationNumber, + }), +}); + +function identityTransform(): ClipTransform { + return { + opacity: 1, + blendMode: 'normal', + position: { x: 0, y: 0, z: 0 }, + scale: { x: 1, y: 1 }, + rotation: { x: 0, y: 0, z: 0 }, + }; +} + +function expectClose(actual: number, expected: number, epsilon = EPSILON) { + expect(Math.abs(actual - expected)).toBeLessThanOrEqual(epsilon); +} + +function expectTransformClose(actual: ClipTransform, expected: ClipTransform) { + expectClose(actual.opacity, expected.opacity); + expect(actual.blendMode).toBe(expected.blendMode); + expectClose(actual.position.x, expected.position.x); + expectClose(actual.position.y, expected.position.y); + expectClose(actual.position.z, expected.position.z); + expectClose(actual.scale.all ?? 1, expected.scale.all ?? 1); + expectClose(actual.scale.x, expected.scale.x); + expectClose(actual.scale.y, expected.scale.y); + expectClose(actual.scale.z ?? 1, expected.scale.z ?? 1); + expectClose(actual.rotation.x, expected.rotation.x); + expectClose(actual.rotation.y, expected.rotation.y); + expectClose(actual.rotation.z, expected.rotation.z); +} + +function cloneTransform(transform: ClipTransform): ClipTransform { + return { + opacity: transform.opacity, + blendMode: transform.blendMode, + position: { ...transform.position }, + scale: { ...transform.scale }, + rotation: { ...transform.rotation }, + }; +} + +function numericFields(transform: ClipTransform) { + return [ + transform.opacity, + transform.position.x, + transform.position.y, + transform.position.z, + transform.scale.all, + transform.scale.x, + transform.scale.y, + transform.scale.z, + transform.rotation.x, + transform.rotation.y, + transform.rotation.z, + ].filter((value): value is number => value !== undefined); +} + +type ParentMapCase = { + ids: string[]; + parents: Record; +}; + +const parentMapArbitrary: fc.Arbitrary = fc + .integer({ min: 2, max: 8 }) + .chain((count) => + fc + .array(fc.integer({ min: 0, max: 100 }), { minLength: count, maxLength: count }) + .map((rawParents) => { + const ids = Array.from({ length: count }, (_, index) => `clip-${index}`); + const parents: Record = {}; + + ids.forEach((id, index) => { + const normalizedParentSlot = rawParents[index] % (index + 1); + parents[id] = + normalizedParentSlot === 0 ? undefined : ids[normalizedParentSlot - 1]; + }); + + return { ids, parents }; + }) + ); + +function parentLookup(parents: Record) { + return (id: string) => parents[id]; +} + +function chainReaches( + parents: Record, + startId: string, + targetId: string +): boolean { + let currentId: string | undefined = startId; + const visited = new Set(); + + while (currentId && !visited.has(currentId)) { + if (currentId === targetId) { + return true; + } + visited.add(currentId); + currentId = parents[currentId]; + } + + return false; +} + +describe('composeTransforms properties', () => { + it('identity parent preserves child transform semantics', () => { + fc.assert( + fc.property(transformArbitrary, (child) => { + const result = composeTransforms(identityTransform(), child); + + expectClose(result.opacity, child.opacity); + expect(result.blendMode).toBe(child.blendMode); + expectClose(result.position.x, child.position.x); + expectClose(result.position.y, child.position.y); + expectClose(result.position.z, child.position.z); + expectClose(result.scale.all ?? 1, child.scale.all ?? 1); + expectClose(result.scale.x, child.scale.x); + expectClose(result.scale.y, child.scale.y); + expectClose(result.scale.z ?? 1, child.scale.z ?? 1); + expectClose(result.rotation.x, child.rotation.x); + expectClose(result.rotation.y, child.rotation.y); + expectClose(result.rotation.z, child.rotation.z); + }), + RUN_OPTIONS + ); + }); + + it('finite input transforms compose to finite numeric fields', () => { + fc.assert( + fc.property(transformArbitrary, transformArbitrary, (parent, child) => { + const result = composeTransforms(parent, child); + + expect(numericFields(result).every(Number.isFinite)).toBe(true); + }), + RUN_OPTIONS + ); + }); + + it('does not mutate parent or child transforms', () => { + fc.assert( + fc.property(transformArbitrary, transformArbitrary, (parent, child) => { + const originalParent = cloneTransform(parent); + const originalChild = cloneTransform(child); + + composeTransforms(parent, child); + + expect(parent).toEqual(originalParent); + expect(child).toEqual(originalChild); + }), + RUN_OPTIONS + ); + }); + + it('is associative for the implemented parent-child composition semantics', () => { + fc.assert( + fc.property(transformArbitrary, transformArbitrary, transformArbitrary, (a, b, c) => { + const left = composeTransforms(composeTransforms(a, b), c); + const right = composeTransforms(a, composeTransforms(b, c)); + + expectTransformClose(left, right); + }), + RUN_OPTIONS + ); + }); + + it('parent scale does not change composed child position', () => { + fc.assert( + fc.property( + transformArbitrary, + scaleNumber, + scaleNumber, + optionalScaleNumber, + (parent, scaleX, scaleY, scaleAll) => { + const scaledParent: ClipTransform = { + ...parent, + scale: { ...parent.scale, all: scaleAll, x: scaleX, y: scaleY }, + }; + + const result = composeTransforms(parent, identityTransform()); + const scaledResult = composeTransforms(scaledParent, identityTransform()); + + expectClose(scaledResult.position.x, result.position.x); + expectClose(scaledResult.position.y, result.position.y); + expectClose(scaledResult.position.z, result.position.z); + } + ), + RUN_OPTIONS + ); + }); +}); + +describe('wouldCreateCycle properties', () => { + it('returns false for safe parent assignments in generated acyclic parent maps', () => { + fc.assert( + fc.property(parentMapArbitrary, ({ ids, parents }) => { + const getParentId = parentLookup(parents); + + ids.forEach((clipId) => { + ids.forEach((parentId) => { + if (parentId !== clipId && !chainReaches(parents, parentId, clipId)) { + expect(wouldCreateCycle(clipId, parentId, getParentId)).toBe(false); + } + }); + }); + }), + RUN_OPTIONS + ); + }); + + it('returns true when the candidate parent chain already reaches the clip', () => { + fc.assert( + fc.property(parentMapArbitrary, ({ ids, parents }) => { + const getParentId = parentLookup(parents); + + ids.forEach((clipId) => { + ids.forEach((parentId) => { + if (parentId !== clipId && chainReaches(parents, parentId, clipId)) { + expect(wouldCreateCycle(clipId, parentId, getParentId)).toBe(true); + } + }); + }); + }), + RUN_OPTIONS + ); + }); + + it('ignores unrelated cycles when the queried parent chain does not reach the clip', () => { + fc.assert( + fc.property(parentMapArbitrary, ({ ids, parents }) => { + const clipId = 'queried-clip'; + const unrelatedCycleA = 'unrelated-cycle-a'; + const unrelatedCycleB = 'unrelated-cycle-b'; + const parentsWithUnrelatedCycle: Record = { + ...parents, + [clipId]: undefined, + [unrelatedCycleA]: unrelatedCycleB, + [unrelatedCycleB]: unrelatedCycleA, + }; + const getParentId = parentLookup(parentsWithUnrelatedCycle); + + ids.forEach((parentId) => { + expect(wouldCreateCycle(clipId, parentId, getParentId)).toBe(false); + }); + }), + RUN_OPTIONS + ); + }); +}); diff --git a/tests/property/transformScale.property.test.ts b/tests/property/transformScale.property.test.ts new file mode 100644 index 00000000..339e7f90 --- /dev/null +++ b/tests/property/transformScale.property.test.ts @@ -0,0 +1,115 @@ +import fc from 'fast-check'; +import { describe, expect, it } from 'vitest'; +import { getEffectiveCameraScale, getEffectiveScale, getScaleAll } from '../../src/utils/transformScale'; + +const propertyConfig = { numRuns: 100, seed: 20260518 }; +const finiteScale = fc.double({ min: -1_000, max: 1_000, noNaN: true, noDefaultInfinity: true }); +const invalidNumber = fc.constantFrom(NaN, Infinity, -Infinity, null, '2', {}, []); + +type ScaleInput = { + all?: unknown; + x?: unknown; + y?: unknown; + z?: unknown; +}; + +type ScaleArg = Parameters[0]; + +function fallbackNumber(value: unknown, fallback: number): number { + return typeof value === 'number' && Number.isFinite(value) ? value : fallback; +} + +function asScaleArg(scale: ScaleInput): ScaleArg { + return scale as ScaleArg; +} + +describe('transformScale property invariants', () => { + it('returns scale.all exactly for finite values and defaults absent or invalid values to 1', () => { + fc.assert( + fc.property(finiteScale, (all) => { + expect(getScaleAll({ all })).toBe(all); + }), + propertyConfig, + ); + + fc.assert( + fc.property(fc.option(invalidNumber, { nil: undefined }), (all) => { + expect(getScaleAll(asScaleArg({ all }))).toBe(1); + }), + propertyConfig, + ); + }); + + it('multiplies finite axis scale by finite uniform scale and falls back per missing or invalid axis', () => { + fc.assert( + fc.property( + fc.record( + { + all: fc.oneof(finiteScale, invalidNumber), + x: fc.oneof(finiteScale, invalidNumber), + y: fc.oneof(finiteScale, invalidNumber), + z: fc.option(fc.oneof(finiteScale, invalidNumber), { nil: undefined }), + }, + { requiredKeys: [] }, + ), + (scale) => { + const all = fallbackNumber(scale.all, 1); + const effective = getEffectiveScale(asScaleArg(scale)); + const expected = { + x: fallbackNumber(scale.x, 1) * all, + y: fallbackNumber(scale.y, 1) * all, + ...(scale.z !== undefined ? { z: fallbackNumber(scale.z, 1) * all } : {}), + }; + + expect(effective).toEqual(expected); + expect(Number.isFinite(effective.x)).toBe(true); + expect(Number.isFinite(effective.y)).toBe(true); + if (scale.z !== undefined) { + expect(Number.isFinite(effective.z)).toBe(true); + } + }, + ), + propertyConfig, + ); + }); + + it('keeps camera x/y consistent with effective scale while leaving camera z unscaled by all', () => { + fc.assert( + fc.property( + fc.record( + { + all: fc.oneof(finiteScale, invalidNumber), + x: fc.oneof(finiteScale, invalidNumber), + y: fc.oneof(finiteScale, invalidNumber), + z: fc.option(fc.oneof(finiteScale, invalidNumber), { nil: undefined }), + }, + { requiredKeys: [] }, + ), + (scale) => { + const effective = getEffectiveScale(asScaleArg(scale)); + const camera = getEffectiveCameraScale(asScaleArg(scale)); + + expect(camera.x).toBe(effective.x); + expect(camera.y).toBe(effective.y); + expect('z' in camera).toBe(scale.z !== undefined); + + if (scale.z !== undefined) { + expect(camera.z).toBe(fallbackNumber(scale.z, 0)); + expect(Number.isFinite(camera.z)).toBe(true); + } + }, + ), + propertyConfig, + ); + }); + + it('treats uniform scale as equivalent x and y axis scale when axes are omitted', () => { + fc.assert( + fc.property(finiteScale, (all) => { + expect(getEffectiveScale({ all })).toEqual({ x: all, y: all }); + expect(getEffectiveCameraScale({ all })).toEqual({ x: all, y: all }); + }), + propertyConfig, + ); + }); +}); diff --git a/tests/stores/timeline/clipSlice.test.ts b/tests/stores/timeline/clipSlice.test.ts index 5348d24d..321b5adc 100644 --- a/tests/stores/timeline/clipSlice.test.ts +++ b/tests/stores/timeline/clipSlice.test.ts @@ -358,6 +358,20 @@ describe('clipSlice', () => { // ========== moveClip ========== describe('moveClip', () => { + it('blocks clip movement while an export is active', () => { + const clip = createMockClip({ id: 'clip-1', trackId: 'video-1', startTime: 0, duration: 5 }); + store = createTestTimelineStore({ + clips: [clip], + isExporting: true, + snappingEnabled: false, + }); + + store.getState().moveClip('clip-1', 10); + + const blocked = store.getState().clips.find(c => c.id === 'clip-1')!; + expect(blocked.startTime).toBe(0); + }); + it('moves a clip to a new start time on the same track', () => { const clip = createMockClip({ id: 'clip-1', trackId: 'video-1', startTime: 0, duration: 5 }); store = createTestTimelineStore({ diff --git a/tests/unit/audioFileEncoder.test.ts b/tests/unit/audioFileEncoder.test.ts new file mode 100644 index 00000000..ed09d069 --- /dev/null +++ b/tests/unit/audioFileEncoder.test.ts @@ -0,0 +1,101 @@ +import { describe, expect, it } from 'vitest'; +import { + encodeAudioBufferToWavBytes, + estimateWavByteSize, + type AudioBufferLike, +} from '../../src/engine/audio/AudioFileEncoder'; +import { + createDefaultExportSettings, + useExportStore, +} from '../../src/stores/exportStore'; + +function createAudioBufferLike(options: { + sampleRate?: number; + channels: Float32Array[]; +}): AudioBufferLike { + const sampleRate = options.sampleRate ?? 48000; + const length = options.channels[0]?.length ?? 0; + + return { + sampleRate, + numberOfChannels: options.channels.length, + length, + getChannelData: (channel) => options.channels[channel] ?? new Float32Array(length), + }; +} + +function ascii(bytes: Uint8Array, offset: number, length: number): string { + return String.fromCharCode(...bytes.slice(offset, offset + length)); +} + +describe('AudioFileEncoder WAV', () => { + it('writes a valid 16-bit PCM WAV header and interleaved samples', () => { + const buffer = createAudioBufferLike({ + sampleRate: 8000, + channels: [ + new Float32Array([0, 1, -1]), + new Float32Array([0.5, -0.5, 2]), + ], + }); + + const bytes = encodeAudioBufferToWavBytes(buffer); + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + + expect(bytes.byteLength).toBe(44 + 3 * 2 * 2); + expect(ascii(bytes, 0, 4)).toBe('RIFF'); + expect(view.getUint32(4, true)).toBe(bytes.byteLength - 8); + expect(ascii(bytes, 8, 4)).toBe('WAVE'); + expect(ascii(bytes, 12, 4)).toBe('fmt '); + expect(view.getUint16(20, true)).toBe(1); + expect(view.getUint16(22, true)).toBe(2); + expect(view.getUint32(24, true)).toBe(8000); + expect(view.getUint32(28, true)).toBe(8000 * 2 * 2); + expect(view.getUint16(32, true)).toBe(4); + expect(view.getUint16(34, true)).toBe(16); + expect(ascii(bytes, 36, 4)).toBe('data'); + expect(view.getUint32(40, true)).toBe(12); + + expect(view.getInt16(44, true)).toBe(0); + expect(view.getInt16(46, true)).toBe(16384); + expect(view.getInt16(48, true)).toBe(32767); + expect(view.getInt16(50, true)).toBe(-16384); + expect(view.getInt16(52, true)).toBe(-32768); + expect(view.getInt16(54, true)).toBe(32767); + }); + + it('estimates WAV size without allocating the file', () => { + const buffer = createAudioBufferLike({ + channels: [new Float32Array(10), new Float32Array(10)], + }); + + expect(estimateWavByteSize(buffer)).toBe(44 + 10 * 2 * 2); + }); + + it('rejects RIFF files above the 4 GB WAV limit', () => { + const buffer: AudioBufferLike = { + sampleRate: 48000, + numberOfChannels: 2, + length: 0xffffffff, + getChannelData: () => new Float32Array(0), + }; + + expect(() => encodeAudioBufferToWavBytes(buffer)).toThrow('4 GB'); + }); +}); + +describe('audio-only export settings', () => { + it('defaults audio-only export to WAV', () => { + expect(createDefaultExportSettings().audioOnlyFormat).toBe('wav'); + }); + + it('persists browser audio mode and sanitizes invalid values', () => { + useExportStore.getState().reset(); + useExportStore.getState().setSettings({ audioOnlyFormat: 'browser' }); + expect(useExportStore.getState().settings.audioOnlyFormat).toBe('browser'); + + useExportStore.getState().replaceSettings({ + audioOnlyFormat: 'mp3' as never, + }); + expect(useExportStore.getState().settings.audioOnlyFormat).toBe('wav'); + }); +}); diff --git a/tests/unit/cloudApiTimeout.test.ts b/tests/unit/cloudApiTimeout.test.ts new file mode 100644 index 00000000..aac0398c --- /dev/null +++ b/tests/unit/cloudApiTimeout.test.ts @@ -0,0 +1,39 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { cloudApi } from '../../src/services/cloudApi'; + +describe('cloudApi AI chat timeouts', () => { + afterEach(() => { + vi.useRealTimers(); + vi.unstubAllGlobals(); + }); + + it('allows hosted chat requests to run longer than the default JSON timeout', async () => { + vi.useFakeTimers(); + + let signal: AbortSignal | undefined; + const fetchMock = vi.fn((_path: RequestInfo | URL, init?: RequestInit) => { + signal = init?.signal ?? undefined; + + return new Promise((_resolve, reject) => { + signal?.addEventListener('abort', () => reject(new Error('aborted')), { once: true }); + }); + }); + + vi.stubGlobal('fetch', fetchMock); + + const request = cloudApi.ai.chat.create({ + messages: [{ content: 'make vhs filter', role: 'user' }], + model: 'gpt-5.1', + }); + const rejection = expect(request).rejects.toThrow('Request to /api/ai/chat timed out after 90000ms.'); + + await vi.advanceTimersByTimeAsync(10_000); + expect(signal?.aborted).toBe(false); + + await vi.advanceTimersByTimeAsync(79_999); + expect(signal?.aborted).toBe(false); + + await vi.advanceTimersByTimeAsync(1); + await rejection; + }); +}); diff --git a/tests/unit/exportLayerBuilder.test.ts b/tests/unit/exportLayerBuilder.test.ts index d1a8816b..3267a50c 100644 --- a/tests/unit/exportLayerBuilder.test.ts +++ b/tests/unit/exportLayerBuilder.test.ts @@ -9,6 +9,7 @@ import { useMediaStore } from '../../src/stores/mediaStore'; import { useTimelineStore } from '../../src/stores/timeline'; import { lottieRuntimeManager } from '../../src/services/vectorAnimation/LottieRuntimeManager'; import type { TimelineClip, TimelineTrack } from '../../src/stores/timeline/types'; +import type { ParallelDecodeManager } from '../../src/engine/ParallelDecodeManager'; describe('ExportLayerBuilder', () => { beforeEach(() => { @@ -93,6 +94,82 @@ describe('ExportLayerBuilder', () => { expect(layers[0]?.source?.webCodecsPlayer).toBe(clipStates.get('clip-1')?.webCodecsPlayer); }); + it('uses export lookup tolerance for parallel decoded frames', () => { + const track = { + id: 'track-1', + type: 'video', + visible: true, + solo: false, + } as unknown as TimelineTrack; + + const videoElement = document.createElement('video'); + const parallelFrame = { + displayWidth: 1920, + displayHeight: 1080, + } as VideoFrame; + + const clip = { + id: 'clip-1', + name: 'Clip 1', + trackId: 'track-1', + startTime: 0, + duration: 5, + inPoint: 0, + outPoint: 5, + source: { + type: 'video', + videoElement, + }, + transform: {}, + } as unknown as TimelineClip; + + const clipStates = new Map([ + ['clip-1', { + clipId: 'clip-1', + webCodecsPlayer: null, + lastSampleIndex: 0, + isSequential: false, + preciseVideoElement: videoElement, + }], + ]); + + const parallelDecoder = { + hasClip: vi.fn(() => true), + getFrameForClip: vi.fn(() => parallelFrame), + } as unknown as ParallelDecodeManager; + + const ctx: FrameContext = { + time: 0.5, + fps: 30, + frameTolerance: 50_000, + clipsAtTime: [clip], + trackMap: new Map([[track.id, track]]), + clipsByTrack: new Map([[track.id, clip]]), + getInterpolatedTransform: () => ({ + position: { x: 0, y: 0, z: 0 }, + scale: { x: 1, y: 1 }, + rotation: { x: 0, y: 0, z: 0 }, + opacity: 1, + blendMode: 'normal', + }), + getInterpolatedEffects: () => [], + getSourceTimeForClip: () => 0.5, + getInterpolatedSpeed: () => 1, + }; + + initializeLayerBuilder([track]); + + const layers = buildLayersAtTime(ctx, clipStates, parallelDecoder, true); + + expect(layers).toHaveLength(1); + expect(layers[0]?.source?.videoFrame).toBe(parallelFrame); + expect(parallelDecoder.getFrameForClip).toHaveBeenCalledWith( + 'clip-1', + 0.5, + { toleranceMultiplier: 3 }, + ); + }); + it('forces gaussian splats onto the native scene path while keeping full-quality export settings', () => { const track = { id: 'track-1', diff --git a/tests/unit/parallelDecodeTimestamps.test.ts b/tests/unit/parallelDecodeTimestamps.test.ts new file mode 100644 index 00000000..9a01cf93 --- /dev/null +++ b/tests/unit/parallelDecodeTimestamps.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; +import { + getNormalizedSampleSourceTime, + getNormalizedSampleTimestampMicroseconds, + getPresentationOffsetSeconds, +} from '../../src/engine/ParallelDecodeManager'; + +describe('parallel decode timestamp normalization', () => { + it('maps a positive first CTS offset to source time zero', () => { + const samples = [ + { cts: 5_000, timescale: 30_000 }, + { cts: 6_000, timescale: 30_000 }, + ]; + + const offset = getPresentationOffsetSeconds(samples); + + expect(offset).toBeCloseTo(1 / 6); + expect(getNormalizedSampleSourceTime(samples[0], offset)).toBe(0); + expect(getNormalizedSampleSourceTime(samples[1], offset)).toBeCloseTo(1 / 30); + expect(getNormalizedSampleTimestampMicroseconds(samples[1], offset)).toBe(33_333); + }); + + it('leaves zero-based sample timestamps unchanged', () => { + const samples = [ + { cts: 0, timescale: 90_000 }, + { cts: 3_000, timescale: 90_000 }, + ]; + + const offset = getPresentationOffsetSeconds(samples); + + expect(offset).toBe(0); + expect(getNormalizedSampleSourceTime(samples[1], offset)).toBeCloseTo(1 / 30); + expect(getNormalizedSampleTimestampMicroseconds(samples[1], offset)).toBe(33_333); + }); +}); diff --git a/tests/unit/playbackDebugStats.test.ts b/tests/unit/playbackDebugStats.test.ts index 6e78e4dc..dc2f86e2 100644 --- a/tests/unit/playbackDebugStats.test.ts +++ b/tests/unit/playbackDebugStats.test.ts @@ -175,6 +175,47 @@ describe('playback debug stats', () => { expect(stats.avgAudioDriftMs).toBe(72); }); + it('marks severe target-moving preview freezes as bad without requiring decoder events', () => { + const vfTimeline: VFPipelineEvent[] = Array.from({ length: 13 }, (_, index) => ({ + type: 'vf_preview_frame', + t: index * 60, + detail: { + changed: 'false', + targetMoved: 'true', + previewPath: 'webcodecs', + clipId: 'clip-freeze', + targetTimeMs: index * 33, + displayedTimeMs: 0, + driftMs: index * 33, + }, + })); + + const stats = buildPlaybackDebugStats({ + decoder: 'WebCodecs', + now: 800, + windowMs: 1000, + vfTimeline, + healthVideos: [ + { + clipId: 'clip-freeze', + src: 'demo.mp4', + currentTime: 0, + readyState: 4, + seeking: false, + paused: true, + played: 1, + warmingUp: false, + gpuReady: true, + }, + ], + }); + + expect(stats.previewFreezeEvents).toBe(1); + expect(stats.stalePreviewWhileTargetMoved).toBe(13); + expect(stats.longestPreviewFreezeMs).toBe(720); + expect(stats.status).toBe('bad'); + }); + it('marks VF playback unhealthy when readyState drops and audio drift show up', () => { const vfTimeline: VFPipelineEvent[] = [ { type: 'vf_capture', t: 0 }, diff --git a/tests/unit/playbackSliceGate.test.ts b/tests/unit/playbackSliceGate.test.ts index d3d5df72..3ccb58f3 100644 --- a/tests/unit/playbackSliceGate.test.ts +++ b/tests/unit/playbackSliceGate.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it, vi, beforeEach } from 'vitest'; +import { describe, expect, it, vi, beforeEach, afterEach } from 'vitest'; import { createPlaybackSlice } from '../../src/stores/timeline/playbackSlice'; import { playheadState } from '../../src/services/layerBuilder/PlayheadState'; import type { TimelineStore } from '../../src/stores/timeline/types'; @@ -46,6 +46,10 @@ describe('playbackSlice HTML readiness gate', () => { playheadState.isUsingInternalPosition = false; }); + afterEach(() => { + vi.useRealTimers(); + }); + it('skips HTML readiness warmup for full WebCodecs clips', async () => { const htmlVideo = { readyState: 0, @@ -83,6 +87,93 @@ describe('playbackSlice HTML readiness gate', () => { expect(htmlVideo.pause).not.toHaveBeenCalled(); }); + it('exposes playback warmup state while HTML video readiness is pending', async () => { + vi.useFakeTimers(); + getRuntimeFrameProvider.mockReturnValue(null); + + const htmlVideo = { + readyState: 0, + play: vi.fn(), + pause: vi.fn(), + }; + htmlVideo.play.mockImplementation(() => { + htmlVideo.readyState = 3; + return Promise.resolve(); + }); + + const state = createPlaybackTestStore({ + clips: [ + { + id: 'clip-1', + startTime: 0, + duration: 10, + source: { + videoElement: htmlVideo, + }, + }, + ], + playheadPosition: 1, + duration: 60, + isPlaying: false, + playbackWarmup: null, + } as Partial); + + const playPromise = state.play(); + + expect(state.isPlaying).toBe(false); + expect(state.playbackWarmup).toMatchObject({ + targetTime: 1, + pendingVideoCount: 1, + totalVideoCount: 1, + }); + + await vi.advanceTimersByTimeAsync(60); + await playPromise; + + expect(state.playbackWarmup).toBeNull(); + expect(state.isPlaying).toBe(true); + expect(htmlVideo.pause).toHaveBeenCalled(); + }); + + it('does not start playback when a pending warmup was canceled', async () => { + vi.useFakeTimers(); + getRuntimeFrameProvider.mockReturnValue(null); + + const htmlVideo = { + readyState: 0, + play: vi.fn().mockResolvedValue(undefined), + pause: vi.fn(), + }; + + const state = createPlaybackTestStore({ + clips: [ + { + id: 'clip-1', + startTime: 0, + duration: 10, + source: { + videoElement: htmlVideo, + }, + }, + ], + playheadPosition: 1, + duration: 60, + isPlaying: false, + playbackWarmup: null, + } as Partial); + + const playPromise = state.play(); + expect(state.playbackWarmup).not.toBeNull(); + + state.pause(); + htmlVideo.readyState = 3; + await vi.advanceTimersByTimeAsync(60); + await playPromise; + + expect(state.playbackWarmup).toBeNull(); + expect(state.isPlaying).toBe(false); + }); + it('keeps the internal playhead in sync when moving the playhead while paused', () => { const state = createPlaybackTestStore({ clips: [],