FrameTrellis has three layers of automated tests, all driven by CTest and Qt Test:
| Label | Where | What it covers |
|---|---|---|
unit |
tests/unit/ |
rational time, frame rates and drop-frame timecode; the timeline model and every edit's boundary rules; project serialization, invalid and hostile documents, moved folders, failed saves; undo/redo, merging, the saved state and media commands; export validation, the helper protocol and the render-job state machine |
integration |
tests/integration/ |
MLT/FFmpeg media probing (video, audio, stills, broken files), thumbnails, frame-accurate graph building, compositing, pillarboxing, opacity, fades, picture adjustments, gain/mute, offline media, producer caching; rendering to MP4 and verifying the result, cancellation, replacing existing files, refusing to overwrite sources, the helper process end to end; real-time playback, seeking, pausing and live updates |
gui |
tests/gui/ |
the real QML interface driven by mouse and keyboard: first-run guide, importing with duplicate detection, appending, dragging and trimming clips, refused overlapping moves, split/undo/redo/delete/duplicate shortcuts, title PNG generation, Space playback, exporting from the dialog, save and reopen |
Run them with:
ctest --preset linux # all tests
ctest --preset linux -L unit # only unit tests
ctest --preset linux -R engine # tests whose name matches "engine"Use the v1.4.0 release preflight and v1.4 editing acceptance matrix to record manual multi-track, precision editing, batch properties, preview/export parity, project-format migration, media recovery and clean-host package checks. This is a pending manual checklist, not an assertion that those checks have passed. Automated CI and hands-on testing provide different evidence.
Subtitle regressions cover exact SRT roundtrips, format-v6 persistence, undo/redo, atomic rejection, rational frame sampling, preview layout, encoded burn-in and real editor draft protection. See the subtitle acceptance matrix.
No media files are stored in the repository. tests/support/media_fixtures.cpp generates them
with MLT at test time: MP4 files with one solid colour per second and a sine tone, WAV files and
PNG stills. Because the colours and tones are known, tests assert on real pixel colours and
audio levels (for example, that the clip trimmed to the "green second" really shows green, or
that a muted clip is silent).
The tst_engine_threads integration test opens media on multiple threads
while video profiles are being prepared. MLT 7.22 may update process-global
profile normalization settings during Mlt::Profile construction; concurrent
initialization was observed to trigger an AddressSanitizer heap-use-after-free.
The engine's createProfile() factory now serializes construction and
initial configuration, while normal work on independently owned profiles
continues on the worker threads.
An additional stress test constructs many profiles in parallel under the
existing sanitizers. This guards the known profile-construction path but
does not establish that all third-party MLT services are thread-safe; any
new direct Mlt::Profile constructions from workers should be reviewed.
All tests run without a display: CTest sets QT_QPA_PLATFORM=offscreen, the GUI test also uses
QT_QUICK_BACKEND=software, and SDL_AUDIODRIVER=dummy gives playback an audio clock without
a sound card. The GUI test isolates QSettings and application data from your profile.
The GUI test can save screenshots of the running application:
FT_SCREENSHOT_DIR=/tmp/shots ctest --preset linux -L guiscripts/make-screenshots.sh does this with more varied media generated by FFmpeg and writes
the optimised images used in the documentation and on the website.
On pull requests, the Linux CI workflow captures selected feature screenshots as an ephemeral workflow artifact (14-day retention). These images are not committed to the repository. The feature screenshots do not replace automated assertions.
cmake --preset linux-asan builds with AddressSanitizer and UBSan; CI runs the unit and
integration tests that way. clang-tidy (configured in .clang-tidy) and clang-format run in
CI as well.
The GUI test verifies that the Create Title dialog disables generation when all video tracks are locked, then becomes available again when an unlocked video track exists. This prevents the normal UI path from leaving a generated title image without a timeline clip.
The render integration suite also exports an MP4 containing a transparent PNG above video. It decodes the result and verifies that opaque title pixels appear while transparent regions reveal footage underneath. This protects export compositing, not only the editor preview.
The editing unit tests verify that copying selected clips between multiple video tracks keeps their relative timing and clip properties intact. The same suite checks that undo and redo restore exact timeline states, including when the edit is rejected because a track is locked or the requested clip ids are invalid.
Clip-attribute copy/paste tests check the behaviour for destination clips shorter than the source, including clamped audio fades. Batch mute/unmute and picture reset tests confirm that unrelated clip attributes are preserved and that a failed edit makes no partial changes. A real QML GUI test drives Ctrl+D and the clip attribute clipboard shortcuts and inspects the resulting timeline and undo state.
The batch picture adjustment editing unit tests check that brightness, contrast
and saturation presets affect exactly the chosen video properties and reject
invalid selections atomically. The tst_engine integration regression also
renders two separate video clips after a shared saturation preset, checks the
actual grayscale pixels and confirms that Undo and Redo restore the exact
rendered images. The tst_render integration regression runs the real export
helper and inspects decoded MP4 frames from two graded video tracks. Together
these test both the preview-graph path and the exported picture rather than
relying only on the serialized property values.
These tests need MLT's avfilter.eq capability. If that optional filter is
missing, the targeted tests skip with an explicit reason; that does not
count as positive color-grading verification for a release. The additional
manual v1.4 matrix remains necessary for interactive performance, GPU
preview and complete output-quality validation.
The Windows package workflow builds a portable ZIP and an installer, then opens the editor with a generated test project and captures a screenshot on clean Windows 2022, 2025 and Windows 11 ARM64 runners (the last executes the x64 package via emulation).
A v1.2.0 post-release main workflow encountered an intermittent portable editor
access violation in Qt6Core.dll on Windows 11 ARM64. The installed version, renderer
and other hosts passed; rerunning the failed ARM64 job passed. The root cause remains
unknown and must not be reported as resolved merely because a rerun succeeded.
In the v1.2.1 development branch, the ARM64 smoke test executes the portable GUI twice
as independent tests. Both must pass, with separately named screenshots and Qt stdout/
stderr logs in the verification-windows-11-arm Actions artifact. A crash prints the
relevant Windows Application Error event and saves a copy of the application's own log.
The verifier remains strict about the portable package's library boundaries; other
runner environments retain one normal GUI launch. A failed inspection of loaded DLLs
is itself a failed check, not a silent skip. Each launch deletes old screenshots and
Qt logs before starting, so a screenshot from an earlier attempt cannot be accepted.
The Windows package build job runs the PowerShell-only regression suite before building packages:
pwsh -NoProfile -File packaging/tests/test_verify_module_paths.ps1This checks allowed Windows/package module paths as well as sibling-directory prefix spoofing, case-insensitive paths and directory traversal. It needs Windows PowerShell 7 but does not require compiled editor binaries.
For manual diagnosis on a Windows test machine with the release files available:
./packaging/windows/verify.ps1 -Zip dist/FrameTrellis-1.2.1-windows-x64.zip `
-Installer dist/FrameTrellis-1.2.0-windows-x64-setup.exe `
-WorkDir "$env:TEMP/frametrellis-verify" -PortableGuiRuns 2Keep the run's full log and screenshots when investigating a failure. A passing emulated runner does not replace testing on physical ARM64 machines.
Timeline insertion now checks both source and sequence endpoints before adding
signed frame counts. A clip must have a positive duration and its source and timeline
ranges must end at or before kMaxTimelineFrames. The isRangeFree query also
rejects negative starts, zero durations and endpoints beyond that limit before
calculating the exclusive end frame.
tests/unit/tst_timeline.cpp exercises the maximum signed FrameCount, values just
past the allowed limit, the exact valid endpoint and unknown tracks. These checks
combine the overlapping changes originally proposed in
#31 and
#32; they belong together so
that the compiler and sanitizers validate the same final timeline model. These are
unreleased changes until merged and verified on the integrated branch.
- Real audio output devices (tests use SDL's dummy driver).
- GPU-accelerated Qt Quick backends (tests use the software renderer).
- Windows packaging on physical machines and macOS in general.
- Manual exploratory testing of drag-and-drop from the operating system's file manager.