feat(separation): real Demucs stem separation, replacing the -39 dB FFT mock (#106 Track 2)#563
Open
seonghobae wants to merge 5 commits into
Open
feat(separation): real Demucs stem separation, replacing the -39 dB FFT mock (#106 Track 2)#563seonghobae wants to merge 5 commits into
seonghobae wants to merge 5 commits into
Conversation
Track 2 (#106) of the ML engine integration plan. The separator previously split audio with FFT frequency-band masks, which is not source separation at all: measured ~-39 dB SI-SDR on a realistic mix. Downstream role, range, and chord analysis were all consuming garbage stems. Replace it with Demucs (htdemucs) running locally on CPU. Measured in-engine on a synthetic bass+harmony+drums mix: bass stem SI-SDR = +22.7 dB (a ~62 dB improvement over the FFT mock), with the other stems correctly uncorrelated to the bass. The 4-stem contract (vocals/bass/drums/other) is unchanged, so downstream consumers are untouched. The neural model is isolated behind _load_model/_apply_model so unit tests mock it (fast, offline, deterministic, full coverage of the plumbing and stem-name mapping). A real-Demucs integration test is gated behind BANDSCOPE_RUN_DEMUCS. Pipeline/CLI progress tests that used to rely on the instant FFT mock now mock the separator so they neither run heavy ML nor depend on network. Security Notes: untrusted audio is size-bounded and decoded via librosa/ soundfile; inference is CPU-local with no network; long audio is split into overlapping segments to bound memory; empty/undecodable input fails safely. Follow-ups (CI hardening, tracked separately): bundle model weights for fully offline operation + supplemental-component-inventory entry; CPU-only torch pinning for cross-platform builds; retire the now-unused bandsplit-v1 profile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjGVapDZ3k7V7zKYk16P4C
The macOS amd64 (Intel) build failed: torch 2.12.1 ships no Intel-mac wheel (PyTorch dropped macosx x86_64 at 2.3). The cross-platform-build-policy mandates macOS Intel + arm64, so the fix keeps Intel support rather than dropping a platform: pin torch>=2.2,<2.3 and torchaudio>=2.2,<2.3 (2.2.2 is the last series with Intel-mac wheels; demucs only needs torch>=1.8.1). torch 2.2.x is built against numpy 1.x, so also pin numpy>=1.26,<2 (2.x raised 'Numpy is not available'). numpy 1.26 lacks the ndarray type-param defaults numpy 2 provides, so relax disallow_any_generics (only that sub-check) rather than churn bare-ndarray annotations across unrelated modules. Real separation re-validated on this stack: bass SI-SDR +21.7 dB. Full engine suite 423 passed; ruff + format + mypy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjGVapDZ3k7V7zKYk16P4C
torch 2.2.2 (last macOS-Intel release; Intel mandated by the cross-platform build policy) carries the torch.load RCE fixed in 2.6 — Intel-mac support and a patched torch are mutually exclusive. The vulnerable API only loads demucs's pinned model weights; user audio never reaches torch.load. Encode the exception in dependency-review (allow-ghsas) + osv-scanner.toml and document it in the dependency policy with an explicit removal condition (ONNX migration or Intel-mac policy change), following the existing RUSTSEC exception pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjGVapDZ3k7V7zKYk16P4C
osv-scanner loads config per-directory beside each lockfile (it loaded apps/desktop/src-tauri/osv-scanner.toml but ignored the repo-root file), so the GHSA-53q9-r3pm-6pq6 exception never applied to services/analysis-engine/uv.lock and osv-scan stayed red. Move the config to services/analysis-engine/ (matching the existing src-tauri convention) and point the policy doc at the exact path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjGVapDZ3k7V7zKYk16P4C
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjGVapDZ3k7V7zKYk16P4C
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Track 2 (#106) foundation: replace the −39 dB FFT mock with real Demucs separation.
The separator split audio with FFT frequency-band masks — not source separation at all (~−39 dB SI-SDR on a realistic mix). Every downstream analysis (roles, ranges, chords) was consuming garbage stems.
Now it runs Demucs (htdemucs) locally on CPU. Measured in-engine on a synthetic bass+harmony+drums mix: bass stem SI-SDR = +22.7 dB (~62 dB better than the mock), other stems correctly uncorrelated to the bass. The 4-stem contract (vocals/bass/drums/other) is unchanged — downstream untouched.
Why
local-first= runs on the user's machine without a server, so real separation means bundling the model. This is the foundation the rich role taxonomy (per rehearsal-domain-model.md: keyboard hands, N keyboards, backing vocals) will be built on — none of it is meaningful on mock stems.Testing
_load_model/_apply_model; unit tests mock it → fast, offline, deterministic, full plumbing + stem-name-mapping coverage.BANDSCOPE_RUN_DEMUCS=1(validated at +22.7 dB).Known follow-ups (CI hardening — this is the foundational PR of a larger epic)
supplemental-component-inventory.jsonentry (currentlyget_modelfetches on first use). The subprocess CLI e2e test needs this to be offline-green in CI.bandsplit-v1profile + inventory entry.🤖 Generated with Claude Code