Skip to content

feat(fs): hardlink import/export assets, harden sync init#3388

Merged
gantoine merged 4 commits into
masterfrom
hardlink-resources-gamelist
May 19, 2026
Merged

feat(fs): hardlink import/export assets, harden sync init#3388
gantoine merged 4 commits into
masterfrom
hardlink-resources-gamelist

Conversation

@gantoine
Copy link
Copy Markdown
Member

Summary

  • Hardlink media assets on same-filesystem import/export, fall back to copy otherwise. The gamelist/launchbox importer (file:// URIs) and the gamelist.xml / metadata.pegasus.txt local exporters now try os.link() first, falling back to shutil.copy2 on EXDEV/EPERM/EOPNOTSUPP/EMLINK/EACCES. Same-filesystem operations become effectively instant and use no extra disk space — a big win for videos and manuals. Cross-filesystem and unsupported-filesystem (FAT32/exFAT) behavior is unchanged.
  • Covers stay as real copies. _store_cover resizes the small cover in place via PIL.Image.save, which would truncate the shared inode and corrupt the user's source image. copy_file got an allow_link: bool = True parameter and the cover path passes allow_link=False.
  • Sync handler tolerates a missing/unwritable /romm/sync. Currently a PermissionError from FSSyncHandler() at module import time takes down the whole app. FSHandler.__init__ now accepts tolerate_missing_base=True, which FSSyncHandler uses to log a warning and continue. Failures still surface at use time, which is the right place.

Test plan

  • pytest tests/utils/test_filesystem.py — 8 new tests for the link_or_copy_file helper (hardlink, EXDEV/EPERM fallback, ENOSPC re-raise, mutation semantics, EEXIST propagation)
  • pytest tests/handler/filesystem/test_base_handler.py — 4 new copy_file tests (default hardlinks, allow_link=False real copy, EXDEV fallback, missing source) + 4 new tolerate_missing_base tests
  • pytest tests/handler/filesystem/test_sync_handler.py — new regression test that FSSyncHandler() construction does not raise on PermissionError from mkdir
  • pytest tests/utils/test_gamelist_exporter.py tests/utils/test_pegasus_exporter.py — existing 37 exporter tests still pass
  • pytest tests/handler/metadata/test_launchbox_handler.py — existing 127 importer-side tests still pass
  • Manual check on a real RomM install with same-filesystem ROM library + resources path: verify exported assets/videos/<rom>.mp4 shares an inode with the source (stat -c %i / ls -li)
  • Manual check on a cross-filesystem setup: verify export falls back to a real copy, content matches, source untouched

🤖 Generated with Claude Code

Importer (gamelist/launchbox file:// flows) and exporters (gamelist.xml,
metadata.pegasus.txt local exports) now hardlink media assets when source
and destination share a filesystem, falling back transparently to a copy
on EXDEV / EPERM / EOPNOTSUPP / EMLINK / EACCES (cross-device, FAT32,
exFAT, network mounts, etc.). Saves disk space and is effectively
instantaneous on large files (videos, manuals, miximages).

Covers keep a real copy (allow_link=False) because _store_cover resizes
the small cover in place via PIL.Image.save, which would truncate the
shared inode and corrupt the user's source image.

Also makes FSSyncHandler tolerate a missing/unwritable /romm/sync at
startup: an OSError from mkdir now logs a warning instead of crashing
the whole app at module-import time. Sync calls still fail at use time
if the mount remains broken — the right place to surface the error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 18, 2026 11:41
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds hardlink-based optimization to media import/export paths and hardens FSSyncHandler so a missing/unwritable /romm/sync no longer takes down app startup.

Changes:

  • New utils.filesystem.link_or_copy_file helper that tries os.link first and falls back to shutil.copy2 on a curated set of errnos (EXDEV/EPERM/EOPNOTSUPP/ENOTSUP/EMLINK/EACCES).
  • FSHandler.copy_file gains an allow_link flag (defaulting to hardlinking); gamelist/pegasus exporters and the cover storage path are wired up accordingly (allow_link=False for covers because PIL resizes in place).
  • FSHandler.__init__ gains tolerate_missing_base, used by FSSyncHandler to log and continue instead of crashing when the sync directory can't be created.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
backend/utils/filesystem.py Adds link_or_copy_file helper and the errno fallback set.
backend/utils/gamelist_exporter.py Routes asset writes through link_or_copy_file.
backend/utils/pegasus_exporter.py Routes asset writes through link_or_copy_file.
backend/handler/filesystem/base_handler.py Adds allow_link to copy_file, tolerate_missing_base to __init__.
backend/handler/filesystem/sync_handler.py Passes tolerate_missing_base=True so startup tolerates an unwritable sync dir.
backend/handler/filesystem/resources_handler.py Forces real copy for small cover (avoids PIL in-place truncating shared inode).
backend/tests/utils/test_filesystem.py New tests for the link/copy helper.
backend/tests/handler/filesystem/test_base_handler.py New tests for copy_file allow_link and tolerate_missing_base.
backend/tests/handler/filesystem/test_sync_handler.py Regression test for tolerant FSSyncHandler() construction.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread backend/handler/filesystem/base_handler.py
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Test Results (postgresql)

    1 files      1 suites   4m 6s ⏱️
1 349 tests 1 349 ✅ 0 💤 0 ❌
1 351 runs  1 351 ✅ 0 💤 0 ❌

Results for commit 584f35b.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Test Results (mariadb)

    1 files      1 suites   4m 36s ⏱️
1 349 tests 1 349 ✅ 0 💤 0 ❌
1 351 runs  1 351 ✅ 0 💤 0 ❌

Results for commit 584f35b.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
16401 11448 70% 0% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
backend/handler/filesystem/base_handler.py 94% 🟢
backend/handler/filesystem/resources_handler.py 45% 🟢
backend/handler/filesystem/sync_handler.py 97% 🟢
backend/tasks/manual/cleanup_orphaned_resources.py 34% 🟢
backend/utils/filesystem.py 98% 🟢
backend/utils/gamelist_exporter.py 89% 🟢
backend/utils/pegasus_exporter.py 69% 🟢
TOTAL 75% 🟢

updated for commit: 584f35b by action🐍

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

@gantoine gantoine merged commit 405f678 into master May 19, 2026
12 checks passed
@gantoine gantoine deleted the hardlink-resources-gamelist branch May 19, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants