feat(fs): hardlink import/export assets, harden sync init#3388
Conversation
Importer (gamelist/launchbox file:// flows) and exporters (gamelist.xml, metadata.pegasus.txt local exports) now hardlink media assets when source and destination share a filesystem, falling back transparently to a copy on EXDEV / EPERM / EOPNOTSUPP / EMLINK / EACCES (cross-device, FAT32, exFAT, network mounts, etc.). Saves disk space and is effectively instantaneous on large files (videos, manuals, miximages). Covers keep a real copy (allow_link=False) because _store_cover resizes the small cover in place via PIL.Image.save, which would truncate the shared inode and corrupt the user's source image. Also makes FSSyncHandler tolerate a missing/unwritable /romm/sync at startup: an OSError from mkdir now logs a warning instead of crashing the whole app at module-import time. Sync calls still fail at use time if the mount remains broken — the right place to surface the error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds hardlink-based optimization to media import/export paths and hardens FSSyncHandler so a missing/unwritable /romm/sync no longer takes down app startup.
Changes:
- New
utils.filesystem.link_or_copy_filehelper that triesos.linkfirst and falls back toshutil.copy2on a curated set of errnos (EXDEV/EPERM/EOPNOTSUPP/ENOTSUP/EMLINK/EACCES). FSHandler.copy_filegains anallow_linkflag (defaulting to hardlinking); gamelist/pegasus exporters and the cover storage path are wired up accordingly (allow_link=Falsefor covers because PIL resizes in place).FSHandler.__init__gainstolerate_missing_base, used byFSSyncHandlerto log and continue instead of crashing when the sync directory can't be created.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| backend/utils/filesystem.py | Adds link_or_copy_file helper and the errno fallback set. |
| backend/utils/gamelist_exporter.py | Routes asset writes through link_or_copy_file. |
| backend/utils/pegasus_exporter.py | Routes asset writes through link_or_copy_file. |
| backend/handler/filesystem/base_handler.py | Adds allow_link to copy_file, tolerate_missing_base to __init__. |
| backend/handler/filesystem/sync_handler.py | Passes tolerate_missing_base=True so startup tolerates an unwritable sync dir. |
| backend/handler/filesystem/resources_handler.py | Forces real copy for small cover (avoids PIL in-place truncating shared inode). |
| backend/tests/utils/test_filesystem.py | New tests for the link/copy helper. |
| backend/tests/handler/filesystem/test_base_handler.py | New tests for copy_file allow_link and tolerate_missing_base. |
| backend/tests/handler/filesystem/test_sync_handler.py | Regression test for tolerant FSSyncHandler() construction. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Test Results (postgresql) 1 files 1 suites 4m 6s ⏱️ Results for commit 584f35b. ♻️ This comment has been updated with latest results. |
Test Results (mariadb) 1 files 1 suites 4m 36s ⏱️ Results for commit 584f35b. ♻️ This comment has been updated with latest results. |
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
Summary
file://URIs) and the gamelist.xml / metadata.pegasus.txt local exporters now tryos.link()first, falling back toshutil.copy2onEXDEV/EPERM/EOPNOTSUPP/EMLINK/EACCES. Same-filesystem operations become effectively instant and use no extra disk space — a big win for videos and manuals. Cross-filesystem and unsupported-filesystem (FAT32/exFAT) behavior is unchanged._store_coverresizes the small cover in place viaPIL.Image.save, which would truncate the shared inode and corrupt the user's source image.copy_filegot anallow_link: bool = Trueparameter and the cover path passesallow_link=False./romm/sync. Currently aPermissionErrorfromFSSyncHandler()at module import time takes down the whole app.FSHandler.__init__now acceptstolerate_missing_base=True, whichFSSyncHandleruses to log a warning and continue. Failures still surface at use time, which is the right place.Test plan
pytest tests/utils/test_filesystem.py— 8 new tests for thelink_or_copy_filehelper (hardlink, EXDEV/EPERM fallback, ENOSPC re-raise, mutation semantics, EEXIST propagation)pytest tests/handler/filesystem/test_base_handler.py— 4 newcopy_filetests (default hardlinks,allow_link=Falsereal copy, EXDEV fallback, missing source) + 4 newtolerate_missing_basetestspytest tests/handler/filesystem/test_sync_handler.py— new regression test thatFSSyncHandler()construction does not raise onPermissionErrorfrommkdirpytest tests/utils/test_gamelist_exporter.py tests/utils/test_pegasus_exporter.py— existing 37 exporter tests still passpytest tests/handler/metadata/test_launchbox_handler.py— existing 127 importer-side tests still passassets/videos/<rom>.mp4shares an inode with the source (stat -c %i/ls -li)🤖 Generated with Claude Code