Initial reimplementation of composefs-c#225
Draft
cgwalters wants to merge 4 commits intocomposefs:mainfrom
Draft
Initial reimplementation of composefs-c#225cgwalters wants to merge 4 commits intocomposefs:mainfrom
cgwalters wants to merge 4 commits intocomposefs:mainfrom
Conversation
Collaborator
Author
|
There's definitely some sub-tasks to this and pieces that we need to break out. One that I'm realizing is that the dumpfile format is hardcoded to sha256-12. I guess we can just auto-detect from length (like we're doing in other places) but the more I think about this the more I feel we need to formalize it (as is argued in #224 ) So how about a magic comment in the dumpfile like or so? |
cgwalters
force-pushed
the
composefs-c-compat
branch
from
e4c819c to
4d43b61
Compare
Change the inode collection algorithm from recursive depth-first to queue-based breadth-first traversal. This matches the C mkcomposefs behavior where all nodes at depth N are processed before any nodes at depth N+1, ensuring bit-for-bit identical output. The C implementation uses a linked-list queue to process directories, adding children to the queue as each directory is visited. This produces a specific inode ordering that our depth-first recursion didn't match. Also adds comprehensive proptests for: - Nested directory structures (test_nested_directories, test_deep_nesting) - Multiple subdirectories with files at various levels - Overlay opaque xattr (trusted.overlay.opaque) - SELinux xattr (security.selinux) - Files in subdirectories - Proptest variants for directory depth and file count
The C mkcomposefs implementation has specific xattr ordering requirements: 1. Per-inode xattrs (local and shared references) are written in ascending alphabetical order by full key name, as sorted by cmp_xattr() using strcmp(na->key, nb->key). 2. The shared xattr table is written in descending alphabetical order, as sorted by xattrs_ht_sort() using strcmp(v2->key, v1->key). This commit fixes the Rust implementation to match: - Implement custom Ord for XAttr that compares by full key name (prefix + suffix) in ascending order, matching cmp_xattr behavior. - Sort local xattrs before processing in share_xattrs() to ensure both local xattrs and shared xattr indices are written in ascending order. - Return shared xattrs in descending order from share_xattrs() to match the xattrs_ht_sort ordering in the shared xattr table. These changes ensure bit-for-bit compatibility with C mkcomposefs output.
…dering Add test_nested() to mkfs.rs which creates a multi-level directory structure (/a/b/c/deep-file, /a/b/mid-file, /a/shallow-file, /x/y/z-file, /root-file) to establish the baseline snapshot for V1_1 format with subdirectories. The BFS inode ordering used in this change is intentional - it aligns V1_1 format with C mkcomposefs behavior for consistency. While V1_1 is Rust-native and could use different ordering, using the same BFS approach: 1. Simplifies implementation (single code path) 2. Ensures predictable behavior across format versions 3. Enables bit-for-bit compatibility with C mkcomposefs in V1_0 Add detailed documentation in writer.rs explaining the BFS algorithm with an example showing the ordering difference vs DFS.
Basically starting on composefs/composefs#423 3 key goals: - Compatible CLI interfaces - Compatible EROFS output format (this is a big deal!) - Next: Compatible C shared library (ugly and messy) Assisted-by: OpenCode (Claude Sonnet 4) Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
force-pushed
the
composefs-c-compat
branch
from
4d43b61 to
1871128
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Basically starting on composefs/composefs#423
3 key goals:
Assisted-by: OpenCode (Claude Sonnet 4)