Merged
Conversation
Move FUSE integration tests from sharness shell scripts (t0030, t0031, t0032) and test/cli/fuse_test.go into a dedicated test/cli/fuse/ Go sub-package, ensuring all FUSE test cases run in CI. - git mv test/cli/fuse_test.go to test/cli/fuse/ (package fuse) - convert all sharness FUSE tests to Go subtests under TestFUSE: mount failure, IPNS symlink, IPNS NS map resolution, MFS file/dir creation, xattr (Linux), files write, add --to-files, file removal, nested dirs, publish-while-mounted block, sharded directory reads - add xattr helpers with build tags (linux/other) using unix.Getxattr - split make test_fuse into test_fuse_unit (./fuse/...) and test_fuse_cli (./test/cli/fuse/...) sub-targets - set TEST_FUSE=0 in test_cli so FUSE tests skip in cli-tests CI job - increase fuse-tests CI timeout from 5m to 10m for CLI tests - delete sharness t0030, t0031, t0032 (were always skipped in CI)
Add cross-reference comments between the unit tests in fuse/readonly/, fuse/ipns/, fuse/mfs/ and the end-to-end CLI tests in test/cli/fuse/. Also fix AGENTS.md to use a temp dir for fusermount symlink instead of sudo.
On shared self-hosted runners, leftover mount points from previous runs can exhaust the kernel FUSE mount limit. - add job-level concurrency group so only one fuse-tests runs at a time - lazy-unmount stale /tmp/fusetest* mounts before running tests
The Flush handler wrapped fi.fi.Flush() in a goroutine so it could return early when the FUSE context was canceled. But the goroutine kept running in the background, and when Release arrived it called Close on the same file descriptor concurrently. The two paths both entered DagModifier.Sync, racing on its internal write buffer and causing a nil pointer panic. The fix is to call Flush directly without a goroutine. The MFS flush cannot be safely canceled mid-operation anyway, so the goroutine only added the illusion of cancellation while leaking work and masking the real error. Also bumps boxo to pick up the matching defense-in-depth fix that serializes FileDescriptor.Flush and Close with a mutex.
lidel
force-pushed
the
feat/consolidate-fuse-tests
branch
from
d60bb04 to
a588c23
Compare
bazil/fuse dispatches each FUSE request in its own goroutine. The IPNS File handle had no synchronization, so concurrent Read/Write/Flush/Release calls could overlap on the underlying DagModifier which is not safe for concurrent use. Add sync.Mutex to File, matching the pattern already used by the MFS FileHandler.
bazil/fuse only dispatches Forget to nodes via the NodeForgetter interface. File is a handle, not a node, so this method was never called. The /mfs mount has no equivalent.
The /mfs mount flushes the directory after Unlink and Rename so changes propagate to the MFS root immediately. The /ipns mount did not, leaving mutations pending until an unrelated flush. Also add an empty-directory check before removing directories, matching the /mfs mount's safety check.
New files created via the /ipns FUSE mount now inherit the CID builder from their parent directory, preventing CIDv0 nodes from appearing inside a CIDv1 tree. The directory is also flushed after AddChild so the new entry propagates to the MFS root immediately, matching the /mfs mount.
Cover the file removal path and the empty-directory safety check added in the previous commit. TestRemoveFile verifies a created file can be removed and is gone afterwards. TestRemoveNonEmptyDirectory verifies that rmdir on a directory with children fails, and succeeds once the children are removed first.
All three FUSE mounts now read mode and mtime from UnixFS metadata when present, falling back to POSIX defaults when absent. Most IPFS data does not include this optional metadata. Writing mode and mtime is opt-in via two new config flags: - Mounts.StoreMtime: persist mtime on file create and open-for-write - Mounts.StoreMode: persist mode on chmod Other changes in this commit: - align default file/dir modes across /ipns and /mfs to 0644/0755 - share mode constants via fuse/mount/mode.go - convert Mounts.FuseAllowOther from bool to Flag for consistency - add Setattr to /ipns FileNode and /mfs File for chmod and touch - move dead File.Setattr from IPNS handle to FileNode (node) - bump boxo for Directory.Mode() and Directory.ModTime() getters
All three FUSE mounts now expose the node's CID via the ipfs.cid extended attribute on both files and directories. The /mfs mount also accepts the old ipfs_cid name for backward compatibility. The /ipfs mount previously had a stub that returned nil for all xattrs; it now returns the correct CID. The xattr name follows the convention used by CephFS (ceph.*), Btrfs (btrfs.*), and GlusterFS (glusterfs.*).
lidel
force-pushed
the
feat/consolidate-fuse-tests
branch
from
03dac69 to
6ba3437
Compare
Member
Author
Status updateStarted as test consolidation, grew into a broader FUSE fix-up after CI exposed a race condition panic. What changedFixed the panic: The IPNS Flush handler leaked a background goroutine that raced with Release on the same file descriptor, causing a nil-pointer crash. Removed the goroutine (the flush can't be canceled anyway) and added a mutex in boxo as safety net. Brought /ipns up to /mfs quality: added mutex on file handle ops, flushing after directory mutations, CID builder inheritance on file creation, rmdir safety check. Removed dead code that bazil/fuse never called. UnixFS mode and mtime: all three mounts now show POSIX mode and mtime from UnixFS when present. New opt-in config flags (
Tests and CI: dedicated fuse-tests job, coverage for remove, chmod, mtime, default modes, xattr. What we can't fix with the current FUSE librarybazil.org/fuse (unmaintained since 2020) dispatches
Why hanwen/go-fuse would fix thishanwen/go-fuse v2 passes the file handle to It's actively maintained (v2.9.0, Oct 2025), used by gocryptfs and rclone. The stability issues rclone hit in 2021 were fixed in v2.5.1 and don't apply to our use case. I'll see if we can refactor without investing too much time. |
lidel
changed the title
Replace the unmaintained bazil.org/fuse (last commit 2020) with hanwen/go-fuse v2.9.0, fixing two architectural issues that could not be solved with the old library. ftruncate now works: hanwen/go-fuse passes the open file handle to NodeSetattrer, so Setattr can truncate through the existing write descriptor instead of trying to open a second one (which deadlocks on MFS's single-writer lock). fsync now works: FileFsyncer runs on the handle directly, flushing the write buffer through the open descriptor. Previously a no-op because bazil dispatched Fsync to the inode only. mount package: - NewMount takes (InodeEmbedder, mountpoint, *fs.Options) instead of (fs.FS, mountpoint, allowOther) - mount/unmount collapses to a single fs.Mount call - fusermount3 tried before fusermount in ForceUnmount all three mounts: - structs embed fs.Inode (hanwen's InodeEmbedder pattern) - Remove split into Unlink + Rmdir (separate FUSE interfaces) - ReadDirAll replaced with Readdir returning DirStream - fillAttr helper shared between Getattr and Lookup responses - kernel cache invalidation via NotifyContent after Flush - 1s entry/attr timeout for writable mounts (matches go-fuse default, gocryptfs, rclone) - O_APPEND tracked on file handle, writes seek to end - build tags standardized to (linux || darwin || freebsd) && !nofuse tests: - replaced bazil fstestutil.MountedT with shared fusetest.TestMount - fixed TestConcurrentRW: channel drain mismatch and missing sync between write Close and read start - added TestFsync, TestFtruncate, TestReadlink, TestSeekRead, TestLargeFile, TestRmdir, TestCrossDirRename, TestUnknownXattr - added StoreMtime disabled/enabled subtests
MFS enforces a single-writer lock, so a leaked write descriptor blocks all subsequent opens of that file until GC.
Without this, IsActive stays true after `fusermount -u` and Unmount returns nil instead of ErrNotMounted.
After confirming the child exists, an Unlink failure could be an IO error. Returning ENOENT would hide the real cause.
Readonly Open now returns a file handle holding a DagReader instead of recreating one per Read call. Sequential reads no longer re-traverse the DAG from the root on each kernel request. All three mounts now use CtxReadFull with the kernel's per-request context so killing a process mid-read cancels in-flight block fetches instead of letting them complete uselessly.
- remove dead `_ = mntDir` in TestXattrCID - comment why immutableAttrCacheTime and mutableCacheTime are var - add TODO for using IPNS record TTL as cache timeout
The old check tried to verify OSXFUSE >= 2.7.2 to avoid a kernel panic from 2015. It used sysctl, tried to `go install` a third-party tool at runtime, and referenced paths that no longer exist. Replace with a simple check for the macFUSE mount helper, matching the same paths go-fuse looks for. If neither macFUSE nor OSXFUSE is found, point the user to the install page. Also standardize build tags to (linux || darwin || freebsd) && !nofuse and use strings.ReplaceAll.
Closed
go-fuse's fusermount errors don't include the path, so tools that check error messages for the mountpoint name couldn't tell which mount failed.
go-fuse finds fusermount3 natively, no symlink needed. The stale mount cleanup was for bazil's fstestutil which we no longer use.
lidel
force-pushed
the
feat/consolidate-fuse-tests
branch
from
13f9e11 to
05bdd59
Compare
Replace panic with log.Errorf + syscall.EIO in IPNS Directory.Lookup for unexpected MFS node types. Also remove duplicate comment block on File.Flush.
- fuse.md: replace stale OSXFUSE section with macFUSE, remove obsolete go-fuse-version tool, fix broken FreeBSD sudo echo, update xattr example to ipfs.cid with CIDv1, add mode/mtime section, add unixfs-v1-2025 tip, add debug logging section, add TOC, link to hanwen/go-fuse - changelog: refine bullet wording, link to fuse.md - config.md: fix double space, update fuse.md link text - experimental-features.md: fix double space, soften wording - README.md: add FUSE to features list and docs table
Extract duplicated code from fuse/mfs and fuse/ipns into a shared fuse/writable package, and consolidate duplicated tests into a reusable suite in fuse/fusetest. - fuse/writable: Dir, FileInode, FileHandle, Symlink types with all FUSE interface methods, shared by both mounts - fuse/fusetest: RunWritableSuite with helpers, exercised by both mfs and ipns via mount-specific factories - fix cache invalidation race: NotifyContent in Flush (synchronous) in addition to Release (async), so stat after close sees new size - drop deprecated ipfs_cid xattr, log error guiding users to ipfs.cid - mfs_unix.go: 632 -> 19 lines (thin wrapper over writable.Dir) - ipns_unix.go: 795 -> 170 lines (Root + key resolution only) - mfs_test.go: 1183 -> 95 lines (factory + persistence test) - ipns_test.go: 1309 -> 162 lines (factory + IPNS-specific tests) - tests that were only in one mount now run on both
Set volname, noapplexattr, and noappledouble on macOS via PlatformMountOpts, applied in NewMount so all three mounts benefit automatically. - volname: shows mount name in Finder instead of "macfuse Volume 0" - noapplexattr: suppresses Finder's com.apple.* xattr probes - noappledouble: prevents ._ resource fork sidecar files
Readdir on writable mounts now checks the underlying DAG node type for TFile entries, reporting S_IFLNK for symlinks instead of regular file. This makes ls -l and find -type l work correctly. - writable: Readdir checks SymlinkTarget for TFile entries - writablesuite: add SymlinkReaddir regression test - readonly: add TestReaddirSymlink regression test - test/cli/fuse: fix stale bazil.org/fuse reference in doc comment
Getxattr for the old "ipfs_cid" name now returns the CID instead of ENOATTR, keeping existing tooling working during the deprecation period. A log error is emitted on each access to nudge migration.
Member
Author
|
Changes since b8a0823 (reviewed by @guillaumemichel):
I think this is pretty solid as-is, amount of fixed bugs is just crazy. If we have any remaining bugs, we can fix them in future PRs, but this is good enough for 0.41 RC1. |
The go-fuse server dispatches each FUSE request in its own goroutine. On files larger than 128 KB the kernel issues concurrent readahead Read requests on the same file handle, racing on the shared DagReader's Seek+CtxReadFull sequence and corrupting its internal state. Add sync.Mutex to roFileHandle (matching the existing pattern in writable.FileHandle) and lock in Read and Release. - fuse/readonly/readonly_unix.go: add mu sync.Mutex to roFileHandle - fuse/readonly/ipfs_test.go: add TestConcurrentLargeFileRead - fuse/fusetest/writablesuite.go: add LargeFileConcurrentRead to shared writable suite (exercised by both /mfs and /ipns tests)
MFS uses an RWMutex (desclock) that holds RLock for the lifetime of a read descriptor and requires exclusive Lock for writes. Tools like rsync --inplace open the same file for reading and writing from separate processes, deadlocking on this mutex. For O_RDONLY opens, create a DagReader directly from the current DAG node instead of going through MFS. The reader gets a point-in-time snapshot and never touches desclock, so writers proceed independently. - fuse/writable/writable.go: add roFileHandle with DagReader for read-only opens, add DAG field to Config - fuse/mfs/mfs_unix.go: pass ipfs.DAG to writable Config - fuse/ipns/ipns_unix.go: pass ipfs.Dag() to writable Config - fuse/fusetest/writablesuite.go: add ConcurrentReadWrite test exercising simultaneous read and write on the same file
Open a temporary write descriptor in Setattr when the kernel sends a size change without a file handle (the truncate(2) syscall, as opposed to ftruncate(fd) which passes the handle). Previously this returned ENOTSUP. - fuse/writable: open, truncate, flush, close in Setattr else branch - fuse/fusetest: add TruncatePath to the shared writable suite - test/cli/fuse: add end-to-end truncation test covering ftruncate(fd), syscall.Truncate(path), and open(O_TRUNC) through a real daemon
The fuse-tests job was being silently cancelled by GitHub at 10min because Go's per-test timeout (5m) was the same order as the job timeout, and GOTRACEBACK=single hid the hung goroutines anyway. - shrink TEST_FUSE_TIMEOUT to 4m so Go's panic fires first - shrink job timeout-minutes to 6 (normal run is ~3min) - set GOTRACEBACK=all so the panic dumps every goroutine, not just the timer
…se-tests # Conflicts: # docs/changelogs/v0.41.md
…se-tests # Conflicts: # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum # test/dependencies/go.mod # test/dependencies/go.sum
Without this, the kernel could cache zero attrs after a chmod, touch, or ftruncate until AttrTimeout (1s) expired. Dir.Setattr and Symlink.Setattr already fill out.Attr; FileInode.Setattr now matches.
Only directories backed by keys the node holds are writable. All other names resolve via IPNS to read-only symlinks into the /ipfs mount.
Final pass on #11272 addressing review feedback. - writable: panic in NewDir if Config.DAG is nil. Both call sites already supply it, but a nil value silently fell back to the MFS path in FileInode.Open, re-introducing the rsync --inplace deadlock the read-only fast path was added to fix. - writable: document Dir.Rename non-atomicity. Source unlink happens before destination add, so any failure between the two loses the source. An atomic fix requires changes in boxo/mfs. - writable: add unit test locking in that Symlink.Setattr accepts a mode-only request without erroring and does not store the requested mode (POSIX symlinks have no meaningful permission bits). - docs/config: correct StoreMode default modes; the previous text listed 0666 for files, which the code never uses.
Without this, fstat on the file handle returned by Create reports mode 0 and size 0 for up to AttrTimeout (1s), because the kernel caches the empty attrs from the Create response. Path-based stat goes through Lookup which already fills attrs, so the bug only shows up via fstat. Mirrors the same fix already applied to FileInode.Setattr. Dir.Mkdir gets the same fillAttr treatment for consistency, plus a TODO noting that boxo's mfs.Directory.Mkdir accepts no mode arg so the caller's mode is dropped on creation. Adds CreateAttrsImmediate and MkdirAttrsImmediate to the shared writable suite to guard both paths against future regressions.
When a userspace process is killed mid-read (Ctrl-C, SIGKILL on a stuck cat) the kernel sends FUSE_INTERRUPT and go-fuse cancels the per-request context. fs.ToErrno does not recognise context.Canceled and falls through to "function not implemented", which the kernel cannot act on. Map context.Canceled and DeadlineExceeded to EINTR so the syscall is correctly aborted. - mount/errno.go: new ReadErrno helper used by all context-aware read paths in both readonly and writable mounts - readonly: applied to Node.Open, Node.Readdir, roFileHandle.Read - writable: applied to FileInode.Open, FileHandle.Read, roFileHandle.Read - readonly/ipfs_test.go: TestReadCancellationUnblocks guards the contract via a blocking DagReader fake; without ReadErrno the test reports "function not implemented" instead of EINTR
Coverage gaps in the shared writable suite: - OExcl: lock files and atomic-create patterns rely on the second open with O_CREATE|O_EXCL failing with EEXIST - DirRename: previously only file rename and cross-dir file rename were tested; this exercises Rename on a directory inode - SparseWrite: WriteAt past the end of an empty file must report the correct size and return zeros for the gap - FsyncCrossHandle: a reader on a fresh fd must see data flushed by fsync on the writer fd, not just after close
Previously TestExternalUnmount only exercised /ipfs, leaving the goroutine that watches fuse.Server.Wait() untested for the other two mounts. Refactor into a table-driven test that runs the same fusermount/umount-then-IsActive flow against all three mounts. Switch to coremock.NewMockNode so the node is online: doMount only attaches the /ipns mount when node.IsOnline is true, and the table needs all three populated.
MountCmd's LongDescription has "MFS mounted at:" with two spaces so the column lines up with the 4-char "IPFS" and "IPNS" rows above, but the runtime encoder and the daemon's startup print used a single space and produced misaligned output. Bring both runtime sites in line with the help text, and update the two existing test fixtures (test/cli/fuse and the sharness test-lib helper that t0040-add-and-cat.sh still uses) to expect the aligned form.
FileHandle.Fsync only flushed the MFS file descriptor and left the kernel's cached attrs and content for the inode untouched. A fresh reader on the same path then saw the size cached from the original Create response (zero), reading zero bytes regardless of how much the writer had synced. Mirror the cache invalidation already done in Flush via inode.NotifyContent(0, 0) so a writer that fsyncs while another process opens the file (vim then a follow-up cat, IDE then a language server) sees consistent state. Sharpen the FsyncCrossHandle assertion to report the size delta on failure; the bug surfaced as got=0/want=500 only after switching from bytes.Equal to require.Equal.
The new test_fuse_unit and test_fuse_cli make targets emit test/fuse/fuse-unit-tests.json and test/fuse/fuse-cli-tests.json respectively, the same gotestsum --jsonfile pattern that test_unit and test_cli already use. Add them to the same .gitignore section so a local test run does not leave the working tree dirty.
Adds TestFUSERealWorld in test/cli/fuse/realworld_test.go: a single shared-daemon test with 18 subtests that exercise the writable /mfs mount through the actual binaries users invoke (sh, cat, seq, wc, ls, stat, cp, mv, rm, ln, readlink, find, dd, sha256sum, tar, rsync, vim). Each subtest verifies the result both via the FUSE filesystem and via 'ipfs files read|stat|ls' so both views agree. Synthetic payloads default to 1 MiB + 1 byte so multi-chunk read/write paths are exercised, not just single-chunk fast paths. External tools are required, not optional: a missing binary fails the test loudly so a CI image change cannot silently turn the suite green. The whole-suite TEST_FUSE gate is the only place a developer is allowed to skip. runCmd forces LC_ALL=C so locale-sensitive tool output (date formats in 'ls -l', decimal separators in 'wc', localized error messages, find/ls collation) is deterministic regardless of the runner's locale settings. One shared daemon across all 18 subtests keeps total runtime under two seconds; isolation comes from per-subtest subdirectories under the mount.
lidel
force-pushed
the
feat/consolidate-fuse-tests
branch
from
38bd4e2 to
0c79ec6
Compare
Member
Author
|
This grew well beyond the initial test consolidation, but the bug fix list speaks for itself: over a decade of FUSE issues knocked out, plus end-to-end coverage with real POSIX tools (cat, cp, mv, ls, find, dd, tar, rsync, vim) so we know it actually works the way the changelog claims, and catch regressions going forward. Merging now and shipping in 0.41-rc1 for wider testing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR migrates to hanwen/go-fuse for reasons noted in #11249 (review) and #11272 (comment)
The library is much better, and we can leverage modern FUSE/kernel APIs to map IPFS/MFS/UnixFS abstractions more closely to a real filesystem.
It also fixes over a decade of FUSE bugs, making it much more useful.
Test consolidation
Consolidate all FUSE tests so they actually run in CI.
test/cli/fuse_test.gointo atest/cli/fuse/sub-packagemake test_fuseintotest_fuse_unit(./fuse/...) andtest_fuse_cli(./test/cli/fuse/...)TEST_FUSE=0intest_cliso FUSE tests skip in thecli-testsCI jobfuse-testsCI jobFUSE library migration
Replaced unmaintained
bazil.org/fusewith actively maintainedhanwen/go-fusev2. This fixes two architectural deadlocks that bazil could never solve:ftruncateandfsyncnow work because go-fuse passesthe open file handle to Setattr/Fsync, letting us use the existing write descriptor.
Fixes
rsync --inplaceand tools that shrink/grow files via ftruncate no longer get ENOTSUPln -sworks on/mfsand/ipns, stored as UnixFS TSymlink nodesls -landfind -type lsee the right file typecatworks: interrupting a read cancels in-flight block fetchesfusermount -ufrom outside the daemon marks the mount inactiveipfs_cidxattr normalized: returns the CID (same asipfs.cid) with an error log, instead of ENOATTRNew features
Mounts.StoreMtimeandMounts.StoreMode: opt-in config flags to persist mtime and POSIX mode in UnixFS metadata on writable mountsipfs.cidxattr on all mounts: extended attribute exposing the node's CIDCAP_ATOMIC_O_TRUNC: kernel sends O_TRUNC in Open instead of a separate SETATTR, avoiding a second write descriptor deadlockvolname,noapplexattr,noappledoubleset automaticallyArchitecture
Dir,FileInode,FileHandle,Symlink) intofuse/writable/package, used by both/mfsand/ipnsfusetest.RunWritableSuite) with ~30 scenarios run by both mountstest/cli/fuse/sub-package with new test cases (sharded dirs, IPNS resolution, publish-while-mounted)(linux || darwin || freebsd) && !nofuseto match actual go-fuse platform supportfusermount3handled natively by go-fuse, CI symlink workaround removedmake test_fusesplit intotest_fuse_unitandtest_fuse_clitest_cliviaTEST_FUSE=0, run in dedicatedfuse-testsCI jobReferences
O_TRUNC) does not work on/mfsand/ipnsmounts #11268echo/cptruncation fixed (same root cause as FUSE: file truncation (O_TRUNC) does not work on/mfsand/ipnsmounts #11268)fuse.hheader check removedfuse-testsjob (unit + CLI)ipfs mountreturns "could not resolve name" #7099 - ipfs mount no longer errors with "could not resolve name" when the local IPNS key has no published record; CreateRoot catches namesys.ErrResolveFailed and falls back to an empty directoryLikely fixed (no way to reproduce reporter's exact environment)
server.Wait()server.Unmount(), CI uses lazy unmount for cleanupcancellation fix means a stuck access can now be killed
each run. (No dedicated ipfs mount clean command was added, so this is partial.)