feat: add custom walker support by tobim · Pull Request #698 · numtide/treefmt

tobim · 2026-05-15T06:45:01Z

As the title says.

Example: Format a git repo + a specific submodule:

walk = "repoPlusSubmodule"

[walker.repoPlusSubmodule]
command = "bash"
options = [
  "-c",
  '''
    submodule="path/to/submodule"
    {
      git ls-files --cached --others --exclude-standard --full-name
      git -C "$submodule" ls-files --cached --others --exclude-standard --full-name \
        | sed "s#^#$submodule/#"
    }
  ''',
]

Example: Format a pijul repo:

walk = "pijul"

[walker.pijul]
command = "pijul"
options = ["ls"]

Add documentation for selecting a custom walker with the global walk option and defining its command and options under [walker.<name>]. Document that custom walker commands run from the tree root, emit one relative path per line, and do not need to handle path arguments.

jfly

@tobim, thanks for the contribution! I am supportive of this change, but haven't had a chance to read the code yet.

I wanted to warn you that #694 touches some of the same code you've touched here. It may land before this PR, apologies if the conflicts are annoying to deal with.

jfly

Thanks! This looks like a solid approach.

Not a blocker, but I'm not in love with a 3rd copy of the walk code. I didn't read it particularly closely. I think 2 copies (jj and git) is OK. 3 is when it probably makes sense to to think of an abstraction. Did you think about that at all?

Accept documented camel-case walker names even when Viper normalizes table keys

What is camel-case? That's kebab-case. This is camelCase (or CamelCase).

Fix custom walker pipe handling

I did not have time to grok this commit during my review. I'll need some more handholding before I can approve it. Does whatever bug you're fixing here apply to the git and jj walkers as well? Is it related to this bug @Mic92 found in #694 (commit labeled "walk: unblock producers when Close() is called before EOF").

jfly · 2026-05-27T04:46:19Z

+When you pass directory paths to `treefmt`, the walker command still runs for the tree root.
+`treefmt` filters the command output to the requested directories.
+The walker command doesn't need to implement path argument handling.


While I appreciate the simplicity of this decision, I'm not sure it's the right decision. I imagine people are most inclined to use the "format a directory" feature in very large repos, where the time spent for some VCS to determine which files are in the entire repo might be a lot slower than discovering just the files in a specific directory. #694 is an example of someone reporting just how slow it is for git to report all files in a repo.

How crazy would it be for the walk command to have to implement path argument handling? IIUC, it would be straightforward for git.

I admit that I didn't pay too much attention to optimizing for subtrees. It's no concern for my own use cases.

For a regular git walker this might not be a problem, but as soon as you want a more custom file list - like in the first example from the PR description - implementing the filtering yourself becomes error-prone rather quickly.

Maybe we'll add a toggle so the user can switch between both methods?

jfly · 2026-05-27T04:52:11Z

+# You can also set this to the name of a configured custom walker
 # Env $TREEFMT_WALK
 # walk = "filesystem"
+# walk = "myWalker"


I wonder if we should instead do walker.myWalker (or perhaps custom-walker.myWalker) here. Then we wouldn't have to worry about (or detect) conflicts with the builtin walkers.

jfly · 2026-05-27T05:14:37Z

-		reader, err = NewReader(Git, root, path, db, statz)
-		if err != nil {
-			reader, err = NewReader(Jujutsu, root, path, db, statz)
+	if selector.Custom != nil {


This diff would be a lot simpler, and the if selector.Custom == nil && selector.Type == Stdin change below wouldn't have had to happen if we introduced a new selector.Type == Custom. Did you consider that?

tobim · 2026-05-27T09:13:45Z

Thank you for the review. I hope to be able to address the individual comments in the next few days.

Not a blocker, but I'm not in love with a 3rd copy of the walk code. I didn't read it particularly closely. I think 2 copies (jj and git) is OK. 3 is when it probably makes sense to to think of an abstraction. Did you think about that at all?

I didn't really consider that because I wanted to keep the new path somewhat isolated.
We could also replace git and jj with built in custom walkers now that the machinery is there.

Does whatever bug you're fixing here apply to the git and jj walkers as well? Is it related to this bug @Mic92 found in #694 (commit labeled "walk: unblock producers when Close() is called before EOF").

I'll have to look into that.

jfly · 2026-05-27T12:55:18Z

I didn't really consider that because I wanted to keep the new path somewhat isolated.
We could also replace git and jj with built in custom walkers now that the machinery is there.

This sounds reasonable to me

Parse [walker.<name>] tables into the configuration model and validate walker names, commands, and walk values. Accept documented camelCase walker names even when Viper normalizes table keys.

Introduce a command-backed walker reader that runs from the tree root, reads one path per stdout line, filters configured subpaths inside treefmt, and converts accepted paths into walk.File values. Route custom walk names through the format command without changing built-in walkers.

Cover selecting a custom walker from treefmt.toml, passing configured walker options, and filtering walker output when treefmt is invoked with a directory path argument.

Use bash -c for custom walker test commands instead of temporary scripts with /usr/bin/env shebangs, so the tests also run inside the Nix build sandbox.

Make the custom walker own its stdout and stderr pipes instead of combining StdoutPipe with a concurrent Wait call. This avoids surfacing a spurious file-closed read error at process exit and preserves command failures for Close().

Refactor custom walker configuration lookup to satisfy nesting limits, remove an unused gosec suppression, and adjust test spacing to satisfy golangci-lint in the Nix check derivation.

Remove the case-insensitive custom walker lookup and rely on exact configured names. Move the default walk value into NewViper so FromViper no longer silently rewrites an empty walk setting. Use lowercase custom walker names in examples and tests, and use --clear-cache in the custom walker CLI test.

Represent selector variants separately from the built-in walker enum. Keep selector fields private and expose IsBuiltin for callers that need to test for stdin.

Build the uncached reader in a helper and apply the cache wrapper once in NewReader. Keep auto fallback inside uncached construction so trying built-in walkers does not recurse through cache setup.

Use PathStreamConfig.PathFilters directly when post-filtering emitted paths. Remove the redundant filters field from PathStreamReader.

Store filesystem path filters as a slice and walk them sequentially in the filesystem reader. Remove the generic CompositeReader that was only used to fan out multiple filesystem paths. Add coverage for multi-path filesystem walking.

tobim · 2026-06-01T08:06:35Z

So I did somewhat of a refactor of the walker component now. Now there is a generic PathStreamReader abstraction that consumes a stream of paths that are separated by either a newline or a NULL character. jj and git are now constructed in terms of that PathStreamReader.
After that are a couple of clean-up commits that simplify the state representation and related control flow.

Record whether the walker command returned after its context was canceled. Treat those wait errors as expected during Close so platforms that report cancellation as signal termination do not fail the unblock test.

This was referenced May 15, 2026

feat: Allow recursing into submodules when using git walker #507

Closed

Darcs walker #656

Open

jfly reviewed May 18, 2026

View reviewed changes

jfly reviewed May 27, 2026

View reviewed changes

tobim added 16 commits May 31, 2026 19:30

Add custom walker configuration

d99461c

Parse [walker.<name>] tables into the configuration model and validate walker names, commands, and walk values. Accept documented camelCase walker names even when Viper normalizes table keys.

Test custom walker integration

cb50b18

Cover selecting a custom walker from treefmt.toml, passing configured walker options, and filtering walker output when treefmt is invoked with a directory path argument.

Make custom walker tests sandbox-compatible

232df26

Use bash -c for custom walker test commands instead of temporary scripts with /usr/bin/env shebangs, so the tests also run inside the Nix build sandbox.

Fix custom walker pipe handling

5f64510

Make the custom walker own its stdout and stderr pipes instead of combining StdoutPipe with a concurrent Wait call. This avoids surfacing a spurious file-closed read error at process exit and preserves command failures for Close().

Fix custom walker lint issues

36f6675

Refactor custom walker configuration lookup to satisfy nesting limits, remove an unused gosec suppression, and adjust test spacing to satisfy golangci-lint in the Nix check derivation.

Document custom walker path filters

4258265

Use explicit walker selectors

7b717d2

Add shared path stream walker

f3b518f

Use path streams for built-in walkers

6fc73ba

Unblock path stream walkers on close

59274e1

Simplify walker selector state

94c0a56

Represent selector variants separately from the built-in walker enum. Keep selector fields private and expose IsBuiltin for callers that need to test for stdin.

Split walker reader construction

5b81b3b

Build the uncached reader in a helper and apply the cache wrapper once in NewReader. Keep auto fallback inside uncached construction so trying built-in walkers does not recurse through cache setup.

Avoid duplicate path stream filter state

f231482

Use PathStreamConfig.PathFilters directly when post-filtering emitted paths. Remove the redundant filters field from PathStreamReader.

Let filesystem walker handle multiple paths

69c5a22

Store filesystem path filters as a slice and walk them sequentially in the filesystem reader. Remove the generic CompositeReader that was only used to fan out multiple filesystem paths. Add coverage for multi-path filesystem walking.

tobim force-pushed the push-spzzmzrpkzzq branch from 7e5f3af to 69c5a22 Compare June 1, 2026 07:53

Handle signal exits after walker close

63f7764

Record whether the walker command returned after its context was canceled. Treat those wait errors as expected during Close so platforms that report cancellation as signal termination do not fail the unblock test.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add custom walker support#698

feat: add custom walker support#698
tobim wants to merge 18 commits into
numtide:mainfrom
tobim:push-spzzmzrpkzzq

tobim commented May 15, 2026

Uh oh!

jfly left a comment

Uh oh!

jfly left a comment

Uh oh!

jfly May 27, 2026

Uh oh!

tobim May 27, 2026

Uh oh!

jfly May 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jfly May 27, 2026

Uh oh!

Uh oh!

tobim commented May 27, 2026

Uh oh!

jfly commented May 27, 2026 •

edited

Loading

Uh oh!

tobim commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tobim commented May 15, 2026

Uh oh!

jfly left a comment

Choose a reason for hiding this comment

Uh oh!

jfly left a comment

Choose a reason for hiding this comment

Uh oh!

jfly May 27, 2026

Choose a reason for hiding this comment

Uh oh!

tobim May 27, 2026

Choose a reason for hiding this comment

Uh oh!

jfly May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jfly May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tobim commented May 27, 2026

Uh oh!

jfly commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobim commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfly commented May 27, 2026 •

edited

Loading