Parallelize (multithread) ILSpyCmd Project Decompilation

[petercrabtree]

Why I’m opening this

• ILSpyCmd is single-threaded, so decompiling large solutions is slower than it needs to be.
• There’s no progress output, which makes it hard to tell whether the tool is still working or has hung.

What I’m proposing

I’ve built a multithreaded, progress-emitting version of ILSpyCmd and done some light refactors to make the new concurrency easier to reason about.

• Draft PR with the full change: Implement Project Parallelism in IlSpyCmd (Stacked PR #5) petercrabtree/ILSpy#3

• It currently stacks on:

  • SDK-style <ProjectReference> fix (in-progress upstream PR: Fix SDK-style (modern) inter-project references #3502)
  • Minor cleanup PR that’s already in my fork: dev: Clean up .editorconfig petercrabtree/ILSpy#2

How I can help

@christophwille @siegfriedpammer – just flagging this for visibility (let me know if there’s a better contact path).

I used these code changes to learn Jujutsu (jj, which is roughly a more modern git), so I'd be happy to slice these commits/PRs however fits the preferred merge/test process.

Please let me know how I can best work with y'all on this. Thanks!

[siegfriedpammer]

Thank you for your interest in improving ilspycmd! Your proposed ideas make a lot of sense. I think you can just open one or more PRs.

[siegfriedpammer]

I had a look at this again: I now realized that you want to parallelize over all projects. Currently the decompilation of each project is performed sequentially, but the WholeProjectDecompiler already parallelizes over the top-level types in each assembly.
Is adding more parallelism still beneficial? If this is added we need to make sure that we do not cause starvation issues by properly distributing cores between the two "parallelizers".

[petercrabtree]

@siegfriedpammer Thanks for the review! I hadn’t realized WholeProjectDecompiler was already parallelized.

You’re right that we should avoid the extra context-switching costs of oversubscribing threads to cores. I’ll look into aligning the two approaches so we get closer to optimal parallelism—likely aiming for roughly one active task per logical core by default.

On whether additional parallelism helps: in my case, I saw a huge speedup (around 10X). Before my changes, CPU usage in "steady state" barely exceeded a single core without project-level parallelism—possibly due to the structure of the DLLs I was decompiling.

There’s also the I/O angle: if threads spend time waiting on I/O, it may make sense to oversubscribe slightly so otherwise-idle cores can do useful work.

[petercrabtree]

Context: While decompiling a large C# game with no project-level parallelism, overall CPU stays far below saturation (around 10% most of the time). (Total time from start to end is ~5 minutes.)

Environment: Win 11, 32 logical cores, 64GB RAM, fast NVMe SSD

(Green is kernel time.)

Hypotheses

• Straggler types: In this scenario, some types are enormous (e.g., autogenerated, XML-serializing classes with thousands of methods). With type-level batching, the batch cannot complete (and thus the next batch cannot start) until the slowest type finishes.
• Synchronous I/O: In all scenarios, writes block the worker thread, assuming I've understood the code correctly. With one worker per logical core, any pending I/O suspends useful work (so unless storage is extremely fast, this will prevent CPU saturation).

Ideal but invasive: A full async pipeline with custom task schedulers and a producer/consumer queue would keep the CPU busy across I/O waits and reduce context-switch overhead. I’ve built similar systems before, but this would be a significant architectural change.

Practical proposal:

1. Enable project-level parallelism (as in my current branch) and intentionally oversubscribe by a factor K per core (guessing we should default K = 4, but I'll benchmark). Make this configurable in ILSpyCmd with clear help text about trade-offs.

2. At IlSpyCmd startup, set a higher worker baseline: ThreadPool.SetMinThreads(cores * K, [...]) to shorten ramp-up. (But benchmark and make sure this helps!)

Pros

• Keeps a "runnable backlog" (~cores × K total threads/workers), so idle cores pick up work immediately despite any blocking IO or stragglers. This should translate to higher throughput in most cases, with the cost of the extra threads more than made up for by higher CPU utilization.
• Low risk surface; minimal churn to existing code.
• Should always help whenever (A) multiple projects are decompiled and (B) any project has slow or lopsided workloads.

Cons

• Higher default memory usage; could OOM on low-RAM machines. (Users can set K to 1 to get the old behavior.)
• In certain scenarios where the current parallelism sustains near-100% CPU usage, the extra context switches and cache misses and whatnot may not be sufficiently compensated for by increased CPU utilization, causing a small perf decrease. (But again, users can tune K, and I expect this would be <10% in the worst case with K <= 4.)

Notes

• Slow storage scenarios will increase IO queue depth (roughly by a factor of K) under this model, which can help or hurt depending on the storage.
• We could alternatively mitigate IO stalls by increasing buffering: We could fully buffer each file before writing, for example, and even move the actual write to a dedicated IO thread. However, this doesn't address the tail-latency (straggler) issue, which I'm thinking is the actual major issue (edit: if it were primarily IO, I expect we wouldn't see the stair-step pattern in the CPU usage).

Thoughts? @siegfriedpammer

[siegfriedpammer]

Thank you for the analysis and your proposal. I am a bit confused by your observations, because I would think that we are not IO, but CPU-bound therefore the cores should work at 100% utilization most of the time.

If we are indeed IO-bound then this might indicate that we are doing IO wrong. Looking at our code the StreamWriter/TextWriter is created here:

ILSpy/ICSharpCode.Decompiler/CSharp/ProjectDecompiler/WholeProjectDecompiler.cs

Lines 191 to 194 in ff24716

	protected virtual TextWriter CreateFile(string path)
	{
	return new StreamWriter(path);
	}

Looking at the implementation of StreamWriter..ctor(string):

https://github.com/dotnet/runtime/blob/1d1bf92fcf43aa6981804dc53c5174445069c9e4/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs#L132-L135

The value for DefaultBufferSize is 1024 bytes, which is probably too low for our use-case.

Can you run some more tests with different buffer sizes?

[dgrunwald]

I doubt we're IO-bound.
It's more likely that most of the time is spent in one particularly big class.

I'm also not sure if .NET's work-stealing is fine-grained -- if one is class a straggler taking a long time, will the other threads take care of all other classes, or will there be classes that won't start processing until the straggler is done?

[petercrabtree]

The current IO is blocking, and we have 1 thread per logical core, so any time spent waiting on a write syscall directly corresponds to an idle logical core. (That said, the OS file system write cache should make writes super fast, unless the drive is under enough write pressure or we're writing to something that doesn't get write caching (like removable drives in Windows by default) or something is causing flushes.)

That said, I think IO is a secondary issue (if it is one at all).

I'm about 95% sure that the relevant partitioner uses chunks even when "load balancing" is enabled, with chunk size doubling every 3 times a chunk gets processed, and the max being ~128, which definitely would explain some of the non-optimal behavior. Good point!

I'll see what replacing the partitioner with one with a max chunk size of 1 via the "NoBuffering" option does to perf.

[petercrabtree]

Short summary of what I found, will write up tomorrow:

• The partitioner change had negligible impact for me. (Probably still a good change.)
• A small number of extremely large types is definitely causing low CPU utilization.
• At least one current solution decompilation path already nests two Parallel.ForEach loops, which will oversubscribe cores. Active work items grow to N^2 for N cores.
• I'm excited about an idea I had on how to solve this; I think there's a pretty simple hybrid approach with no obvious downside.

[petercrabtree]

Findings so Far

Huge Types is an Issue

• In one particular assembly that takes ≈ 50 s to decompile, all but two types complete by 2 s.
• The two stragglers run in parallel and emit large outputs (≈ 7 MB and ≈ 22 MB).

Net effect: Low CPU utilization when serially decompiling such assemblies (how ilspycmd works today).

Current Parallel Solution Decompilation

• SolutionWriter.CreateSolution() runs Parallel.ForEach over projects with max degree Environment.ProcessorCount.
• Each task calls CSharpLanguage.DecompileAssembly(), which constructs an ILSpyWholeProjectDecompiler (subclass of WholeProjectDecompiler).
• That decompiler also uses Parallel.ForEach with the same max degree for types.
• Note: This path was identified by following uses of Partitioner.Create() and is presumed in use; I didn’t check.
• My parallelized ilspycmd branch currently uses the same strategy.

Net effect: A max of O(N^2) queued tasks, which is undesirable (even though N is fixed per machine), and on top of that, N thread pool threads spend most of their lifetimes blocked, waiting for Parallel.ForEach() calls to finish, which then limits concurrency until the thread pool's hill climbing algorithm catches up.

Proposal Draft

Changes

• Add ACTIVE_TASK_LIMIT (L) and BACKLOG_LOW_WATERMARK (W) config (default both to Environment.ProcessorCount (N)) (TODO: I don't love these names.).

• In the solution decompilation code (async orchestrator):

  • Create an unbounded, prioritized channel for storing types-to-decompile.
  • Spin up L async tasks on the thread pool, each of which reads from the channel until it's closed, doing synchronous decompilation of each type.
  • Monitor the channel's size. If it is <= (W - L), start a new project (synchronously, so there's no race condition when we get back to monitoring the channel size). (TODO: How do we efficiently check for this without spinning? Interlocked + TaskCompletionSource?)

• WholeProjectDecompiler:

  • Take a channel parameter.

  • Instead of Parallel.ForEach() over the type batch:

    • Queue the entire batch to the channel
    • Wait for queued types to finish (TODO: Interlocked + TaskCompletionSource? CountdownEvent?)

Why This Approach

• Keeps every core active (until there are <L types left in all remaining projects), despite any configuration of slow-to-decompile types.

• Perfectly controlled concurrency

  • Oversubscription shouldn't be necessary to keep cores mostly active
  • Configurable oversubscription (can oversubscribe to keep cores 100% active).
  • Configurable "start-next-project"

• Prioritized tasks keep the work focused on particular projects (good for cache locality).

• Doesn't change any thread safety properties, because the changes are surgical and only change how the work gets kicked off, the work is still done in the same way.

• No threads wasted, blocked waiting on Parallel.ForEach(), no additional need for the thread pool to grow.

Secondary Perf Ideas

Edit: These aren't fully baked, but this seemed like the best place to put some notes on how to improve perf further.

• In UI contexts, instead of the thread pool, create dedicated, low-priority threads for WholeProjectDecompiler tasks (may keep ILSpy and the user's machine more responsive).

• Sort work to do the biggest first:

  • Projects by file size.
  • Types by ... IL Size? (TODO)

• Instrument WholeProjectDecompiler to allow using dotnet-trace to collect parallelism issues.

  • I have a prototype implementation of this I've been using for digging into this issue, might be worth merging.

• Optionally fully cache decompilation output files (write to a RecyclableMemoryStream  during decompilation, write to disk when done with each file).

  • Pros:

    • Gives the file system the least amount of total work (no file resizing, for example), so should never be significantly slower (assuming sufficient RAM, the only cost is increased GC pressure).

      • Also minimizes SSD write amplification.

    • Reduces file fragmentation

      • Guarantees a maximally sequential write (and usefully, read) of output
      • Matters less these days, but even high end NVMe drives are usually faster sequentially.

    • Significantly improves performance in some scenarios

      • The OS write cache usually hides write latency, but that doesn't work for fairly common encrypted/ compressed/ removable-drive/ spinning-disk/ network-file-system/ cache-pressure scenarios.)

  • Cons:

    • Higher memory usage.

      • I'm not very concerned. RAM is cheap, source code files are tiny compared with the memory used in decompiling them.
      • We can make it an option for low-memory users.

• Tune limit defaults:

  • I’d bet Ceiling(Environment.ProcessorCount * 1.1) is a better default for ACTIVE_TASK_LIMIT, allowing for a couple of extra ready-to-run tasks/threads to “cover up” any blocking operations on the threads.
  • There may be a BACKLOG_LOW_WATERMARK > N which is a better default (2N?); waiting until there are free cores to start the next project might be too late to utilize all available CPU time.

[petercrabtree]

I implemented most of the above as a hack to prototype; it's working well but the code will need a refactoring pass (or two!) -- CPU usage fits the pattern we want (full until remaining_files < cores), sorting the types and files to start with the biggest first works well to get the giant types started early in the process.

Screen.Recording.2025-08-29.145833.mp4