Rearchitect frontend and improve documentation and performance

[pascalecu]

Summary

Reduce the compiler's peak memory usage by eliminating unnecessary in-memory storage during token processing.
Also, I improved how the diagnostic engine works by doing less work, as well as the output of said diagnostics (they now take the severity into account to use an appropriate foreground color). The filesystem logic has also been migrated to use coroutines, which will be very helpful later on when dealing with multiple files and incremental parsing contexts such as LSPs. That filesystem logic is now also integrated into the source manager, where it was previously unused and a lot of duplication happened between them.

Results

Benchmarks across GCC/Clang and Debug/Release builds show:

• Peak memory reduced from ~3.2 GB to ~130 MB (~25× reduction)
• Lower runtime when stdout is redirected to /dev/null (which represents raw token processing without printing)
  • Debug GCC: ~11% faster
  • Debug Clang: ~29% faster
  • Release GCC: ~16% faster
  • Release Clang: ~19% faster
• Runtime with token printing enabled (stdout) remains largely unchanged, indicating that terminal I/O dominates execution time in that mode, but there is a measurable difference.

Why

The previous implementation retained significantly more data in memory than necessary while processing tokens. This change makes the compiler process data in a more streaming-oriented manner, reducing the memory footprint without affecting observable behavior.

Benchmark configuration

Benchmarks were run:

• 5 iterations per configuration
• GCC and Clang
• Debug and Release builds
• Both with token output enabled and with output redirected to /dev/null

Notes

The memory reduction is consistent across all tested configurations. Redirecting output to /dev/null also reveals measurable runtime improvements, while interactive stdout runs remain primarily limited by output overhead rather than compiler performance.

These are the raw results:

Before

Azin Compiler Performance Report
--------------------------------
Benchmarking: debug/gcc (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 167.65 s | User: 85.55 s | Sys: 68.51 s | Peak RAM: 3327480 KB
    Run 2: Real: 159.87 s | User: 82.29 s | Sys: 66.05 s | Peak RAM: 3327532 KB
    Run 3: Real: 158.09 s | User: 81.66 s | Sys: 65.23 s | Peak RAM: 3327880 KB
    Run 4: Real: 159.86 s | User: 82.49 s | Sys: 66.25 s | Peak RAM: 3327696 KB
    Run 5: Real: 160.14 s | User: 82.39 s | Sys: 66.51 s | Peak RAM: 3327532 KB
  Mode: redirected to /dev/null
    Run 1: Real: 39.34 s | User: 37.24 s | Sys: 2.08 s | Peak RAM: 3327920 KB
    Run 2: Real: 39.79 s | User: 37.77 s | Sys: 1.99 s | Peak RAM: 3327360 KB
    Run 3: Real: 37.96 s | User: 35.96 s | Sys: 1.98 s | Peak RAM: 3327556 KB
    Run 4: Real: 40.52 s | User: 38.39 s | Sys: 2.08 s | Peak RAM: 3327524 KB
    Run 5: Real: 38.87 s | User: 36.76 s | Sys: 2.08 s | Peak RAM: 3327340 KB

Benchmarking: debug/clang (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 178.46 s | User: 103.47 s | Sys: 64.99 s | Peak RAM: 3278136 KB
    Run 2: Real: 180.93 s | User: 105.20 s | Sys: 65.17 s | Peak RAM: 3278376 KB
    Run 3: Real: 176.32 s | User: 102.10 s | Sys: 64.35 s | Peak RAM: 3278264 KB
    Run 4: Real: 177.28 s | User: 103.05 s | Sys: 64.41 s | Peak RAM: 3278280 KB
    Run 5: Real: 179.93 s | User: 103.68 s | Sys: 66.19 s | Peak RAM: 3278176 KB
  Mode: redirected to /dev/null
    Run 1: Real: 64.27 s | User: 62.16 s | Sys: 2.03 s | Peak RAM: 3278168 KB
    Run 2: Real: 64.25 s | User: 62.11 s | Sys: 2.10 s | Peak RAM: 3277956 KB
    Run 3: Real: 63.54 s | User: 61.46 s | Sys: 2.05 s | Peak RAM: 3277824 KB
    Run 4: Real: 62.22 s | User: 60.12 s | Sys: 2.06 s | Peak RAM: 3277916 KB
    Run 5: Real: 62.75 s | User: 60.64 s | Sys: 2.07 s | Peak RAM: 3277904 KB

Benchmarking: release/gcc (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 111.62 s | User: 30.68 s | Sys: 64.66 s | Peak RAM: 3327756 KB
    Run 2: Real: 119.23 s | User: 33.86 s | Sys: 68.03 s | Peak RAM: 3327720 KB
    Run 3: Real: 116.74 s | User: 32.05 s | Sys: 67.48 s | Peak RAM: 3327464 KB
    Run 4: Real: 114.94 s | User: 31.44 s | Sys: 66.29 s | Peak RAM: 3327652 KB
    Run 5: Real: 118.23 s | User: 32.46 s | Sys: 68.19 s | Peak RAM: 3327808 KB
  Mode: redirected to /dev/null
    Run 1: Real: 8.05 s | User: 6.04 s | Sys: 2.01 s | Peak RAM: 3327884 KB
    Run 2: Real: 8.26 s | User: 6.24 s | Sys: 2.01 s | Peak RAM: 3327684 KB
    Run 3: Real: 8.15 s | User: 6.13 s | Sys: 2.01 s | Peak RAM: 3327800 KB
    Run 4: Real: 8.14 s | User: 6.12 s | Sys: 2.01 s | Peak RAM: 3327644 KB
    Run 5: Real: 8.20 s | User: 6.16 s | Sys: 2.03 s | Peak RAM: 3327860 KB

Benchmarking: release/clang (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 113.20 s | User: 29.98 s | Sys: 66.18 s | Peak RAM: 3277844 KB
    Run 2: Real: 116.62 s | User: 31.00 s | Sys: 67.91 s | Peak RAM: 3278164 KB
    Run 3: Real: 115.24 s | User: 30.48 s | Sys: 67.56 s | Peak RAM: 3277880 KB
    Run 4: Real: 112.81 s | User: 29.97 s | Sys: 65.98 s | Peak RAM: 3278224 KB
    Run 5: Real: 112.82 s | User: 29.83 s | Sys: 66.22 s | Peak RAM: 3277888 KB
  Mode: redirected to /dev/null
    Run 1: Real: 7.62 s | User: 5.55 s | Sys: 2.06 s | Peak RAM: 3277864 KB
    Run 2: Real: 8.20 s | User: 5.89 s | Sys: 2.30 s | Peak RAM: 3278240 KB
    Run 3: Real: 8.27 s | User: 5.96 s | Sys: 2.29 s | Peak RAM: 3278196 KB
    Run 4: Real: 7.47 s | User: 5.40 s | Sys: 2.07 s | Peak RAM: 3277904 KB
    Run 5: Real: 7.57 s | User: 5.50 s | Sys: 2.07 s | Peak RAM: 3278176 KB

After

Azin Compiler Performance Report
--------------------------------
Benchmarking: debug/gcc (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 166.82 s | User: 90.45 s | Sys: 67.35 s | Peak RAM: 133064 KB
    Run 2: Real: 156.92 s | User: 84.93 s | Sys: 63.48 s | Peak RAM: 133076 KB
    Run 3: Real: 154.86 s | User: 85.25 s | Sys: 64.45 s | Peak RAM: 133104 KB
    Run 4: Real: 164.34 s | User: 88.81 s | Sys: 66.46 s | Peak RAM: 133124 KB
    Run 5: Real: 160.97 s | User: 84.89 s | Sys: 65.87 s | Peak RAM: 132960 KB
  Mode: redirected to /dev/null
    Run 1: Real: 36.82 s | User: 36.53 s | Sys: 0.18 s | Peak RAM: 133048 KB
    Run 2: Real: 36.16 s | User: 35.91 s | Sys: 0.19 s | Peak RAM: 133084 KB
    Run 3: Real: 33.23 s | User: 33.00 s | Sys: 0.16 s | Peak RAM: 133076 KB
    Run 4: Real: 34.51 s | User: 34.32 s | Sys: 0.15 s | Peak RAM: 133120 KB
    Run 5: Real: 33.32 s | User: 33.07 s | Sys: 0.16 s | Peak RAM: 133040 KB

Benchmarking: debug/clang (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 184.71 s | User: 106.17 s | Sys: 67.88 s | Peak RAM: 132752 KB
    Run 2: Real: 174.49 s | User: 102.77 s | Sys: 64.42 s | Peak RAM: 132828 KB
    Run 3: Real: 175.31 s | User: 104.24 s | Sys: 64.09 s | Peak RAM: 132784 KB
    Run 4: Real: 173.93 s | User: 103.27 s | Sys: 63.99 s | Peak RAM: 132732 KB
    Run 5: Real: 174.39 s | User: 104.00 s | Sys: 64.37 s | Peak RAM: 132652 KB
  Mode: redirected to /dev/null
    Run 1: Real: 47.04 s | User: 46.82 s | Sys: 0.17 s | Peak RAM: 132700 KB
    Run 2: Real: 44.17 s | User: 43.99 s | Sys: 0.16 s | Peak RAM: 132696 KB
    Run 3: Real: 45.91 s | User: 45.69 s | Sys: 0.17 s | Peak RAM: 132660 KB
    Run 4: Real: 44.08 s | User: 43.92 s | Sys: 0.15 s | Peak RAM: 132704 KB
    Run 5: Real: 44.49 s | User: 44.33 s | Sys: 0.14 s | Peak RAM: 132720 KB

Benchmarking: release/gcc (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 111.92 s | User: 35.75 s | Sys: 66.13 s | Peak RAM: 132952 KB
    Run 2: Real: 112.60 s | User: 35.59 s | Sys: 66.41 s | Peak RAM: 132888 KB
    Run 3: Real: 119.56 s | User: 37.04 s | Sys: 70.49 s | Peak RAM: 132868 KB
    Run 4: Real: 118.28 s | User: 36.35 s | Sys: 69.23 s | Peak RAM: 132876 KB
    Run 5: Real: 122.04 s | User: 37.34 s | Sys: 72.10 s | Peak RAM: 132944 KB
  Mode: redirected to /dev/null
    Run 1: Real: 6.97 s | User: 6.79 s | Sys: 0.17 s | Peak RAM: 132932 KB
    Run 2: Real: 7.24 s | User: 7.07 s | Sys: 0.15 s | Peak RAM: 132928 KB
    Run 3: Real: 6.38 s | User: 6.20 s | Sys: 0.17 s | Peak RAM: 132880 KB
    Run 4: Real: 6.57 s | User: 6.40 s | Sys: 0.16 s | Peak RAM: 132880 KB
    Run 5: Real: 7.01 s | User: 6.85 s | Sys: 0.15 s | Peak RAM: 132928 KB

Benchmarking: release/clang (5 runs)
  Mode: with stdout (printing tokens)
    Run 1: Real: 120.95 s | User: 33.92 s | Sys: 72.83 s | Peak RAM: 132332 KB
    Run 2: Real: 131.81 s | User: 35.34 s | Sys: 77.89 s | Peak RAM: 132324 KB
    Run 3: Real: 127.45 s | User: 35.55 s | Sys: 76.37 s | Peak RAM: 132416 KB
    Run 4: Real: 124.88 s | User: 33.91 s | Sys: 73.96 s | Peak RAM: 132332 KB
    Run 5: Real: 117.51 s | User: 33.06 s | Sys: 70.54 s | Peak RAM: 132348 KB
  Mode: redirected to /dev/null
    Run 1: Real: 5.68 s | User: 5.48 s | Sys: 0.17 s | Peak RAM: 132304 KB
    Run 2: Real: 6.28 s | User: 6.05 s | Sys: 0.16 s | Peak RAM: 132348 KB
    Run 3: Real: 6.88 s | User: 6.67 s | Sys: 0.18 s | Peak RAM: 132352 KB
    Run 4: Real: 6.43 s | User: 6.26 s | Sys: 0.16 s | Peak RAM: 132376 KB
    Run 5: Real: 6.65 s | User: 6.47 s | Sys: 0.17 s | Peak RAM: 132352 KB

Final notes

Sorry for the bad docs, I am tired as hell and I really want to get this over with.

[pascalecu]

I didn't realize the file IO stuff isn't implemented for non-{Windows,Linux} platforms. I'll roll up my own abstraction, but not now, I'll go to sleep. I could just maintain our own fork here, but I'd rather contribute to upstream.