feat(cli): auto-download buf binary when not on PATH

[Swimburger]

Description

Refs: Follow-up to #13667 (protoc-gen-openapi auto-download)

Auto-downloads the buf CLI binary (v1.66.1) from GitHub Releases when it is not found on PATH. This eliminates the requirement for users to pre-install buf before running proto-based SDK generation.

Changes Made

• New BufDownloader.ts: Downloads buf from GitHub Releases, caches at ~/.fern/bin/ with versioned filenames, filesystem locking (mkdir-based), and atomic file operations (write-to-temp + rename). Follows the same pattern as ProtocGenOpenAPIDownloader.ts.
• New shared ensureBufCommand() in utils.ts: Extracted the buf resolution logic (check PATH via which, fall back to auto-download) into a standalone helper function. Both generators delegate to this shared utility, avoiding duplication.
• Modified ProtobufIRGenerator.ts and ProtobufOpenAPIGenerator.ts: Both generators now call ensureBufResolved() at the start of generateLocal(). Each generator's ensureBufResolved() is a thin wrapper that caches the result on the instance and delegates to the shared ensureBufCommand(). The resolved command (either "buf" or the full cached path) is passed through to all buf invocations.
• Modified ProtobufOpenAPIGenerator.ts: Added ensureProtocGenOpenAPIResolved() — the which protoc-gen-openapi check now runs once per generation instead of once per proto file. Previously this check ran inside doGenerateLocal() (called per file), producing 17+ repeated "not found on PATH" log lines.
• Promoted key download/cache log messages to info level in both BufDownloader.ts and ProtocGenOpenAPIDownloader.ts — cache hit, download start, download complete, and version update messages are now visible without --log-level debug.
• Modified utils.ts: detectAirGappedModeForProtobuf() accepts an optional bufCommand parameter so it uses the resolved buf binary instead of assuming "buf" is on PATH.
• Fixed lock retry race condition in both BufDownloader.ts and ProtocGenOpenAPIDownloader.ts: After a stale-lock force-break, the previously unguarded mkdir is now replaced with a bounded retry loop. The retry budget is Math.max(LOCK_RETRY_INTERVAL_MS * 5, deadline - Date.now()), guaranteeing a minimum of ~1s of retries even after the original deadline expires, preventing both unhandled exceptions and dead-code retry loops.
• New BufDownloader.test.ts: 11 unit tests (mocked fetch) + 2 real e2e tests that download the actual buf binary from GitHub Releases.
• Updated ProtocGenOpenAPIDownloader.test.ts: Test assertions updated to check logger.info instead of logger.debug for promoted messages; added missing vi.resetModules().
• versions.yml: Added 4.37.0 entry.
• Updated test snapshots (url-form-encoded.json): Regenerated to include TokenRequest/TokenResponse/get_token types added on main.

Supported platforms: linux/x64, linux/arm64, darwin/x64, darwin/arm64, win32/x64, win32/arm64.

Testing

• Unit tests added (11 tests covering fresh download, cache hit, version upgrade, download failure, lock safety, concurrent access, permissions, URL format)
• Real e2e tests added (download actual binary, verify ELF/Mach-O magic bytes, verify cache reuse)
• All 25 tests passing locally (13 buf + 12 protoc-gen-openapi)
• Lint passing (pnpm run check)
• Manual e2e test: built CLI locally, ran fern check --api csharp-grpc-proto --log-level debug — verified fresh download, cache reuse, and proto generation

⚠️ Human Review Checklist

• resolveBuf returns a full binary path (not a directory like resolveProtocGenOpenAPI). Verify the integration in both generators correctly passes this as the command to runExeca and createLoggingExecutable.
• One-time protoc-gen-openapi resolution: The which check moved from doGenerateLocal() to ensureProtocGenOpenAPIResolved() uses a protocGenOpenAPIResolved boolean flag. If PATH were to change mid-generation (unlikely), the cached result would be stale.
• Duplicated ensureBufResolved() method in both generators → Addressed: Extracted shared ensureBufCommand() into utils.ts. Each generator's ensureBufResolved() is now a thin wrapper (caches result + calls failAndThrow on error).
• ensureBufCommand uses process.cwd() as cwd for the which call — this should be fine since which searches PATH regardless of cwd, but differs from the previous inline code which used protobufGeneratorConfigPath. Worth a quick sanity check.
• Platform/arch mapping (getPlatformInfo): buf uses aarch64 for Linux ARM64 but arm64 for macOS ARM64. Verify this matches actual release assets.
• Lock retry budget after stale-lock break: Uses Math.max(LOCK_RETRY_INTERVAL_MS * 5, deadline - Date.now()). This guarantees a minimum ~1s retry window (5 × 200ms) even when the original deadline has already expired. Confirm this is a reasonable minimum — too short risks failing prematurely if the competing process is still downloading; too long delays the error path.
• which fallback: The catch block in ensureBufCommand() / ensureProtocGenOpenAPIResolved() catches all errors from which (not just "not found"). Same pattern as existing code but worth noting.
• E2e tests download ~50MB binary — could be slow/flaky in CI depending on GitHub availability.
• Snapshot changes in url-form-encoded.json are from merging with main (adds TokenRequest/TokenResponse/get_token), not from this PR's code changes.

Link to Devin session: https://app.devin.ai/sessions/9aa3a5abd77d4e069bcfade48423e986
Requested by: @Swimburger

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[claude]

⚠️ Code review skipped — your organization's overage spend limit has been reached.

Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit in Settings → Usage.

Once credits are available, reopen this pull request to trigger a review.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 4 additional findings.

[graphite-app]

Critical race condition in lock retry logic after timeout. If multiple processes hit the timeout simultaneously, they all call rm(lockPath) followed by mkdir(lockPath). The final mkdir at line 139 is not wrapped in error handling, causing an unhandled exception when another process wins the race.

Impact: The function throws unexpectedly instead of either acquiring the lock or gracefully failing, causing resolveBuf() to return undefined.

Fix: The retry logic should loop back to the main acquisition loop rather than attempting a single unguarded retry:

try {
    await mkdir(lockPath, { recursive: false });
} catch (err) {
    logger.debug(`Failed to re-acquire buf lock after break, retrying acquisition loop`);
    // Loop back to normal acquisition with remaining time
    const remainingTime = Math.max(0, LOCK_TIMEOUT_MS - (Date.now() - deadline));
    if (remainingTime <= 0) {
        throw new Error(`Failed to acquire buf lock after timeout and retry`);
    }
    // Recursive call or loop back to start
}

Spotted by Graphite



Is this helpful? React 👍 or 👎 to let us know.

[devin-ai-integration]

Good catch — fixed in 7f76e72. Replaced the unguarded final mkdir with a bounded retry loop in both BufDownloader.ts and ProtocGenOpenAPIDownloader.ts. If another process wins the race after the stale lock is broken, we now loop with LOCK_RETRY_INTERVAL_MS until a retry deadline instead of throwing an unhandled error.

[graphite-app]

Incorrect calculation of remaining timeout. The code adds LOCK_TIMEOUT_MS to deadline (which is already Date.now() + LOCK_TIMEOUT_MS), resulting in double the intended timeout.

What breaks: After breaking a stale lock, the retry will wait for 2 * LOCK_TIMEOUT_MS (240 seconds) instead of the remaining time from the original deadline.

Fix:

const remaining = Math.max(0, deadline - Date.now());

This correctly calculates how much time remains from the original 120-second deadline.

Suggested change

	const remaining = Math.max(0, deadline + LOCK_TIMEOUT_MS - Date.now());
	const remaining = Math.max(0, deadline - Date.now());

Spotted by Graphite



Is this helpful? React 👍 or 👎 to let us know.

[devin-ai-integration]

Good catch — fixed in 9b565ef. Changed deadline + LOCK_TIMEOUT_MS - Date.now() to deadline - Date.now() in both BufDownloader.ts and ProtocGenOpenAPIDownloader.ts. The retry now correctly uses the remaining time from the original 120s deadline instead of doubling it.