Queue Pilot

Queue Pilot is a Node.js workspace for inspecting queue pressure on the SiFive Slurm/Jenkins EDA farm. Today it is primarily a read-heavy triage tool: it helps engineers understand why jobs are pending, where regression flows are logjammed, and which parents or external flows are blocking progress. Queue-management functions are planned later, but mutating Slurm actions are currently gated off behind ENABLE_ACTIONS=false.

It ships with:

• a React web dashboard
• a Fastify REST API backed by cached Slurm snapshots in SQLite
• an MCP server so agents can query the same diagnostics surfaces

What It Does Today

• Summarizes queue pressure by account and partition.
• Splits diagnostics into focused views for logjams, pending, running, and control plane traffic.
• Groups jobs by flow, WCKey, and workdir root so related runs can be traced together.
• Surfaces fan-out srun logjams where running parent flows are waiting on re-queued children.
• Annotates logjams with external queue pressure: higher-priority jobs from other flows ahead in the same scheduling lane and an estimated drain latency.
• Provides watchlist matching for jobs of interest by user, account, WCKey, workdir, name, or job id.
• Estimates ETA-to-start / ETA-to-finish from historical bucket statistics and live queue shape.

Current UI

The web app currently exposes these pages:

• Pressure: account and partition hotspot summary from the latest collector snapshot.
• Logjams: grouped D3 graph view of blocked flows, origin parents, active runners, external queue pressure, and blocked reason buckets.
• Control Plane: isolates /root and nullish orchestration flows from normal verification traffic.
• Pending: aggregated graph or list view of waiting jobs, with WCKey grouping, parent blockers, and clickable workdir links.
• Running: aggregated graph or list view of active jobs by flow and WCKey.
• Watchlist: saved matchers with diagnosis and ETA context.

UI Snapshots

Pressure view:

Logjams view:

Repo Layout

• packages/shared - REASON taxonomy, ETA math, shared types and helpers
• packages/server - Fastify API, Slurm adapters, collector, diagnostics, watchlist, SQLite
• packages/mcp - MCP server exposing queue diagnostics tools over stdio
• packages/web - Vite + React dashboard
• docs - architecture notes, Slurm query recipes, ETA notes, triage workflow references
• okf - Open Knowledge Format bundle for repo-aware agents and knowledge retrieval
• AGENTS.md - authoritative implementation spec and operating manual for this repo

Runtime Model

• Slurm access is transport-agnostic: cli, restd, or mock.
• The diagnostics endpoints read from the latest cached snapshot when available instead of hitting live Slurm on every page refresh.
• SQLite stores snapshots and historical rollups used by the ETA heuristic.
• Queue actions are not implemented in the shipped code paths today; ENABLE_ACTIONS remains a safety gate for future work.

Quick Start

1. Install dependencies

npm install

2. Create .env

cp .env.example .env

Common settings:

• SLURM_ADAPTER=mock for offline development
• SLURM_ADAPTER=cli for live reads via local Slurm commands or SSH
• SLURM_SSH_HOST, SLURM_SSH_USER, SLURM_SSH_KEY when reading through a login node
• DB_PATH to control where the SQLite snapshot cache is stored
• ENABLE_ACTIONS=false to keep all Slurm access read-only

3. Run with mock data

In one shell:

npm run dev:mock

In another shell:

npm run dev:web

Optional MCP server:

npm run dev:mcp

4. Run against a real Slurm environment

Set the adapter and connection details in .env, then start the server and web app:

npm run dev

npm run dev:web

The server defaults to port 8080.

Main API Surfaces

• GET /api/clusters
• GET /api/pressure
• GET /api/diagnose
• GET /api/jobs/:id
• GET /api/eta/:id
• GET|POST|DELETE /api/watch
• GET /api/watch/:id/status

The diagnose endpoint powers the page-specific views and supports:

• section=summary|logjams|control|pending|running
• view=graph|list for pending/running
• search=... for job ids, WCKeys, blockers, users, accounts, and workdirs

Verification

Run the shipped tests:

npm test

Build the web app:

npm run build

Reference Docs

• docs/SLURM-QUERIES.md - exact read-only Slurm commands used by the app
• docs/REGRESSION-SLURM-JOB-TRIAGE-WORKFLOW.md - reference workflow for stalled regression jobs
• docs/ETA-MODEL.md - ETA heuristic notes and caveats
• docs/ARCHITECTURE.md - service and transport overview
• okf/index.md - agent-oriented knowledge bundle spanning architecture, UI, cache logic, metrics, and operations

For Implementers

Read AGENTS.md first. It is the authoritative spec for the Slurm queries, diagnostics rules, ETA methodology, watchlist behavior, and safety constraints in this repo.