Skip to content

⚡ Bolt: Optimize Dashboard KPIs by replacing in-memory counting with SQL aggregates #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bobdivx wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

⚡ Bolt: Optimize Dashboard KPIs by replacing in-memory counting with SQL aggregates #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
## 2024-05-27 - Drizzle ORM Full Table Fetch Bottlenecks

**Learning:** When retrieving aggregate counts or checking conditions for KPIs or dashboards, performing `db.select().from(Table)` reads all records into application memory, leading to an O(N) memory and processing bottleneck. This is evident in `src/pages/api/dashboard-kpis.ts` which loads all projects, tasks, open requests, and issues.
**Action:** Replace `db.select().from(Table)` full table fetches with native SQL aggregates. Astro DB exposes the `count()`, `gte()`, `eq()`, `inArray()` helpers inside `loadAstroDb()` and they should be used to push counting to the SQL engine `db.select({ count: count() }).from(Table)`.
38 changes: 15 additions & 23 deletions src/pages/api/dashboard-kpis.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,6 @@ function countRunningSessions(sessions: unknown[]): number {
}).length;
}

function isOpenQueueStatus(st: string): boolean {
const s = String(st).toLowerCase();
return s === 'open' || s === 'in_progress';
}

export const GET: APIRoute = async () => {
const base = {
projectCount: 0,
Expand All @@ -42,30 +37,27 @@ export const GET: APIRoute = async () => {
};

try {
const { db, Project, AgentTask, Request, AgentAppIssue, AgentDependencyRequest, eq } =
const { db, Project, AgentTask, Request, AgentAppIssue, AgentDependencyRequest, eq, count, gte, inArray } =
await loadAstroDb();
const today = new Date();
today.setHours(0, 0, 0, 0);

const [projects, tasksAll, openRequests, issuesAll, depsAll] = await Promise.all([
db.select().from(Project),
db.select().from(AgentTask),
db.select().from(Request).where(eq(Request.status, 'pending')),
db.select().from(AgentAppIssue),
db.select().from(AgentDependencyRequest),
// Bolt Optimization: Replace full table fetches with SQL count aggregates to reduce memory and processing overhead
const [projectsCount, tasksAllCount, tasksTodayCount, openRequestsCount, issuesAllCount, depsAllCount] = await Promise.all([
db.select({ count: count() }).from(Project),
db.select({ count: count() }).from(AgentTask),
db.select({ count: count() }).from(AgentTask).where(gte(AgentTask.createdAt, today)),
db.select({ count: count() }).from(Request).where(eq(Request.status, 'pending')),
db.select({ count: count() }).from(AgentAppIssue).where(inArray(AgentAppIssue.status, ['open', 'in_progress'])),
db.select({ count: count() }).from(AgentDependencyRequest).where(inArray(AgentDependencyRequest.status, ['open', 'in_progress'])),
Comment on lines +51 to +52
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The previous implementation of status checking (isOpenQueueStatus) was case-insensitive because it converted the status to lowercase before comparing (String(st).toLowerCase()).\n\nThe new SQL query uses inArray(AgentAppIssue.status, ['open', 'in_progress']), which is case-sensitive in SQLite/Astro DB. If there are any legacy or externally inserted records with different casing (e.g., 'Open' or 'In_Progress'), they will be missed by this count.\n\nTo preserve the case-insensitive behavior, you can use the sql helper to lowercase the column in the query.\n\nNote: You will also need to add sql to the destructured imports from loadAstroDb() on line 40.

      db.select({ count: count() }).from(AgentAppIssue).where(inArray(sql`lower(${AgentAppIssue.status})`, ['open', 'in_progress'])),\n      db.select({ count: count() }).from(AgentDependencyRequest).where(inArray(sql`lower(${AgentDependencyRequest.status})`, ['open', 'in_progress'])),

]);

const tasksTodayCount = tasksAll.filter((t) => {
const d = t.createdAt instanceof Date ? t.createdAt : new Date(t.createdAt as Date);
return d >= today;
}).length;

base.projectCount = projects.length;
base.tasksTotal = tasksAll.length;
base.tasksToday = tasksTodayCount;
base.openRequests = openRequests.length;
base.openAppIssues = issuesAll.filter((r) => isOpenQueueStatus(String(r.status))).length;
base.openDependencyRequests = depsAll.filter((r) => isOpenQueueStatus(String(r.status))).length;
base.projectCount = projectsCount[0]?.count || 0;
base.tasksTotal = tasksAllCount[0]?.count || 0;
base.tasksToday = tasksTodayCount[0]?.count || 0;
base.openRequests = openRequestsCount[0]?.count || 0;
base.openAppIssues = issuesAllCount[0]?.count || 0;
base.openDependencyRequests = depsAllCount[0]?.count || 0;
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
base.dbError = msg;
Expand Down