Performance: Optimize historical performance fetching & implement backend caching to avoid rate-limiting

[yashvi-3106]

Problem Description

Currently, when a user accesses a student's analytics profile, the server triggers the /api/student/:username endpoint, calling the fetchStudentHistory function in scripts/fetch-student-info.js.

To generate a 365-day graph, this function fetches daily JSON snapshots directly from the raw GitHub contents of the data repository (leetcode-ranking-data) via fetch() in chunks of 100 requests.

This leads to several critical issues:

1. Excessive Network Requests: Up to 365 HTTP requests are dispatched to raw.githubusercontent.com for a single page view.
2. GitHub API Rate Limiting: The server will quickly get rate-limited by GitHub when multiple profiles are viewed.
3. High Latency: Loading the profile page takes several seconds due to network roundtrips.
4. Memory Leak Risk: The in-memory studentCache in server.js is unbounded, meaning its size grows indefinitely as different usernames are queried.

---

Proposed Solution

1. Server-side Cache Persistence: Store the compiled student histories in a local file-based cache or a lightweight database (e.g., SQLite) on the Express backend server.
2. Incremental Sync: Instead of scanning 365 historical files on every request, fetch the full history once, and then only poll/fetch incremental new daily files during synchronization, appending them to the cached record.
3. Bounded Cache: Implement a size limit or LRU (Least Recently Used) cache strategy for studentCache inside server.js to automatically evict stale entries and prevent memory growth.
4. Alternative - Pre-Compiled History Files: Compile the unified history of each student into a single JSON file (e.g., history/username.json) in the data repository during the scheduled synchronization cron job. The backend can then fetch one file instead of hundreds.

---

Affected Files

• server.js
• scripts/fetch-student-info.js
• scripts/sync-leaderboard.js

[github-actions]

Thank you for opening an issue.

Please review our CONTRIBUTING.md guidelines before getting started.

A maintainer will review this issue shortly. If you would like to work on the issue yourself, please leave a comment requesting assignment.

Issue Assignment Policy

• The issue creator is given the first priority to work on their issue.
• If the creator declines or does not request assignment, the issue will be assigned to others in the order of requests received.
• Please do not submit PRs or begin work until you are officially assigned.

Note: This project is currently maintained by a solo maintainer, so reviews, assignments, and responses may sometimes take a little time. Thanks for your patience.

[yashvi-3106]

@jagdish-15 would worlk on this issue!

[jagdish-15]

I think the cleanest long-term approach here would probably be shifting toward pre-generated per-user history files instead of dynamically rebuilding a user's history from hundreds of daily leaderboard snapshots on every profile request.

Something like:

historical-user-data/
username.json

where each file stores the user's historical snapshots over time.

The overall flow could become:

• create a one-time migration/bootstrap script initially which scans the existing historical leaderboard datasets and generates the per-user history files,
• after that, during the regular sync process, simply append the latest snapshot/state for each user into their existing history file instead of rebuilding histories repeatedly,
• then update fetch-student-info.js to directly load the single user history file rather than fetching hundreds of historical leaderboard files in batches.

I’d probably prefer this approach over introducing a separate database layer for now, since it still keeps the overall architecture relatively simple and consistent with the current GitHub-data-driven setup.