Performance: Optimize historical performance fetching & implement backend caching to avoid rate-limiting#187
Open
yashvi-3106 wants to merge 1 commit into
Conversation
Contributor
|
Thank you for submitting a pull request. Please ensure your changes comply with the project's contribution guidelines and that all workflow checks pass successfully. Formatting and Branching
|
Author
|
@jagdish-15 i have linked this pr you can review it! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR optimizes the student history loading process. Previously, loading a student's analytics profile required fetching up to 365 daily snapshot files from the remote data repository in chunks of 100 HTTP requests. By transitioning to a single, pre-generated historical user data file ($O(\text{days})$ to a single $O(1)$ JSON request.
historical-user-data/username.json) per student, we reduce the network load fromLinked Issue
Fixes #154
Changes Made
scripts/bootstrap-user-history.js: A one-time bootstrap script that parses all existing daily snapshot files underdata/daily/and compiles historical records into individual user history files underdata/historical-user-data/.scripts/sync-leaderboard.js: Integrates incremental history updates into the sync cron job. It appends the latest solved counts (easy, medium, hard) to each user's history file during daily synchronization.scripts/fetch-student-info.js: Refactored the retrieval logic to load the single user history file directly. It searches the local directory first for development environments, and falls back to a raw GitHub query in production.Type of Change
Testing
I successfully tested the modifications locally:
Migration test: Ran the bootstrap script on a local clone of the data repository to successfully generate user history files for 121 users from 342 daily snapshots.
Sync test: Executed the synchronization script and confirmed the stats correctly appended to the user-specific history files under the new directory.
API Fetch test: Verified the refactored
fetchStudentHistorysuccessfully queries the Vercel API and reads history files in ~1.1 seconds with zero console errors.Tested locally
Tested on mobile viewport (if applicable)
No console errors introduced
Checklist
npx prettier --write .before submittingmainbranch