[musashi-api] Archive analyzed tweets to Supabase#13
Open
TianyiRnj wants to merge 2 commits into
Open
Conversation
Production movers endpoint timed out 100% of the time for 8 days starting 5/4 16:00 UTC. Root cause: handler did 1,600 sequential kv.get() calls inside detectMovers() (one per tracked market), plus 1,600 KV writes via recordPriceSnapshots(), all in the request path. Once @vercel/kv's Upstash migration added a small amount of per-request latency, total runtime crossed the 25s function timeout and never recovered. Changes: - Extract snapshot/movers logic to api/lib/price-snapshots.ts. detectMoversBatch() uses kv.mget() in chunks of 100, replacing 1,600 individual gets with ~16 batched calls. recordPriceSnapshots() now also reads via mget and caps each market's snapshot array at 300 entries so values stop bloating until 7-day TTL. - Add api/cron/refresh-markets.ts. Runs every 2 minutes, fetches markets, records snapshots, precomputes movers for buckets 0.02/0.05/0.1/0.2, and writes them to movers:precomputed:<bucket> with a 5-minute TTL. Writes freshness timestamps to meta:last_snapshot_run and meta:last_movers_run for /api/health to consume. - Rewrite api/markets/movers.ts as a read-only endpoint: single kv.get() of the precomputed bucket, in-memory filter for limit/category, plus a 20s in-process response cache. Returns 503 if the cron hasn't run. - Rewrite api/health.ts as a 4-check honest health probe: real market counts (Poly >= 800, Kalshi >= 200), KV read+write probe, snapshot freshness (< 5min), movers freshness (< 5min). Returns 503 on any degraded check so external monitors actually fire. - Update vercel.json with the new cron entry and rewrite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…health Fix /api/markets/movers timeout and make /api/health honest
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/api/supabase-server-client.ts— singleton Supabase client (server-side, service role) with typed schema for theanalyzed_tweetstablesrc/api/analyzed-tweet-archive.ts— batched upsert ofAnalyzedTweetrecords after each cron run, with per-batch error capture and anArchiveResultsummaryapi/cron/collect-tweets.tsto:500and skip rotation ifSUPABASE_ARCHIVE_REQUIRED=trueand any rows failarchivestats in cron metadata responsesupabase/migrations/20260512000000_analyzed_tweets.sqlwithtweet_idas the upsert conflict keyNew env vars
SUPABASE_URLSUPABASE_SERVICE_ROLE_KEYSUPABASE_ARCHIVE_REQUIREDtrueto make archive failure block cron rotationMigration
Run
supabase/migrations/20260512000000_analyzed_tweets.sqlagainst the target project before deploying.Test plan
pnpm typecheckpassespnpm testpasses (analyzed-tweet-archive,supabase-server-client,collect-tweets)analyzed_tweetstablearchivefieldSUPABASE_ARCHIVE_REQUIRED=true: simulate upsert failure → cron returns500, rotation does not advance