feat: enhanced scrapeLikedTweets scraper with rich data extraction#13
Closed
nj-io wants to merge 1 commit into
Closed
feat: enhanced scrapeLikedTweets scraper with rich data extraction#13nj-io wants to merge 1 commit into
nj-io wants to merge 1 commit into
Conversation
Replace the broken xeepy-based x_get_likes handler with a proper scraper in src/scrapers/twitter/index.js, following the same pattern as scrapeBookmarks. The new scraper extracts rich data per tweet: - text, author, handle, timestamp, link - images (attributed to correct author) - quoted tweets (detected via multiple UserAvatar-Container elements) - X Articles (title, description, cover image via article-cover-image) - link cards (via card.wrapper) - engagement stats (replies, retweets, likes, views from role="group") Also fixes: - "Show more" expansion — clicks buttons one at a time since X re-renders the DOM after each click, detaching other button refs - Scroll-based pagination with deduplication - Removes x_get_likes from xeepyTools, routes through local-tools.js Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
@nj-io is attempting to deploy a commit to the kaivocmenirehtacgmailcom's projects Team on Vercel. A member of the Team first needs to authorize it. |
5 tasks
Author
|
Superseded by updated PR with timestamp filtering, auth checks, and proper scraper pattern. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the broken xeepy-based
x_get_likeshandler with a properscrapeLikedTweets()scraper, following the same pattern asscrapeBookmarks().What changed
scrapeLikedTweets()insrc/scrapers/twitter/index.js— scroll-based scraper with rich data extractionsrc/scrapers/index.js→src/mcp/local-tools.js(same pattern as bookmarks)x_get_likesfromxeepyToolsarray and deleted the brokenexecuteXeepyToolhandler that called the non-existentlocalTools.getPage()Rich data per tweet
textauthor,handleUser-Name+ firsta[href]timestamp,linktime[datetime], first/status/linkimagesa[href*="/photo/"]attributed to correct author by handle matchingquotedTweetUserAvatar-Container-*elements (X doesn't always usequoteTweettestid)articlearticle-cover-image+nextElementSiblingfor title/descriptioncardcard.wrapperfor link previewsreplies,retweets,likes,viewsrole="group"aria-labelBug fixes
Node is detached from documenterrors.article.urlfor direct articles (not quoted tweets where the article belongs to a different author). Always includesarticle.tweetUrlfor reliable resolution.Relation to #7
This overlaps with the
scrapeLikedTweetsportion of #7. The scraper here is more comprehensive (quote tweets, articles, cards, engagement stats, image attribution) vs #7's basic fields. Both follow the same architectural pattern requested by the maintainer: scraper intwitter/index.js, routed throughlocal-tools.js, removed fromxeepyTools.Test plan
x_get_likesreturns rich data with quote tweets, articles, cards, engagement stats🤖 Generated with Claude Code