Add feed-type endpoints for all/videos/shorts/live#22
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughAdds feed_type filtering (all, videos, shorts, live) end-to-end: scanner scraping, get_rss_feed signature, Flask routing (typed /feed/<feed_type>/<channel_url> with validation and caching), UI selectors, and README examples updated. ChangesFeed Type Support throughout Stack
Sequence Diagram(s)sequenceDiagram
participant Client
participant Flask API
participant rss_scanner
participant YouTube Page
Client->>Flask API: GET /feed/<feed_type>/<channel_url>
Flask API->>Flask API: validate feed_type
Flask API->>rss_scanner: get_rss_feed(url, feed_type="all"|"videos"|"shorts"|"live")
rss_scanner->>rss_scanner: get_channel_videos(channel_id, feed_type)
rss_scanner->>YouTube Page: scrape /channel/<id>/<feed_type>
YouTube Page-->>rss_scanner: HTML with video IDs
rss_scanner->>rss_scanner: extract unique 11-char video IDs via regex
rss_scanner-->>Flask API: Atom feed + api_endpoints (feed/all/, videos_feed, shorts_feed, live_feed)
Flask API-->>Client: Atom XML response
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. Fixes Applied SuccessfullyFixed 3 file(s) based on 3 unresolved review comments. Files modified:
Commit: The changes have been pushed to the Time taken: |
Fixed 3 file(s) based on 3 unresolved review comments. Co-authored-by: CodeRabbit <noreply@coderabbit.ai>
There was a problem hiding this comment.
🧹 Nitpick comments (2)
api/app.py (2)
171-171: ⚡ Quick winRemove redundant
channel_id or channel_idexpression.The expression
channel_id or channel_idalways evaluates tochannel_id. This appears to be a copy-paste error where the second occurrence should likely just bechannel_id, or the entire expression was meant to provide a fallback.♻️ Suggested fix
atom_feed = atom_feed.replace( - f">{channel_id or channel_id} - YouTube Videos", + f">{channel_id} - YouTube Videos", f">{channel_name} - YouTube Videos" )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@api/app.py` at line 171, The f-string currently uses a redundant expression `channel_id or channel_id`; replace it with a single `channel_id` (i.e., `f"{channel_id} - YouTube Videos"`) in the same place where the f-string is constructed so the title uses the actual channel_id value (or, if a fallback is desired, use a proper fallback like `channel_id or "<unknown>"` instead).
165-165: ⚡ Quick winReplace unused
video_countwith_.The
video_countvariable is unpacked but never used in this function. Replace it with_to signal the value is intentionally ignored.♻️ Suggested fix
- _, channel_id, channel_name, atom_feed, video_count, _, _ = rss_scanner.get_rss_feed(full_url, feed_type=feed_type) + _, channel_id, channel_name, atom_feed, _, _, _ = rss_scanner.get_rss_feed(full_url, feed_type=feed_type)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@api/app.py` at line 165, Unpack from rss_scanner.get_rss_feed already binds an unused video_count variable; change the tuple unpack to replace video_count with _ so the call becomes: _, channel_id, channel_name, atom_feed, _, _, _ when calling rss_scanner.get_rss_feed(full_url, feed_type=feed_type) — update the unpacking in the function that calls rss_scanner.get_rss_feed to use _ for the unused value to signal intentional ignore.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@api/app.py`:
- Line 171: The f-string currently uses a redundant expression `channel_id or
channel_id`; replace it with a single `channel_id` (i.e., `f"{channel_id} -
YouTube Videos"`) in the same place where the f-string is constructed so the
title uses the actual channel_id value (or, if a fallback is desired, use a
proper fallback like `channel_id or "<unknown>"` instead).
- Line 165: Unpack from rss_scanner.get_rss_feed already binds an unused
video_count variable; change the tuple unpack to replace video_count with _ so
the call becomes: _, channel_id, channel_name, atom_feed, _, _, _ when calling
rss_scanner.get_rss_feed(full_url, feed_type=feed_type) — update the unpacking
in the function that calls rss_scanner.get_rss_feed to use _ for the unused
value to signal intentional ignore.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 04bc8b30-ef51-40db-9dff-f36766a16144
📒 Files selected for processing (6)
README.mdapi/app.pyapi/index.htmlindex.htmlrss_scanner.pytemplates/index.html
🚧 Files skipped from review as they are similar to previous changes (2)
- README.md
- rss_scanner.py
| "atom_feed_path": f"{base_url.rstrip('/')}/feed/all/{encoded_url}", | ||
| "atom_feed_query": f"{base_url.rstrip('/')}/feed/all/{urllib.parse.quote(url, safe='')}", |
There was a problem hiding this comment.
🟡 atom_feed_path and atom_feed_query produce identical URLs
The PR changed atom_feed_query from a distinct query-parameter-based URL format (/feed/?channel_url=...) to the same path-based format as atom_feed_path. Since encoded_url = urllib.parse.quote(url, safe="") at rss_scanner.py:326 and the inline urllib.parse.quote(url, safe='') at line 330 produce the same result, both keys now generate the exact same URL. The CLI at rss_scanner.py:416-417 still prints both as "Atom Feed (path)" and "Atom Feed (query)" — showing identical URLs with different labels, which is confusing to users.
Prompt for agents
The atom_feed_path and atom_feed_query entries in the api_endpoints dictionary (rss_scanner.py:329-330) now produce identical URLs since both use the same path-based format. The old code had atom_feed_query using a query-parameter format (/feed/?channel_url=...) which was a distinct alternative.
Since the query-parameter route was removed in api/app.py, either:
1. Remove the atom_feed_query key entirely and update all references (rss_scanner.py:417 CLI print, and any UI references).
2. Or, if the intent is to show the currently-selected feed type's URL rather than always 'all', replace atom_feed_query with a URL that uses the selected feed_type parameter, e.g. f"{base_url}/feed/{feed_type}/{encoded_url}".
Also update rss_scanner.py:416-417 (the CLI output) to stop printing the duplicate entry.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
||
|
|
||
| @app.route('/feed/<feed_type>/<path:channel_url>') | ||
| @cache.cached(timeout=300, key_prefix=lambda: f"feed_{request.path}") |
There was a problem hiding this comment.
🟡 Cache stores transient error responses (404/500) for 5 minutes
The @cache.cached decorator at api/app.py:146 caches all responses from get_feed, including Response("No videos found", status=404) at line 180 and Response(f"Error: {str(e)}", status=500) at line 182. If YouTube scraping temporarily fails (e.g., due to rate limiting or network issues), the error response is served from cache for 5 minutes, even after the transient issue resolves. The old code effectively had no working cache (the cached route get_cached_feed was shadowed by the identically-patterned uncached get_feed route), so this is a new behavior introduced by the PR.
| @cache.cached(timeout=300, key_prefix=lambda: f"feed_{request.path}") | |
| @cache.cached(timeout=300, key_prefix=lambda: f"feed_{request.path}", response_filter=lambda response: response.status_code == 200) |
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Motivation
all,videos,shorts, orlivefeeds while keeping existing combined behavior.channel,@handle,/c/,/user/, video URLs) and expose typed feed links in API output.Description
feed_typesupport toget_rss_feed(...)and wired it into scraping so the scanner fetches the appropriate YouTube tab; signature changed toget_rss_feed(..., feed_type: str = "all")andget_channel_videos(...)now acceptsfeed_typeand uses apage_path_by_typemap to hitvideos,shorts, orstreamspages and a shared_extract_video_ids_from_page(...)helper.api/app.pyrouting to new endpointsGET /feed/all/<channel>,/feed/videos/<channel>,/feed/shorts/<channel>,/feed/live/<channel>with feed-type validation returning400for invalid types and adjusted cached route to include the request path in the cache key so each feed type caches separately (key_prefix=lambda: f"feed_{request.path}").api_endpointsreturned byget_rss_feed(...)wheninclude_api_endpointsis enabled to advertise typed feed URLs (atom_feed_path,videos_feed,shorts_feed,live_feed).README.mdto document the new/feed/<type>/<channel_url>pattern and example endpoints.Testing
python -m py_compile rss_scanner.py api/app.py(succeeded).app.test_client()using a stubbedrss_scanner.get_rss_feedto confirm valid feed types (all,videos,shorts,live) return200and invalid type returns400(succeeded)._extract_video_ids_from_page) to confirm thelimitparameter works (10IDs returned) (succeeded).app.test_client()against real YouTube pages but outbound requests failed in this environment (Tunnel connection failed: 403 Forbidden), so full live fetch validation could not complete (network-limited failure).Codex Task
Summary by CodeRabbit
New Features
Documentation