[BACKEND] fix(ingest): warning for 429 error youtube subtitles for issue 219 #222
+68
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In _fetch_transcripts is now a differentiation between subtitles that are manual, or with -orig, or auto-translated. In this original priority is how subtitles are selected anyways. \nIf a different language than native is selected AND the result is a translated transcript, then a warning is sent in logs, and I've tried my best to make it easy to connect this to a future frontend fix that isn't here yet.
Collaborator
|
Hi @LucasImpulse, I tested your patch while ingesting a video, and it failed. I believe this is related to the recent changes in the YouTube ingestion code, where we switched from VTT to SRT parsing. Could you please take a look? 
|
Line 234, otherwise couldn't merge, we switched from VTT to SRT in the same day.
Author
|
It was exactly that, I was working on the code already on the same day when we switched to SRT, just needed to point to correct method again and think it's good now. 0 mentions of "vtt" in new commit version. 
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In _fetch_transcripts, there is now a bool tag called is_translated, always false for manual subtitles, always false for subtitles with language code having "-orig" on the end, and true for auto-captions IF the caption is not native. In this original priority is how subtitles were selected anyways.
If a different language than native is selected AND the result is a translated transcript, the bool is_translated is true, so then a warning is sent in logs, and I've tried my best to make it easy to connect this to a future frontend fix that isn't here yet.
This is only a partial fix to issue #219 and the fix has been tied in a knot so it doesn't cause errors but ready to connect to a potential frontend user facing warning. When an Error 429 happens, this is caught in lines 416-418 and the process gracefully fails and allows the continuing of other videos.