feat(cra): presigned S3 URLs + progress tracking#538
Open
jirkamotejl wants to merge 26 commits intomasterfrom
Open
feat(cra): presigned S3 URLs + progress tracking#538jirkamotejl wants to merge 26 commits intomasterfrom
jirkamotejl wants to merge 26 commits intomasterfrom
Conversation
Instead of downloading the video to the server and re-uploading via SFTP, the ingest manifest now includes a presigned S3 URL. CRA downloads the video directly from S3, eliminating disk/memory pressure on the application server. Also adds safety guards for destroying files that were never sent to CRA (early return in DeleteMediaJob and file_processing concern).
jirkamotejl
force-pushed
the
feature/cra-encoding-improvements
branch
from
9931257 to
31dce63
Compare
jirkamotejl
changed the title
- Parse CRA encoding messages for phase-level progress (validation, audio, video, thumbnails, packaging) and estimate completion time - Extract JobResolver to deduplicate job resolution logic across CreateMediaJob, CheckProgressJob, and MonitorProcessingJob - Fix CheckProgressJob to properly save and broadcast on FAILED status - Guard process_output_hash against nil HLS/DASH entries - Set progress_percentage to 100.0 on DONE for consistent final state - Add dedicated encoding_progress MessageBus event for UI updates - Add processing_failed locale translations
Listen for the dedicated CraMediaCloud encoding_progress MessageBus event (not the generic file_update) to avoid reacting to unrelated file changes like in-place input edits or metadata extraction. During processing: update state label text inline (no page reload). On state change (e.g. processing -> ready): full Turbo reload.
jirkamotejl
force-pushed
the
feature/cra-encoding-improvements
branch
from
31dce63 to
8ef7294
Compare
- Add encoding_completed_at timestamp to CheckProgressJob on DONE - New EncodingInfoComponent in file detail meta bar showing progress, phases, ETA during encoding and duration after completion - Expand encoding_progress broadcast with progress/phases/ETA data - Live-update encoding info via MessageBus without page reload - Fix stale state badge: listen for file_update events, Turbo reload frame only when aasm_state actually changes - Clean up FileSerializer: remove progress % hack from aasm_state_human
- Move meta-item wrapper inside component template so it doesn't render an empty div when component is hidden - Render component during all CRA processing, not just when progress_percentage is already present - Handle nil progress_percentage in template
Render encoding info as inline span next to state label instead of a separate meta-item. Shows as "Zpracováváno — 45.1% · audio · ~16 min".
- Translate encoding phases (validace, audio, náhledy, video, balení) - ETA shows as live countdown "zbývá ~9:32" ticking every second - Countdown refreshes on each MessageBus progress update - Phase translations and remaining label passed via data attributes
- Remove encoding phases display (technical detail, not useful for users) - Interpolate progress percentage between 15s MessageBus updates for smooth continuous movement - ETA countdown ticks every second: "zbývá ~9:32" - Progress and ETA share a single 1s ticker interval
Track displayed interpolated value and use Math.max on server update so progress never visually decreases.
Move all encoding progress interpolation and ETA countdown logic from show_component.js into encoding_info_component.js. The ticker JS now only loads when the encoding info component is actually rendered. Also remove unused phases_completed from broadcast payload.
Remove ENCODING_RATIO constant (was 0.35, actual ~0.70) and compute ETA purely from elapsed time and CRA progress fraction.
…l initial state, pause interpolation when ahead
…s, env-unique reference IDs, improve progress UI - Handle CRA REMOVED status in CheckProgressJob to stop infinite polling - Add environment prefix to reference IDs to prevent cross-env collisions - Prevent MonitorProcessingJob from creating duplicate CreateMediaJobs - Add REMOVED and WAITING to JobResolver STATUS_MAP - Anchor-based ETA calculation using CRA phase field - Mock progress 0-25% during phase 0 based on file size - Stale detection (45s threshold) freezes progress when backend stops updating - CSS :has() selector pulses processing state dot - Reduce initial CheckProgressJob delay from 30s to 10s
Remap CRA's nonlinear 0→1 progress to user-friendly 0→100 scale (25→90% for encoding phase), replace anchor-based ETA with simple elapsed/progress extrapolation, handle VALIDATING status, add last_progress_check_at for stale detection, simplify frontend by removing progressSlowCrawl logic.
Mock progress uses hyperbolic curve (asymptotically approaches 25% but never reaches it) instead of linear ramp. ETA clamp ensures estimated_completion_at can only shrink, never push further out.
Replace strict "never grow" clamp with a padding multiplier that starts at ~1.5x when little data is available and shrinks toward 1.0x as progress advances. Produces naturally shrinking estimates.
DISPLAY_ENCODING_START was 25% while frontend MOCK_PROGRESS_CAP was 30%, causing potential backward jumps when server value arrived below mock. Both now start at 30%.
Don't set progress_percentage until CRA status is PROCESSING — keeps frontend in mock mode during WAITING/CREATED/VALIDATING instead of jumping to 30% and stalling. Smooth ETA with exponential moving average (70/30 blend) to prevent oscillation from CRA jitter.
…ress, interpolation Show only what we know: current phase (waiting/encoding/packaging) and raw CRA progress during encoding. No JS tickers, no mock curves, no ETA calculations.
Add `unique :until_and_while_executing` (same pattern as Mux) to prevent duplicate jobs in queue. Wrap state updates with `with_lock` to prevent concurrent writes to remote_services_data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
encoder_sd_profile_groupis defined (e.g."VoDSD"), CreateMediaJob submits two manifests with the samerefId— SD first, then HD (VoDHDauto). SD completion triggers subtitle transcription. Backward compatible: single-phase whenencoder_sd_profile_groupis nil.messagesarray for per-phase milestones (validation, audio, video, thumbnails, packaging), extracts video duration, and estimates completion time. New states:sd_processing → sd_processed → hd_processing → full_media_processed.Remaining tasks (separate PRs)
sd_ready?/sd_mp4_urlhelpers (Task 5)encoder_sd_profile_groupreturning"VoDSD"in production (separate repo)Test plan