Fix/mediatek smart tv hevc audio optimizations#1577
Open
janblade wants to merge 4 commits into
Open
Conversation
MediaTek Smart TV SoCs use the 'omx.ms' codec prefix (e.g. MT5889 in TCL C7xx
series). These are distinct from MediaTek mobile SoCs ('omx.mtk') and were
previously missing proper decoder support in several areas.
Changes:
- Whitelist omx.ms for HEVC decoding on Android 11+ (API 30+). The MT5889
properly declares HEVC Main10 support in its media_codecs.xml.
- Enable direct submit for omx.ms. Fixed-function HW decoders with low
input-buffer latency, matching the behaviour of other C2/HW decoders.
- Remove omx.ms from the adaptive playback blacklist. The original comment
incorrectly grouped omx.ms with omx.mtk (mobile). TCL/MT5889 devices
properly declare adaptive-playback support and are stable with it enabled.
- Enable HEVC RFI (reference frame invalidation) for omx.ms on Android 11+.
Confirmed on TCL C735 (MT5889/Mali-G57); unlike omx.mtk this does not
require GPU-family gating.
- Add mediaTeKTvDecoderPrefixes and a dedicated KEY_LOW_LATENCY +
vdec-lowlatency code path. The MT5889 ACodec layer requires both
KEY_LOW_LATENCY (standard Android R API) and the vendor 'vdec-lowlatency'
key to fully engage low-latency I-frame decoding. The existing
FEATURE_LowLatency early-return is bypassed for these decoders so both
keys are always applied together.
- Fix decoderSupportsFusedIdrFrame() to return false for decoders on the
adaptive playback blacklist. Fused IDR submission requires the codec to
have been configured with KEY_MAX_WIDTH/KEY_MAX_HEIGHT; blacklisted
decoders are not configured that way, so submitting fused IDR buffers
would be incorrect.
Tested on TCL C735 (MT5889/Mali-G57, Android 11) via WiFi and Ethernet
with HEVC Main10 HDR at 4K 60fps.
…oid 11+ On Android 11+ (API 30), Window.preferMinimalPostProcessing maps to HDMI Auto Low Latency Mode (ALLM) on connected displays. This instructs the TV to switch to its Game Mode picture preset, bypassing MEMC (Motion Smoothing / frame interpolation), post-sharpening, and other processing that adds display lag and can introduce periodic visual artefacts during streaming. Some TV vendor overlays (notably TCL/MediaTek) ignore the equivalent AndroidManifest attribute; setting it programmatically on WindowManager LayoutParams at runtime is more reliable. The setAttributes() calls in the display-mode selection paths are also added to ensure the flag is applied to the window regardless of which branch is taken (mode change vs. already-optimal mode).
…itor timestamp Two related rendering fixes: 1. MAX_SMOOTHNESS / CAP_FPS frame queue drain: when multiple output buffers are queued, the previous code called releaseOutputBuffer(index, false) for all intermediate frames, silently discarding them. In MAX_SMOOTHNESS and CAP_FPS modes the intent is to never drop frames; intermediate frames are now released with timestamp=0 (immediate render) so every decoded frame reaches the display. 2. Lowest-latency mode final frame timestamp: the previous code passed System.nanoTime() as the presentation timestamp. On smart TV SoCs (e.g. MediaTek MT5889), this causes SurfaceFlinger to queue frames against future VSync deadlines it cannot meet, leading to constant 2-3 fps drops and periodic compositor stalls. Passing timestamp=0 requests immediate release and avoids compositor-side queue build-up.
…with effects Two audio reliability fixes: 1. Audio queue threshold raised from 40 ms to 120 ms. At 40 ms the queue limit is hit during normal AudioTrack hardware buffering, causing constant packet drops that manifest as periodic audio stuttering. 120 ms safely absorbs the hardware buffer depth and brief network jitter without introducing perceptible latency. 2. Skip buffer size index 2 when audio effects processing is active. With an equalizer or other effects chain enabled, standard-latency AudioTrack paths require a larger minimum buffer to avoid underruns under high CPU/GPU load (e.g. 4K AV1 decode). Index 2 produces a size too small for reliable playback in that case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add MediaTek Smart TV SoC (omx.ms / MT5889) decoder support and display/audio improvements
Overview
This PR adds first-class support for MediaTek Smart TV SoCs that use the
omx.mscodec prefix (e.g. MT5889 in TCL C7xx, Hisense U8x, and similar Android TV 11+
devices). It also includes three independent improvements to frame rendering and
audio that benefit TV devices more broadly.
Commit 1 — Add support for MediaTek Smart TV SoC decoders (
omx.ms/ MT5889)MediaCodecHelper.javaMediaTek Smart TV SoCs use the
omx.mscodec prefix and are entirely distinctfrom MediaTek mobile SoCs (
omx.mtk). They were previously treated identically,which incorrectly denied them several capabilities.
HEVC whitelist —
omx.msdecoders are whitelisted for HEVC on Android 11+(API 30+). The MT5889 properly declares
HEVCProfileMain10HDR10in itsmedia_codecs.xmland is stable at 4K 60fps HDR10.Direct submit —
omx.msadded todirectSubmitPrefixes. These arefixed-function hardware decoders with low input-buffer latency, consistent with
other hardware decoders already in that list.
Adaptive playback —
omx.msremoved fromblacklistedAdaptivePlaybackPrefixes.The original comment incorrectly grouped it with
omx.mtk(mobile). TCL/MT5889devices declare
<Feature name="adaptive-playback" />in their codec XML andare stable with it enabled. The comment is updated to clarify that only
omx.mtk/c2.mtk(mobile) require the blacklist.HEVC RFI —
omx.msadded torefFrameInvalidationHevcPrefixesonAndroid 11+. Confirmed on TCL C735 (MT5889/Mali-G57). Unlike
omx.mtk(which requires GPU-family gating due to GE8xxx hangs),
omx.msTV decodersdo not require this restriction.
Low-latency path —
mediaTeKTvDecoderPrefixeslist added (omx.ms).The MT5889 ACodec layer requires both
KEY_LOW_LATENCY(standard Android RAPI) and the vendor key
vdec-lowlatencyto fully engage low-latencyI-frame decoding. The existing
FEATURE_LowLatencyearly-return insetDecoderLowLatencyOptions()is bypassed for these decoders so both keysare always applied together across retry attempts.
Fused IDR fix —
decoderSupportsFusedIdrFrame()now returnsfalseforany decoder on the adaptive playback blacklist. Fused IDR submission requires
the codec to have been configured with
KEY_MAX_WIDTH/KEY_MAX_HEIGHT;blacklisted decoders are not configured that way, so submitting fused IDR
buffers without adaptive playback would be incorrect. This is a generic
correctness fix.
Tested on: TCL C735 (MT5889/Mali-G57, Android 11) via WiFi and Ethernet,
HEVC Main10 HDR at 4K 60fps.
Commit 2 — Enable Game Mode / ALLM via
preferMinimalPostProcessingon Android 11+Game.javaOn Android 11+ (API 30),
WindowManager.LayoutParams.preferMinimalPostProcessingmaps to HDMI Auto Low Latency Mode (ALLM) on connected displays. This
instructs the TV to switch to its Game Mode picture preset, bypassing MEMC
(motion smoothing / frame interpolation), post-sharpening, and other processing
pipelines that add display lag and can introduce periodic visual artefacts during
streaming.
Some TV vendor overlays (notably TCL/MediaTek) ignore the equivalent
AndroidManifestattribute; setting it programmatically onWindowManager LayoutParamsat runtime is more reliable.setAttributes()calls are also added in the display-mode selection paths toensure the flag is applied regardless of which branch is taken (mode change vs.
already-optimal mode).
Commit 3 — Fix MAX_SMOOTHNESS frame queue rendering and TV compositor timestamp
MediaCodecDecoderRenderer.javaMAX_SMOOTHNESS / CAP_FPS frame queue drain — when multiple output buffers
are queued, the previous code called
releaseOutputBuffer(index, false)forall intermediate frames, silently discarding them. In
MAX_SMOOTHNESSandCAP_FPSmodes the intent is to never drop frames; intermediate frames arenow released with
timestamp=0(immediate render) so every decoded framereaches the display.
Lowest-latency mode final frame timestamp — the previous code passed
System.nanoTime()as the presentation timestamp. On smart TV SoCs (e.g.MediaTek MT5889), this causes SurfaceFlinger to queue frames against future
VSync deadlines it cannot meet, leading to constant 2–3 fps drops and periodic
compositor stalls. Passing
timestamp=0requests immediate release and avoidscompositor-side queue build-up.
Commit 4 — Fix audio queue threshold and buffer selection with effects
AndroidAudioRenderer.javaAudio queue threshold raised from 40 ms to 120 ms. At 40 ms the queue
limit is hit during normal AudioTrack hardware buffering, causing constant
packet drops that manifest as periodic audio stuttering. 120 ms safely absorbs
the hardware buffer depth and brief network jitter without introducing
perceptible latency.
Skip buffer size index 2 when audio effects are active. With an equalizer
or other effects chain enabled, standard-latency
AudioTrackpaths require alarger minimum buffer to avoid underruns under high CPU/GPU load (e.g. 4K AV1
decode). Index 2 produces a size too small for reliable playback with effects
active, so it is skipped in that case.
Testing
Notes for reviewers
omx.msadditions are gated onBuild.VERSION.SDK_INT >= Build.VERSION_CODES.R(Android 11+) where confirmed stable. No changes affect devices below API 30.
releaseOutputBuffer(0)change in commit 3 replacesSystem.nanoTime()onlyin the lowest-latency (non-balanced) code path. The Choreographer/balanced path is
unchanged.
producing IDR frames too small (~41KB at 20Mbps/60fps) for the decoder at 4K.
This is resolved server-side via
nvenc_vbv_increase = 150in Sunshine and isnot related to the client-side changes in this PR.