feat: SSRF-hardened fetch-from-URL input module (remote_input.py)#196
Open
seonghobae wants to merge 1 commit into
Open
feat: SSRF-hardened fetch-from-URL input module (remote_input.py)#196seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
Add remote_input.fetch_media(url, dest_dir, *, max_bytes, timeout) so recordings hosted at an https URL (cloud recording links, podcast episodes) can be downloaded to a local temp path for the existing shrinking engine. Stdlib-only (urllib.request + ipaddress + socket). Defensive design: - https-only scheme allowlist (http/file/ftp/data rejected) - hostname resolved and ALL addresses validated before connecting: private/loopback/link-local/multicast/reserved rejected, including numeric-IP URLs, IPv4-mapped IPv6 literals, and localhost/*.localhost - redirects disabled entirely via a raising redirect handler (a 3xx could bounce the request to an internal address post-validation) - Content-Length pre-check plus streaming 1 MiB byte cap; overruns abort and delete the partial file - filename derived only from the sanitized URL path basename (Content-Disposition deliberately ignored), fallback download.bin - embedded URL credentials rejected; never overwrites existing files - module docstring documents the threat model, including the residual DNS-rebinding TOCTOU window Tests (tests/test_remote_input.py, 33 cases, no network): scheme and credential rejection, private/loopback/link-local/mixed-answer DNS rejection via mocked getaddrinfo, redirect refusal, size-cap aborts deleting partials, filename sanitization, and mocked happy path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJRVbDrp1vGYkJgNHMGPpG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Users often have recordings sitting at an https URL — cloud recording links, podcast episodes — and today codec-carver can only shrink files that are already local. This PR adds a composable, stdlib-only ingestion module that safely downloads a remote media file to a local path so the existing engine can take over. No CLI wiring yet by design; the module (plus a tiny
python -m remote_input <url> <dest_dir>entry point) is the unit.Threat model
The URL is attacker-influenced input, so the module is designed SSRF-first:
httpsonly.http,file,ftp,dataare rejected with explicit errors (local-read / SSRF vectors).socket.getaddrinfoand every resolved address must be globally routable. Private (RFC 1918 / ULA), loopback, link-local (incl.169.254.169.254metadata), multicast, reserved, and unspecified ranges are rejected — for hostnames, numeric-IP URLs, and IPv4-mapped IPv6 literals alike.localhostand*.localhostare rejected by name.302-bounce the client to an internal address after checks passed.Content-Lengthpre-check when present, plus a hard byte counter while streaming in 1 MiB chunks (default cap 5 GiB). Overruns abort and delete the partial file../../separator/NUL/overlong all fall back todownload.bin). The server-controlledContent-Dispositionheader is deliberately ignored. Existing files are never overwritten (open(..., "xb")).user:pass@credentials rejected; socket timeout on connect and each read; every rejection raisesRemoteInputErrorwith an actionable message.What
remote_input.py—fetch_media(url, dest_dir, *, max_bytes=5*1024**3, timeout=60) -> Path, plusRemoteInputErrorand a minimal__main__CLI. Stdlib only (urllib.request,ipaddress,socket). New files only; no existing file touched.Tests
tests/test_remote_input.py— 33 cases, fully offline (mockedgetaddrinfo+ fake opener/response doubles):[::1], IPv4-mapped[::ffff:127.0.0.1], mixed public+private DNS answers, unresolvable hostslocalhostand*.localhostrejection..,%2e%2e,%2Ftraversal, overlong, query strings)Gates:
python3.14 -m unittest discover -s tests→ 146 tests, no new failures (baseline exactly 5 macOSos.listxattrerrors).interrogate→ 100% docstring coverage.🤖 Generated with Claude Code