web-fetch

Fetch web content — auto-convert, chunk, cache with progressive disclosure.

web-fetch <url>

HTML → Markdown, JSON → formatted, PDF → text sections, CSV → table. Image/Video/Audio → metadata extraction + original file. Large content → auto-chunked temp cache. Original binaries preserved.

When to use web-fetch (not a curl replacement)

curl is the right tool when you need raw HTTP control (headers, methods, cookies, full response). web-fetch is a higher-level tool for specific scenarios where you want the content, not the bytes:

• HTML page → Markdown: saves ~60% tokens vs raw HTML for LLM consumption
• Large documents: auto-chunked — first chunk previewed, rest indexed. No terminal flood
• PDF papers: extract text by sections, original preserved for reference
• JSON APIs: auto-formatted and split by top-level keys, no jq needed
• CSV data: rendered as Markdown table, auto-paginated at 100 rows
• Media file checks: inline metadata for images (size/format) and video/audio (duration/codec/resolution)
• Network-restricted envs: auto proxy discovery, or explicit --proxy
• Repeated fetches: built-in cache with 1d/30d expiry policy

Bottom line: if you'd pipe curl into jq, html2text, pdftotext, or ffprobe, try web-fetch instead — one command, progressive disclosure, no pipe chains.

For everything else (binary blobs, custom headers, uploads, full HTTP debugging), stick with curl.

Install

pnpm add -g @lenml/web-fetch
# or
npx @lenml/web-fetch <url>

Image metadata via image-size (built-in, zero native deps) For video/audio metadata: install ffmpeg (provides ffprobe)

Usage

# Small content: inline output
web-fetch https://example.com

# Large content: auto-chunked, first chunk previewed
web-fetch https://example.com/large-doc

# PDF: original preserved + text extracted by sections
web-fetch https://example.com/paper.pdf

# CSV: formatted as markdown table, chunks of 100 rows
web-fetch https://example.com/data.csv

# ZIP: file listing + extracted text files content
web-fetch https://example.com/archive.zip

# Image: saved as original + metadata (image-size)
web-fetch https://example.com/photo.jpg

# Video: saved as original + metadata (ffprobe)
web-fetch https://example.com/video.mp4

# Show cached index
web-fetch --cache <id>

# Show specific chunk
web-fetch --cache <id> --chunk <key>

# Through proxy
web-fetch --proxy http://127.0.0.1:10808 https://example.com

# Raw output (no conversion)
web-fetch -r https://example.com

# Allow downloading files larger than 50MB
web-fetch --dl-big-file https://example.com/large-file.bin

# Custom size limit (200 MB)
web-fetch --max-size 200 https://example.com/file.bin

Options

Flag	Description
--proxy <url>	HTTP/SOCKS proxy (skip auto-discovery)
--never-proxy	Skip proxy auto-discovery, connect directly
--cache <id>	View cached fetch index
--chunk <key>	View chunk from cache (use with --cache <id>)
-r, --raw	Raw binary output, no conversion
-i, --inline	Force inline output, no chunking
-g, --global-cache	Use OS temp dir instead of local .fetch-cache/
--dl-big-file	Allow downloading files larger than --max-size
--max-size <mb>	Big file threshold in MB (default 50)
--disable-banner	Suppress the conversion source banner
-h, --help	Show help
--version	Show version

Design

Progressive Disclosure

Content auto-saved to .fetch-cache/ (local CWD) or $TMPDIR/web-fetch-cache/ (global):

• <50KB — inline output
• >50KB — split into chunks, first chunk previewed, remaining chunks referenced by key
• >1 day — cache marked stale (metadata preserved, content needs re-fetch)
• >30 days — cache auto-deleted on next cleanup

Content Handling

Type	Conversion	Chunking
HTML	Turndown → Markdown	By h1/h2/h3 headings
JSON	Pretty-print (2-space)	By top-level keys / array elements
PDF	pdf-parse v2 text extraction	By detected headings, min 1000 chars per chunk
CSV	Markdown table	100 rows per chunk (header repeated)
ZIP	File listing + text file extraction	Per-file chunks
Image	Original saved + metadata via image-size	Metadata as index chunk
Video	Original saved + metadata via ffprobe	Metadata as index chunk
Audio	Original saved + metadata via ffprobe	Metadata as index chunk
Binary (unknown)	Saved as-is	Single file
Unknown text	Raw	Paragraph split fallback

Large File Protection

• HEAD pre-check sends a HEAD request first to check Content-Length
• If file > --max-size (default 50MB), the download is blocked with a warning
• Add --dl-big-file to bypass the check
• Streaming check also enforces the limit during download
• HEAD failure gracefully falls back to direct GET

Media Metadata (Optional Dependencies)

• Images: image-size (built-in) → width, height, format
• Video/Audio: ffprobe from ffmpeg (system PATH) → duration, codec, resolution, bitrate, sample rate, channels
• Missing dependencies → graceful degradation (only file size and type returned)

Proxy Auto-Discovery

1. Explicit --proxy arg → skip auto-discovery
2. Direct connection attempted first
3. --never-proxy → throw original error
4. os-proxy-config (reads Windows registry / macOS prefs / env vars)
5. Env vars HTTP_PROXY / HTTPS_PROXY / all_proxy
6. Fallback: http://127.0.0.1:10808, http://127.0.0.1:7890

Caching

Default cache is project-local .fetch-cache/. Use --global-cache for OS temp directory. Each fetch creates a subdirectory with index.json (metadata) and chunk files. Stale records (1d+) show metadata but mark content as expired. Very old records (30d+) auto-deleted.

Dev

pnpm install
pnpm test
pnpm lint
pnpm typecheck
pnpm build

Built with:

• vite / vitest
• undici (HTTP fetch)
• turndown (HTML to Markdown)
• jsdom (HTML parsing)
• pdf-parse v2 (PDF text extraction)
• adm-zip (ZIP extraction)
• image-size (image metadata, pure JS)
• commander (CLI framework)
• picocolors (terminal colors)
• eslint + typescript-eslint (strict)
• TypeScript 6 + strict mode