Transparent gateway stall under high normal connection churn (tun + gvisor + fakeip + WG) before host saturation

[richfang]

Version / Environment

• sing-box: 1.13.11
• OS: Ubuntu Linux x86_64
• Deployment role: transparent office gateway
• Config shape at incident time:
  • tun
  • stack: gvisor
  • auto_route: true
  • mtu: 1280
  • fakeip enabled
  • sniff enabled
  • log level debug
  • outbound split through WG-HongKong and nested WG-Singapore

Problem

Under normal office traffic, the gateway intermittently appears to “disconnect” for users for several seconds.

The important part is that this does not look like:

• upstream packet loss
• sing-box restart/crash
• CPU saturation
• link saturation
• OOM

Instead, it looks like sing-box becomes internally congested first under a burst of normal business connections.

Incident window

Observed around:

• 2026-05-18 14:07 to 14:10 (Asia/Shanghai, UTC+8)

Business destination IPs involved

These are confirmed normal company business services:

• 106.75.138.153:443
• 120.55.168.246:443
• 106.75.169.134:443

Why this does not look like a simple host/network bottleneck

Upstream remained healthy

Gateway -> upstream gateway stayed at 0% packet loss.

No service or kernel failure

During the incident window:

• journalctl -k: no NIC reset / link flap / kernel fault
• journalctl -u sing-box: no restart or crash

CPU was not saturated

At 2026-05-18 14:10:00:

• all CPUs: about 22%
• busiest relevant core: about 37%

Memory increased sharply, but host did not OOM

Health log RSS trend:

• 14:07: 4.33G
• 14:08: 1.69G
• 14:09: 1.92G
• 14:10: 5.09G
• 14:11: 6.85G
• 14:12: 7.20G
• 14:13: 8.05G

But there was:

• no OOM kill
• no swap storm
• no reboot
• no sing-box restart

NIC bandwidth was not saturated

Interface utilization was far below 1G saturation.

Evidence that sing-box fell behind on connection handling

Per-minute counters in the critical window:

• 14:07: inbound=6952, packet=972, sniff=948, finished=2883
• 14:08: inbound=7622, packet=1086, sniff=973, finished=2651
• 14:09: inbound=7246, packet=1091, sniff=977, finished=4020
• 14:10: inbound=2924, packet=993, sniff=477, finished=2936

This looks like:

• from 14:07 to 14:09, new connections were arriving much faster than they were finishing
• at 14:10, new inbound count dropped sharply
• later 14:11-14:13 looked like retry/reconnect amplification

This feels more like “new connection handling fell behind” than “traffic disappeared”.

Other notes

• FakeIP pool itself did not appear exhausted
• We saw a missing fakeip record symptom, but not pool depletion
• tun0 qlen=500 did not look like the main cause
• The issue reproduced under normal company traffic, not obviously abnormal traffic

Question

Is this a known weak path or expected limitation for deployments shaped like:

• tun
• gvisor
• fakeip
• debug
• transparent multi-user gateway
• WG-based outbound split

From an operator perspective, sing-box appears to become the bottleneck in its internal connection/state path before the host itself is actually saturated.

Are there known mitigations or recommended deployment changes for this scenario?

[hypressi0n]

Reporting the same pattern on a very different deployment shape — adding data points in case it helps narrow the root cause.

Environment

• sing-box: 1.12.17, linux/arm64, tags with_gvisor,with_quic,with_wireguard,with_clash_api,with_utls,with_tailscale, CGO enabled
• Host: Xiaomi AX3000T router, MediaTek MT7981B (2× Cortex-A53), 256 MB RAM, no swap
• OS: OpenWrt 25.12.2, kernel 6.12.74
• Config shape:
  • inbound tproxy (not tun), managed by podkop
  • fakeip enabled (198.18.0.0/15), store_fakeip: true, sniff enabled
  • outbound: urltest over 3× vless+reality peers
  • urltest_check_interval: 3m
  • rule-sets from itdoginfo/allow-domains

Same internal-state-grows-faster-than-it-shrinks pattern

Even though my setup is the opposite end of yours in scale (256 MB instead of 8+ GB), the trajectory matches what you describe. Captured by a per-minute health snapshot:

Time	MemAvailable	load1	tcp_estab	conntrack	sing-box RSS
00:30	14 MB	0.23	143	728	46 MB
00:31	15 MB	0.36	147	716	45 MB
00:32	4.2 MB	1.32	302	1475	55 MB
00:33	4.0 MB	1.79	195	1128	53 MB
00:34 (+10s)	0.6 MB	5.55	83	911	57 MB
00:35	0.07 MB	7.59	112	889	58 MB
00:36 (+2s)	0 MB	8.54	54	926	58 MB
(after reboot)	60 MB	—	—	—	~38 MB

• Trigger looks identical to your inbound >> finished window: at 00:32 a burst doubled tcp_estab (147→302) and conntrack (728→1475). sing-box RSS grew +10 MB in 60 seconds, then refused to shrink even as the burst subsided.
• After the burst passes the connection counts come back down, but RSS does not — same behaviour you see at scale (RSS climbing 4.3→8 GB and not returning).
• Once MemAvailable hits ≤1 MB, kswapd0 ends up at the top of CPU usage (visible in my per-minute top sampler) — both cores stall in direct-reclaim, userspace freezes, web UI becomes unreachable. Manual power-cycle required.

Slow growth across multi-day uptime

Over a ~3.6-day uptime, baseline MemAvailable drifts from ~60 MB right after boot to ~18-22 MB, then occasional intra-hour dips down to 5-13 MB. OOM-killer had already killed sing-box twice in this uptime before the visible collapse:

[186745.579913] Out of memory: Killed process 23629 (sing-box)
  total-vm:1436964kB, anon-rss:72028kB, file-rss:36kB, shmem-rss:192kB,
  UID:0 pgtables:316kB oom_score_adj:0

i.e. the issue you observe as "memory keeps growing without OOM" I observe as "memory keeps growing until OOM, restart, repeat". The growth-rate side looks the same; I just don't have headroom to mask it.

missing fakeip record

I see the same router: missing fakeip record, try enable 'experimental.cache_file' symptom you mention — even though I do have cache_file.store_fakeip: true enabled. It appears reliably after sing-box restart (whether from OOM or controlled), during the first ~10 seconds while clients try to use IPs from the previous fakeip pool.

Historical context

This recurrence pattern has been reported and closed-as-stale repeatedly across years:

• #1607 (2024-05, OpenWrt, heap profile pointed at canceler.NewTimeoutPacketConn)
• #1736, #2027, #2382 (2024)
• #3172 (2025-07, macOS SFM, tun + vless + urltest, RSS climbs to 1 GB+ over days)
• and others

The common substrate seems to be urltest + fakeip + sniff + sustained real traffic over time, regardless of platform or inbound type (tun/tproxy).

Happy to provide

• Per-minute health log for the 24 hours preceding the collapse
• Full /etc/sing-box/config.json (generated by podkop) and OpenWrt nftables ruleset
• Repro environment is constrained — it depends on a real upstream VPN + real client traffic, so I can't make it self-contained, which I see is a template requirement. Mentioning the data here in case it's useful even without a local repro.