Multi-socket on the same port is slow under load

[edthepurple]

Describe the bug

When running OpenVPN 2.7.0 on Ubuntu 24.04 and configuring a single OpenVPN instance to listen on both TCP and UDP on the same port (8443) using:

local * 8443 tcp
local * 8443 udp

the initial connection establishment becomes extremely slow under load (approximately 400–500 concurrent clients). TLS handshake and session establishment can take 30+ seconds. Post-connection latency is also noticeably increased.

If TCP and UDP are handled by separate OpenVPN instances (each bound to port 8443 but only one protocol per instance), the issue does not occur. In that setup, connection establishment is immediate and latency remains stable under the same load.

The server has sufficient CPU, memory, and network capacity. No resource exhaustion or networking anomalies are observed at the OS level. OpenVPN logs (with verb 0) do not show warnings or errors during the issue.

This setup also uses deferred authentication via auth-script-openvpn:
https://github.com/fac/auth-script-openvpn

To Reproduce

Install OpenVPN 2.7.0 on Ubuntu 24.04.

Configure a single OpenVPN server instance to listen on both TCP and UDP on the same port:

local * 8443 tcp
local * 8443 udp

Enable deferred authentication using:
https://github.com/fac/auth-script-openvpn

Generate load with approximately 400–500 concurrent clients connecting.

Observe connection establishment time and latency.

Expected behavior

Connection establishment should remain fast and consistent under load, similar to when TCP and UDP are handled by separate OpenVPN instances.

There should be no significant degradation in initial handshake time or post-connection latency simply due to serving both protocols from a single instance on the same port.

Version information

OS: Ubuntu 24.04

OpenVPN version: 2.7.0

Clients: Mixed TCP and UDP clients (same server port 8443)

Additional context

The issue is reproducible only when both TCP and UDP are served by the same OpenVPN instance on the same port.

When splitting into two separate OpenVPN processes (one TCP-only, one UDP-only), performance is stable under identical load.

The server is not CPU-bound, memory-bound, or I/O-bound during the issue.

No relevant warnings or errors appear in openvpn.log at verb 0.

[cron2]

Session handshake aside, there is something here which should make it easier to pinpoint - you say "post-connection latency", so "when there are 400 clients connected, ping times to/through this OpenVPN server instance goes up"?

(Obviously this is without DCO, as with DCO "userspace latency" has no direct effect on forwarding latency).

How much latency are we speaking about? What is the 1 client baseline, what is with 500 clients? Is this "500 clients on TCP" or "250 UDP, 250 TCP"? Is it sufficient to have that many clients connected to get extra latency on any one of them, or do they all need to do some traffic? Are only TCP clients affected, or UDP clients as well?

("Many clients on TCP" produce "many different sockets", which could hint at issues with event handling / select() / poll() handling, while "many clients on UDP" all use the same socket, thus it would have to do something different)

Sorry for coming back with so many questions and no solution yet ;-)

pinging @ordex, @ralflici, @itsGiaan - this very much smells like "the new event handling"...

[edthepurple]

Thanks for your reply.
We're talking about roughly 250 udp clients and 250 tcp clients. sometimes udp clients are a little more than tcp ones.
and what I mean by post-connection latency is normal web things like pinging 8.8.8.8 through the openvpn connection from the client's machine. which is normally around 80-90ms and when the slowdown happens it goes as high as 12000ms! and at midnight where there are less clients online, everything automatically becomes normal again. so I think more traffic = more latency. and this happens to both UDP and TCP clients.

and about DCO, on previous versions of openvpn, I could do this

git clone https://github.com/OpenVPN/ovpn-dco
cd ovpn-dco
make
make install
modprobe ovpn-dco-v2
echo "ovpn-dco-v2" | tee /etc/modules-load.d/ovpn-dco.conf
depmod -a
update-initramfs -u

and I could verify that openvpn is running with DCO by doing

openvpn --version

and it would say the DCO version configured with it.
but I could not do that with 2.7.0. I don't know why or what I'm doing wrong.

[cron2]

OpenVPN 2.6 uses the "ovpn-dco-v2" module, which is maintained outside the kernel.

OpenVPN 2.7 uses the "ovpn" module, which is part of the upstream kernel since (I think) 6.16. Backports to older kernels are available via https://github.com/OpenVPN/ovpn-backports - that repo has instructions, and pointers to precompiled packages for some Linux distributions.

As for the latency, I understand that "250+250 clients, all doing some sort of light traffic through the server" will be sufficient to produce a very significant latency increase - not something which might not be noticed ("80ms to 90ms") but very significant. We need to see how to reproduce that...

[ordex]

With 2.7.0 you need the new ovpn kernel module (part of upstream Linux kernel since 6.16, otherwise you can clone the ovpn-backports project and compile it, or install via pkg manager)

[schwabe]

No relevant warnings or errors appear in openvpn.log at verb 0.

verb 0 also shows absolutely critical errors. So warnings are supressed at verb 0.