kube-proxy iptables mode has 5-6x performance regression on aws-k8s-1.34 due to iptables-nft backend

[xqianwang]

Image I'm using:

• Bottlerocket OS 1.54.0 (aws-k8s-1.34) - kernel 6.12.63
• Bottlerocket OS 1.54.0 (aws-k8s-1.32) - kernel 6.1.159 (for comparison)
• Bottlerocket OS 1.54.0 (aws-k8s-1.33) - kernel 6.12.63 (for comparison)

What I expected to happen:

kube-proxy running in iptables mode should have similar rule sync performance on aws-k8s-1.34 nodes as on aws-k8s-1.32/1.33 nodes

What actually happened:
kube-proxy iptables mode sync latency is 5-6x slower on aws-k8s-1.34 nodes compared to aws-k8s-1.32/1.33 nodes.

Node Variant	iptables version	Rule Sync Latency	iptables-save time
aws-k8s-1.32	iptables v1.8.11 (legacy)	32-52ms	21ms
aws-k8s-1.33	iptables v1.8.11 (legacy)	24-34ms	-
aws-k8s-1.34	iptables v1.8.11 (nf_tables)	172-248ms	61ms

Test environment: Same EKS cluster , same kube-proxy version (v1.34.3-eksbuild.2), same workload (877 services, 2057 endpoints).

Root cause: aws-k8s-1.34 switched from legacy iptables to iptables-nft (nftables kernel backend with iptables compatibility layer). Every iptables operation is now translated to nftables, adding significant overhead.

Evidence from the nodes:

aws-k8s-1.32 (fast)

$ iptables --version
iptables v1.8.11 (legacy)

$ ls -la /usr/sbin/iptables*

Points to legacy iptables binaries

aws-k8s-1.34 (slow)

$ iptables --version
iptables v1.8.11 (nf_tables)

$ ls -la /usr/sbin/iptables*
lrwxrwxrwx. 1 root root 7 Jan 23 01:35 /usr/sbin/iptables -> xtables
lrwxrwxrwx. 1 root root 17 Jan 23 01:35 /usr/sbin/iptables-nft -> xtables-nft-multi

How to reproduce the problem:

1. Create an EKS cluster with mixed node groups:
   • Node group A: Bottlerocket aws-k8s-1.32 AMI
   • Node group B: Bottlerocket aws-k8s-1.34 AMI
2. Deploy kube-proxy in iptables mode (EKS default)
3. Deploy enough services to generate meaningful load (100+ services)
4. Check kube-proxy logs on both node types:
   kubectl logs -n kube-system | grep "SyncProxyRules complete"

[mhadimani]

Seeing similar issues on K8s 1.32 with an AL2023 node

[KCSesh]

Seems Related:
kubernetes/kubernetes#135639

[mhadimani]

@KCSesh , the linked issue could be unrelated. The issue being reported is seen with the kube-proxy 'iptables' and not nftables.

[xqianwang]

@KCSesh , the linked issue could be unrelated. The issue being reported is seen with the kube-proxy 'iptables' and not nftables.

@mhadimani That is correct. The performance regression was with 'iptables-nft', not the native nftables mode @KCSesh