nftables with conntrack can break k3s

[lspgn]

Environmental Info:
K3s Version:

$ k3s -v
k3s version v1.30.6+k3s1 (1829eaae)
go version go1.22.8

Node(s) CPU architecture, OS, and Version:

Ubuntu 24.04

$ uname -a
Linux abc 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov  4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: Single node

Describe the bug:

When a certain conntrack rule is added via the nft command, it breaks k3s when it tries to setup the firewall:

Dec 05 08:05:15 abc k3s[10542]: panic: F1205 08:05:15.257193   10542 network_policy_controller.go:413] failed to list rules in filter table INPUT chain due to running [/usr/sbin/iptables -t filter -S INPUT --wait]: exit status 1: iptables v1.8.10 (nf_tables): chain `INPUT' in table `filter' is incompatible, use 'nft' tool.
Dec 05 08:05:15 abc k3s[10542]: goroutine 58811 [running]:
Dec 05 08:05:15 abc k3s[10542]: k8s.io/klog/v2.(*loggingT).output(0xaab94c0, 0x3, 0xc000a50de0, 0xc0017e9880, 0x1, {0x86a37a6?, 0x2?}, 0xc00fc63250?, 0x0)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:965 +0x73d
Dec 05 08:05:15 abc k3s[10542]: k8s.io/klog/v2.(*loggingT).printfDepth(0xaab94c0, 0x3, 0xc000a50de0, {0x0, 0x0}, 0x1, {0x6439f02, 0x37}, {0xc00fc2bde0, 0x2, ...})
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:767 +0x1f0
Dec 05 08:05:15 abc k3s[10542]: k8s.io/klog/v2.(*loggingT).printf(...)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:744
Dec 05 08:05:15 abc k3s[10542]: k8s.io/klog/v2.Fatalf(...)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:1655
Dec 05 08:05:15 abc k3s[10542]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains.func2({0x7298940, 0xc0101be730}, {0x62b5350, 0x5}, {0xc0101f4c00, 0x6, 0x6}, {0xc0101b4b00, 0x10}, 0x1)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:413 +0x316
Dec 05 08:05:15 abc k3s[10542]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains(0xc008b53320)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:467 +0x1be9
Dec 05 08:05:15 abc k3s[10542]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).Run(0xc008b53320, 0xc0065e1320, 0xc001a78a20, 0xc004e1a040)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:168 +0x171
Dec 05 08:05:15 abc k3s[10542]: created by github.com/k3s-io/k3s/pkg/agent/netpol.Run in goroutine 1
Dec 05 08:05:15 abc k3s[10542]:         /go/src/github.com/k3s-io/k3s/pkg/agent/netpol/netpol.go:184 +0xe34
Dec 05 08:05:15 abc systemd[1]: k3s.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 05 08:05:15 abc systemd[1]: k3s.service: Failed with result 'exit-code'.

Steps To Reproduce:

There are two ways of adding the "rule" (eg: allow all established TCP), one with nft and the other with iptables.
The first one breaks k3s but the second does not. There seem to be a translation of established,related into opcodes breaking the iptables -S command, breaking in turn k3s.

sudo nft add rule filter INPUT ct state related,established accept

will be converted into:

$ sudo nft -a -n list table ip filter
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter { # handle 32
	chain INPUT { # handle 1
		type filter hook input priority 0; policy drop;
		ct state 0x2,0x4 accept # handle 163
	}

but

sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

will result into

$ sudo nft -a -n list table ip filter
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter { # handle 32
	chain INPUT { # handle 1
		type filter hook input priority 0; policy drop;
		ct state related,established counter packets 0 bytes 0 accept # handle 164
	}

Expected behavior:

With the iptables command

$ sudo /usr/sbin/iptables -t filter -S INPUT --wait
-P INPUT DROP
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Actual behavior:

With the nft command

$ sudo /usr/sbin/iptables -t filter -S INPUT --wait
iptables v1.8.10 (nf_tables): chain `INPUT' in table `filter' is incompatible, use 'nft' tool.

Thank you for your help

[brandond]

Dec 05 08:05:15 abc k3s[10542]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains.func2({0x7298940, 0xc0101be730}, {0x62b5350, 0x5}, {0xc0101f4c00, 0x6, 0x6}, {0xc0101b4b00, 0x10}, 0x1)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:413 +0x316
Dec 05 08:05:15 abc k3s[10542]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains(0xc008b53320)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:467 +0x1be9
Dec 05 08:05:15 abc k3s[10542]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).Run(0xc008b53320, 0xc0065e1320, 0xc001a78a20, 0xc004e1a040)
Dec 05 08:05:15 abc k3s[10542]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:168 +0x171
Dec 05 08:05:15 abc k3s[10542]: created by github.com/k3s-io/k3s/pkg/agent/netpol.Run in goroutine 1
Dec 05 08:05:15 abc k3s[10542]:         /go/src/github.com/k3s-io/k3s/pkg/agent/netpol/netpol.go:184 +0xe34

This crash is in the kube-router netpol controller. You'll want to open an issue at https://github.com/cloudnativelabs/kube-router

When a certain conntrack rule is added via the nft command, it breaks k3s when it tries to setup the firewall

Maybe don't try to mix native nft and iptables-nft use on the same host? Why are you doing this in the first place?

[lspgn]

@brandond thank you!
Will open an issue there and link to it here.

Maybe don't try to mix native nft and iptables-nft use on the same host? Why are you doing this in the first place?

I mostly wanted to transition to native nft since it's there now but will fallback to iptables-nft.