Asymmetric routing using mirrored network mode?

[wizpresso-steve-cy-fan]

I saw this from my Kubernetes Pod in WSL2, running curl 1.1.1.1 inside a pod as a result:

tcp      6 116 SYN_SENT src=10.244.75.137 dst=1.1.1.1 sport=54562 dport=80 [UNREPLIED] src=1.1.1.1 dst=172.30.0.7 sport=80 dport=4440 mark=0 use=1

10.244.75.137 being the Pod IP and 172.30.0.7 being the local NIC IP on the worker node. It seems like we have a case of asymmetric routing using mirrored network

I also saw it from tcpdump, all the packets initiated from the actions in the pod container will have a packet length of 0.

[root@WIZPRESSO-07 ~]# tcpdump -i eth0 dst 1.1.1.1 or src 1.1.1.1
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:23:47.099223 IP WIZPRESSO-07.15964 > one.one.one.one.http: Flags [S], seq 2816550060, win 64860, options [mss 1410,sackOK,TS val 1817783335 ecr 0,nop,wscale 7], length 0
16:23:48.131793 IP WIZPRESSO-07.15964 > one.one.one.one.http: Flags [S], seq 2816550060, win 64860, options [mss 1410,sackOK,TS val 1817784368 ecr 0,nop,wscale 7], length 0

Capturing the Calico interface (the pod IP change a little, but the result is the same):

[root@WIZPRESSO-07 ~]# tcpdump -i caliaebf3765d75 dst 10.244.75.138 or src 10.244.75.138
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on caliaebf3765d75, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:41:58.521860 ARP, Request who-has 169.254.1.1 tell 10.244.75.138, length 28
16:41:58.521881 ARP, Reply 169.254.1.1 is-at ee:ee:ee:ee:ee:ee (oui Unknown), length 28
16:41:58.521883 IP 10.244.75.138.46224 > one.one.one.one.http: Flags [S], seq 136154146, win 64860, options [mss 1410,sackOK,TS val 1343131119 ecr 0,nop,wscale 7], length 0
16:41:58.974691 IP WIZPRESSO-07.60926 > 10.244.75.138.hostmon: Flags [S], seq 539835790, win 64860, options [mss 1410,sackOK,TS val 590740937 ecr 0,nop,wscale 7,tfo  cookiereq,nop,nop], length 0
16:41:58.974714 IP 10.244.75.138.hostmon > WIZPRESSO-07.60926: Flags [R.], seq 0, ack 539835791, win 0, length 0
16:41:59.570414 IP 10.244.75.138.46224 > one.one.one.one.http: Flags [S], seq 136154146, win 64860, options [mss 1410,sackOK,TS val 1343132168 ecr 0,nop,wscale 7], length 0
16:42:01.650416 IP 10.244.75.138.46224 > one.one.one.one.http: Flags [S], seq 136154146, win 64860, options [mss 1410,sackOK,TS val 1343134248 ecr 0,nop,wscale 7], length 0
16:42:04.050388 ARP, Request who-has 10.244.75.138 tell WIZPRESSO-07, length 28
16:42:04.050426 ARP, Reply 10.244.75.138 is-at 2e:ed:c1:63:77:13 (oui Unknown), length 28
16:42:05.730410 IP 10.244.75.138.46224 > one.one.one.one.http: Flags [S], seq 136154146, win 64860, options [mss 1410,sackOK,TS val 1343138328 ecr 0,nop,wscale 7], length 0

This is the result captured from running curl 1.1.1.1 on the WSL shell (without getting into another k8s pod network namespace):

[root@WIZPRESSO-07 ~]# tcpdump -i eth0 dst 1.1.1.1 or src 1.1.1.1
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:35:06.655968 IP WIZPRESSO-07.60880 > one.one.one.one.http: Flags [S], seq 1791911529, win 64240, options [mss 1460,sackOK,TS val 654083934 ecr 0,nop,wscale 7], length 0
16:35:06.658901 IP one.one.one.one.http > WIZPRESSO-07.60880: Flags [S.], seq 1967045943, ack 1791911530, win 65160, options [mss 1460,sackOK,TS val 1862767259 ecr 654083934,nop,wscale 13], length 0
16:35:06.658982 IP WIZPRESSO-07.60880 > one.one.one.one.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 654083937 ecr 1862767259], length 0
16:35:06.659072 IP WIZPRESSO-07.60880 > one.one.one.one.http: Flags [P.], seq 1:72, ack 1, win 502, options [nop,nop,TS val 654083937 ecr 1862767259], length 71: HTTP: GET / HTTP/1.1
16:35:06.661579 IP one.one.one.one.http > WIZPRESSO-07.60880: Flags [.], ack 72, win 7, options [nop,nop,TS val 1862767262 ecr 654083937], length 0
16:35:06.662625 IP one.one.one.one.http > WIZPRESSO-07.60880: Flags [P.], seq 1:387, ack 72, win 8, options [nop,nop,TS val 1862767262 ecr 654083937], length 386: HTTP: HTTP/1.1 301 Moved Permanently
16:35:06.662667 IP WIZPRESSO-07.60880 > one.one.one.one.http: Flags [.], ack 387, win 501, options [nop,nop,TS val 654083941 ecr 1862767262], length 0
16:35:06.662794 IP WIZPRESSO-07.60880 > one.one.one.one.http: Flags [F.], seq 72, ack 387, win 501, options [nop,nop,TS val 654083941 ecr 1862767262], length 0
16:35:06.665537 IP one.one.one.one.http > WIZPRESSO-07.60880: Flags [F.], seq 387, ack 73, win 8, options [nop,nop,TS val 1862767266 ecr 654083941], length 0
16:35:06.665583 IP WIZPRESSO-07.60880 > one.one.one.one.http: Flags [.], ack 388, win 501, options [nop,nop,TS val 654083944 ecr 1862767266], length 0

---

I ran the same process except the WSL2 host instance is using bridged network mode:
So I ran curl 1.1.1.1 in one of the pod specifically in a k8s node that runs bridged network...

wizpresso@WIZPRESSO-14:~$ sudo tcpdump -i eth0 dst 1.1.1.1 or src 1.1.1.1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:36:46.838311 IP 172.30.14.123.6978 > one.one.one.one.http: Flags [S], seq 503430139, win 64860, options [mss 1410,sackOK,TS val 2414218992 ecr 0,nop,wscale 7], length 0
16:36:46.841148 IP one.one.one.one.http > 172.30.14.123.6978: Flags [S.], seq 1797645387, ack 503430140, win 65160, options [mss 1460,sackOK,TS val 2103919644 ecr 2414218992,nop,wscale 13], length 0
16:36:46.841186 IP 172.30.14.123.6978 > one.one.one.one.http: Flags [.], ack 1, win 507, options [nop,nop,TS val 2414218995 ecr 2103919644], length 0
16:36:46.841206 IP 172.30.14.123.6978 > one.one.one.one.http: Flags [P.], seq 1:72, ack 1, win 507, options [nop,nop,TS val 2414218995 ecr 2103919644], length 71: HTTP: GET / HTTP/1.1
16:36:46.843848 IP one.one.one.one.http > 172.30.14.123.6978: Flags [.], ack 72, win 7, options [nop,nop,TS val 2103919647 ecr 2414218995], length 0
16:36:46.844849 IP one.one.one.one.http > 172.30.14.123.6978: Flags [P.], seq 1:387, ack 72, win 8, options [nop,nop,TS val 2103919647 ecr 2414218995], length 386: HTTP: HTTP/1.1 301 Moved Permanently
16:36:46.844866 IP 172.30.14.123.6978 > one.one.one.one.http: Flags [.], ack 387, win 504, options [nop,nop,TS val 2414218998 ecr 2103919647], length 0
16:36:46.844973 IP 172.30.14.123.6978 > one.one.one.one.http: Flags [F.], seq 72, ack 387, win 504, options [nop,nop,TS val 2414218998 ecr 2103919647], length 0
16:36:46.847791 IP one.one.one.one.http > 172.30.14.123.6978: Flags [F.], seq 387, ack 73, win 8, options [nop,nop,TS val 2103919651 ecr 2414218998], length 0
16:36:46.847815 IP 172.30.14.123.6978 > one.one.one.one.http: Flags [.], ack 388, win 504, options [nop,nop,TS val 2414219001 ecr 2103919651], length 0

tcpdump from the pod NIC created by Calico:

wizpresso@WIZPRESSO-14:~$ sudo tcpdump -i calid062c39e1b4 dst 10.244.218.106 or src 10.244.218.106
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on calid062c39e1b4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:40:04.860218 IP 10.244.218.106.48374 > one.one.one.one.http: Flags [S], seq 1897226260, win 64860, options [mss 1410,sackOK,TS val 2414417019 ecr 0,nop,wscale 7], length 0
16:40:04.863184 IP one.one.one.one.http > 10.244.218.106.48374: Flags [S.], seq 1802784603, ack 1897226261, win 65160, options [mss 1460,sackOK,TS val 2156359339 ecr 2414417019,nop,wscale 13], length 0
16:40:04.863212 IP 10.244.218.106.48374 > one.one.one.one.http: Flags [.], ack 1, win 507, options [nop,nop,TS val 2414417022 ecr 2156359339], length 0
16:40:04.863261 IP 10.244.218.106.48374 > one.one.one.one.http: Flags [P.], seq 1:72, ack 1, win 507, options [nop,nop,TS val 2414417023 ecr 2156359339], length 71: HTTP: GET / HTTP/1.1
16:40:04.865976 IP one.one.one.one.http > 10.244.218.106.48374: Flags [.], ack 72, win 7, options [nop,nop,TS val 2156359342 ecr 2414417023], length 0
16:40:04.866912 IP one.one.one.one.http > 10.244.218.106.48374: Flags [P.], seq 1:387, ack 72, win 8, options [nop,nop,TS val 2156359342 ecr 2414417023], length 386: HTTP: HTTP/1.1 301 Moved Permanently
16:40:04.866927 IP 10.244.218.106.48374 > one.one.one.one.http: Flags [.], ack 387, win 504, options [nop,nop,TS val 2414417026 ecr 2156359342], length 0
16:40:04.867042 IP 10.244.218.106.48374 > one.one.one.one.http: Flags [F.], seq 72, ack 387, win 504, options [nop,nop,TS val 2414417026 ecr 2156359342], length 0
16:40:04.869886 IP one.one.one.one.http > 10.244.218.106.48374: Flags [F.], seq 387, ack 73, win 8, options [nop,nop,TS val 2156359345 ecr 2414417026], length 0
16:40:04.869892 IP 10.244.218.106.48374 > one.one.one.one.http: Flags [.], ack 388, win 504, options [nop,nop,TS val 2414417029 ecr 2156359345], length 0

Then this is pretty normal

[wizpresso-steve-cy-fan]

I switched from VXLAN mode to Wireguard mode in Calico, but I still got the same result. All packets have a length of 0 going out.

Related: #7547, #5810, #7407

[wizpresso-steve-cy-fan]

Closed in favor of #10842