Retina can break connectivity of pods to the Kubernetes Cluster IP in clusters using Cilium

[andreev-io]

This issue is seen in both AKS and GCP. See notes for AKS at #252 (comment)

Describe the bug
Upon installation of Retina, connectivity can be lost for pods in a GKE cluster using managed Cilium.

To Reproduce

1. Go to create a standard GKE cluster.
2. Select the Standard: You manage your cluster option (see screenshot 1).
3. Specify GKE version 1.26.11-gke.105500 in the No channel channel selector (see screenshot 2). We suspect the issue would occur with other versions too, but we used a specific one for reproducability.
4. [Optional] Configure the cluster to run in one AZ with fewer nodes than the default to manage cost.
5. [Important] In the Networking configuration tab for the entire cluster, select Enable Dataplane V2 to enable managed Cilium-powered networking.
6. Create the cluster and wait for all default pods in the cluster to come up.
7. Install Retina and wait for the agent pods to start.

> VERSION=$( curl -sL https://api.github.com/repos/microsoft/retina/releases/latest | jq -r .name)
helm install retina oci://ghcr.io/microsoft/retina/charts/retina \
    --set namespace=kube-system \
    --version $VERSION \
    --namespace kube-system \
    --set image.tag=$VERSION \
    --set operator.tag=$VERSION \
    --set image.pullPolicy=Always \
    --set logLevel=info \
    --set operator.enabled=true \
    --set operator.enableRetinaEndpoint=true \
    --set enabledPlugin_linux="\[packetparser\]" \
    --set enablePodLevel=true \
    --set remoteContext=true

Note: if you are running a cluster with small nodes, you might need to manually edit the retina-agent DaemonSet to lower resource requests. Wait until retina-agent pods start.

8. Identify metrics-server running in the kube-system namespace and check its logs. You will see error logs such as

E0409 15:21:23.378785       1 webhook.go:202] Failed to make webhook authorizer request: Post "https://10.114.192.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": context canceled
E0409 15:21:23.378851       1 errors.go:77] Post "https://10.114.192.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": context canceled

9. Identify the cluster IP and the endpoint IP:

> kubectl get service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.114.192.1   <none>        443/TCP   45m
> kubectl get ep     
NAME         ENDPOINTS        AGE
kubernetes   10.128.0.7:443   45m

10. Connect to another pod and check connectivity to these origins. You'll see that there is connectivity to the endpoint IP but not to the service IP.

> kubectl debug -ti --image="nixery.dev/shell/curl" kube-dns-ff4bbcc87-tvzm7 -n kube-system
bash-5.2# curl https://10.114.192.1 -v -k
...
bash-5.2# curl https://10.128.0.7 -v -k
*   Trying 10.128.0.7:443...
* Connected to 10.128.0.7 (10.128.0.7) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=34.173.138.225
*  start date: Apr  9 14:52:44 2024 GMT
*  expire date: Apr  8 14:54:44 2029 GMT
*  issuer: CN=ca353e3b-048b-4feb-aa93-19a7c8a6aa89
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://10.128.0.7/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 10.128.0.7]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: 10.128.0.7
> User-Agent: curl/8.4.0
> Accept: */*
> 
* received GOAWAY, error=0, last_stream=1
< HTTP/2 403 
< audit-id: 2c7f6280-d595-4ddf-850f-abf1cadd85d8
< cache-control: no-cache, private
< content-type: application/json
< x-content-type-options: nosniff
< x-kubernetes-pf-flowschema-uid: 759447f6-3823-412a-86a3-09c764ef91eb
< x-kubernetes-pf-prioritylevel-uid: 2707b41b-d15c-402a-a039-b0df8aff1c2d
< content-length: 217
< date: Tue, 09 Apr 2024 15:45:36 GMT
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
* Closing connection
* TLSv1.3 (OUT), TLS alert, close notify (256):

Expected behaviour
No connectivity impact when installing Retina.

Screenshots
Step (2). Select Standard: You manage your cluster.

Step (3). No channel when specifying the version, then specify version 1.26.11-gke.1055000.

Step (4). Select Enable Dataplane V2 in the cluster network configuration tab.

Platform (please complete the following information):
See steps to reproduce.

Additional context
N/A

[andreev-io]

We reproduced an identical issue in AKS. We created a cluster using managed Cilium:

az group create --name ilya-experiments --location uksouth
az network vnet create -g ilya-experiments --location uksouth --name ilya-vnet --address-prefixes 10.0.0.0/8 -o table
az network vnet subnet create -g ilya-experiments --vnet-name ilya-vnet --name nodesubnet --address-prefixes 10.240.0.0/16 -o none
az network vnet subnet create -g ilya-experiments --vnet-name ilya-vnet --name podsubnet --address-prefixes 10.241.0.0/16 -o none
az aks create -n ilya-experiments -g ilya-experiments -l uksouth --max-pods 250 --network-plugin azure --vnet-subnet-id /subscriptions/<>/resourceGroups/ilya-experiments/providers/Microsoft.Network/virtualNetworks/ilya-vnet/subnets/nodesubnet --pod-subnet-id /subscriptions/<>/resourceGroups/ilya-experiments/providers/Microsoft.Network/virtualNetworks/ilya-vnet/subnets/podsubnet --network-dataplane cilium

Installed Retina like described above and observed the same symptoms.

[rbtr]

While we start to repro and root-cause this, any additional datapoints from others running on EKS/GKE and/or with Cilium are welcome