Initial troubleshooting¶
Percona Operator for MySQL uses Custom Resources to manage options for the various components of the cluster.
-
PerconaServerMySQL
Custom Resource with Percona Server for MySQL cluster options (it has handyps
shortname also), -
PerconaServerMySQLBackup
andPerconaServerMySQLRestore
Custom Resources contain options for Percona XtraBackup used to backup Percona Server for MySQL and to restore it from backups (ps-backup
andps-restore
shortnames are available for them).
The first thing you can check for the Custom Resource is to query it with kubectl get
command:
$ kubectl get ps
Expected output
NAME REPLICATION ENDPOINT STATE MYSQL ORCHESTRATOR HAPROXY ROUTER AGE
cluster1 group-replication cluster1-haproxy.default ready 3 3 20m
The Custom Resource should have ready
state.
Note
You can check which Percona’s Custom Resources are present and get some information about them as follows:
$ kubectl api-resources | grep -i percona
Expected output
perconaservermysqlbackups ps-backup,ps-backups ps.percona.com/v1alpha1 true PerconaServerMySQLBackup
perconaservermysqlrestores ps-restore ps.percona.com/v1alpha1 true PerconaServerMySQLRestore
perconaservermysqls ps ps.percona.com/v1alpha1 true PerconaServerMySQL
Check the Pods¶
If Custom Resource is not getting ready
state, it makes sense to check
individual Pods. You can do it as follows:
$ kubectl get pods
Expected output
The above command provides the following insights:
READY
indicates how many containers in the Pod are ready to serve the traffic. In the above example,cluster1-haproxy-0
container has all two containers ready (2/2). For an application to work properly, all containers of the Pod should be ready.STATUS
indicates the current status of the Pod. The Pod should be in aRunning
state to confirm that the application is working as expected. You can find out other possible states in the official Kubernetes documentation .RESTARTS
indicates how many times containers of Pod were restarted. This is impacted by the Container Restart Policy . In an ideal world, the restart count would be zero, meaning no issues from the beginning. If the restart count exceeds zero, it may be reasonable to check why it happens.AGE
: Indicates how long the Pod is running. Any abnormality in this value needs to be checked.
You can find more details about a specific Pod using the
kubectl describe pods <pod-name>
command.
$ kubectl describe pods cluster1-mysql-0
Expected output
...
Name: cluster1-mysql-0
Namespace: default
...
Controlled By: StatefulSet/cluster1-mysql
Init Containers:
mysql-init:
...
Containers:
mysql:
...
Restart Count: 0
Limits:
memory: 2G
Requests:
memory: 2G
Liveness: exec [/opt/percona/healthcheck liveness] delay=15s timeout=30s period=10s #success=1 #failure=3
Readiness: exec [/opt/percona/healthcheck readiness] delay=30s timeout=3s period=5s #success=1 #failure=3
Startup: exec [/opt/percona/bootstrap] delay=15s timeout=300s period=10s #success=1 #failure=1
Environment:
...
Mounts:
...
Volumes:
...
Events: <none>
This gives a lot of information about containers, resources, container status and also events. So, describe output should be checked to see any abnormalities.