Chapter 7: Troubleshooting
This chapter describes the error messages of Hitachi Replication Plug-in for Containers.
Note: If you cannot create or delete PVCs or Pods using Storage Plug-in for
Containers, see Troubleshooting chapter in the Hitachi Storage Plug-in for
Containers Quick Reference Guide.
Troubleshooting Replication CR issues
Make sure you have read through this document and understood how Replication CRs are
created. For more details, see Details on HUR pair creation (on page 31).
Creating a Replication CR triggers Hitachi Replication Plug-in for Containers to create a
LocalVolume and a RemoteVolume CR. These CRs are used to gather information about the
primary and secondary sites storage volumes.
When the status of Replication CR does not change to an expected status, then run the
kubectl describe command for the event log.
If an error message is shown in the newest event log, follow the instructions of the error
message to fix the problem. Also, check the event logs for the LocalVolume and
RemoteVolume CRs in the primary and secondary sites.
Collecting information for troubleshooting
Use the logs and suggested procedures to troubleshoot issues with Hitachi Replication Plug-
in for Containers.
Collecting logs for Hitachi Replication Plug-in for Containers
You can retrieve the logs for Hitachi Replication Plug-in for Containers by using kubectl
logs command.
% kubectl logs -n hspc-replication-operator-system deploy/hspc-replication-operator-controller-manager
Note: If necessary, set up cluster-level logging to save logs: https://kubernetes.io/
docs/concepts/cluster-administration/logging/
Chapter 7: Troubleshooting
Hitachi Replication Plug-in for Containers Configuration Guide 27
Collecting logs for Hitachi Replication Plug-in for Containers
To view recent events for Replication CR, run the following command:
% kubectl describe -n <namespace> <crd type> <cr name>
Note:
■ There are three types of CRD: Replication, LocalVolume, and RemoteVolume.
■ The name of these resources is the same as the Replication in the primary
site.
Collecting storage system information for VSP family
If you are using an SVP, collect the regular dump files.
If you are not using an SVP, collect system dumps by using the maintenance utility. For
details about how to collect the dump files of storage systems, see the Hitachi Virtual Storage
Platform System Administrator Guide.
Information needed in case of contacting support
If you must contact customer support, collect the following information before contacting
customer support:
Information Procedure
Cluster information Run the following command:
% kubectl cluster-info dump -A > dump.txt
Hitachi Replication Plug-in for Containers See Collecting logs for Hitachi Replication
logs Plug-in for Containers
Hitachi Replication Plug-in for Containers Manifest files for Hitachi Replication Plug-in
manifests for Containers
Replication manifests Get yaml files for Replication
PVC related manifests Get YAML file for StorageClass, Secret, and
PersistentVolumeClaim
Hitachi Storage Plug-in for Containers logs See Hitachi Storage Plug-in for Containers
Quick Reference Guide
Application manifests Get the YAML files for applications that uses
Storage Plug-in for Containers PVC
Storage logs See Collect storage system information for
VSP family (on page 28).
Chapter 7: Troubleshooting
Hitachi Replication Plug-in for Containers Configuration Guide 28
Restarting Hitachi Replication Plug-in for Containers
Restarting Hitachi Replication Plug-in for Containers
When you make configuration changes, you must restart the Hitachi Replication Plug-in for
Containers. Use the following Kubernetes commands to restart the Hitachi Replication Plug-
in for Containers:
# Restarting HRPC in the primary site
% KUBECONFIG=${KUBECONFIG_P} kubectl rollout restart deployment \
-n hspc-replication-operator-system hspc-replication-operator-
controller-manager
# Restarting HRPC in the secondary site
% KUBECONFIG=${KUBECONFIG_S} kubectl rollout restart deployment \
-n hspc-replication-operator-system hspc-replication-operator-
controller-manager
Deleting replication CRs without connection to a remote site
Deleting replication CRs requires connection to a remote site to gracefully delete all
Kubernetes objects and storage system assets. However, there may be a situation where
your remote site is not accessible, but you must delete the Replication CRs. In such cases,
you can delete the Replication CRs forcibly.
Procedure
1. To enable Hitachi Replication Plug-in for Containers and to delete Replication CRs
without connecting to a remote site, set
SPC_REPLICATION_ALLOW_FORCE_DELETION environment variable to true.
% kubectl set env -n hspc-replication-operator-system \
deployments/hspc-replication-operator-controller-manager \
SPC_REPLICATION_ALLOW_FORCE_DELETION=true
2. Run the following command to delete the Replication CRs forcibly.
% kubectl delete -f hspc_v1_replication.yaml
3. Using Storage Navigator delete the HUR pair corresponding with the Replication CR
since it remains after force deletion of Replication CRs.
4. Reset the SPC_REPLICATION_ALLOW_FORCE_DELETION environment variable to
false.
Chapter 7: Troubleshooting
Hitachi Replication Plug-in for Containers Configuration Guide 29
