Kubernetes Troubleshooting Runbook: Debugging, kubectl Workflows, and Common Failure Modes

## Kubernetes Troubleshooting Runbook Troubleshooting in Kubernetes can seem daunting due to its distributed nature and abstraction layers. However, by adopting a systematic approach and leveraging the powerful `kubectl` command-line tool, you can efficiently diagnose and resolve most common issues. This runbook provides a practical guide to debugging, outlines essential `kubectl` workflows, and highlights common failure modes. ### The Core of Troubleshooting: `kubectl` `kubectl` is your primary interface for interacting with your Kubernetes cluster. Mastering its commands is crucial for effective troubleshooting. **1. Getting the Status of Resources:** * **`kubectl get pods -n <namespace>`**: The first step is always to check the status of your Pods. Look for Pods in `Pending`, `CrashLoopBackOff`, `Error`, or `ImagePullBackOff` states. * **`kubectl get deployments -n <namespace>`**: Verify the status of your Deployments. Ensure the desired number of replicas is available and that there are no scaling issues. * **`kubectl get services -n <namespace>`**: Check if your Services are correctly configured and exposing your applications as expected. Look for `ClusterIP`, `NodePort`, or `LoadBalancer` statuses. * **`kubectl get nodes`**: Ensure all nodes in your cluster are `Ready`. **2. Diving Deeper into Pod Issues:** When a Pod isn't running correctly, you need to inspect it further. * **`kubectl describe pod <pod-name> -n <namespace>`**: This is arguably the most important command. It provides detailed information about the Pod, including its status, events, container states, volumes, and associated controllers. Pay close attention to the `Events` section at the bottom for clues. * **`kubectl logs <pod-name> -n <namespace>`**: View the logs from a container within a Pod. If a Pod has multiple containers, specify the container name using `-c <container-name>`. * **`kubectl logs -p <pod-name> -n <namespace>`**: View the logs of a *previous* instance of a container. This is invaluable when a container has crashed and restarted. * **`kubectl exec -it <pod-name> -n <namespace> -- /bin/bash` (or `/bin/sh`)**: Execute commands inside a running container. This allows you to inspect the filesystem, check running processes, and manually test network connectivity from within the Pod. **3. Network Troubleshooting:** Network issues are common in Kubernetes. * **`kubectl get endpoints -n <namespace>`**: Check if your Service has associated Endpoints (i.e., healthy Pod IPs). If there are no endpoints, the Service won't route traffic. * **`kubectl port-forward <pod-name> <local-port>:<pod-port> -n <namespace>`**: Temporarily forward a local port to a port on a Pod. This is useful for direct testing of an application running inside a Pod without exposing it via a Service. * **Check Network Policies**: If you have NetworkPolicies applied, ensure they are not blocking legitimate traffic between Pods or to/from external services. **4. Storage Troubleshooting:** Persistent storage issues can prevent applications from starting or cause data loss. * **`kubectl get pvc -n <namespace>`**: Check the status of your PersistentVolumeClaims (PVCs). Ensure they are `Bound` to a PersistentVolume (PV). * **`kubectl describe pvc <pvc-name> -n <namespace>`**: Similar to `describe pod`, this command provides details about the PVC and any associated events. * **`kubectl get pv`**: Check the status of your PersistentVolumes. ### Common Failure Modes and How to Address Them **1. Pods Stuck in `Pending`:** * **Cause**: Insufficient resources (CPU, memory) on nodes, or Pod scheduling constraints (e.g., node selectors, taints/tolerations, affinity/anti-affinity rules). * **Troubleshooting**: Use `kubectl describe pod` to check events for messages like `FailedScheduling`. Check cluster resource utilization (`kubectl top nodes`). Review Pod's resource requests and limits. **2. `CrashLoopBackOff`:** * **Cause**: The container is starting, crashing, and then restarting repeatedly. This usually indicates an application error within the container. * **Troubleshooting**: Check `kubectl logs <pod-name>` and `kubectl logs -p <pod-name>` for application error messages. Use `kubectl exec` to investigate the container's state. Ensure the application is correctly configured to run in a containerized environment. **3. `ImagePullBackOff` / `ErrImagePull`:** * **Cause**: Kubernetes cannot pull the container image. This could be due to an incorrect image name/tag, a typo in the registry, or authentication issues with a private registry. * **Troubleshooting**: Verify the image name and tag in the Deployment/Pod spec. Check `kubectl describe pod` events for specific error messages. Ensure `imagePullSecrets` are correctly configured if using a private registry. **4. `ImagePullError` (often related to OCI runtime errors):** * **Cause**: The container runtime on the node is unable to pull the image, possibly due to network issues on the node, or problems with the container registry itself. * **Troubleshooting**: Check node network connectivity. Ensure the container runtime (e.g., Docker, containerd) is healthy on the node. Try pulling the image manually on the node if possible. **5. Application Unreachable / Service Not Working:** * **Cause**: Incorrect Service configuration, NetworkPolicy blocking traffic, Pods not running or unhealthy, or application errors. * **Troubleshooting**: Verify `kubectl get endpoints` for the Service. Check `kubectl logs` for the Pods backing the Service. Ensure the Service selector matches the Pod labels. Review NetworkPolicies. **6. Persistent Volume Issues:** * **Cause**: PVC not bound, PV not available, storage provisioner errors, or incorrect access modes. * **Troubleshooting**: Check `kubectl get pvc` and `kubectl get pv`. Review events in `kubectl describe pvc` and `kubectl describe pod` for storage-related errors. ### Proactive Measures * **Implement Health Checks**: Use `livenessProbe` and `readinessProbe` in your Pod definitions to allow Kubernetes to manage container health and readiness automatically. * **Resource Requests and Limits**: Define appropriate CPU and memory requests and limits for your containers to prevent resource starvation and ensure predictable scheduling. * **Logging and Monitoring**: Set up robust logging and monitoring solutions (e.g., Prometheus, Grafana, ELK stack) to gain visibility into your cluster and applications. * **Automated Deployments**: Use CI/CD pipelines to ensure consistent and repeatable deployments, reducing manual errors. By systematically applying these `kubectl` commands and understanding common failure modes, you can become proficient in maintaining a healthy and stable Kubernetes environment.