Kubernetes Troubleshooting Runbook: Debugging, kubectl Workflows, and Common Failure Modes

Start with the object that is actually unhappy

kubectl get pods -n <namespace> first. Pending, CrashLoopBackOff, ImagePullBackOff, and Error are different problems that feel the same from a status page. Then kubectl describe pod <name> -n <namespace> and read the Events at the bottom. The events are the cluster telling you what it already tried.

kubectl get nodes if nothing is scheduling. A node that is NotReady makes every other theory a hobby. kubectl get deploy,svc,endpoints -n <namespace> if the process is up and the user still cannot reach it. Endpoints empty means the Service selector and the pod labels have stopped speaking.

Pending, CrashLoop, ImagePull

Pending is usually scheduling. Describe the pod. FailedScheduling means CPU, memory, a taint, a node selector, or a PVC that is not bound. kubectl top nodes if you have metrics. A request of 4 CPU on a 2 CPU node will sit there forever, politely.

CrashLoopBackOff means the process started and died. kubectl logs <pod> -n <namespace> and kubectl logs -p for the previous container. The -p is the one people forget, and it is the one that has the stack trace. If the container has no shell, you still have the logs. Do not exec into a crash loop and call it debugging.

ImagePullBackOff is the name or the credentials. Check the image string for a typo before you rebuild the registry. Private registries need an imagePullSecret that the ServiceAccount can see. ErrImagePull that mentions the runtime rather than the name is often the node: DNS, disk, or a registry that is down. Describe the pod. It already has the sentence.

The network is a selector problem until it is not

kubectl get endpoints <service> -n <namespace>. No addresses means no ready pods matched. Fix labels or the readiness probe before you touch a service mesh. kubectl port-forward is how you ask the pod itself, skipping the Service. If port-forward works and the Service does not, you have your answer.

NetworkPolicy is the next place. A default-deny that forgot to allow the app port, or forgot DNS, looks exactly like a crashed dependency. Describe the policies in the namespace before you redeploy the app for the third time.

Ingress: confirm a controller is running, the class matches, and the hostname matches DNS. An Ingress with no controller is a YAML diary.

Disk, and the volume that is not bound

kubectl get pvc -n <namespace>. Pending means the StorageClass, the size, or the access mode does not match anything the cluster can provision. Describe the PVC. The event is usually blunt.

A pod that mounts a PVC and stays ContainerCreating is often the volume, not the image. Multi-attach errors mean two pods want a ReadWriteOnce disk. That is a replica count problem. emptyDir filling the node looks like a random eviction. Check what the process writes, and whether anyone put a database on it.

Commands worth memorising

kubectl describe for events. kubectl logs and kubectl logs -p for the process. kubectl get endpoints for the Service. kubectl auth can-i when you are about to blame RBAC. kubectl rollout history deploy/<name> and kubectl rollout undo when the last deploy is the change. Undo is a Deployment feature. A bare Pod does not have it, which is why the other lesson told you not to ship one.

Keep a scratch namespace for kubectl run and kubectl debug if the cluster allows it. Do not debug by installing curl into the production image and calling it a hotfix.

Write the answer into the alert

The third time you run the same describe, it belongs in the runbook linked from the alert. Symptoms, the command, the reversible next step. 'Restart the deployment' is allowed when you have said what you expect to change. It is not a personality.

If you fixed it by editing a live object, put that edit back into Git before you close the incident. Otherwise the agent, or the next deploy, will undo your heroics. That is the system working. It will not feel like it at the time.