Top 7 Kubernetes Troubleshooting Use Cases: Hands-On Incident Runbook

A node transitions into NotReady when the Kubelet stops updating its heartbeat Lease object with the Kubernetes control plane or when the node's Ready condition evaluates to False. Once the node-monitor-grace-period (default 40s) expires, the Node Controller marks the node unready; after node-eviction-timeout, pods on that node begin evicting.

Forensic Diagnostic Flow

Execute triage commands to isolate the failure domain:

# Step 1: Immediately cordon to protect cluster capacity
kubectl cordon <node-name>

# Step 2: Check conditions and lease heartbeat
kubectl describe node <node-name> | grep -A 10 Conditions:
kubectl get lease -n kube-node-lease <node-name> -o yaml

# Step 3: SSH / SSM into worker node and inspect Kubelet & Containerd
systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager

# Step 4: Validate CRI socket responsiveness
crictl info
crictl ps

# Step 5: Check client TLS certificate expiration
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

Permanent Solution & Cluster Return

# Restart containerd and kubelet if deadlocked
systemctl restart containerd && systemctl restart kubelet

# Wait 30 seconds and verify status flips to Ready
kubectl get node <node-name>

# Uncordon the node to resume scheduling
kubectl uncordon <node-name>

When available disk drops below configured eviction thresholds (default: imagefs.available < 15% or nodefs.available < 10%), Kubelet flags DiskPressure=True, halts pod scheduling, and immediately starts evicting BestEffort and Burstable pods.

Filesystem Partition Forensics (nodefs vs imagefs)

Never run rm -rf on containerd storage layers. Identify the saturated partition first:

# On the worker node: check mount usage
df -hT / /var/lib/containerd /var/log

# If imagefs is full: safely prune unreferenced images
crictl rmi --prune

# If nodefs is full: identify runaway container logs in /var/log/pods
du -sh /var/log/pods/* | sort -hr | head -n 10

# Safely truncate runaway log file without breaking open file descriptors
truncate -s 0 /var/log/pods/<namespace_pod_uid>/<container>/*.log

# Check for bloated emptyDir volumes
du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* | sort -hr | head -n 10

Permanent Kubelet Garbage Collection Configuration

In /var/lib/kubelet/config.yaml, enforce proactive automated garbage collection:

imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 65
imageMinimumGCAge: 2m
containerLogMaxSize: 50Mi
containerLogMaxFiles: 5
evictionHard:
  nodefs.available: 10%
  imagefs.available: 15%
evictionSoft:
  nodefs.available: 15%
  imagefs.available: 20%
evictionSoftGracePeriod:
  nodefs.available: 2m
  imagefs.available: 2m

In CreateContainerError, the container image was pulled successfully, but when containerd or CRI-O calls the Linux kernel to create the container sandbox (attaching mounts, setting up namespaces, configuring cgroups, or enforcing AppArmor/SELinux profiles), the system call fails. Because application code never ran, kubectl logs returns nothing.

Extracting the Container Runtime Error

# Step 1: Capture the exact CRI RPC failure message
kubectl describe pod <pod-name> | grep -A 8 'State: Waiting'

# Step 2: Audit stuck PersistentVolume attachments (EBS Multi-Attach trap)
kubectl get volumeattachment | grep <pv-name>

# Step 3: Check host kernel audit logs for AppArmor / SELinux rejections
dmesg -T | grep -iE 'apparmor|selinux|audit|denied'

# Step 4: Check host inotify watches limit
cat /proc/sys/fs/inotify/max_user_watches

Permanent Solution & Fixes

CreateContainerConfigError happens earlier than CreateContainerError. Before Kubelet even touches the container runtime, it must assemble environment variables and projections from API server objects. If a referenced ConfigMap or Secret is missing, or a key does not exist inside it, Kubelet stops immediately with this error.

Triage Workflow & Diagnosis

# Describe pod to reveal the exact missing resource name and key
kubectl describe pod <pod-name>

# Typical output:
# Error: configmap "api-config" not found
# Error: couldn't find key DB_PASSWORD in Secret default/db-credentials

# Check existing ConfigMaps and Secrets in the workload namespace
kubectl get configmap,secret -n <namespace>

# Check External Secrets Operator (ESO) synchronization health
kubectl get externalsecret -n <namespace>
kubectl describe externalsecret <name> -n <namespace>

The optional: true Resilience Pattern

Prevent pod startup crashes for non-essential configuration variables using optional: true:

env:
  - name: FEATURE_TOGGLES
    valueFrom:
      configMapKeyRef:
        name: app-toggles
        key: feature_flags
        optional: true   # Pod starts gracefully even if ConfigMap is absent!
envFrom:
  - configMapRef:
      name: non-critical-tuning
      optional: true

ErrImagePull is the initial error returned when Kubelet fails to fetch a container image layer or manifest. Kubernetes then places the pod into ImagePullBackOff, enforcing an exponential backoff retry loop (5s, 10s, 20s... capped at 5 minutes).

Decoding the 4 Primary Root Causes

# Step 1: Decode exact RPC error message from describe events
kubectl describe pod <pod-name> | grep Events -A 10

# Error 1: 404 Not Found / manifest unknown -> Typo in image repository or tag
# Error 2: 401 Unauthorized / 403 Forbidden -> Missing/expired registry credentials
# Error 3: 429 Too Many Requests -> Docker Hub rate limit reached (100 pulls/6hr)
# Error 4: i/o timeout -> Private VPC subnet missing NAT Gateway or ECR Endpoints
# Error 5: no match for platform in manifest -> ARM64 vs AMD64 architecture mismatch

# Step 2: Test image pull directly on the worker node using crictl
crictl pull <image-name>:<tag>

Private Subnet AWS VPC & S3 Gateway Endpoint Fix

In air-gapped or private AWS VPC subnets, AWS ECR requires both interface endpoints AND an Amazon S3 Gateway Endpoint to download layer blobs:

# Verify VPC Endpoints in AWS CLI
aws ec2 describe-vpc-endpoints --filters "Name=vpc-id,Values=<vpc-id>"   --query "VpcEndpoints[*].[ServiceName,VpcEndpointType,State]" --output table

# Required Endpoints for ECR:
# 1. com.amazonaws.<region>.ecr.api (Interface)
# 2. com.amazonaws.<region>.ecr.dkr (Interface)
# 3. com.amazonaws.<region>.s3      (Gateway - CRITICAL: layer blobs live in S3!)

Unlike CrashLoopBackOff (where the application started and crashed), in RunContainerError the operating system kernel failed to start the container task (execve(2) system call failed) or a postStart lifecycle hook failed with a non-zero exit code.

Diagnostic Triage Checklist

# Step 1: Inspect event logs for kernel execve failure
kubectl describe pod <pod-name> | grep Events -A 10

# Symptom A: "exec: "/entrypoint.sh": stat /entrypoint.sh: no such file or directory"
# Root Cause: Windows CRLF line endings (
) in shebang line (#!/bin/sh
)
# Fix: Run dos2unix entrypoint.sh before building Dockerfile

# Symptom B: "exec: "/bin/bash": stat /bin/bash: no such file or directory"
# Root Cause: Distroless or scratch base image has no shell!
# Fix: Execute binary directly: command: ["/app/server"]

# Symptom C: "permission denied"
# Root Cause: Missing executable bit or runAsNonRoot: true with root-owned binary
# Fix: RUN chmod +x /entrypoint.sh in Dockerfile

# Symptom D: "FailedPostStartHook"
# Root Cause: A lifecycle.postStart exec command exited with non-zero code.
# Kubernetes contract: If postStart fails, the container is killed instantly!

Exit Code 137 is 128 + 9 (SIGKILL). It indicates that the process was unconditionally terminated by the Linux kernel Out-Of-Memory killer. A container is OOMKilled when its memory working set crosses the Linux cgroup limit (memory.max in cgroups v2 or memory.limit_in_bytes in cgroups v1), even if the host node has 64GB of free RAM.

Prometheus Working Set vs Process RSS

Kubernetes does NOT evaluate Resident Set Size (RSS) alone—it tracks container_memory_working_set_bytes, which includes active page cache:

working_set = memory.current - inactive_file

# PromQL Query to detect containers nearing OOM limit
(container_memory_working_set_bytes{container!=""} /
 kube_pod_container_resource_limits{resource="memory"}) * 100 > 85

# Differentiate Container OOM from Node OOM
kubectl describe pod <pod-name>   # Look for Last State: Terminated, Exit Code 137
kubectl describe node <node-name>  # Look for MemoryPressure: True/False

JVM / Java Container Memory Blindness Solution

Older JVMs allocate heap based on host physical RAM instead of container cgroup limits. Configure modern container-aware JVM flags:

# Recommended production JVM flags in Kubernetes
JAVA_OPTS="-XX:+UseContainerSupport            -XX:MaxRAMPercentage=75.0            -XX:InitialRAMPercentage=50.0            -XX:+ExitOnOutOfMemoryError"