Skip to content

Understanding the Kubernetes Log Universe: A Guide to Efficient Troubleshooting

Understanding the Kubernetes Log Universe: A Guide to Efficient Troubleshooting

Logs are the lifeblood of any production system, and in a Kubernetes cluster, they are the definitive story of what’s happening inside your nodes, containers, and control plane. Without a clear logging strategy, identifying the root cause of failed deployments, performance degradation, or cluster instability becomes a daunting task, like searching for a needle in a haystack.

However, the distributed nature of Kubernetes means logs are spread across different components and locations. Knowing where to look and which log to use is the key to swift and effective troubleshooting. This knowledge directly translates into faster incident response, as it minimizes the time spent mapping cryptic log entries back to tangible cluster events, significantly reducing the time to resolve issues. This guide breaks down the Kubernetes log structure, explains how to retrieve different types of logs, and walks through a real-world case study to put theory into practice.

The Three Pillars of Kubernetes Logging

Kubernetes logs can be broadly grouped into three categories, each answering a different question: what the application did, where it ran, and why the cluster made certain decisions.

## 1. Application Logs (The ‘What’)

These logs are generated by your own applications running inside containers. They are the most common starting point for debugging.

  • Location: Kubernetes captures the standard output (stdout) and standard error (stderr) streams from every container. On the node itself, the container runtime redirects these streams to files, typically located in directories like /var/log/pods/ and /var/log/containers/.
    • /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/: Contains the log files for each container within a specific pod.
    • /var/log/containers/: This directory contains helpful symlinks to the pod logs, named in a more human-readable format: <pod-name>_<namespace>_<container-name>-<container-id>.log. Log collection agents like Fluentd or Filebeat often monitor this directory.
  • How to Retrieve: The primary tool for accessing these logs is kubectl. Direct SSH access to nodes is rarely necessary.
    • kubectl logs <pod-name>: Fetches logs from a pod. If the pod has multiple containers, you must specify the container name with -c <container-name>.
    • kubectl logs -f <pod-name>: Streams logs in real-time (follow).
    • kubectl logs --previous <pod-name>: Crucial for debugging crashes. This command shows the logs from the previous instance of a restarted container.

## 2. Node-Level Logs (The ‘Where’)

These logs are generated by the node’s operating system and the Kubernetes components running on it, providing context about the environment where your pods are running. You typically need to SSH into a node to view these.

  • Kubelet Logs: The kubelet is the primary Kubernetes agent on each node responsible for managing the pod lifecycle. Its logs are critical for diagnosing pod startup issues, health check failures, and problems with storage volumes.
    • Location: On modern Linux systems using systemd, the logs are best accessed via journalctl -u kubelet. You may also find them in /var/log/kubelet.log.
  • Container Runtime Logs: These logs come from the container runtime (for example, containerd, CRI-O) and detail container-level events like pulling images, starting/stopping containers, and resource-related errors.
    • Location: Use journalctl -u containerd or check files like /var/log/docker.log.
  • System Logs: These are the OS-level logs, essential for finding underlying hardware, kernel, or networking issues that could affect all pods on a node.
    • Location: Found in /var/log/syslog or /var/log/messages. Look for kernel panics, disk pressure warnings, or network interface errors here.

## 3. Control Plane Logs (The ‘Why’)

The control plane is the brain of the Kubernetes cluster. Its logs tell you why the cluster is in its current state.

  • Components:
    • API Server (kube-apiserver): The gateway to the cluster. Logs API requests, authentication issues, and admission controller decisions.
    • Scheduler (kube-scheduler): Decides which node each pod should run on. Its logs are invaluable for troubleshooting why a pod is stuck in a Pending state.
    • Controller Manager (kube-controller-manager): Runs controllers that manage cluster state (for example, ReplicationControllers, Deployments).
    • etcd: The key-value store that holds all cluster state. Its logs are important for diagnosing cluster instability.
  • Location:
    • Self-Managed Clusters: In clusters set up with tools like kubeadm, control plane components run as static pods. You can find their logs on the control-plane nodes in /var/log/, with names like /var/log/kube-apiserver.log.
    • Managed Clusters (GKE, EKS, AKS): Cloud providers manage the control plane for you and typically expose these logs through their own logging services (for example, Google Cloud Logging, AWS CloudWatch).

Case Study: Solving the Dreaded CrashLoopBackOff

Let’s walk through a common troubleshooting scenario. An engineer receives an alert that the api-gateway service is unhealthy.

The Scenario: A critical api-gateway pod is flapping, and its status is CrashLoopBackOff. This means Kubernetes is trying to start the pod, but the container is crashing repeatedly.

Step 1: Initial Triage with kubectl

First, we check the status of the pods in the production namespace.

Bash

$ kubectl get pods -n production

Output:

NAME                           READY   STATUS             RESTARTS   AGE
api-gateway-5d8b9f4c7f-z2x8c     0/1     CrashLoopBackOff   8          20m
redis-cache-6b7f9c8d5c-9w7l4     1/1     Running            0          1h
database-7f6c5b8d4d-plm2q        1/1     Running            0          1h

The output confirms that api-gateway is restarting frequently.

Step 2: Checking the Current Container’s Logs

Our first instinct is to check the logs of the crashing container.

Bash

$ kubectl logs api-gateway-5d8b9f4c7f-z2x8c -n production

Output:

INFO: Starting API Gateway version 1.2.3...
INFO: Configuration loaded successfully.
INFO: Attempting to connect to Redis cache at redis-cache:6379...

This log is not very helpful. It shows the application starting up but cuts off before any error occurs. This is typical for a crashing container, the log buffer might not get flushed before the process dies.

Step 3: Uncovering the Real Story with --previous

The key to solving CrashLoopBackOff is often in the logs of the previous failed container instance. Let’s retrieve those.

Bash

$ kubectl logs --previous api-gateway-5d8b9f4c7f-z2x8c -n production

Output:

INFO: Starting API Gateway version 1.2.3...
INFO: Configuration loaded successfully.
INFO: Attempting to connect to Redis cache at redis-cache:6379...
ERROR: Connection to Redis cache failed after 3 retries.
panic: failed to establish critical cache connection
goroutine 1 [running]:
main.main()
	/app/main.go:42 +0x2c5

Aha! 💡 We have our root cause. The application is panicking because it cannot connect to its dependency, the redis-cache service.

Step 4: Following the Lead and Investigating the Dependency

Now we know the problem lies in the connection between api-gateway and redis-cache. Why can’t it connect?

  1. Is the Redis service configured correctly? Let’s describe the service to check its selector and port.Bash$ kubectl describe svc redis-cache -n production Output:Name: redis-cache Namespace: production Labels: <none> Selector: app=redis Type: ClusterIP IP: 10.100.200.15 Port: cache-port 6379/TCP TargetPort: 6380/TCP Endpoints: <none> Here’s the problem. The service is listening on port 6379, but it’s trying to forward traffic to TargetPort 6380 on the pods. The Endpoints field shows <none>, which means the service selector app=redis is not matching any running pods.
  2. Let’s check the labels on the Redis pod.Bash$ kubectl get pod redis-cache-6b7f9c8d5c-9w7l4 --show-labels -n production Output:NAME READY STATUS RESTARTS AGE LABELS redis-cache-6b7f9c8d5c-9w7l4 1/1 Running 0 1h app=redis-cache We’ve found the misconfiguration. The service is selecting pods with the label app=redis, but the pod itself has the label app=redis-cache.

The Resolution

The fix is to align the service selector with the pod’s label. By editing the redis-cache service YAML to use app=redis-cache as the selector and reapplying it, the service will correctly find the Redis pod, establish an endpoint, and allow the api-gateway to connect, resolving the CrashLoopBackOff loop.

Conclusion

Troubleshooting in Kubernetes is a systematic process of peeling back layers. By understanding the three pillars of logging, application, node, and control plane, you can form a complete picture of any issue. Start with the application logs using kubectl logs, dig deeper into node-level issues with journalctl when necessary, and consult control plane logs for cluster-wide problems.

Mastering this structured approach will transform you from a reactive debugger into a proactive problem-solver, capable of keeping your cluster healthy and resilient. In short: know your logs, and you’ll know your cluster.