KubePodCPUUsage

Resolving High CPU Usage in a Kubernetes Pod

Overview

Resolve high CPU usage in a Kubernetes Pod to ensure stable application performance and efficient resource utilization.

Initial Response

  • Alert received indicating high CPU usage of a pod.

  • Acknowledge the alert and assign yourself as the incident owner.

  • Notify the team about the ongoing incident using the primary communication channel.

  • Update the incident status on the incident tracking system.

Detailed Steps

1) Monitor and Verify CPU Usage

Use kubectl to list all Pods in the target namespace

kubectl get pods -n <namespace>

Identify the Pod(s) with high CPU usage by looking at the %CPU column. Note the name of the affected Pod.

2) Identify the Cause

2.1 Access the shell of the affected Pod for further investigation

kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

2.2 Use utilities like top or htop to identify the process(es) consuming CPU resources:

top

2.3 Note down the process(es) and their corresponding PID(s).

3) Diagnose the Issue

3.1 Exit the Pod shell

exit

3.2 Identify the container name within the Pod causing high CPU usage

kubectl describe pod <pod-name> -n <namespace>

3.3 Check the Pod's logs for any errors, excessive logging, or patterns that might indicate the root cause of the high CPU usage

kubectl logs <pod-name> -c <container-name> -n <namespace>

Solutions Details

1) Scale the Deployment

If the high CPU usage is caused by legitimate load, consider scaling the Deployment to distribute the load across multiple Pods.

kubectl scale deployment <deployment-name> --replicas=<new-replica-count> -n <namespace>

2) Optimize Application Code

Work with the development team to optimize the application's code and queries that might be causing the CPU spikes.

3) Resource Limit and Requests

Ensure that the Pod's resource requests and limits are appropriately set to avoid resource contention. Adjust the values in the Pod's YAML definition.

Escalation:

If the issue persists or is severe, escalate to a senior SRE engineer for additional support and guidance.

Further Information

PreviousKubePodNotHealthyNextNodeHighCPULoad

Last updated