CreateContainerError

This issue happens when Kubernetes tries to create a container (a precursor to the ‘run’ phase of the container) but fails.

Overview

This generally implies some kind of issue with the container runtime, but can also indicate a problem starting up the container, such as the command not existing.

This symptom can also lead to CrashLoopBackOff errors, so it may be worth looking at that runbook if this one does not resolve your issue.

Initial Response

  • Alert received indicating create container error.

  • Acknowledge the alert and assign yourself as the incident owner.

  • Notify the team about the ongoing incident using the primary communication channel.

  • Update the incident status on the incident tracking system.

Check RunBook Match

When running a kubectl get pods command, you will see a line like this in the output for your pod:

NAME                     READY     STATUS                RESTARTS   AGE
nginx-7ef9efa7cd-qasd2   0/1       CreateContainerError  2          1m

Detailed Steps

1) Gather information

To determine the root cause here, first gather relevant information that you may need to refer back to later:

kubectl describe pod -n [NAMESPACE] -p [POD_NAME] > /tmp/runbooks_describe_pod.txt

2) Examine pod Events output

Look at the Events section of your /tmp/runbooks_describe_pod.txt file.

2.1) If you see no command specified

Like this:

Warning  Failed     116s    kubelet, dali      Error: failed to generate container "be3707fcf873ecc052ff73d3ffe7ed3a51eec0756b84839f908f7c9ab730ae74" spec: no command specified

then the error is caused by both the image and pod specification not specifying a command to run. See the CrashLoopBackOff runbook solution B.

2.2) If you see starting container process caused

Then the message following this explains what happened on the node when the container process was run

For example, if the error looks like this:

Warning  Failed 8s (x3 over 22s)  kubelet, nginx-7ef9efa7cd-qasd2  Error: container create failed: container_linux.go:296: starting container process caused "exec: \"mycommand\": executable file not found in $PATH"

Then starting command might not be available on the image. Correct the container start command - or the image’s contents - accordingly. See the CrashLoopBackOff runbook solution C.

2.3) If you see container name [...] already in use by container

For example:

The container name "/k8s_kube-apiserver_kube-apiserver-master_kube-system_ce0f74ad5fcbf28c940c111df265f4c8_24" is already in use by container 14935b714aee924aa42295fa5d252c760264d24ee63ea74e67092ccc3fb2b530. You have to remove (or rename) that container to be able to reuse that name.

Then it is likely there is a problem with the container runtime on that host not cleaning up old containers. If you have admin access, check the Kubelet logs on the node the container was assigned to.

2.4) If you see is waiting to start

In the error message, like this:

container "nginx" in pod "nginx-1234567890-12345" is waiting to start: CreateContainerConfigError

then the issue may be to do with

  • Volume/secrets mounting.

If there is an Init container, then look at the logs for that, as it may be responsible for provisioning the volume.

Check any secrets and/or configmaps in the pod are available to your pods in your namespace.

Check Resolution

If the pod starts up with status RUNNING according to the output, then the issue has been resolved.

If there is a different status, then it may be that this particular issue is resolved, but a new issue has been revealed, and the runbook needs to be re-followed.

Escalation:

If the issue persists or is severe, escalate to senior engineering and development teams for additional support and guidance.

Further Information

Init containers

Kubelet logs

PreviousCrashLoopBackOffNextKubeStatefulsetDown

Last updated