Skip to content

K8S – pods stuck on ‘Terminating’

Kubernetes - Pod Stuck In Terminating State K8S

Hey, today a quick post on a bug that sometimes happens in kubernetes while removing pods. If someone is unlucky, when removing the kubernetes, they can “get stuck” in the Terminating state and hang like that for a few days.

k8s stuck kubernetes terminating
Kubectl delete pods --all -n NAMESPACE_NAME
Kubectl delete pods POD_NAME -n NAMESPACE_NAME
Kubectl delete deployments DEPLOYMENT_NAME -n NAMESPACE_NAME

Deleting PODS in the ‘Terminating’ State

Manual removal of pod from namespace will not help, even if we specify that we want to delete a specific pod and give its exact name. Removing the ‘deployment’ will also do nothing. POD on kubernetes will be stubbornly stuck in the state of ‘Terminating’.

If we want to fix this, we have to use some force. The command given below always helps me, it removes it almost immediately. Of course, if you have a POD in a dedicated namespace, add its name to the command.

kubectl delete pod POD_NAME --grace-period=0 --force
kubectl delete pod POD_NAME --grace-period=0 --force -n NAMESPACE_NAME

There is one more solution, a bit more drastic, that I didn’t have to use. If forcibly removing the pod doesn’t work, you can restart the entire ‘worker’.

A little bit of automation

A bit of automation from bash if someone is bored of removing blocked pods manually:

namespace="enter_the_name"

delpods=$(kubectl get pods -n ${namespace} | grep -i 'Terminating' | awk '{print $1 }')

for i in ${delpods[@]}; do
  kubectl delete pod $i --force=true --wait=false --grace-period=0  -n ${namespace}
done
  • At the beginning you give the name of the namespace from which you want to remove blocked pods.
  • In the next part, the names of all pods in the terminating state are saved in the variable. Thanks to the ‘grep‘ command, I only print those in the terminating state. Thanks to the ‘awk‘ command, I only save the names of pods.
  • Finally, using a simple loop ‘for‘, I delete pods whose names are stored in the variable ‘delpods’.

You can find more information about Pods and Kubernetes in the k8s documentation.

You can find more articles about Kubernetes in the Kubernetes category.

Simple way to copy data in kubernetes containers
Integration ACR with AKS
Tags:

6 thoughts on “K8S – pods stuck on ‘Terminating’”

  2. Jerome

    Thanks for this article, helped me greatly, after struggling with awk not returning an array I noticed that you can also do a oneliner using xargs:

    kubectl get pods -n | grep -i ‘Terminating’ | awk ‘{print $1 }’ | xargs -I{} kubectl delete pod {} –force –grace-period=0 -n

  4. Abderrahman Ziani

    Hello thank you for the article,
    I beleive the grep -i ‘Terminated’ should be replaced with grep -i ‘Terminating’.

    Regards

  6. Abderrahman Ziani

    Sometimes, when the “-force –grace-period” is not doing the job, you can use the below command instead:

    kubectl get pods | grep -i ‘Terminating’ | awk ‘{print $1}’ | xargs -I kubectl patch pod –type=json -p ‘[{“op”: “remove”, “path”: “/metadata/finalizers” }]’

    Or
    kubectl get pods | grep -i ‘Terminating’ | awk ‘{print $1}’ | xargs -I kubectl patch pod -p ‘{“metadata”:{“finalizers”:null}}’

    That was very helpful to remove “perminating” pods from a node .

    Good luck!

Comments are closed.