Hey, today a quick post on a bug that sometimes happens in kubernetes while removing pods. If someone is unlucky, when removing the kubernetes, they can “get stuck” in the Terminating state and hang like that for a few days.
Kubectl delete pods --all -n NAMESPACE_NAME
Kubectl delete pods POD_NAME -n NAMESPACE_NAME
Kubectl delete deployments DEPLOYMENT_NAME -n NAMESPACE_NAME
Deleting PODS in the ‘Terminating’ State
Manual removal of pod from namespace will not help, even if we specify that we want to delete a specific pod and give its exact name. Removing the ‘deployment’ will also do nothing. POD on kubernetes will be stubbornly stuck in the state of ‘Terminating’.
If we want to fix this, we have to use some force. The command given below always helps me, it removes it almost immediately. Of course, if you have a POD in a dedicated namespace, add its name to the command.
kubectl delete pod POD_NAME --grace-period=0 --force
kubectl delete pod POD_NAME --grace-period=0 --force -n NAMESPACE_NAME
There is one more solution, a bit more drastic, that I didn’t have to use. If forcibly removing the pod doesn’t work, you can restart the entire ‘worker’.
A little bit of automation
A bit of automation from bash if someone is bored of removing blocked pods manually:
namespace="enter_the_name"
delpods=$(kubectl get pods -n ${namespace} | grep -i 'Terminating' | awk '{print $1 }')
for i in ${delpods[@]}; do
kubectl delete pod $i --force=true --wait=false --grace-period=0 -n ${namespace}
done
- At the beginning you give the name of the namespace from which you want to remove blocked pods.
- In the next part, the names of all pods in the terminating state are saved in the variable. Thanks to the ‘grep‘ command, I only print those in the terminating state. Thanks to the ‘awk‘ command, I only save the names of pods.
- Finally, using a simple loop ‘for‘, I delete pods whose names are stored in the variable ‘delpods’.
You can find more information about Pods and Kubernetes in the k8s documentation.
You can find more articles about Kubernetes in the Kubernetes category.
I added a little bit of automation, a simple script that can be run from eg cron and it will automatically kill all pods that are in the “Terminating” state.
Thanks for this article, helped me greatly, after struggling with awk not returning an array I noticed that you can also do a oneliner using xargs:
kubectl get pods -n | grep -i ‘Terminating’ | awk ‘{print $1 }’ | xargs -I{} kubectl delete pod {} –force –grace-period=0 -n
Looks nice in line. Thanks for sharing
Hello thank you for the article,
I beleive the grep -i ‘Terminated’ should be replaced with grep -i ‘Terminating’.
Regards
yes that’s right, instead of copy paste. I typed it wrong. thank you
Sometimes, when the “-force –grace-period” is not doing the job, you can use the below command instead:
kubectl get pods | grep -i ‘Terminating’ | awk ‘{print $1}’ | xargs -I kubectl patch pod –type=json -p ‘[{“op”: “remove”, “path”: “/metadata/finalizers” }]’
Or
kubectl get pods | grep -i ‘Terminating’ | awk ‘{print $1}’ | xargs -I kubectl patch pod -p ‘{“metadata”:{“finalizers”:null}}’
That was very helpful to remove “perminating” pods from a node .
Good luck!
Comments are closed.