Kubernetes: Namespace Stuck in Terminating State

 Recently I was testing the RabbitMQ cluster operators and creating RabbitMQ clusters to test their HA features. The cluster operators create a namespace called rabbitmq-system. Upon deleting the RabbitMQ stateful set and the cluster operator the rabbitmq-system namespace got stuck in terminating state.

The environment is a k3s cluster with one master node and two worker nodes. I checked for any resources using the namespace rabbitmq-system with the following command.

Then I proceeded to delete the resource

However, the resource was not deleted as it was a custom resource. Next I checked for the namespace rabbitmq-system in the YAML output format.

In the above YAML output we can see that the status of the namespace is in Terminating phase and the reason is NamespaceFinalizersRemaining. In the spec we can see the finalizers is set as kubernetes. This means, kubernetes will wait for a criteria to be met before deleting some resources and when the criteria does not meet, the namespace will be stuck in terminating state.

So we will delete the finalizers. We will export the namespace in JSON format and delete the finalizers.

The JSON file after deleting the finalizers:

After deleting the finalizers section, we start the kubectl proxy.

Then in a new terminal we run the below command to delete the namespace. Please note that the format should be http://127.0.0.1:8001/api/v1/namespaces/rabbitmq-system/finalize where rabbitmq-system is the namespace we are going to delete.

The namespace has been deleted as shown below.

Sometimes the finalizers are seen in the metadata as well. So we need to delete the content of the finalizers in the metadata and spec. Below is shown finalizers in metadata.

After deleting the finalizers in the JSON file as shown below, we follow the previous process similarly. Notice the empty finalizers.