Recently I was testing the RabbitMQ cluster operators and creating RabbitMQ clusters to test their HA features. The cluster operators create a namespace called rabbitmq-system. Upon deleting the RabbitMQ stateful set and the cluster operator the rabbitmq-system namespace got stuck in terminating state.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@master operators]# kubectl get ns rabbitmq-system | |
| NAME STATUS AGE | |
| rabbitmq-system Terminating 20d |
The environment is a k3s cluster with one master node and two worker nodes. I checked for any resources using the namespace rabbitmq-system with the following command.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@master operators]# kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n rabbitmq-system | |
| NAME ALLREPLICASREADY RECONCILESUCCESS AGE | |
| rabbitmqcluster.rabbitmq.com/rabbitmq Unknown False 16d |
Then I proceeded to delete the resource
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@master operators]# kubectl delete rabbitmqcluster.rabbitmq.com/rabbitmq -n rabbitmq-system |
However, the resource was not deleted as it was a custom resource. Next I checked for the namespace rabbitmq-system in the YAML output format.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| kind: Namespace | |
| metadata: | |
| annotations: | |
| cattle.io/status: '{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2022-05-24T11:47:53Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2022-05-24T11:47:54Z"}]}' | |
| kubectl.kubernetes.io/last-applied-configuration: | | |
| {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"rabbitmq-operator","app.kubernetes.io/name":"rabbitmq-system","app.kubernetes.io/part-of":"rabbitmq"},"name":"rabbitmq-system"}} | |
| lifecycle.cattle.io/create.namespace-auth: "true" | |
| creationTimestamp: "2022-05-24T11:47:52Z" | |
| finalizers: | |
| - controller.cattle.io/namespace-auth | |
| labels: | |
| app.kubernetes.io/component: rabbitmq-operator | |
| app.kubernetes.io/name: rabbitmq-system | |
| app.kubernetes.io/part-of: rabbitmq | |
| kubernetes.io/metadata.name: rabbitmq-system | |
| name: rabbitmq-system | |
| resourceVersion: "539684" | |
| uid: 5fa5b330-3bc5-415c-ad2f-d55e60ebbf76 | |
| spec: | |
| finalizers: | |
| - kubernetes | |
| status: | |
| conditions: | |
| - lastTransitionTime: "2022-05-24T08:50:31Z" | |
| message: 'Some content in the namespace has finalizers remaining: deletion.finalizers.rabbitmqclusters.rabbitmq.com in 1 resource instances' | |
| reason: SomeFinalizersRemain | |
| status: "True" | |
| type: NamespaceFinalizersRemaining | |
| phase: Terminating |
In the above YAML output we can see that the status of the namespace is in Terminating phase and the reason is NamespaceFinalizersRemaining. In the spec we can see the finalizers is set as kubernetes. This means, kubernetes will wait for a criteria to be met before deleting some resources and when the criteria does not meet, the namespace will be stuck in terminating state.
So we will delete the finalizers. We will export the namespace in JSON format and delete the finalizers.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@master operators]# kubectl get ns rabbitmq-system -o json > rabbitmq.json |
The JSON file after deleting the finalizers:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "apiVersion": "v1", | |
| "kind": "Namespace", | |
| "metadata": { | |
| "annotations": { | |
| "cattle.io/status": "{\"Conditions\":[{\"Type\":\"ResourceQuotaInit\",\"Status\":\"True\",\"Message\":\"\",\"LastUpdateTime\":\"2022-05-04T10:06:28Z\"},{\"Type\":\"InitialRolesPopulated\",\"Status\":\"True\",\"Message\":\"\",\"LastUpdateTime\":\"2022-05-04T10:06:28Z\"}]}", | |
| "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/component\":\"rabbitmq-operator\",\"app.kubernetes.io/name\":\"rabbitmq-system\",\"app.kubernetes.io/part-of\":\"rabbitmq\"},\"name\":\"rabbitmq-system\"}}\n", | |
| "lifecycle.cattle.io/create.namespace-auth": "true" | |
| }, | |
| "creationTimestamp": "2022-05-04T10:06:27Z", | |
| "deletionTimestamp": "2022-05-24T08:50:19Z", | |
| "labels": { | |
| "app.kubernetes.io/component": "rabbitmq-operator", | |
| "app.kubernetes.io/name": "rabbitmq-system", | |
| "app.kubernetes.io/part-of": "rabbitmq", | |
| "kubernetes.io/metadata.name": "rabbitmq-system" | |
| }, | |
| "name": "rabbitmq-system", | |
| "resourceVersion": "516744", | |
| "uid": "dd01816f-f46c-4005-9834-9aa5a313f7e2" | |
| }, | |
| "spec": { | |
| }, | |
| "status": { | |
| "conditions": [ | |
| { | |
| "lastTransitionTime": "2022-05-24T09:16:36Z", | |
| "message": "All resources successfully discovered", | |
| "reason": "ResourcesDiscovered", | |
| "status": "False", | |
| "type": "NamespaceDeletionDiscoveryFailure" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-05-24T08:50:31Z", | |
| "message": "All legacy kube types successfully parsed", | |
| "reason": "ParsedGroupVersions", | |
| "status": "False", | |
| "type": "NamespaceDeletionGroupVersionParsingFailure" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-05-24T08:50:31Z", | |
| "message": "All content successfully deleted, may be waiting on finalization", | |
| "reason": "ContentDeleted", | |
| "status": "False", | |
| "type": "NamespaceDeletionContentFailure" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-05-24T08:50:31Z", | |
| "message": "Some resources are remaining: rabbitmqclusters.rabbitmq.com has 1 resource instances", | |
| "reason": "SomeResourcesRemain", | |
| "status": "True", | |
| "type": "NamespaceContentRemaining" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-05-24T08:50:31Z", | |
| "message": "Some content in the namespace has finalizers remaining: deletion.finalizers.rabbitmqclusters.rabbitmq.com in 1 resource instances", | |
| "reason": "SomeFinalizersRemain", | |
| "status": "True", | |
| "type": "NamespaceFinalizersRemaining" | |
| } | |
| ], | |
| "phase": "Terminating" | |
| } | |
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@master operators]# kubectl proxy | |
| Starting to serve on 127.0.0.1:8001 |
Then in a new terminal we run the below command to delete the namespace. Please note that the format should be http://127.0.0.1:8001/api/v1/namespaces/rabbitmq-system/finalize where rabbitmq-system is the namespace we are going to delete.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@master operators]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @rabbitmq.json http://127.0.0.1:8001/api/v1/namespaces/rabbitmq-system/finalize |
The namespace has been deleted as shown below.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@master operators]# kubectl get ns rabbitmq-system | |
| Error from server (NotFound): namespaces "rabbitmq-system" not found |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| kind: Namespace | |
| metadata: | |
| annotations: | |
| cattle.io/status: '{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2022-06-28T09:06:08Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2022-06-28T09:06:14Z"}]}' | |
| field.cattle.io/projectId: c-74n2g:p-t4zl9 | |
| kubectl.kubernetes.io/last-applied-configuration: | | |
| {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"cattle-system"}} | |
| lifecycle.cattle.io/create.namespace-auth: "true" | |
| creationTimestamp: "2022-06-28T09:06:03Z" | |
| finalizers: | |
| - controller.cattle.io/namespace-auth | |
| labels: | |
| field.cattle.io/projectId: p-t4zl9 | |
| managedFields: | |
| - apiVersion: v1 | |
| fieldsType: FieldsV1 | |
| fieldsV1: | |
| f:metadata: | |
| f:annotations: | |
| .: {} | |
| f:kubectl.kubernetes.io/last-applied-configuration: {} | |
| f:status: | |
| f:phase: {} | |
| manager: kubectl-client-side-apply | |
| operation: Update | |
| time: "2022-06-28T09:06:03Z" | |
| - apiVersion: v1 | |
| fieldsType: FieldsV1 | |
| fieldsV1: | |
| f:metadata: | |
| f:annotations: | |
| f:cattle.io/status: {} | |
| f:field.cattle.io/projectId: {} | |
| f:lifecycle.cattle.io/create.namespace-auth: {} | |
| f:finalizers: | |
| .: {} | |
| v:"controller.cattle.io/namespace-auth": {} | |
| manager: rancher | |
| operation: Update | |
| time: "2022-06-28T09:06:09Z" | |
| - apiVersion: v1 | |
| fieldsType: FieldsV1 | |
| fieldsV1: | |
| f:metadata: | |
| f:labels: | |
| .: {} | |
| f:field.cattle.io/projectId: {} | |
| manager: Go-http-client | |
| operation: Update | |
| time: "2022-06-28T09:06:13Z" | |
| name: cattle-system | |
| resourceVersion: "7824" | |
| uid: 17c2a802-6cd4-4bc9-b163-2cd7d92dde77 | |
| spec: | |
| finalizers: | |
| - kubernetes | |
| status: | |
| phase: Terminating |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "apiVersion": "v1", | |
| "kind": "Namespace", | |
| "metadata": { | |
| "annotations": { | |
| "cattle.io/status": "{\"Conditions\":[{\"Type\":\"ResourceQuotaInit\",\"Status\":\"True\",\"Message\":\"\",\"LastUpdateTime\":\"2021-06-23T08:41:34Z\"} ,{\"Type\":\"InitialRolesPopulated\",\"Status\":\"True\",\"Message\":\"\",\"LastUpdateTime\":\"2021-06-23T08:41:41Z\"}]}", | |
| "field.cattle.io/projectId": "c-6nhvd:p-s5ft9", | |
| "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"cattle-s ystem\"}}\n", | |
| "lifecycle.cattle.io/create.namespace-auth": "true" | |
| }, | |
| "creationTimestamp": "2021-06-23T08:40:57Z", | |
| "deletionGracePeriodSeconds": 0, | |
| "deletionTimestamp": "2022-06-26T19:56:17Z", | |
| "finalizers": [ | |
| ], | |
| "labels": { | |
| "field.cattle.io/projectId": "p-s5ft9" | |
| }, | |
| "managedFields": [ | |
| { | |
| "apiVersion": "v1", | |
| "fieldsType": "FieldsV1", | |
| "fieldsV1": { | |
| "f:metadata": { | |
| "f:annotations": { | |
| ".": {}, | |
| "f:kubectl.kubernetes.io/last-applied-configuration": {} | |
| } | |
| } | |
| }, | |
| "manager": "kubectl-client-side-apply", | |
| "operation": "Update", | |
| "time": "2021-06-23T08:40:57Z" | |
| }, | |
| { | |
| "apiVersion": "v1", | |
| "fieldsType": "FieldsV1", | |
| "fieldsV1": { | |
| "f:metadata": { | |
| "f:annotations": { | |
| "f:cattle.io/status": {}, | |
| "f:field.cattle.io/projectId": {}, | |
| "f:lifecycle.cattle.io/create.namespace-auth": {} | |
| }, | |
| "f:finalizers": { | |
| } | |
| } | |
| }, | |
| "manager": "rancher", | |
| "operation": "Update", | |
| "time": "2021-06-23T08:41:34Z" | |
| }, | |
| { | |
| "apiVersion": "v1", | |
| "fieldsType": "FieldsV1", | |
| "fieldsV1": { | |
| "f:metadata": { | |
| "f:labels": { | |
| ".": {}, | |
| "f:field.cattle.io/projectId": {} | |
| } | |
| } | |
| }, | |
| "manager": "agent", | |
| "operation": "Update", | |
| "time": "2021-06-23T08:41:44Z" | |
| }, | |
| { | |
| "apiVersion": "v1", | |
| "fieldsType": "FieldsV1", | |
| "fieldsV1": { | |
| "f:metadata": { | |
| "f:labels": { | |
| "f:field.cattle.io/projectId": {} | |
| } | |
| } | |
| }, | |
| "manager": "Go-http-client", | |
| "operation": "Update", | |
| "time": "2022-06-26T09:05:52Z" | |
| }, | |
| { | |
| "apiVersion": "v1", | |
| "fieldsType": "FieldsV1", | |
| "fieldsV1": { | |
| "f:status": { | |
| "f:phase": {} | |
| } | |
| }, | |
| "manager": "k3s", | |
| "operation": "Update", | |
| "time": "2022-06-26T19:56:33Z" | |
| } | |
| ], | |
| "name": "cattle-system", | |
| "resourceVersion": "125212837", | |
| "uid": "b7b84729-de84-4d4e-89f7-2a06f4a056d2" | |
| }, | |
| "spec": {}, | |
| "status": { | |
| "conditions": [ | |
| { | |
| "lastTransitionTime": "2022-06-26T19:56:22Z", | |
| "message": "All resources successfully discovered", | |
| "reason": "ResourcesDiscovered", | |
| "status": "False", | |
| "type": "NamespaceDeletionDiscoveryFailure" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-06-26T19:56:22Z", | |
| "message": "All legacy kube types successfully parsed", | |
| "reason": "ParsedGroupVersions", | |
| "status": "False", | |
| "type": "NamespaceDeletionGroupVersionParsingFailure" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-06-26T19:56:22Z", | |
| "message": "All content successfully deleted, may be waiting on finalization", | |
| "reason": "ContentDeleted", | |
| "status": "False", | |
| "type": "NamespaceDeletionContentFailure" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-06-26T19:56:33Z", | |
| "message": "All content successfully removed", | |
| "reason": "ContentRemoved", | |
| "status": "False", | |
| "type": "NamespaceContentRemaining" | |
| }, | |
| { | |
| "lastTransitionTime": "2022-06-26T19:56:22Z", | |
| "message": "All content-preserving finalizers finished", | |
| "reason": "ContentHasNoFinalizers", | |
| "status": "False", | |
| "type": "NamespaceFinalizersRemaining" | |
| } | |
| ], | |
| "phase": "Terminating" | |
| } | |
| } |
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.