Kubernetes: Externalized Configuration with Spring Boot

Why Externalize Configurations in Spring Boot ?

Most applications have different properties and resources that need to be configured before or after deployment. It is also important that the application is not environment specific. The application must be able to run on any environment be it Dev/Test/QA/Staging or Production. So, how are we able to achieve this? The answer is externalized configurations. These externalized configurations include the environment the application is running, the credentials for database and other applications in microservices and various other configurations that can be easily changed pre or post deployment. So we do not have to build the application every time we have to make minor changes.

So, It is preferred in microservices architecture to externalize configurations. 

Java Spring Boot is a popular, open source tool that helps make web application and microservices with Spring Framework easier with these core capabilities:

1. Autoconfiguration
2. An opinionated approach to configuration
3. The ability to create standalone applications

In this blog, we are going to build a sample application with jenkins and docker and then run it with external configurations in kubernetes.

How to Externalize Spring Boot Configurations ?

Spring boot application loads application properties or YAML configuration files from the locations mentioned below:

1. /config directory of the current directory
2. The current directory
3. The classpath /config package
4. The classpath root (Resource folder)

It is also important to know the priority for each location listed above. 

Location 1, /config directory of the current directory has the highest priority. If the configurations are found in this location other locations will be discarded.

Location 2, the current directory has the second priority. If the configurations are not found in the first location then the configurations will be loaded from this location and locations 3 and 4 will be discarded.

Location 3. the classpath /config package has the third priority. If the configurations are not found in the locations 1 and 2, they will be loaded from this location. Location 4 will be discarded.

Location  4, the classpath root which is the resource folder will be prioritized last. If the configurations are not found in all the other three locations then they will be loaded from this location. If they are not found even in this location, then the application will fail to start.

So for the demo application we will remove the YAML config files from the project repository and resource directory. Then we will load the configs from Kubernetes configmap in the location /app/config and the application will be running from the /app directory. We will be running a jar application located in /app in Docker alpine image.

Building the Application with Jenkins and Dockerfile

We have used maven for building the jar file from the java source code and Docker to build container image from a Dockerfile.

The Jenkinsfile:

The Dockerfile:

I have already build a alpine base image (alpine_base:1.0) with latest packages and java 17 installed, that is why I have used "FROM alpine_base:1.0" in the Dockerfile. After the build is successful we will run the application in Kubernetes.

Running the Application in Kubernetes

First we will create the deployment and configmap in Kubernetes and then run the application.

The deployment:

The configmap:

Now we run the application by scaling the deployment replicas to 1.

We can find the config files in /app/config location as shown below.

The application processes running in the pod container are shown below.

Finally, we can verify the application is running by checking the logs.

Kubernetes: Add and Manage External Clusters

Background

We've all wanted to make our work easier and handling all the Kubernetes clusters from a single instance will definitely help. In this post, we look at how we can add external Kubernetes cluster and manage it from a single Kubernetes cluster. The original idea was to deploy applications to multiple Kubernetes clusters from a single ArgoCD instance. For that, we had to add multiple clusters to a single cluster where the ArgoCD was hosted.

For this post I have setup two clusters and we will add the second cluster to the first cluster's kubeconfig and manage it from there. We will authenticate to the external cluster using a service account token. However, there are other ways of authenticating such as certificates and username/password credentials.

Configuring the External Cluster

These configurations will be done in the external cluster.

First, we will create a new serviceaccount for the external cluster.

Second, we will create clusterrolebinding.

The service account has been created as shown below:

The clusterrolebinding has been created as shown below:

Now, we check the cluster admin token cluster-admin-token-asldf created white creating the serviceaccount.

Adding the External Cluster

First, we add the external cluster. Here I have used the IP and Port for the external cluster,  named it test-cluster and skipped TLS verification.

Second, we set the cluster context.

Third, we set the credentials for the external cluster authentication.

The final contents of kubeconfig are as shown below from the command: kubectl config view

Adding the External Cluster in ArgoCD

We can only add the external cluster in ArgoCD from commandline. So, we are going to run the below commands in the argocd-server pod. 

First, we login as admin.

Second, we add the cluster.

Third, we can list the clusters available for deployment.

Finally, we can see the added external cluster in the ArgoCD UI.

Kubernetes: Namespace Stuck in Terminating State

 Recently I was testing the RabbitMQ cluster operators and creating RabbitMQ clusters to test their HA features. The cluster operators create a namespace called rabbitmq-system. Upon deleting the RabbitMQ stateful set and the cluster operator the rabbitmq-system namespace got stuck in terminating state.

The environment is a k3s cluster with one master node and two worker nodes. I checked for any resources using the namespace rabbitmq-system with the following command.

Then I proceeded to delete the resource

However, the resource was not deleted as it was a custom resource. Next I checked for the namespace rabbitmq-system in the YAML output format.

In the above YAML output we can see that the status of the namespace is in Terminating phase and the reason is NamespaceFinalizersRemaining. In the spec we can see the finalizers is set as kubernetes. This means, kubernetes will wait for a criteria to be met before deleting some resources and when the criteria does not meet, the namespace will be stuck in terminating state.

So we will delete the finalizers. We will export the namespace in JSON format and delete the finalizers.

The JSON file after deleting the finalizers:

After deleting the finalizers section, we start the kubectl proxy.

Then in a new terminal we run the below command to delete the namespace. Please note that the format should be http://127.0.0.1:8001/api/v1/namespaces/rabbitmq-system/finalize where rabbitmq-system is the namespace we are going to delete.

The namespace has been deleted as shown below.

Sometimes the finalizers are seen in the metadata as well. So we need to delete the content of the finalizers in the metadata and spec. Below is shown finalizers in metadata.

After deleting the finalizers in the JSON file as shown below, we follow the previous process similarly. Notice the empty finalizers.

Kubernetes: Update Service Account with Registry Secrets

Background

 Recently, I faced some issues in my Kubernetes test environment where I had deleted the secret that contained my private registry (harbor) credentials. So, the deployments could not pull the images from harbor. Even after adding the secret containing harbor credentials, the deployments could not pull the images. 

After some troubleshooting I figured out the deployments did not include the registry credential secret in the YAML manifests. So instead of editing each deployment in the namespace, I edited the service account and added the secret so all the deployments in that namespace would use the registry secret by default.

The Secret

The YAML file:

Updating Service Account

Since we only have default service account we will be updating the default service account in the namespace "testbed".  

There are several ways to update the service account: 

1. Patch the service account
2. Replace the service account
3. Edit the service account

Patch Service Account

Replace Service Account

First we export the default service account in YAML format.

After exporting the service account in YAML format we edit the YAML file and add the following

The final default service account after editing the serviceaccount.yaml file and adding imagePullSecrets.

Now we replace the serviceaccount with the serviceaccount.yaml file.

Edit Service Account

We directly edit the service account and add the imagePullSecrets and save.

The final service account:

Kubernetes: Set Local Time Zone on Deployments and Pods

  In Kubernetes Pods, the time zone is set to UTC by default and if you require logs and data to have precise local time then we need to make changes to the pods or deployments. In this post we will be changing the date and time to our local zone Asia/Kathmandu. 

Set Time Zone on Deployments

First we are going to deploy an Nginx image and make changes to the deployment.

[root@master ~]#  kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

The deployment has been created and we can verify as following.

[root@master ~]# kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
nginx                1/1     1            1           32s

The pod has also been created.

[root@master ~]# kubectl get pods -n default
NAME                                  READY   STATUS    RESTARTS            AGE
nginx-6799fc88d8-lnpqt                1/1     Running   0                   81s

[root@master ~]# kubectl get pod nginx-6799fc88d8-lnpqt -n default -o wide
NAME                     READY   STATUS    RESTARTS   AGE    IP           NODE       NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-lnpqt   1/1     Running   0          110s   10.42.2.37   worker01              

We need to change the local timezone as below i.e. Asia/Kathmandu. The /etc/localtime information is of the master node.

[root@master ~]# file /usr/share/zoneinfo/Asia/Kathmandu
/usr/share/zoneinfo/Asia/Kathmandu: timezone data, version 2, 3 gmt time flags, 3 std time flags, no leap seconds, 2 transition times, 3 abbreviation chars


[root@master ~]# ll /etc/localtime
lrwxrwxrwx. 1 root root 36 Dec 28 19:24 /etc/localtime -> ../usr/share/zoneinfo/Asia/Kathmandu

We edit the deployment and add environment variable TZ with value "Asia/Kathmandu".

[root@master ~]# kubectl edit deployment nginx

spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - env:
        - name: TZ
          value: Asia/Kathmandu
        image: nginx
        imagePullPolicy: IfNotPresent
        name: nginx
        resources: {}

[root@master ~]# kubectl exec  -it nginx-8766wj383u-grxpq -- date
Wed Apr 27 14:45:43 +0545 2022

We can verify above that the time zone has been changed to +05:45 GMT after adding the TZ environment variable. 

 Set Time Zone on Pods

Below we have created a YAML file for POD deployment with Nginx image and also set environment

variable named TZ with value "Asia/Kathmandu".

[root@master ~]# cat timezone-Nginx.yaml
apiVersion: v1
kind: Pod
metadata:
  name: timezone-demo
  labels:
    purpose:  demonstrate-localpodtime
spec:
  containers:
  - name: nginx
    image: nginx
    env:
    - name: TZ
      value:  Asia/Kathmandu

Now we apply the YAML file and deploy the POD.

[root@master ~]# kubectl apply -f timezone-Nginx.yaml
pod/timezone-demo created

We verify the POD deployment as below where timezone-demo is the POD name. Then we verify the timezone and date/time.

[root@master ~]# kubectl get pod/timezone-demo
NAME            READY   STATUS    RESTARTS   AGE
timezone-demo   1/1     Running   0          10s      

[root@master ~]# kubectl exec  -it timezone-demo  -- date
Wed Apr 27 15:05:43 +0545 2022

Thus, we have successfully changed the timezone to the timezone as desired. 

We can also change the timezone using configmap and volumes.

kubectl create configmap tz --from-file=/usr/share/zoneinfo/Asia/Kathmandu

  
[root@master ~]# cat timezone-Nginx.yaml
apiVersion: v1
kind: Pod
metadata:
  name: timezone-demo
  labels:
    purpose:  demonstrate-localpodtime
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
      - name: tz
        mountPath: /etc/localtime
    volumes:
    - name: tz
      configMap:
        name: tz