Time synchronization in Kubernetes

Configure time synchronization in Kubernetes.

Configure all containers in a Kubernetes cluster to regularly and reliably synchronize with a Network Time Protocol (NTP) server to ensure consistency among all containers in the cluster. When times are out of sync, this can lead to unexpected behavior. This then results in poor performance, among other things.

It can also lead to errors that are not necessarily caused by the timing problem. For example, if you want to make a system update or update Kubernetes. Just there are then also problems with certificates.

Problems or confusion can also arise when reading log files if a different time is suddenly displayed in the log.

Problems can also arise with the so-called cronjobs during execution.

Install NTP on each machine in your cluster:

Here for CentOS / RHEL:

sudo yum install ntp ntpdate

sudo systemctl start ntpd

sudo systemctl enable ntpd

sudo systemctl status ntpd

sudo ntpdate -u -s 0.centos.pool.ntp.org

sudo systemctl restart ntpd

If you then check with “ timedatectl “, it should look like this:

How to change the time zone in Kubernetes

First, we should find out what time zone our pods use.

Containers do not inherit time zones from hosts and only have access to the clock through the kernel. The default time zone for most images is UTC, but it is not guaranteed and can vary from container to container.

Imagine we have the following pod description:

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"

In the above mentioned file the command "sleep" is executed 1000000 times within the OS Busybox.

Now save this file as timecheck.yaml. After that please create this pod and we will check which timezone is used by the pod.

$ kubectl apply -f timecheck.yaml

You should now get a response that the pod was successfully created.

Now we use this command to check which time zone the Pod is using:

$ kubectl exec busybox-sleep date

The Pod uses the UTC time zone.

So, now we change this with additional information (red box) in the yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
      volumeMounts:
    - name: tz-config
      mountPath: /etc/localtime
  volumes:
    - name: tz-config
      hostPath:
        path: /usr/share/zoneinfo/Europe/Berlin

Save this file as timecheck-new.yaml and then start it with

$ kubectl apply -f timecheck-new.yaml

Now we check again with the command which time zone the Pod uses:

$ kubectl exec busybox-sleep -- date

As we can see now, we have the desired CEST time zone in our pod.

If you want to change the time zone for all pods, you need to include this extension with the volumes and volumeMounts in their pod description. This ensures that each pod will then use the desired time zone in the future.

Automated setting of time zones:

The k8tz tool is a Kubernetes Admission Controller and a "CLI" tool to inject time zones into pods and CronJobs.

The tool and documentation can be found here.