FAQs

  18 minute read  

FAQ - Kubeopsctl

Critical security vulnerabilities in NGINX

What is the issue?

Critical security vulnerabilities in NGINX (Rift and PoolSlip) affect the version used in KubeOps COMPLIANCE. These vulnerabilities may allow remote code execution or denial-of-service attacks.

Who is affected?

KubeOps COMPLIANCE users running affected NGINX versions, including:

KubeOps 2.1.0 KubeOps 2.0.3–2.0.6 KubeOps 1.7.6–1.7.8

What action is required?

Install the provided hotfix as soon as possible.

Script Instuctions for V 1.7.6 – 1.7.8

  • Step 1: Define the KOSI Package Set the package name for the Nginx hotfix:

    kosinginx="kubeops/ingress-nginx: 2.0.6"

    Create the filename that will be used when the package is pulled:

    kosinginxfile="ingress-nginx-$(echo $kosinginx | awk -F: '{print $NF}').tgz"

    This creates a file name based on the package version

  • Step 2: Identify the latest kubeopsctl configuration file.

    Locate the most recent file matching $KUBEOPSROOT/kubeopsctl_*.yaml.

    kubeopsctllatest=$(ls $KUBEOPSROOT/kubeopsctl_*.yaml | sort | tail -n 1)
echo "The latest kubeopsctl.yaml is: $kubeopsctllatest"

  • Step 3: Prepare the KOSI Directory

    Temporarily enable housekeeping in the KOSI configuration, run kosi version, and then disable housekeeping again:

    sed -i "s/  housekeeping: false/  housekeeping: true/g" $KUBEOPSROOT/kosi/config.yaml
kosi version
sed -i "s/  housekeeping: true/  housekeeping: false/g" $KUBEOPSROOT/kosi/config.yaml

  • Step 4: Prepare the patch workspace and configuration

    During this part of the process, you verify the current installation, review the patch package details, and prepare the tools-values.yaml file that is required for the update. Depending on your environment, you can either reuse an existing values file, generate one from the current kubeopsctl configuration, or provide a custom file.

mkdir -p ~/nginx-1.15.6-patch

helm list -A | grep ingress-nginx
echo 
echo "=================================================================================================================================="
echo "WARNING!!! The following kosi package will be used for the patch to nginx-1.15.6: $kosinginx"
echo "Since this is a kosi package for kubeops 2.0, the tools-values.yaml must be created"
echo "according to the format of kubeops 2.0"
echo "=================================================================================================================================="
echo 
sleep 20

if [ $# -eq 0 ]; then
    echo "no file provided"
        if [ -f ~/nginx-1.15.6-patch/tools-values.yaml ]; then
        echo "A file ~/nginx-1.15.6-patch/tools-values.yaml already exists. Should it be used?"
        echo "---------------------------------------------------------------------"
        cat ~/nginx-1.15.6-patch/tools-values.yaml
        echo "---------------------------------------------------------------------"
        echo -ne "[y/n] "
        read answer
        if [[ "$answer" == "y" ]]
        then
            create="false"
        else
            create="true"
        fi
    else
        create="true"
    fi

    if [[ "$create" == "true" ]]
    then
            echo "Attempting to create ~/nginx-1.15.6-patch/tools-values.yaml."
# --------------------------------------------------------------------------------------------------------
# determine entry after ingressValues:
nachiv=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | sed '1d' | sed -n '/^[^ ]/ {p; q;}')
ivblock=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | grep -B1000 "^$nachiv" | sed '$d')
voradvanced=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | grep -B1000 "^$nachiv" | sed '$d' | sed "s/^ingressValues:/- name: ingress-nginx\n  enabled: true\n  values:\n    standard:/g" | sed 's/  namespace:/      namespace:/g' | sed 's/  externalIPs:/      externalIPs:/g' | sed 's/  advancedParameters:/    advanced:/g' | grep -B100 "^    advanced:" | sed '$d')
nachadvanced=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | grep -B1000 "^$nachiv" | sed '$d' | grep -A1000 "^  advancedParameters:" | sed '1d' | sed 's/^    /      /')

cat << EOF > ~/nginx-1.15.6-patch/tools-values.yaml
packages:
$voradvanced
    advanced:
$nachadvanced
EOF
# --------------------------------------------------------------------------------------------------------
        echo "Should this generated tools-values.yaml be used?"
                echo "---------------------------------------------------------------------"
                cat ~/nginx-1.15.6-patch/tools-values.yaml
                echo "---------------------------------------------------------------------"
                echo -ne "[y/n] "
                read answer
                if [[ "$answer" != "y" ]]
                then
                        echo "Please adjust the values in ~/nginx-1.15.6-patch/tools-values.yaml or provide an appropriate yaml file to the script."
            exit 1
                fi

    else
        echo "using ~/nginx-1.15.6-patch/tools-values.yaml"
    fi
else
    if [ -f "$1" ]; then
            echo "File [$1] exists. Copying $1 to ~/nginx-1.15.6-patch/tools-values.yaml"
        cp $1 ~/nginx-1.15.6-patch/tools-values.yaml
    else
        echo "File [$1] not found."
        exit 1
    fi
fi    

cd ~/nginx-1.15.6-patch
```Step 5: Log In to KOSI

Log in to KOSI with a valid user account. The following loop prompts for a username and retries until the login is successful:

```bash
while true; do
    read -rp "$(printf "Username for kosi login: ")" KOSI_USER

    echo "Please enter the password for user '$KOSI_USER'"

    if kosi login -u "$KOSI_USER"; then
        echo "Login successful"
        break
    else
        echo "Login failed. Please try again."
        sleep 1
    fi
done
  • Step 5: Determine the Harbor Configuration

Retrieve the Harbor namespace, endpoint, port, and password from the KubeOps configuration.

harborns=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep -m 1 "namespace:" | awk '{print $2}')
harborep=$(kubectl get cm -n"$harborns" harbor-core -oyaml | grep "EXT_ENDPOINT" | awk '{print $2}')
harborepohnehttp=$(echo "$harborep" | awk -F"//" '{print $2}')
harborport=$(echo "$harborepohnehttp" | awk -F: '{print $NF}')
harborpw=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep "harborpass" | awk '{print $NF}' | sed 's/"//g')

Display the Harbor URL and port:

echo "Harbor URL : $harborepohnehttp"
echo "Harbor PORT: $harborport"

Step 6: Log In to Harbor with Podman

Log in to Harbor as the admin user by using the password retrieved in the previous step:

if podman login "$harborepohnehttp" -u admin -p "$harborpw" --tls-verify=false; then
    echo "Podman login successful"
else
    echo "Podman login to $harborepohnehttp as admin with password $harborpw was not successful"
    exit 1
fi

  • Step 6: Pull the Nginx Hotfix Package

    Pull the ingress-nginx package from the KOSI hub and save it as the local package archive:

    kosi pull --hub kubeops "$kosinginx" -o "$kosinginxfile" -r "$harborepohnehttp/kubeops" -t "localhost:$harborport/kubeops"

  • Step 7: Determine the Deployment Name for ingress-nginx

    Retrieve the deployment name for the existing ingress-nginx package:

    dname=$(kosi list | grep ingress-nginx | awk '{print $2}')

  • Step 8: Update ingress-nginx

    Run the KOSI update command by using the deployment name, the pulled package file, and the KubeOps values file:

    kosi update --dname="$dname" -p "$kosinginxfile" -f ./tools-values.yaml

    This applies the hotfix package to the existing ingress-nginx deployment.

  • Step 9: Remove Local Podman Images

    After the update completes, remove the local Podman images related to ingress-nginx and defaultbackend:

    for i in $(podman images | grep "ingress-nginx\|defaultbackend" | awk '{print $3}'); do
    podman rmi -f $i
done

    This cleans up the local images from the administration host.

 

Script Details for V 1.7.6 – 1.7.8 
#!/bin/bash

# kosi package
kosinginx="kubeops/ingress-nginx:2.0.6"

# filename for kosi pull
kosinginxfile="ingress-nginx-$(echo $kosinginx | awk -F: '{print $NF}').tgz"

kubeopsctllatest=$(ls $KUBEOPSROOT/kubeopsctl_*.yaml | sort | tail -n 1)
echo "The latest kubeopsctl.yaml is: $kubeopsctllatest"

# clean up kosi directory
sed -i "s/  housekeeping: false/  housekeeping: true/g" $KUBEOPSROOT/kosi/config.yaml
kosi version
sed -i "s/  housekeeping: true/  housekeeping: false/g" $KUBEOPSROOT/kosi/config.yaml

# create patch directory
mkdir -p ~/nginx-1.15.6-patch

helm list -A | grep ingress-nginx
echo 
echo "=================================================================================================================================="
echo "WARNING!!! The following kosi package will be used for the patch to nginx-1.15.6: $kosinginx"
echo "Since this is a kosi package for kubeops 2.0, the tools-values.yaml must be created"
echo "according to the format of kubeops 2.0"
echo "=================================================================================================================================="
echo 
sleep 20

if [ $# -eq 0 ]; then
    echo "no file provided"
        if [ -f ~/nginx-1.15.6-patch/tools-values.yaml ]; then
        echo "A file ~/nginx-1.15.6-patch/tools-values.yaml already exists. Should it be used?"
        echo "---------------------------------------------------------------------"
        cat ~/nginx-1.15.6-patch/tools-values.yaml
        echo "---------------------------------------------------------------------"
        echo -ne "[y/n] "
        read answer
        if [[ "$answer" == "y" ]]
        then
            create="false"
        else
            create="true"
        fi
    else
        create="true"
    fi

    if [[ "$create" == "true" ]]
    then
            echo "Attempting to create ~/nginx-1.15.6-patch/tools-values.yaml."
# --------------------------------------------------------------------------------------------------------
# determine entry after ingressValues:
nachiv=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | sed '1d' | sed -n '/^[^ ]/ {p; q;}')
ivblock=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | grep -B1000 "^$nachiv" | sed '$d')
voradvanced=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | grep -B1000 "^$nachiv" | sed '$d' | sed "s/^ingressValues:/- name: ingress-nginx\n  enabled: true\n  values:\n    standard:/g" | sed 's/  namespace:/      namespace:/g' | sed 's/  externalIPs:/      externalIPs:/g' | sed 's/  advancedParameters:/    advanced:/g' | grep -B100 "^    advanced:" | sed '$d')
nachadvanced=$(cat $kubeopsctllatest | grep -A1000 "^ingressValues:" | grep -B1000 "^$nachiv" | sed '$d' | grep -A1000 "^  advancedParameters:" | sed '1d' | sed 's/^    /      /')

cat << EOF > ~/nginx-1.15.6-patch/tools-values.yaml
packages:
$voradvanced
    advanced:
$nachadvanced
EOF
# --------------------------------------------------------------------------------------------------------
        echo "Should this generated tools-values.yaml be used?"
                echo "---------------------------------------------------------------------"
                cat ~/nginx-1.15.6-patch/tools-values.yaml
                echo "---------------------------------------------------------------------"
                echo -ne "[y/n] "
                read answer
                if [[ "$answer" != "y" ]]
                then
                        echo "Please adjust the values in ~/nginx-1.15.6-patch/tools-values.yaml or provide an appropriate yaml file to the script."
            exit 1
                fi

    else
        echo "using ~/nginx-1.15.6-patch/tools-values.yaml"
    fi
else
    if [ -f "$1" ]; then
            echo "File [$1] exists. Copying $1 to ~/nginx-1.15.6-patch/tools-values.yaml"
        cp $1 ~/nginx-1.15.6-patch/tools-values.yaml
    else
        echo "File [$1] not found."
        exit 1
    fi
fi    

cd ~/nginx-1.15.6-patch

# kosi login

while true; do
    read -rp "$(printf "Username for kosi login: ")" KOSI_USER

    echo "Please enter the password for user '$KOSI_USER'"

    if kosi login -u "$KOSI_USER"; then
        echo "Login successful"
        break
    else
        echo "Login failed. Please try again."
        sleep 1
    fi
done

# determine Harbor values
nextblock=$(cat $kubeopsctllatest | grep -A50 "^harborValues:" | sed '1d' | sed -n '/^[^ ]/ {p; q;}')
harborblock=$(cat $kubeopsctllatest | grep -A1000 "^harborValues:" | sed '1d' | grep -B1000 "$nextblock" | sed '$d')
harborns=$(echo "$harborblock" | grep "namespace:" | awk '{print $2}')
harborep=$(kubectl get cm -n"$harborns" harbor-core -oyaml | grep "EXT_ENDPOINT" | awk '{print $2}')
harborepohnehttp=$(echo "$harborep" | awk -F"//" '{print $2}')
harborport=$(echo "$harborepohnehttp" | awk -F: '{print $NF}')
harborpw=$(echo "$harborblock" | grep "harborpass" | awk '{print $NF}' | sed 's/"//g')
echo "harbor URL : $harborepohnehttp"
echo "harbor PORT: $harborport"

# Podman login
if podman login "$harborepohnehttp" -u admin -p "$harborpw" --tls-verify=false; then
    echo "podman login successful"
else
    echo "podman login to $harborepohnehttp as admin with $harborpw was not successful"
    exit 1
fi

# kosi pull 
kosi pull --hub kubeops "$kosinginx" -o "$kosinginxfile" -r "$harborepohnehttp/kubeops" -t "localhost:$harborport/kubeops"

# determine dname for ingress-nginx 
dname=$(kosi list | grep ingress-nginx | awk '{print $2}')

# kosi update
kosi update --dname="$dname" -p "$kosinginxfile" -f ./tools-values.yaml

# clean up admin Podman images
for i in $(podman images | grep "ingress-nginx\|defaultbackend" | awk '{print $3}') ; do  podman rmi -f $i  ; done

helm list -A | grep ingress-nginx

 

Known Issues

ImagepullBackoffs in Cluster

If you have imagepullbackoffs in your cluster, p.e. for prometheus, you can just use the kubeopsctl change registry command again. e.g.

kubeopsctl change registry -r <your master ip>:30002/library -t localhost:30002/library -f kubeopsctl.yaml

FAQ - KubeOps KOSI

Error Messages

There is an error message regarding Remote-Certificate

  • Error: http://hub.kubernative.net/dispatcher?apiversion=3&vlientversion=2.X.0 : 0
  • X means per version
  • CentOS 7 cannot update the version by itself (ca-certificates-2021.2.50-72.el7_9.noarch).
    • Fix: yum update ca-certificates -y or yum update
  • Manual download and install of ca-certificates RPM:
    • Download: curl http://mirror.centos.org/centos/7/updates/x86_64/Packages/ca-certificates-2021.2.50-72.el7_9.noarch.rpm -o ca-certificates-2021.2.50-72.el7_9.noarch.rpm
    • Install: yum install ca-certificates-2021.2.50-72.el7_9.noarch.rpm -y

KOSI Usage

Can I use KOSI with sudo?

  • At the moment, KOSI has no sudo support.
  • Docker and Helm, which are required, need sudo permissions.

I get an error message when I try to search an empty Hub?

  • Known bug, will be fixed in a later release.
  • Need at least one package in the Hub before you can search.

Package Configuration

In my package.yaml, can I use uppercase characters as a name?

  • Currently, only lowercase characters are allowed.
  • This will be fixed in a later release.

I have an error message that says “Username or password contain non-Latin characters”?

  • Known bug, may occur with incorrect username or password.
  • Please ensure both are correct.

In my template.yaml, can I just write a value without an associated key?

  • No, a YAML file requires a key-value structure.

Do I have to use the template plugin in my KOSI package?

  • No, you don’t have to use the template plugin if you don’t want to.

I have an error message that says “reference not set to an instance of an object”?

  • Error from our tool for reading YAML files.
  • Indicates an attempt to read a value from a non-existent key in a YAML file.

I try to template but the value of a key stays empty.

  • Check the correct path of your values.
  • If your key contains “-”, the template plugin may not recognize it.
  • Removing “-” will solve the issue.

FAQ - KubeOps LIMA

Error Messages

LIMA Cluster not ready

  • You have to apply the calico.yaml in the $LIMAROOT folder:
kubectl apply -f $LIMAROOT/calico.yaml

read header failed: Broken pipe

for lima version >= 0.9.0

  • Lima stops in line

ansible Playbook : COMPLETE : Ansible playbooks complete.

  • Search for
$LIMAROOT/dockerLogs/dockerLogs_latest.txt

in the path Broken pipe. From the line with Broken pipe check if the following lines exist:

debug3: mux_client_read_packet: read header failed: Broken pipe

debug2: Received exit status from master 1

Shared connection to vli50707 closed.

<vli50707> ESTABLISH SSH CONNECTION FOR USER: demouser

<vli50707> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)

(ControlPersist=60s)

If this is the case, line /etc/ansible/ansible.cfg

in the currently running lima container in file ssh_args =-C -o ControlMaster=auto -o ControlPersist=60s must be commented out or removed.

Example:

docker container ls

CONTAINER ID IMAGE COMMAND

CREATED STATUS PORTS NAMES

99cabe7133e5 registry.preprod.kubernative.net/kubeops/lima:v0.8.0 "/bin/bash" 6 days

ago Up 6 days lima-v0.8.0

docker exec -it 99cabe7133e5 bash

vi /etc/ansible/ansible.cfg

Change the line ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s to #ssh_args = -C-o ControlMaster=auto -o ControlPersist=60s or delete the line.

I want to delete the cluster master node and rejoin the cluster. When trying to rejoin the node a problem occurs and rejoining fails. What can be done?

To delete the cluster master, we need to set the cluster master to a different master machine first.

  1. On the admin machine: change the IP-Address from the current to new cluster master in:

    1. $LIMAROOT/<name_of_cluster>/clusterStorage.yaml
    2. ~/.kube/config

  2. Delete the node

  3. Delete the images to prevent interference: ctr -n k8s.io i rm $(ctr -n k8s.io i ls -q)

  4. Change IP on new cluster master in

/etc/kubernetes/admin.conf

  1. Change IPs in config maps:

    1. kubectl edit cm kubeadm-config -n kube-system
    2. kubectl edit cm kube-proxy -n kube-system
    3. kubectl edit cm kubeadm-config -n kube-system
    4. kubectl edit cm cluster-info -n kube-public

  2. Restart kubelet

  3. Rejoin the node

Using LIMA on RHEL8 fails to download metadata for repo “rhel-8-for-x86_64-baseos-rpms”. What should I do?

This is a common problem which happens now and then, but the real source of error is difficult to identify. Nevertheless, the workaround is quick and easy: clean up the current repo data, refresh the subscription-manager and update the whole operating system. This can be done with the following commands:

dnf clean all

rm -frv /var/cache/dnf

subscription-manager refresh

dnf update -y

How does LIMA handle SELinux?

SELinux will be temporarily deactivated during the execution of LIMA tasks. After the execution is finished, SELinux is automatically reactivated. This indicates you are not required to manually enable SELinux every time while working with LIMA.

My pods are stuck: CONFIG-UPDATE 0/1 CONTAINERCREATING

  1. They are responsible for updating the loadbalancer, you can update them manualy and delete the pod

  2. You can try redeploying the deamonset to the kube-system namespace

My master can not join, it fails when creating /ROOT/.KUBE

try the following commands on the master

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Some nodes are missing the loadbalancer

  1. Check if the Loadbalancer staticPod file can be found in the manifest folder of the node.

  2. If it isn’t there please copy it from another node.

Some nodes didn’t upgrade. What to do now?

  1. Retry to upgrade your cluster.

  2. If LIMA thinks you are already on the target version edit the stored data of your cluster at $LIMAROOT/myClusterName/clusterStorage.yaml.

    Set the Key kubernetesVersion to the lowest kubernetes Version present on a Node in your cluster.

Could not detect a supported package manager from the followings list: [‘PORTAGE’, ‘RPM’, ‘PKG’, ‘APT’], or the required PYTHON library is not installed. Check warnings for details.

  1. Check if you got a package manager.

  2. You have to install python3 with yum install python and then create a symlink from python to python3 with update-alternatives --config python.

Aborting, target uses SELINUX but PYTHON bindings (LIBSELINUX-PYTHON) aren’t installed!

You have to install libselinux-python on your cluster machine so you can install a firewall via LIMA.

FAQ - KubeOps PIA

The httpd service is terminating too long. How can I force the shut down?

  1. Use following command to force shut down httpd service:
kubectl delete deployment pia-httpd –grace-period=0 –force
  1. Most deployments got a networking service like our httpd does.

Delete the networking service with the command:

kubectl delete svc pia-httpd-service –grace-period=0 –force

I get the error that some nodes are not ‘Ready’. How do I fix the problem?

  1. Use kubectl get nodes command to find out first which node is not ready.

  2. To identify the problem, get access to the shell of the non-ready node . Use systemctl status kubelet to get status information about state of kubelet.

  3. The most common cause of this error is that the kubelet has the problem of not automatically identify the node. In this case, the kubelet must be restarted manually on the non-ready machine. This is done with systemctl enable kubelet and systemctl start kubelet. maybe you need to restart containerd: systemctl stop containerd and systemctl restart containerd

  4. If the issue persists, reason behind the error can be evaluated by your cluster administrators.

I checked my clusterStorage.yaml after the clustercrartion and there is only a entry for master1

This error occurs sporadically and will be fixed in a later version. The error has no effect.

FAQ KubeOps PLATFORM

Support of S3 storage configuration doesn’t work

At the moment, the kosi-package rook-ceph:1.1.2 (utilized in kubeOps 1.1.3) is employing a Ceph version with a known bug that prevents the proper setup and utilization of object storage via the S3 API. If you require the functionality provided by this storage class, we suggest considering the use of kubeOps 1.0.7. This particular version does not encounter the aforementioned issue and provides comprehensive support for S3 storage solutions.

Change encoding to UTF-8

Please make sure that your uservalues.yaml is using UTF-8 encoding.

If you get issues with encoding, you can change your file to UTF-8 with the following command:

iconv -f UTF-8 -t ISO-8859-1 uservalues.yaml > uservalues.yaml

How to update Calico Multus?

  1. Get podSubnet located in clusterStorage.yaml ($LIMAROOT/<clustername>/clusterStorage.yaml)

  2. Create a values.yaml with key=>podSubnet an value=>

    Example:

    podSubnet: 192.168.0.0/17

  3. Get the deployment name of the current calicomultus installation with the kosi list- command

Example:

| Deployment | Package | PublicHub | Hub |

|-------------|--------------------------------------|--------------|----------|

| 39e6da | local/calicomultus:0.0.1 |        | local |
  1. Update the deployment with
kosi update lima/calicomultus:0.0.2 --dname <yourdeploymentname> --hub=public -f values.yaml

–dname: important parameter mandatory for the update command.

-f values.yaml: important that the right podSubnet is being used.

Known issue:

error: resource mapping not found for name: calico-kube-controllers namespace:co.yaml: no matches for kind PodDisruptionBudget in version policy/v1beta1

ensure CRDs are installed first

Create Cluster-Package with firewalld:

If you want to create a cluster with firewalld and the kubeops/clustercreate:1.0. - package you have to manually pull the firewalld - maintenance - package for your OS first, after executing the kubeops/setup:1.0.1 - package.

Opensearch pods do not start:

If the following message appears in the Opensearch pod logs, the vm.max_map_count:

ERROR: [1] bootstrap checks failed

[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

On all control-plane and worker nodes the line vm.max_map_count=262144 must be added to the file /etc/sysctl.conf.

After that the following command must be executed in the console on all control-plane and worker nodes: sysctl -p

Finally, the Opensearch pods must be restarted.

Does the KubeOps Platform have a vendor lock-in?

No, the KubeOps Platform does not have a vendor lock-in. It is built entirely on open standards and Kubernetes technologies, ensuring you retain full control over your infrastructure at all times.

If you decide to discontinue using the KubeOps Platform, you can:

  • Export data and configurations: All data and settings are available in standardized formats.
  • Migrate workloads: Your applications can run on any Kubernetes environment without requiring modifications.
  • Replace modules: Features like monitoring, security, or lifecycle management can be gradually replaced with alternative tools.

Your infrastructure will remain fully operational throughout the transition. Additionally, we provide comprehensive documentation and optional support to ensure a smooth migration process.

FAQ - KubeOps KUBEOPSCTL

Known issue:

The namespace for packages must remain consistent. For example, if you deploy a package in the “monitoring” namespace, all Kosi updates should also be applied within the same namespace.

HA capability only after 12h, for earlier HA capability manually move the file /etc/kubernetes/manifest/haproxy.yaml out of the folder and back in again

After upgrading a node or zone it is possible that the lima container is still running. Please confirm with podman ps -a if a lima container is running. Remove the lima container with podman rm -f <container id>. After that you can start another upgrade of node or zone.

Sometimes the rook-ceph PDBs are blocking the kubernetes upgrade if you have 3 workers, so you have to delete the rook-ceph PDBs so that the nodes can be drained in the kubernetes upgrade process. the PDBs are created dynamically, so you have to the PDBs could be created after some time.

if the calico or the multus images have a imagepullbackoff, you need toe execute kosi pull --hub public lima/calicomultus:0.0.3 -o calico.tgz -r masternode:5000 -t localhost:5001 for all masternodes.

even if you have the updateRegistry parameter in your yaml file set to true, the images will not be rewritten. you can use lima update -r (clustername from the yaml file).

The rook-ceph dashboard inaccessable with kubeopsctl v1.6.2

An additional worker or master is not added to the existing cluster. In kubeopsctl 1.5.0 an additional worker or master is not added to the existing cluster. We faced that issue with kubeopsctl 1.5.0. After the cluster creation an additional master or worker node should be joined. The kubeopsctl logs are showing that the additional node couldn’t be found. In $KUBEOPSROOT/lima/dockerLogs/dockerLogs_latest.txt at the bottom of the file we found the Error Variable useInsecureRegistry is not defined. After checking $KUBEOPSROOT/lima/test/clusterStorage.yaml (test is the name of our cluster, in your case its the clustername you gave in the kubeopsctl.yaml file) we found out that there is the entry useInsecureRegistry: without value. After we changed it to useInsecureRegistry: false and tried to add the additional node it worked.

ImagepullBackoffs in Cluster If you have imagepullbackoffs in your cluster, p.e. for prometheus, you can just use the kubeopsctl change registry command again. e.g.

kubeopsctl change registry -r <your master ip>:30002/library -t localhost:30002/library -f kubeopsctl.yaml

Update the kubeopsctl.yaml from 1.6.X to 1.7.X you need to change change the ipAdress parameter to ipAddress in your kubeopsctl.yaml file and all generated yaml files in $KUBEOPSROOT

list elements not templateable in advanced parameters list elements are not templateable on rhel 8 machines. so if you want to use lists, then you need to upügrade to rhel9.