Setup Cluster

Setup Cluster

  7 minute read  

Setup Cluster

After completing the cluster preparation steps, you are ready to configure the kubeops tooling and create the Kubernetes cluster from your cluster admin host. This Setup Cluster document describes how to install and configure KOSI and kubeopsctl, prepare the cluster configuration, and then trigger the automated cluster deployment.

Important: the following commands have to be executed on your cluster admin

1. Install KOSI

# KOSI-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kosi=<kosi-version>

# KOSI-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kosi-version>


# KOSI-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kosi-version>.deb
sudo dpkg --install <kosi-version>.deb

# KOSI-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm -e kosi
wget https://packagerepo.kubeops.net/rpm/<kosi-version>.rpm
sudo rpm --install -v <kosi-version>.rpm

2. Set the KUBEOPSROOT env var

Set KUBEOPSROOT and XDG_RUNTIME_DIR in ~/.bashrc

# file ~/.bashrc
# Append these values to the end of your ~/.bashrc file
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

Source .bashrc to apply the values

source ~/.bashrc
echo $KUBEOPSROOT
echo $XDG_RUNTIME_DIR

As a result you should see your KUBEOPSROOT-path two times.

3. Adjust KOSI Configuration

This creates a kubeops directory in your home directory and transfers all necessary files, e.g., the kosi-config and the plugins, to it.

mkdir ~/kubeops
cd ~/kubeops
cp -R /var/kubeops/kosi/ .
cp -R /var/kubeops/plugins/ .

The config.yaml is in your KUBEOPSROOT-path (typically in ~/kubeops/kosi)

  • Set hub in your kosi config to hub: https://dispatcher.kubeops.net/v4/dispatcher/
  • Set the “plugins”-entry in your kosi config to plugins: /home/<yourUser>/kubeops/plugins, where is changed to your username
# file $KUBEOPSROOT/kosi/config.yaml
apiversion: kubernative/sina/config/v2

spec:
  hub: https://dispatcher.kubeops.net/v4/dispatcher/ # <-- set hub url
  plugins: <your kubeopsroot>/kubeops/plugins/ # <-- set the path to your plugin folder (~ for home or $KUBEOPSROOT don't work, it has to be the full path)
  workspace: /tmp/kosi/process/
  logging: info
  housekeeping: false
  proxy: false

4. Install KOSI enterprise plugins

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.0.0

5. Login with your user

kosi login -u <yourUser>

At this point it is normal if you get the following error message:
Error: The login to registry is temporary not available. Please try again later.
The reason for this is that podman is not yet installed.

6. Install kubeopsctl

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm -e kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install -v <kubeopsctl-version>.rpm

7. Create a cluster-values.yaml configuration file

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local                    # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: true
zones:
# IMPORTANT: The following part has to be adapted so that every one of your masternodes and workernodes is included
# This file only includes the minimum requirements for the amount of masters and workers and an example usage of zones
# You should adapt this part to your amount of masters and workers and cluster them into as many zones as you like
- name: zone1
  nodes:
  - name: <your master1 hostname>
    iPAddress: <your master1 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker1 hostname>
    iPAddress: <your worker1 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone2
  nodes:
  - name: <your master2 hostname>
    iPAddress: <your master2 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker2 hostname>
    iPAddress: <your worker2 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone3
  nodes:
  - name: <your master3 hostname>
    iPAddress: <your master3 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker3 hostname>
    iPAddress: <your worker3 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>

Full Example 
# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: myuser
kubernetesVersion: 1.32.2
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local                 # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: dev07-master1-ubuntu2404
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker1-ubuntu2404
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2
- name: zone2
  nodes:
  - name: dev07-master2-ubuntu2404
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker2-ubuntu2404
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2
- name: zone3
  nodes:
  - name: dev07-master3-ubuntu2404
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker3-ubuntu2404
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

7.1 Using KubeVip in your Cluster (optional)

KubeVip provides a highly available virtual IP for the Kubernetes control plane, giving the cluster a stable API endpoint. In multi-control-plane clusters, the virtual IP can move to another control-plane node if the active one fails. For more information, refer to the official KubeVip documentation here.

To enable KubeVip, set kubeVipEnabled to true and configure virtualIP with an unused IP address in the same network as your control-plane nodes.

If KubeVip is disabled, set kubeVipEnabled to false and use the IP address of the first control-plane node as virtualIP in your cluster-values.yaml in the Setup.

Examples:

kubeVipEnabled: true
virtualIP: <IP in your cluster ip range which is not given yet>

or

kubeVipEnabled: false
virtualIP: <master1 ip>

For highly available clusters, enabling KubeVip is recommended. For test environments or single-control-plane clusters, disabling it is usually sufficient.

8. Pull required KOSI packages

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion <your wanted Kubernetesversion> you can pull an older Kubernetes version.
Available Kubernetes versions are 1.34.7 , 1.35.3 , 1.36.0 . 

kubeopsctl pull

or

kubeopsctl pull --kubernetesVersion <x.xx.x>

9. Install podman

kosi install -p $KUBEOPSROOT/lima/podman_5.2.2.tgz -f cluster-values.yaml

10. Install helm

kosi install -p $KUBEOPSROOT/lima/helm_v3.16.4.tgz

11. Install kubernetes tools (kubectl)

Make sure the kubernetes version matches the one you pulled before.

kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_<your kubernetes version>.tgz -f cluster-values.yaml

This command also installs kubelet and kubeadm. You can either mask or delete them on your admin as they are not necessary for the cluster creation process.

Full Example 
kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_1.32.2.tgz -f cluster-values.yaml

12. Cluster Setup

Make sure that you are logged in on hub and registry.

kosi login -u <your username>

Now the login for hub and registry should be successful!

Make sure that you changed the kosi config.yaml.

cat $KUBEOPSROOT/kosi/config.yaml

Make sure that you pulled all required packages.

ll -1 $KUBEOPSROOT/lima

Install Kubernetes Cluster with kubeopsctl. Cluster setup takes about 10 to 15 minutes.

kubeopsctl apply -f cluster-values.yaml