Installation

17 minute read

KubeOps Installation and Setup

Welcome to the very first step to getting started with KubeOps. In this section, you will get to know about

hardware, software and network requirements
steps to install the required software
key configurations for KubeOps

Prerequisites

A total of 7 machines are required:

one admin
three master
three worker

You can choose between Red Hat Enterprise Linux 8 or OpenSUSE 15 . All of your machines need the same os.
Below you can see the minimal requirements for CPU, memory and disk storage:

OS	Minimum Requirements
Red Hat Enterprise Linux 8	8 CPU cores, 16 GB memory, 50GB disk storage
OpenSUSE 15	8 CPU cores, 16 GB memory, 50GB disk storage

For each working node, an additional unformatted hard disk with 50 GB each is required. For more information about the harddrives for rook-ceph, visit the rook-ceph prerequisites page

Requirements on admin

The following requirements must be fulfilled on the admin machine.

All the utilized users require sudo privileges. If you are using Kubeops as a user, you need a user with sudo rights, so for Opensuse and RHEL 8 Environments it is the wheel group the user should be added to. Make sure that you change your user with:

su -l <user>

Admin machine must be synchronized with the current time.
You need an internet connection to use the default KubeOps registry registry1.kubernative.net/lima.

A local registry can be used in the Airgap environment. KubeOps only supports secure registries.
It is important to list your registry as an insecure registry in registry.conf (/etc/containers/registries.conf for podman, /etc/docker/deamon.json for docker), in case of insecure registry usage.

Now you can create your own registry instead of using the default. Checkout how to Guide Create a new Repository. for more info.

it is recommended that runc is uninstalled
```
dnf remove -y runc
```
```
zypper remove -y runc
```

tc should be installed.

dnf install -y tc
dnf install -y libnftnl

zypper install -y iproute2
zypper install -y libnftnl

for opensearch, the /etc/sysctl.conf should be configured, the line

vm.max_map_count=262144

should be added. also the command

 sysctl -p

should be executed after that.

Podman must be installed on your machine.
```
sudo dnf install -y podman
```
```
sudo zypper install -y podman
```

Warning

There can be an issue with conflicts with containerd, so it is recommended that containerd.io is removed before installing the podman package.

$KUBEOPSROOT and $LIMAROOT must be set.

echo 'export KUBEOPSROOT=<home folder of user>/kubeops' >> $HOME/.bashrc
echo 'export LIMAROOT=<home folder of user>/kubeops/lima' >> $HOME/.bashrc
source $HOME/.bashrc

Requirements for each node

The following requirements must be fulfilled on each node.

All the utilized users require sudo privileges. If you are using Kubeops as a user, you need a user with sudo rights, so for Opensuse and RHEL 8 Environments it is the wheel group the user should be added to.
Every machine must be synchronized with the current time.
You have to assign lowercase unique hostnames for every machine you are using.

We recommended using self-explanatory hostnames.

To set the hostname on your machine use the following command:
```
hostnamectl set-hostname <name of node>
```
- Example
  Use the commands below to set the hostnames on each machine as admin, master, node1 node2.
```
hostnamectl set-hostname admin
hostnamectl set-hostname master 
hostnamectl set-hostname node1
hostnamectl set-hostname node2
```
Requires sudo privileges
It is recommended that a dns service is running, or if you don’t have a nds service, you can change the /etc/hosts file. an example for a entry in the /etc/hosts file could be:
```
10.2.10.12 admin
10.2.10.13 master1
10.2.10.14 master2
10.2.10.15 master3
10.2.10.16 node1
10.2.10.17 node2
10.2.10.18 node3
```
To establish an SSH connection between your machines, you either need an SSH key or you need to install sshpass.
1. Generate an SSH key on admin machine using following command
```
ssh-keygen
```
  There will be two keys generated in ~/.ssh directory.
  The first key is the id_rsa(private) and the second key is the id_rsa.pub(public).
2. Copy the ssh key from admin machine to your node machine/s with following command
```
ssh-copy-id <ip address or hostname of your node machine>
```
3. Now try establishing a connection to your node machine/s
```
ssh <ip address or hostname of your node machine>
```

Installing KubeOpsCtl

Create a kubeopsctl.yaml file with respective information as shown in kubeopsctl.yaml parameters, in order to use the KubeOps package.
Install the kubeops*.rpm on your admin machine.

Working with KubeOpsCtl

Before starting with KubeOps cluster, it is important to check if podman is running.

To verify that podman is running, use the following command:
```
systemctl status podman
```

To start and enable podman use the following commands:

systemctl enable podman

systemctl start --now podman

Note: This must be done with the root user or with a user with sudo privileges.

Run kubeopsctl on your commandline like:

kubeopsctl apply -f kubeopsctl.yaml

kubeopsctl.yaml parameters

the names of the nodes should be the same as the hostnames of the machines.

### General values for registry access ###
imagePullRegistry: "registry1.kubernative.net/lima" # mandatory, the registry from which the images for the cluster are pulled
localRegistry: false # mandatory, set to true if you use a local registry, if you want to set the value to true then you also have to set harbor to true

the imagePullRegistry parameter is for the registry, from which the images for the platform softeware is pulled. the localRegistry is a parameter for using a insecure, local registry for pulling images.

### Values for setup configuration ###
apiVersion: kubeops/kubeopsctl/alpha/v3 # mandatory
clusterName: "example" # mandatory
clusterUser: "root" # mandatory
kubernetesVersion: "1.28.2" # mandatory, check lima documentation
#masterHost: optional if you have an hostname, default value in "masterIP"
masterIP: 10.2.10.12 # mandatory
firewall: "nftables" # mandatory, default "nftables"
pluginNetwork: "calico" # mandatory, default "calico"
containerRuntime: "containerd" # mandatory
clusterOS: "Red Hat Enterprise Linux" # mandatory, can be "Red Hat Enterprise Linux" or "openSUSE Leap"
### Additional values for cluster configuration
useInsecureRegistry: false # optional, default is false
ignoreFirewallError: false # optional, default is false
serviceSubnet: 192.168.128.0/17 # optional, default "192.168.128.0/17"
podSubnet: 192.168.0.0/17 # optional, default "192.168.0.0/17"
debug: false # optional, default is false
logLevel: vvvvv # optional, default "vvvvv"
systemCpu: "1" # optional, default "1"
systemMemory: "2G" # optional, default "2G"
sudo: true # optional, default is true
tmpCopyDir: "/tmp" # optional, default is /tmp
createCluster: true # optional, default is true
updateRegistry: true # optional, default is true

the parameter clusterName is used to interact with and manage the cluster later on, p.e. if you want to change the runtime, you need the clusterName parameter.
the clusteruser is the linux user for using the cluster. the clusterOS is the linux distribution of the cluster.
masterIP is the ip-adress of the clustermaster or the first master, which is later used for interacting with the cluster.
useInsecureRegistry is for using a local and insecure registry for pulling images for the lima software.
ignoreFirewallError is a parameter for ignoring firewall errors while the cluster is created (not while operating on the cluster).
serviceSubnet is the subnet for all kubernetes service IP-adresses.
podSubnet is the subnet for all kubernetes pod IP-adresses.
systemCpu is the maximum of cpu that the kube-apiserver is allowd to use.
sudo is a parameter for using sudo for commands that need sudo rights, if you use a non-root linux-user.
tmpCopyDir is a parameter for templating the folder on the cluster nodes, where images of lima will be copied to.
createCluster is a parameter with which you can specify whether you want to create a cluster or not.
updateRegistry is a parameter for updating the docker registry.

zones:
  - name: zone1 
    nodes:
      master:
        - name: cluster1master1
          ipAdress: 10.2.10.11
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.28.2
        - name: cluster1master2
          ipAdress: 10.2.10.12
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.28.2
      worker:
        - name: cluster1worker1
          ipAdress: 10.2.10.14
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.28.2
        - name: cluster1worker2
          ipAdress: 10.2.10.15
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.28.2
  - name: zone2
    nodes:
      master:
        - name: cluster1master3
          ipAdress: 10.2.10.13
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.28.2  
      worker:
        - name: cluster1worker3
          ipAdress: 10.2.10.16
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.28.2

This YAML content is mandatory and describes a configuration for managing multiple zones in a Kubernetes cluster. Let’s break it down step by step:

zones: This is the top-level key in the YAML file, representing a list of zones within the Kubernetes cluster.
- zone1 and zone2: These are two zones within the cluster, each with its own configuration.
  - nodes: This is a sub-key under each zone, indicating the different types of nodes within that zone.
    - master: This is a sub-key under nodes, representing the master nodes in the zone.
      - cluster1master1, cluster1master2, and cluster1master3: These are individual master nodes in the cluster, each with its own configuration settings. They have attributes like name (node name has to be equal to host name), ipAdress (IP address), user (the user associated with the node), systemCpu (CPU resources allocated to the system), systemMemory (system memory allocated), status (the status of the node, can be either “active” or “drained”), and kubeversion (the Kubernetes version running on the node). Those kubernetes versions in the zone are for the nodes. NOTE: If you drain too many nodes, you may have too few OSDs for Rook.
    - worker: This is another sub-key under nodes, representing the worker nodes in the zone.
      - cluster1worker1, cluster1worker2, and cluster1worker3: Similar to the master nodes, these are individual worker nodes in the cluster, each with its own configuration settings, including name, IP address, user, system resources, status, and Kubernetes version.

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true # if localRegistry is set to true, harbor also needs to be set to true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
headlamp: true
certman: true
ingress: true 
keycloak: true
velero: true

this values are booleans for deciding which applications are later installed into the cluster.

# Global values, will be overwritten by the corresponding values of the individual packages
namespace: "kubeops"
storageClass: "rook-cephfs"

These global values will be used for installing the packages, but will be overwritten by the corresponding package-level settings.

namespace defines the kubernetes-namespace in which the applications are deployed
storageClass defines the name of StorageClass-Ressource that will be used by the applications.

rookValues:
  namespace: kubeops
  cluster:
    spec:
      dataDirHostPath: "/var/lib/rook" # optional, default is /var/lib/rook
    storage:
      useAllNodes: true # optional, default value: true
      useAllDevices: true # optional, default value: true
      deviceFilter: "^sd[a-b]" # optional, will only be used if useAllDevices is set to false
      config:
        metadataDevice: "sda" # optional, only set this value, if there is a device available
      nodes: # optional if useAllNodes is set to true, otherwise mandatory
        - name: "<ip-adress of node_1>"
          devices:
            - name: "sdb" 
        - name: "<ip-adress of node_2>"
          deviceFilter: "^sd[a-b]"
          config:
            metadataDevice: "sda" # optional
    resources:
      mgr:
        requests:
          cpu: "500m" # optional, default is 500m, limit: 1000m
          memory: "512Mi" # optional, default is 1Gi, limit: 1Gi
      mon:
        requests:
          cpu: "1" # optional, default is 1, limit: 2000m
          memory: "1Gi" # optional, default is 1Gi, limit: 2Gi
      osd:
        requests:
          cpu: "1" # optional, default is 1, limit: 2
          memory: "1Gi" # optional, default is 4Gi, limit: 4Gi
      cephFileSystems:
        requests:
          cpu: "1" # optional, default is 1, limit: 2
          memory: "1Gi" # optional, default is 1, limit: 4Gi
      cephObjectStores:
        requests:
          cpu: "1" # optional, default is 1, limit: 2
          memory: "1Gi" # optional, default is 1Gi, limit: 2Gi
  operator:
    data:
      rookLogLevel: "DEBUG" # optional, default is DEBUG

the namespace parameter is important for the apllications, because this parameter decides, in which namespace the individual applications are deployed.
dataDirHostPath is for setting the path of the configuration fiules of rook.
useAllNodes is parameter of rook-ceph and if it is set to true, all worker nodes will be used for rook-ceph.
useAllDevices is parameter of rook-ceph and if it is set to true, all possible devices will be used for rook-ceph.
deviceFilter is a Global filter to only select certain devicesnames. This example matches names starting with sda or sdb. it will only be used if useAllDevices is set to false and will be ignored if individual devices have been specified on a node.
metadataDevice: Name of a device or lvm to use for the metadata of OSDs(daemons for storing data on the local file system) on each node Performance can be improved by using a low latency device (SSD or NVMe) as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. This global setting will be overwritten by the corresponding node-level setting.
nodes: Names of individual nodes in the cluster that should have their storage included. Will only be used if useAllNodes is set to false. Specific configurations of the individual nodes will overwrite global settings.
resources refers to the cpu and memory that the parts of rook-ceph will be requesting. In this case it is the manager, the monitoring pods and the OSDs (they have the job of managing the local storages of the nodes and together they form the distributed storage) as well as the filesystem and object-store pods (they manage the respecting storage solution).
rookLogLevel: the loglevel of rook-ceph. this provides the most informative logs.

harborValues: 
  namespace: kubeops # optional, default is kubeops
  harborpass: "password" # mandatory: set password for harbor access
  databasePassword: "Postgres_Password" # mandatory: set password for database access
  redisPassword: "Redis_Password" # mandatory: set password for redis access
  externalURL: http://10.2.10.13:30002 # mandatory, the ip address and port, from which harbor is accessable outside of the cluster
  nodePort: 30002 # mandatory
  hostname: harbor.local # mandatory
  harborPersistence:
    persistentVolumeClaim:
      registry:
        size: 5Gi # mandatory, depending on storage capacity
        storageClass: "rook-cephfs" #optional, default is rook-cephfs
      jobservice:
        jobLog:
          size: 1Gi # mandatory: Depending on storage capacity
          storageClass: "rook-cephfs" #optional, default is rook-cephfs
      database:
        size: 1Gi # mandatory, depending on storage capacity
        storageClass: "rook-cephfs" #optional, default is rook-cephfs
      redis:
        size: 1Gi # mandatory, depending on storage capacity
        storageClass: "rook-cephfs" #optional, default is rook-cephfs
      trivy: 
        size: 5Gi # mandatory, depending on storage capacity
        storageClass: "rook-cephfs" #optional, default is rook-cephfs

You can set the root password for the postgres-database, redis and harbor itself. For the persistant volumes of harbor, the sizes and the storageclass are also templatable. So all applications of harbor. p.e. trivy for the image-scanning or the chart museum for helm charts.

###Values for Logstash deployment###
##For detailed explaination for each key see: https://github.com/elastic/helm-charts/releases/tag/v7.16.3###
logstashValues:
  namespace: kubeops
  volumeClaimTemplate:
    accessModes: 
      - ReadWriteMany #optional, default is [ReadWriteMany]
    resources:
      requests:
        storage: 1Gi # mandatory, depending on storage capacity
    storageClass: "rook-cephfs" #optional, default is rook-cephfs

for logstash the pvc size is also templateable

###Values for OpenSearch-Dashboards deployment###
##For detailed explaination for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch-dashboards###
openSearchDashboardValues:
  namespace: kubeops
  nodePort: 30050
###Values for OpenSearch deployment###
##For detailed explaination for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch###
openSearchValues:
  namespace: kubeops
  opensearchJavaOpts: "-Xmx512M -Xms512M" # optional, default is -Xmx512M -Xms512M
  resources:
    requests:
      cpu: "250m" # optional, default is 250m
      memory: "1024Mi" # optional, default is 1024Mi
    limits:
      cpu: "300m" # optional, default is 300m
      memory: "3072Mi" # optional, default is 3072Mi
  persistence:
    size: 4Gi # mandatory
    enabled: "true" # optional, default is true
    enableInitChown: "false" # optional, default is false
    labels:
      enabled: "false" # optional, default is false
    storageClass: "rook-cephfs" # optional, default is rook-cephfs
    accessModes:
      - "ReadWriteMany" # optional, default is {ReadWriteMany}
  securityConfig:
    enabled: false # optional, default value: false
    ### Additional values can be set, if securityConfig is enabled:
    # path: "/usr/share/opensearch/plugins/opensearch-security/securityconfig"
    # actionGroupsSecret:
    # configSecret:
    # internalUsersSecret: internal-users-config-secret
    # rolesSecret:
    # rolesMappingSecret:
    # tenantsSecret:
    # config:
    #   securityConfigSecret: ""
    #   dataComplete: true
    #   data: {}
  replicas: "3" # optional, default is 3
#--------------------------------------------------------------------------------------------------------------------------------

opensearchJavaOpts is the size of the java heap.
enableInitChown is the procedure of changing the owner of the opensearch configuration files, so non-root users can change the configuration files.
if you want labels for the opensearch pods in the cluster, you can enable the labels with the enabled parameter under the labels subtree.
if you want to use a custom security config, you can enable it and use then paramters like the path to the file. if you want more info, you can find it here
the replicas are 3 by default, but you can template it for better scaling.


###Values for Prometheus deployment###
prometheusValues:
  namespace: kubeops # optional, default is kubeops
  privateRegistry: false # optional, default is false
  grafanaUsername: "user" # optional, default is user
  grafanaPassword: "password" # optional, default is password
  retentionSize: "24GB" # optional, default is 24GB
  grafanaResources:
    nodePort: 30211 # optional, default is 30211
    storageClass: "rook-cephfs" # optional, default is rook-cephfs
    storage: 5Gi # optional, default is 5Gi
    grafanaUsername: "admin" # optional, default is admin
    grafanaPassword: "admin" # optional, default is admin
    retention: 10d # mandatory
    retentionSize: "24GB" # mandatory
    storageClass: "rook-cephfs" # optional, default is rook-cephfs
    storage: 25Gi # optional, default is 25Gi

  prometheusResources:
    retention: 10d # optional, default is 10d
    retentionSize: "24GB" # optional, default is "24GB"
    storageClass: "rook-cephfs" # optional, default is rook-cephfs
    storage: 25Gi # optional, default is 25Gi

the nodePort is 30211, so you can visit the grafana apllication on every master with :30211, but you can template and thus change it.

###Values for OPA deployment###
opaValues:
  namespace: kubeops

namespace value specifies the Kubernetes namespace where OPA will be deployed.

###Values for Headlamp deployment###
headlampValues:
  namespace: kubeops
  hostname: kubeops-dashboard.local
  service:
    nodePort: 30007

namespace value specifies the Kubernetes namespace where Headlamp will be deployed.
hostname is for accessing the Headlamp service.
the nodePort value specifies the node port for accessing Headlamp.

###Values for cert-manager deployment###
certmanValues:
  namespace: kubeops
  replicaCount: 3
  logLevel: 2

namespace value specifies the Kubernetes namespace where cert-manager will be deployed.
replicaCount specifies the number of replicas for the cert-manager deployment.
logLevel specifies the logging level for cert-manager.

###Values for ingress-nginx deployment###
ingressValues:
  namespace: kubeops
  externalIPs: []

namespace value specifies the Kubernetes namespace where ingress-nginx will be deployed.
externalIPs value specifies a list of external IP addresses that will be used to expose the ingress-nginx service. This allows external traffic to reach the ingress controller. The value for this key is expected to be provided as a list of IP addresses.

###Values for keycloak deployment###
keycloakValues:
  namespace: "kubeops" # Optional, default is "keycloak"
  storageClass: "rook-cephfs" # Optional, default is "rook-cephfs"
  nodePort: "30180" # Optional, default is "30180"
  hostname: keycloak.local
  keycloak:
    auth:
      adminUser: admin # Optional, default is admin
      adminPassword: admin # Optional, default is admin
      existingSecret: "" # Optional, default is ""
  postgresql:
    auth:
      postgresPassword: "" # Optional, default is ""
      username: bn_keycloak # Optional, default is "bn_keycloak"
      password: "" # Optional, default is ""
      database: bitnami_keycloak # Optional, default is "bitnami_keycloak"
      existingSecret: "" # Optional, default is ""

namespace value specifies the Kubernetes namespace where keycloak will be deployed.
storageClass value specifies the storage class to be used for persistent storage in Kubernetes. If not provided, it defaults to “rook-cephfs”.
nodePort value specifies the node port for accessing Keycloak. If not provided, it defaults to “30180”.
hostname value specifies the hostname for accessing the Keycloak service.
adminUser value specifies the username for the Keycloak admin user. Defaults to “admin”.
adminPassword value specifies the password for the Keycloak admin user. Defaults to “admin”.
existingSecret value specifies an existing Kubernetes secret to use for Keycloak admin authentication. Defaults to an empty string.
postgresPassword value specifie the password for the PostgreSQL database. Defaults to an empty string.
username value specifies the username for the PostgreSQL database. Defaults to “bn_keycloak”.
password value specifies the password for the PostgreSQL database. Defaults to an empty string.
database value specifies the name of the PostgreSQL database. Defaults to “bitnami_keycloak”.
existingSecret value specifies an existing Kubernetes secret to use for PostgreSQL authentication. Defaults to an empty string.

veleroValues:
  namespace: "velero"
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"
  useNodeAgent: false
  defaultVolumesToFsBackup: false
  provider: "aws"
  bucket: "velero"
  useVolumeSnapshots: false
  backupLocationConfig:
    region: "minio"
    s3ForcePathStyle: true
    s3Url: "http://minio.velero.svc:9000"

namespace: Specifies the Kubernetes namespace where Velero will be deployed.
accessKeyId: Your access key ID for accessing the S3 storage service.
secretAccessKey: Your secret access key for accessing the S3 storage service.
useNodeAgent: Indicates whether to use a node agent for backup operations. If set to true, Velero will use a node agent.
defaultVolumesToFsBackup: Specifies whether to default volumes to file system backup. If set to true, Velero will use file system backup by default.
provider: Specifies the cloud provider where the storage service resides.
bucket: The name of the S3 bucket where Velero will store backups.
useVolumeSnapshots: Indicates whether to use volume snapshots for backups. If set to true, Velero will use volume snapshots.
backupLocationConfig: Configuration for the backup location.
region: Specifies the region where the S3 storage service is located.
s3ForcePathStyle: Specifies whether to force the use of path-style URLs for S3 requests.
s3Url: The URL for accessing the S3-compatible storage service.