This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Reference

In the reference you will find articles on the Kubeopsctl Commands, Fileformats, KubeOps Version and the Glossary

1 - KubeOpsCtl CLI Commands

This documentation shows all commands of the kubeopsctl and how to use them.

General commands

Overview of all KUBEOPSCTL commands

Description:
  kubeopsctl is a kubernetes cluster manager

Usage:
  kubeopsctl [command] [options]

Options:
  --version       Show version information
  -?, -h, --help  Show help and usage information

Commands:
  version  kubeopsctl version information
  login    Login to kubeops hub and registry
  logout   Logout from kubeops hub
  pull     Pull kosi packages for kubernetes cluster setup and plattform tools
  apply    Apply values on kubernetes cluster

Command ‘kubeopsctl –version’

The kubeopsctl --version command shows you the current version of kubeopsctl.

kubeopsctl --version

The output should be:

2.0.3

Command ‘kubeopsctl –help’

The command kubeopsctl --help gives you an overview of all available commands:

kubeopsctl --help

Alternatively, you can also enter kubeopsctl or kubeopsctl -? in the command line.

Command ‘kubeopsctl login’

The command kubeopsctl login performs a login against the KOSI HUB. A valid login session is neccessary to pull the packages.

Description:
  Login to kubeops hub and registry

Usage:
  kubeopsctl login [options]

Options:
  -u, --username <username> (REQUIRED)  Username
  -p, --password <password>             Password
  -?, -h, --help                        Show help and usage information

Example:

kubeopsctl login -u <username> -p <password>

Command ‘kubeopsctl logout’

The command kubeopsctl logout performs a logout from the KOSI HUB.

Description:
  Logout from kubeops hub

Usage:
  kubeopsctl logout [options]

Options:
  -?, -h, --help  Show help and usage information

Example:

kubeopsctl logout

Command ‘kubeopsctl pull’

The command kubeopsctl pull downloads all necessary KOSI packages to the admin node:

Description:
  Pull kosi packages for kubernetes cluster setup and plattform tools

Usage:
  kubeopsctl pull [options]

Options:
  -k, --kubernetesVersion <kubernetesVersion>  Kubernetes version
  -f, --tools <tools>                          Tools values file
  -?, -h, --help                               Show help and usage information

Example:

kubeopsctl pull

If you do not specify a parameter, the latest from kubeopsctl supported Kubernetes version will be pulled.
With parameter --kubernetesVersion 1.30.8 you can pull an older Kubernetes version.

Example:

kubeopsctl pull --kubernetesVersion 1.30.8

Command ‘kubeopsctl apply’

The command kubeopsctl apply is used to set up the kubeops platform with a configuration file.

Description:
  Apply values on kubernetes cluster

Usage:
  kubeopsctl apply [options]

Options:
  -f, --file <file> (REQUIRED)  Values files for cluster, tools and user
  -?, -h, --help                Show help and usage information

-f flag

The -f parameter is used to use value parameter yaml-files

Example:

kubeopsctl apply -f cluster-values.yaml

To install your KubeOps Compliance Applications, you have to use a second value file. It is called enterprise-value.yaml

Example:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

–delete flag

The -- delete parameter is used to perform a delete action.

This flag delete all nodes which are not present in the cluster-values.yaml - file Example:

kubeopsctl apply --delete -f cluster-values.yaml

2 - FAQs

KubeOps COMPLIANCE Security Hotfix

What is the issue?

Critical security vulnerabilities in NGINX (Rift and PoolSlip) affect the version used in KubeOps COMPLIANCE. These vulnerabilities may allow remote code execution or denial-of-service attacks.

Who is affected?

KubeOps COMPLIANCE users running affected NGINX versions, including:

KubeOps 2.1.0 KubeOps 2.0.3–2.0.6 KubeOps 1.7.6–1.7.8

What action is required?

Install the provided hotfix as soon as possible.

Script Details for KubeOps V2.x.x
  • Step 1: Load the shell environment & define the KOSI Package.

Load the shell environment from the current user’s bashrc file &

source ~/.bashrc

Set the package name for the Nginx hotfix:

kosinginx="kubeops/ingress-nginx:2.1.0_Beta1"

Create the filename that will be used when the package is pulled:

kosinginxfile="ingress-nginx-$(echo $kosinginx | awk -F: '{print $NF}').tgz"

This creates a file name based on the package version, in this example for KubeOps 2.1.0:

ingress-nginx-2.1.0_Beta1.tgz

  • Step 2: Prepare the KOSI Directory

    Temporarily enable housekeeping in the KOSI configuration, run kosi version, and then disable housekeeping again:

    sed -i "s/  housekeeping: false/  housekeeping: true/g" $KUBEOPSROOT/kosi/config.yaml
kosi version
sed -i "s/  housekeeping: true/  housekeeping: false/g" $KUBEOPSROOT/kosi/config.yaml

    This prepares the KOSI directory before the package update.

  • Step 3: Log In to KOSI

    Log in to KOSI with a valid user account. The following loop prompts for a username and retries until the login is successful:

    while true; do
    read -rp "$(printf "Username for kosi login: ")" KOSI_USER

    echo "Please enter the password for user '$KOSI_USER'"

    if kosi login -u "$KOSI_USER"; then
        echo "Login successful"
        break
    else
        echo "Login failed. Please try again."
        sleep 1
    fi
done

  • Step 4: Determine the Harbor Configuration

    Retrieve the Harbor namespace, endpoint, port, and password from the KubeOps configuration.

  harborns=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep -m 1 "namespace:" | awk '{print $2}')
  harborep=$(kubectl get cm -n"$harborns" harbor-core -oyaml | grep "EXT_ENDPOINT" | awk '{print $2}')
  harborepohnehttp=$(echo "$harborep" | awk -F"//" '{print $2}')
  harborport=$(echo "$harborepohnehttp" | awk -F: '{print $NF}')
  harborpw=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep "harborpass" | awk '{print $NF}' | sed 's/"//g')

Display the Harbor URL and port:

  echo "Harbor URL : $harborepohnehttp"
  echo "Harbor PORT: $harborport"

  • Step 5: Log In to Harbor with Podman

    Log in to Harbor as the admin user by using the password retrieved in the previous step:

    if podman login "$harborepohnehttp" -u admin -p "$harborpw" --tls-verify=false; then
    echo "Podman login successful"
else
    echo "Podman login to $harborepohnehttp as admin with password $harborpw was not successful"
    exit 1
fi

  • Step 6: Pull the Nginx Hotfix Package

    Pull the ingress-nginx package from the KOSI hub and save it as the local package archive:

    kosi pull --hub kubeops "$kosinginx" -o "$kosinginxfile" -r "$harborepohnehttp/kubeops" -t "localhost:$harborport/kubeops"

  • Step 7: Determine the Deployment Name for ingress-nginx

    Retrieve the deployment name for the existing ingress-nginx package:

    dname=$(kosi list | grep ingress-nginx | awk '{print $2}')

  • Step 8: Update ingress-nginx

    Run the KOSI update command by using the deployment name, the pulled package file, and the KubeOps values file:

    kosi update --dname="$dname" -p "$kosinginxfile" -f $KUBEOPSROOT/kubeopsctl/tools-values.yaml

    This applies the hotfix package to the existing ingress-nginx deployment.

  • Step 9: Remove Local Podman Images

    After the update completes, remove the local Podman images related to ingress-nginx and defaultbackend:

    for i in $(podman images | grep "ingress-nginx\|defaultbackend" | awk '{print $3}'); do
    podman rmi -f $i
done

    This cleans up the local images from the administration host.

Complete Script for 2.1.0 
#!/bin/bash
source ~/.bashrc

# kosi package
kosinginx="kubeops/ingress-nginx:2.1.0_Beta1"

# Filename for kosi pull
kosinginxfile="ingress-nginx-$(echo $kosinginx | awk -F: '{print $NF}').tgz"

# Clean up kosi directory
sed -i "s/  housekeeping: false/  housekeeping: true/g" $KUBEOPSROOT/kosi/config.yaml
kosi version
sed -i "s/  housekeeping: true/  housekeeping: false/g" $KUBEOPSROOT/kosi/config.yaml

# kosi login

while true; do
    read -rp "$(printf "Username for kosi login: ")" KOSI_USER

    echo "Please enter the password for user '$KOSI_USER'"

    if kosi login -u "$KOSI_USER"; then
        echo "Login successful"
        break
    else
        echo "Login failed. Please try again."
        sleep 1
    fi
done

# Determine Harbor values
harborns=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep -m 1 "namespace:" | awk '{print $2}')
harborep=$(kubectl get cm -n"$harborns" harbor-core -oyaml | grep "EXT_ENDPOINT" | awk '{print $2}')
harborepohnehttp=$(echo "$harborep" | awk -F"//" '{print $2}')
harborport=$(echo "$harborepohnehttp" | awk -F: '{print $NF}')
harborpw=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep "harborpass" | awk '{print $NF}' | sed 's/"//g')

echo "Harbor URL : $harborepohnehttp"
echo "Harbor PORT: $harborport"

# Podman login
if podman login "$harborepohnehttp" -u admin -p "$harborpw" --tls-verify=false; then
    echo "Podman login successful"
else
    echo "Podman login to $harborepohnehttp as admin with password $harborpw was not successful"
    exit 1
fi

# kosi pull
kosi pull --hub kubeops "$kosinginx" -o "$kosinginxfile" -r "$harborepohnehttp/kubeops" -t "localhost:$harborport/kubeops"

# Determine dname for ingress-nginx
dname=$(kosi list | grep ingress-nginx | awk '{print $2}')

# kosi update
kosi update --dname="$dname" -p "$kosinginxfile" -f $KUBEOPSROOT/kubeopsctl/tools-values.yaml

# Clean up admin Podman images
for i in $(podman images | grep "ingress-nginx\|defaultbackend" | awk '{print $3}'); do
    podman rmi -f $i
done

 
Complete Script for 2.0.3-2.0.6 
#!/bin/bash
source ~/.bashrc

# kosi package
kosinginx="kubeops/ingress-nginx:2.0.6"

# Filename for kosi pull
kosinginxfile="ingress-nginx-$(echo $kosinginx | awk -F: '{print $NF}').tgz"

# Clean up kosi directory
sed -i "s/  housekeeping: false/  housekeeping: true/g" $KUBEOPSROOT/kosi/config.yaml
kosi version
sed -i "s/  housekeeping: true/  housekeeping: false/g" $KUBEOPSROOT/kosi/config.yaml

# kosi login

while true; do
    read -rp "$(printf "Username for kosi login: ")" KOSI_USER

    echo "Please enter the password for user '$KOSI_USER'"

    if kosi login -u "$KOSI_USER"; then
        echo "Login successful"
        break
    else
        echo "Login failed. Please try again."
        sleep 1
    fi
done

# Determine Harbor values
harborns=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep -m 1 "namespace:" | awk '{print $2}')
harborep=$(kubectl get cm -n"$harborns" harbor-core -oyaml | grep "EXT_ENDPOINT" | awk '{print $2}')
harborepohnehttp=$(echo "$harborep" | awk -F"//" '{print $2}')
harborport=$(echo "$harborepohnehttp" | awk -F: '{print $NF}')
harborpw=$(cat $KUBEOPSROOT/kubeopsctl/tools-values.yaml | grep -A50 "^- name: harbor" | grep -B50 "^- name:" | grep "harborpass" | awk '{print $NF}' | sed 's/"//g')

echo "Harbor URL : $harborepohnehttp"
echo "Harbor PORT: $harborport"

# Podman login
if podman login "$harborepohnehttp" -u admin -p "$harborpw" --tls-verify=false; then
    echo "Podman login successful"
else
    echo "Podman login to $harborepohnehttp as admin with password $harborpw was not successful"
    exit 1
fi

# kosi pull
kosi pull --hub kubeops "$kosinginx" -o "$kosinginxfile" -r "$harborepohnehttp/kubeops" -t "localhost:$harborport/kubeops"

# Determine dname for ingress-nginx
dname=$(kosi list | grep ingress-nginx | awk '{print $2}')

# kosi update
kosi update --dname="$dname" -p "$kosinginxfile" -f $KUBEOPSROOT/kubeopsctl/tools-values.yaml

# Clean up admin Podman images
for i in $(podman images | grep "ingress-nginx\|defaultbackend" | awk '{print $3}'); do
    podman rmi -f $i
done

WireGuard Installation with Local Package Repository

What is the issue?

WireGuard installation may fail when the local package repository is used on unsupported Red Hat Enterprise Linux versions.

Who is affected?

KubeOps COMPLIANCE users installing WireGuard with packagerepo: local.

What action is required?

When using packagerepo: local, ensure that all cluster nodes are running Red Hat Enterprise Linux 9.6.

3 - Fileformats in kubeopsctl

This documentation shows you all the different kind of fileformats kubeopsctl uses and how to use them.

There are currently 2 different files which can be handled by KubeOpsCtl:

cluster-values.yaml

The cluster-values.yaml defines every aspect of the cluster itself. It has no influence over which applications get installed.

apiVersion: kubeops/kubeopsctl/cluster/beta/v1 # required
imagePullRegistry: registry.kubeops.net/kubeops/kubeops # required
airgap: true # optional, default: true
clusterName: myCluster # required 
clusterUser: root # optional, default: root
kubernetesVersion: 1.32.2 # required
kubeVipEnabled: false # optional, default: true
virtualIP: 10.2.10.110 # required
firewall: nftables # optional, default: nftables
pluginNetwork: calico # optional, default: calico | possible alternative: cilium
containerRuntime: containerd # optional, default: containerd
kubeOpsRoot: /var/kubeops # optional, default: /var/kubeops
serviceSubnet: 192.168.128.0/17 # optional, default: 192.168.128.0/17
podSubnet: 192.168.0.0/17 # optional, default: 192.168.0.0/17
debug: false # optional, default: false
systemCpu: 250m # optional, default: 250m
systemMemory: 256Mi # optional, default: 256Mi
packageRepository: local # optional, default: local
changeCluster: true # optional, default: true

advanced:
  localRegistry:
    mountPath: /mnt/registry

zones: # required
- name: zone1 # required
  nodes: # required
  - name: master1 # required
    iPAddress: 10.2.10.110 # required
    type: controlplane # required
    kubeVersion: 1.32.2 # required
Detailed Parameter Information
Key Possible Values Additional Info
pluginNetwork Calico, Cilium

enterprise-values.yaml

The enterprise-values.yaml defines all enterprise applications currently available for you to install in your cluster via kubeopsctl.
You can append multiple of them into a single enterprise-values.yaml as shown in the first example.

For each application you have 2 ways to change its values:

  • the standard values
  • the advanced values

While the standard values only cover predefined keys, the advanced values let you change every key available in the helm chart. Keep in mind, that the standard values overwrite the advanced values if both are set.

Each as optional marked line can be skipped unless otherwise stated. If a optional line is skipped, its default value will be used instead. If there is no default value, it can just be omitted and won’t affect the cluster and/or the application

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
  - name: opa-gatekeeper
    enabled: true
    values:
      standard:
        namespace: gatekeeper-system # optional, default is gatekeeper-system
      advanced:
  - name: filebeat-os
    enabled: false
    values:
      standard:
        namespace: logging # optional, default is logging
      advanced:

  ### Values for Rook-Ceph ###
  ### For detailed explanation for each key see: https://artifacthub.io/packages/helm/rook/rook-ceph?modal=values ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: rook-ceph
    enabled: true
    values:
      standard:
        namespace: rook-ceph # optional, default is rook-ceph
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route the ceph dashboard via Traefik
        cluster:
          spec:
            dataDirHostPath: "/var/lib/rook" # optional, default is /var/lib/rook
          resources:
            mgr:
              requests:
                cpu: "500m" # optional, default is 500m, limit: 1000m
                memory: "512Mi" # optional, default is 1Gi, limit: 1Gi
            mon:
              requests:
                cpu: "1" # optional, default is 1, limit: 2000m
                memory: "1Gi" # optional, default is 1Gi, limit: 2Gi
            osd:
              requests:
                cpu: "1" # optional, default is 1, limit: 2
                memory: "1Gi" # optional, default is 4Gi, limit: 4Gi
        operator:
          data:
            rookLogLevel: "DEBUG" # optional, default is DEBUG
      advanced: 
        cluster: # All values from https://artifacthub.io/packages/helm/rook/rook-ceph-cluster?modal=values are overwritable
        operator: # All values from https://artifacthub.io/packages/helm/rook/rook-ceph?modal=values are overwritable

  ### Values for Harbor deployment ###
  ### For detailed explanation for each key see: https://artifacthub.io/packages/helm/harbor/harbor/1.8.1#configuration ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: harbor
    enabled: true
    values:
      standard:
        namespace: harbor # optional, default is harbor
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        harborpass: "password" # required: set password for harbor access
        databasePassword: "Postgres_Password" # required: set password for database access
        redisPassword: "Redis_Password" # required: set password for redis access
        externalURL: http://10.2.10.11:30002 # required, the ip address and port, from which harbor is accessable outside of the cluster
        nodePort: 30002 # required
        hostname: harbor.local # required
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi # optional, default is 40Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            jobservice:
              jobLog:
                size: 1Gi # optional, default is 1Gi
                storageClass: "rook-cephfs" #optional, default is rook-cephfs
            database:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            redis:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            trivy: 
              size: 5Gi # optional, default is 5Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
      advanced: #  All values from https://artifacthub.io/packages/helm/harbor/harbor/1.8.1#configuration are overwritable

  ### Values for cert-manager deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: cert-manager
    enabled: true
    values:
      standard:
        namespace: cert-manager # optional, default is cert-manager
        replicaCount: 3
        logLevel: 2
        secretName: root-secret
      advanced: # All values from https://artifacthub.io/packages/helm/cert-manager/cert-manager?modal=values are overwritable
      ## add helm values here
      ca: # LetsEncrypt ClusterIssuer / ACME HTTP01 solver settings
        # emailLetsEncrypt: <your_email@domain.com> # default: example@example.com --> must configure
        # ingressName: <ingress_name> # default: nginx; set to "traefik" to solve ACME HTTP01 via Traefik

  ### Values for ingress-nginx deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: ingress-nginx
    enabled: true
    values:
      standard:
        namespace: ingress-nginx # optional, default is ingress
      advanced: # All values from https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx?modal=values are overwritable

### Values for Traefik deployment ###
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false 
localRegistry: false
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik # optional default is traefik
    advanced: # All values from https://artifacthub.io/packages/helm/traefik/traefik/39.0.0?modal=values are overwritable

### Values for keycloak deployment ###
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: keycloak # Optional, default is "keycloak"
      ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
      storageClass: "rook-cephfs" # Optional, default is "rook-cephfs"
      keycloak:
        auth:
          adminUser: admin # Optional, default is admin
          adminPassword: admin # Optional, default is admin, only used during initial installation
          existingSecret: "" # Optional, default is ""
      postgresql:
        auth:
          postgresPassword: "" # Optional, default is ""
          username: bn_keycloak # Optional, default is "bn_keycloak"
          password: "" # Optional, default is ""
          database: bitnami_keycloak # Optional, default is "bitnami_keycloak"
          existingSecret: "" # Optional, default is ""
    advanced: # All values from https://artifacthub.io/packages/helm/bitnami/keycloak?modal=values are overwritable, except values that are only applied during the installation, such as the Keycloak admin password

keycloak.auth.adminPassword is only applied during the installation. After Keycloak has been installed, changing this value in enterprise-values.yaml does not change the Keycloak password. To change the Keycloak password after installation, change it directly in Keycloak.

  ### Values for filebeat deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: filebeat-os
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging   
      advanced: # All values from https://artifacthub.io/packages/helm/elastic/filebeat?modal=values are overwritable

  ### Values for Logstash deployment ###
  ### For detailed explanation for each key see: https://github.com/elastic/helm-charts/releases/tag/v7.16.3 ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: logstash-os
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging
        volumeClaimTemplate:
          accessModes: 
            - ReadWriteMany #optional, default is [ReadWriteMany]
          resources:
            requests:
              storage: 1Gi # required, depending on storage capacity
          storageClass: "rook-cephfs" #optional, default is rook-cephfs
      advanced: # All values from https://artifacthub.io/packages/helm/elastic/logstash?modal=values are overwritable
      

  ### Values for OpenSearch-Dashboards deployment ###
  ### For detailed explanation for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch-dashboards ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: opensearch-dashboards
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        nodePort: 30050
        hostname: opensearch-dashboard.local # optional, default is opensearch-dashboard.local; external ingress hostname, determines the Keycloak SSO URL
        adminPassword: "Password@@123456" # optional, default is Password@@123456; must match opensearch-os adminPassword
      advanced: # All values from https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch-dashboards?modal=values are overwritable

  ### Values for OpenSearch deployment ###
  ### For detailed explanation for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: opensearch-os
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging
        hostname: opensearch-dashboard.local # optional, default is opensearch-dashboard.local; Keycloak host for SSO, must match opensearch-dashboards
        adminPassword: "Password@@123456" # optional, default is Password@@123456; OpenSearch admin password, enforced after deploy
        opensearchJavaOpts: "-Xmx512M -Xms512M" # optional, default is -Xmx512M -Xms512M
        resources:
          requests:
            cpu: "250m" # optional, default is 250m
            memory: "1024Mi" # optional, default is 1024Mi
          limits:
            cpu: "300m" # optional, default is 300m
            memory: "3072Mi" # optional, default is 3072Mi
        persistence:
          size: 4Gi # required
          enabled: "true" # optional, default is true
          enableInitChown: "false" # optional, default is false
          labels:
            enabled: "false" # optional, default is false
          storageClass: "rook-cephfs" # optional, default is rook-cephfs
          accessModes:
            - "ReadWriteMany" # optional, default is {ReadWriteMany}
        securityConfig:
          enabled: false # optional, default value: false
          ### Additional values can be set, if securityConfig is enabled:
          # path: "/usr/share/opensearch/plugins/opensearch-security/securityconfig"
          # actionGroupsSecret:
          # configSecret:
          # internalUsersSecret: internal-users-config-secret
          # rolesSecret:
          # rolesMappingSecret:
          # tenantsSecret:
          # config:
          #   securityConfigSecret: ""
          #   dataComplete: true
          #   data: {}
        replicas: "3" # optional, default is 3
      advanced: # All values from https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch?modal=values are overwritable

  ### Values for Prometheus deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: kube-prometheus-stack
    enabled: true
    values:
      standard:
        namespace: monitoring # optional, default is monitoring
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        privateRegistry: false # optional, default is false
        grafanaUsername: "user" # optional, default is user
        grafanaPassword: "password" # optional, default is password
        grafanaResources:
          storageClass: "rook-cephfs" # optional, default is rook-cephfs
          storage: 5Gi # optional, default is 5Gi
          nodePort: 30211 # optional, default is 30211

        prometheusResources:
          storageClass: "rook-cephfs" # optional, default is rook-cephfs
          storage: 25Gi # optional, default is 25Gi
          retention: 10d # optional, default is 10d
          retentionSize: "24GB" # optional, default is 24GB
          nodePort: 32090
      advanced: # All values from https://artifacthub.io/packages/helm/prometheus-community/prometheus?modal=values-schema are overwritable

  ### Values for OPA deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: opa-gatekeeper
    enabled: true
    values:
      standard:
        namespace: gatekeeper-system # optional, default is gatekeeper-system
      advanced: # All values from https://artifacthub.io/packages/helm/gatekeeper/gatekeeper/3.1.1?modal=values are overwritable

  ### Values for KubeOps-Dashboard (Headlamp) deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: kubeops-dashboard
    enabled: true
    values:
      standard:
        namespace: monitoring # optional, default is monitoring
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        service:
          nodePort: 30007
      advanced: # All values from https://artifacthub.io/packages/helm/headlamp/headlamp?modal=values are overwritable

  ### Values for velero deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: velero
    enabled: true
    values:
      standard:
        namespace: velero # Optional, default is "velero"
        accessKeyId: "your_s3_storage_username"
        secretAccessKey: "your_s3_storage_password"
        useNodeAgent: false
        defaultVolumesToFsBackup: false
        provider: "aws"
        bucket: "velero"
        useVolumeSnapshots: false
        backupLocationConfig:
          region: "minio"
          s3ForcePathStyle: true
          s3Url: "http://minio.velero.svc:9000"
      advanced: # All values from https://artifacthub.io/packages/helm/vmware-tanzu/velero?modal=values are overwritable

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
# used in deleteLogstash.sh
deleteNs: false # TODO undocumented values! dangerous if other charts are installed in the namespace :-o
# used in some packages
localRegistry: false # create pull secret for registry 
packages:
- name: kubevirt-manager
  enabled: true
  values:
    standard:
      namespace: kubevirt # optional default is kubevirt
      hostname: kubeops-vms.local
      nodeport: 30733 # optional default is 30733
      ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
    advanced: {}

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
# used in deleteLogstash.sh
deleteNs: false # TODO undocumented values! dangerous if other charts are installed in the namespace :-o
# used in some packages
localRegistry: false # create pull secret for registry 
packages:
- name: kubevirt-manager
  enabled: true
  values:
    standard:
      namespace: kubevirt # optional default is kubevirt
      hostname: kubeops-vms.local
      nodeport: 30733 # optional default is 30733
      ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
    advanced: {}
- name: kubevirt
  enabled: true
  hub: kubeops-vm
  values:
    standard:  
      namespace: kubevirt
      imagePullSecretCred:
        registry: registry.preprod.kubeops.net
        user: kubeops
        password: changeme # must be CLI Token from harbor 
      kubevirtobject:
        spec:
          configuration:
            developerConfiguration:
              logVerbosity:
                virtLauncher: 2
                virtHandler: 3
                virtController: 4
                virtAPI: 5
                virtOperator: 6
              featureGates:
              - ExpandDisks
              - CPUManager
              - ExperimentalIgnitionSupport
              - HypervStrictCheck
              - Sidecar
              - HostDevices
              - Snapshot
              - VMExport
              - HotplugVolumes
              - HostDisk
              - ExperimentalVirtiofsSupport
              - DownwardMetricsFeatureGate
              - Root
              - ClusterProfiler
              - WorkloadEncryptionSEV
              - VSOCK
              - DisableCustomSELinuxPolicy
              - KubevirtSeccompProfile
              - DisableMDEVConfiguration
              - PersistentReservation
              - VMPersistentState
              - MultiArchitecture
              - AutoResourceLimitsGate
              - AlignCPUs
              - VolumesUpdateStrategy
              - VolumeMigration
              #- NodeRestriction # Forbidden setting: Do not use!
              - InstancetypeReferencePolicy
              useEmulation: false
      cdiobject:
        spec:
          config:
            insecureRegistries:
            - registry.preprod.kubeops.net # Need to test if this is really necessary
            scratchSpaceStorageClass: rook-cephfs # Optional; Default: rook-cephfs

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: rook-ceph
 enabled: true
 values:
   standard:
     namespace: rook-ceph
     cluster:
       resources:
         mgr:
           requests:
             cpu: "500m"
             memory: "512Mi"
         mon:
           requests:
             cpu: "1"
             memory: "1Gi"
         osd:
           requests:
             cpu: "1"
             memory: "1Gi"
       dashboard:
         enabled: true
     operator:
       data:
         rookLogLevel: "DEBUG"
- name: harbor
 enabled: true
 values:
   standard:
     namespace: harbor
     harborpass: "password"
     databasePassword: "password"
     redisPassword: "password"
     externalURL: http://10.2.10.11:30002
     nodePort: 30002
     hostname: harbor.local
     harborPersistence:
       persistentVolumeClaim:
         registry:
           size: 40Gi
           storageClass: "rook-cephfs"
         jobservice:
           jobLog:
             size: 1Gi
             storageClass: "rook-cephfs"
         database:
           size: 1Gi
           storageClass: "rook-cephfs"
         redis:
           size: 1Gi
           storageClass: "rook-cephfs"
         trivy: 
           size: 5Gi
           storageClass: "rook-cephfs"
   advanced:
- name: cert-manager
 enabled: true
 values:
   standard:
     namespace: cert-manager
     replicaCount: 3
     logLevel: 2
     secretName: root-secret
   advanced:
- name: ingress-nginx
 enabled: true
 values:
   standard:
     namespace: ingress-nginx
   advanced:
- name: keycloak
 enabled: true
 values:
   standard:
     namespace: keycloak
     storageClass: "rook-cephfs"
     keycloak:
       auth:
         adminUser: admin
         adminPassword: admin
         existingSecret: ""
     postgresql:
       auth:
         postgresPassword: ""
         username: bn_keycloak
         password: ""
         database: bitnami_keycloak
         existingSecret: ""
   advanced:
- name: filebeat-os
 enabled: true
 values:
   standard:
     namespace: logging
   advanced:
- name: logstash-os
 enabled: true
 values:
   standard:
     namespace: logging
     volumeClaimTemplate:
       accessModes: 
         - ReadWriteMany
       resources:
         requests:
           storage: 1Gi
       storageClass: "rook-cephfs"
   advanced:
- name: opensearch-dashboards
 enabled: true
 values:
   standard:
     namespace: logging
     nodePort: 30050
     hostname: opensearch-dashboard.local
     adminPassword: "Password@@123456"
   advanced:
- name: opensearch-os
 enabled: true
 values:
   standard:
     namespace: logging
     hostname: opensearch-dashboard.local
     adminPassword: "Password@@123456"
     opensearchJavaOpts: "-Xmx512M -Xms512M"
     resources:
       requests:
         cpu: "250m"
         memory: "1024Mi"
       limits:
         cpu: "300m"
         memory: "3072Mi"
     persistence:
       size: 4Gi
       enabled: "true"
       enableInitChown: "false"
       labels:
         enabled: "false"
       storageClass: "rook-cephfs"
       accessModes:
         - "ReadWriteMany"
     securityConfig:
       enabled: false
     replicas: "3"
   advanced:
- name: kube-prometheus-stack
 enabled: true
 values:
   standard:
     namespace: monitoring
     privateRegistry: false
     grafanaUsername: "user"
     grafanaPassword: "password"
     grafanaResources:
       storageClass: "rook-cephfs"
       storage: 5Gi
       nodePort: 30211
     prometheusResources:
       storageClass: "rook-cephfs"
       storage: 25Gi
       retention: 10d
       retentionSize: "24GB"
       nodePort: 32090
   advanced:
- name: opa-gatekeeper
 enabled: true
 values:
   standard:
     namespace: opa-gatekeeper
   advanced:
- name: kubeops-dashboard
 enabled: true
 values:
   standard:
     namespace: monitoring
     service:
       nodePort: 30007
   advanced:
- name: velero
 enabled: true
 values:
   standard:
     namespace: velero
     accessKeyId: "your_s3_storage_username"
     secretAccessKey: "your_s3_storage_password"
     useNodeAgent: false
     defaultVolumesToFsBackup: false
     provider: "aws"
     bucket: "velero"
     useVolumeSnapshots: false
     backupLocationConfig:
       region: "minio"
       s3ForcePathStyle: true
       s3Url: "http://minio.velero.svc:9000"
   advanced:
- name: kubevirt-manager
 enabled: true
 values:
   standard:
     namespace: kubevirt
     hostname: kubeops-vms.local
     nodeport: 30733
     ingressClassName: nginx
   advanced: {}
- name: kubevirt
 enabled: true
 hub: kubeops-vm
 values:
   standard:  
     namespace: kubevirt
     imagePullSecretCred:
       registry: registry.preprod.kubeops.net
       user: kubeops
       password: changeme
     kubevirtobject:
       spec:
         configuration:
           developerConfiguration:
             logVerbosity:
               virtLauncher: 2
               virtHandler: 3
               virtController: 4
               virtAPI: 5
               virtOperator: 6
             featureGates:
             - ExpandDisks
             - CPUManager
             - ExperimentalIgnitionSupport
             - HypervStrictCheck
             - Sidecar
             - HostDevices
             - Snapshot
             - VMExport
             - HotplugVolumes
             - HostDisk
             - ExperimentalVirtiofsSupport
             - DownwardMetricsFeatureGate
             - Root
             - ClusterProfiler
             - WorkloadEncryptionSEV
             - VSOCK
             - DisableCustomSELinuxPolicy
             - KubevirtSeccompProfile
             - DisableMDEVConfiguration
             - PersistentReservation
             - VMPersistentState
             - MultiArchitecture
             - AutoResourceLimitsGate
             - AlignCPUs
             - VolumesUpdateStrategy
             - VolumeMigration
             - InstancetypeReferencePolicy
             useEmulation: false
     cdiobject:
       spec:
         config:
           insecureRegistries:
           - registry.preprod.kubeops.net
           scratchSpaceStorageClass: rook-cephfs

4 - Glossary

KOSI package

KOSI package is the .tgz file packaged by bundling package.kosi and other essential yaml files and artifacts. This package is ready to install on your Kubernetes Clusters.

KubeOps Hub

KubeOps Hub is a secure repository where published KOSI packages can be stored and shared. You are welcome to contribute and use public hub also at the same time KubeOps provides you a way to access your own private hub.

Installation Address

It is the distinctive address automatically generated for each published package on KubeOps Hub. It is constructed using name of package creator, package name and package version.
You can use this address at the time of package installation on your Kubernetes Cluster.

It is indicated by the install column in KubeOps Hub.

Deployment name

When a package is installed, KOSI creates a deployment name to track that installation. Alternatively, KOSI also lets you specify the deployment name of your choice during the installation.
A single package may be installed many times into the same cluster and create multiple deployments.
It is indicated by Deployment column in the list of package deployments.

Tasks

As the name suggests, “Tasks” in package.yaml are one or more sets of instructions to be executed. These are defined by utilizing Plugins.

Plugins

KOSI provides many functions which enable you to define tasks to be executed using your package. These are called Plugins. They are the crucial part of your package development.

KUBEOPSROOT Variable

The environment variable KUBEOPSROOT stores the location of the KOSI plugins and the config.yaml. To use the variable, the config.yaml and the plugins have to be copied manually.

apiVersion

It shows the supported KubeOps tool API version. You do not need to change it unless otherwise specified.

Registry

As the name suggests, it is the location where docker images can be stored. You can either use the default KubeOps registry or specify your own local registry for AirGap environments. You need an internet connection to use the default registry provided by KubeOps.

Maintenance Package

KubeOps provides a package for the supported Kubernetes tools. These packages help you update the Kubernetes tools to the desired versions on your clusters along with the dependencies.

Cluster

In computing, a cluster refers to a group of interconnected computers or servers that work together as a single system.

These machines, or nodes, are typically networked and collaborate to execute tasks or provide services. Clusters are commonly used in various fields such as distributed computing, high-performance computing, and cloud computing to improve reliability, scalability, and performance. In the context of technologies like Kubernetes, a cluster consists of multiple nodes managed collectively to deploy, manage, and scale containerized applications.

Container

A container is a lightweight, standalone package that includes everything needed to run a piece of software, including the code, runtime, libraries, and dependencies.

Containers are isolated from each other and from the underlying infrastructure, providing consistency and portability across different environments. Kubernetes manages containers, orchestrating their deployment, scaling, and management across a cluster of nodes. Containers are often used to encapsulate microservices or individual components of an application, allowing for efficient resource utilization and simplified deployment processes.

Drain-node

A Drain Node is a feature in distributed systems, especially prevalent in Kubernetes, used for gracefully removing a node from a cluster.

It allows the system to evict all existing workload from the node and prevent new workload assignments before shutting it down, ensuring minimal disruption to operations.

Kube-proxy

Kube-Proxy, short for Kubernetes Proxy, is a network proxy that runs on each node in a Kubernetes cluster. Its primary responsibility is to manage network connectivity for Kubernetes services. Its main tasks include service proxying and load balancing.

Kubelet

Kubelet is a crucial component of Kubernetes responsible for managing individual nodes in a cluster. It ensures that containers are running in pods as expected, maintaining their health and performance.

Kubelet communicates with the Kubernetes API server to receive instructions about which pods should be scheduled and executed on its node. It also monitors the state of these pods, reporting any issues back to the API server. Kubelet plays a vital role in the orchestration and management of containerized workloads within a Kubernetes cluster.

Node

A Kubernetes node oversees and executes pods.

It serves as the operational unit (virtual or physical machine) for executing assigned tasks. Similar to how pods bring together multiple containers to collaborate, a node gathers complete pods to work in unison. In large-scale operations, the goal is to delegate tasks to nodes with available pods ready to handle them.

Pod

In Kubernetes, a pod groups containers and is the smallest unit managed by the system.

Each pod shares an IP address among its containers and resources like memory and storage. This allows treating the containers as a single application, similar to traditional setups where processes run together on one host. Often, a pod contains just one container for simple tasks, but for more complex operations requiring collaboration among multiple processes with shared data, multi-container pods simplify deployment.

For example, in an image-processing service creating JPEGs, one pod might have containers for resizing images and managing background tasks or data cleanup, all working together.

Registry

Helm registry serves as a centralized repository for Helm charts, facilitating the discovery, distribution, and installation of Kubernetes applications and services.

It allows users to easily find, share, and consume pre-packaged Kubernetes resources, streamlining the deployment process in Kubernetes environments.

Zone

A “zone” typically refers to a subset of the overall cluster that shares certain characteristics, such as geographic location or hardware specifications. Zoning helps distribute resources strategically and can enhance fault tolerance by ensuring redundancy within distinct zones.

5 - Software Bill of Materials (SBOM)

KubeOps publishes a Software Software Bill of Materials (SBOM) for the kubeopsctl command-line tooling that ships in the package repositories. It lists every bundled component with its exact version and license, in CycloneDX format. The published document is the authoritative source — link to it rather than copying its contents, so the list never drifts out of date.

Download

The SBOM is served at a fixed path in each package repository. Both repositories ship the same SBOM document, so the only difference is the retrieval URL for your platform.

curl -fsSLO https://packagerepo.kubeops.net/rpm/sbom/kubeopsctl-2.2.0.0_Beta0-0.x86_64.xml

You can checkout all SBOMs here sbom

curl -fsSLO https://packagerepo.kubeops.net/deb/pool/main/sbom/kubeopsctl-2.2.0.0_Beta0-0.x86_64.xml

You can checkout all SBOMs here sbom

Using the SBOM

The CycloneDX document can be fed directly into SBOM-aware vulnerability scanners and license-compliance tooling without re-resolving dependencies, for example:

grype sbom:./kubeopsctl-bom.xml   # vulnerability scan
trivy sbom ./kubeopsctl-bom.xml   # vulnerability scan

Scanner output reflects advisories at the time you run the scan, not at the time the SBOM was generated. Re-run scans regularly rather than relying on a single point-in-time result.