Resilience Kubernetes components

Ensuring the resilience of Kubernetes components is critical for maintaining a stable and robust cluster environment. Kubernetes supports clusters with up to 5000 nodes, but it is essential to adhere to specific criteria and use advanced features like the HorizontalPodAutoscaler to maintain optimal performance and avoid resource exhaustion.

Recommended Limits for Cluster Resilience

To ensure resilience, it is recommended to adhere to the following limits:

No more than 100 pods per node
No more than 5000 nodes
No more than 150,000 pods in total
No more than 300,000 containers in total

Scaling the cluster by adding or removing nodes as needed helps maintain performance and availability.

Measure M039: Replicas & Autoscaler

HorizontalPodAutoscaler (HPA)

Kubernetes provides a HorizontalPodAutoscaler (HPA) to automatically scale the number of pods in a ReplicaController, deployment, ReplicaSet, or StatefulSet based on observed CPU utilization. The HPA helps prevent excessive resource usage and offers protection against forkbombing attacks.

Protecting Against Forkbombing

Attackers can use commands like kubectl scale to create an excessive number of pods, potentially paralyzing the cluster. For example:

kubectl scale deployment <Name> --replicas=10000000000000000000000000000000

This command would likely crash the cluster. By using the HPA, you can define a maximum number of replicas, preventing such attacks. For example:

kubectl autoscale deployment test --max=3

Enabling the Metrics Server

To fully utilize the HPA, a metrics server is required to provide CPU and memory data. The metrics server can be installed and configured as follows:

Install the Metrics Server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/<find latest one>/components.yaml

Configure the Metrics Server:

Edit the deployment to add necessary arguments

kubectl edit deployment metrics-server -n kube-system

Add the following entries in the args section:

containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP
- --kubelet-use-node-status-port
- --kubelet-insecure-tls

With the Metrics Server enabled, the HPA can function correctly.

Example of HorizontalPodAutoscaler

Create a Deployment and Service:

apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
spec:
selector:
matchLabels:
run: php-apache
replicas: 1
template:
metadata:
labels:
run: php-apache
spec:
containers:
- name: php-apache
image: k8s.gcr.io/hpa-example
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
labels:
run: php-apache
spec:
ports:
- port: 80
selector:
run: php-apache

Apply the above YAML to create the deployment and service:

kubectl apply -f /path/to/deployment.yaml

Create an Autoscaler:

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

Check the autoscalers in the default namespace:

kubectl get hpa

Generate Load to Test the Autoscaler:

Use a BusyBox pod to generate load on the php-apache deployment:

kubectl run -it load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

Monitor the HPA in action:

watch kubectl get hpa

Conclusion

Adhering to recommended limits and using tools like the HorizontalPodAutoscaler and Metrics Server is essential for maintaining the resilience of Kubernetes clusters. By configuring these components correctly, you can prevent resource exhaustion, protect against attacks, and ensure your cluster remains stable and performant. Implementing these best practices helps safeguard your Kubernetes environment against potential threats and operational challenges.

Risks