Blogs

Kubernetes operators can be useful in stateless applications – however, a few critical security aspects must be considered. 

For example, considering an operator with a certificate manager role, if a hacker is able to exploit a vulnerability in a payload container, he might be able to then attack other containers in the same namespace as well. One of our customers was concerned about the potential security implications, so our team determined that we needed to explore operator security risks further.   

This article focuses on the use of operators in a production ready environment and the associated security aspects, which are worth considering before you use Kubernetes operators. We will also review a practical example using a certificate manager operator.  

How operators work  

Before we discuss security, let’s talk about how operators work. Simply put, operators streamline and simplify working with Kubernetes. An operator-pod is initially deployed, which contains the secrets of the specific user as required for effective communication between the operator-pod and the etcd. This read and write access of information within the etcd is possible (given the specific user access rights).  

In certain circumstances, a user might create custom resource definitions (CRD) within the etcd, which may contain schemes to deploy resources in the cluster. With the kubectl command, a user can activate those CRDs, storing the files in the etcd. From there, operators can be triggered from the new file in the etcd and create any deployment through the kube API into the cluster.  

The functionality is shown in the picture below. 

Operator risks and preventative measures  

The main security risk regarding operator pods is their innate ability to deploy other pods. Even an operator pod with low rights may deploy a pod with higher rights than itself. Given the way Kubernetes works, pods may communicate with each other directly and trigger an escalation of security risks. In a worst case scenario, even the host machine might be affected. That's why a strict separation between the application being deployed by the Operator and the Operator itself is necessary.  

Basic prevention 

Operators are pods that can directly communicate with the Kubernetes API. Operators should have no, or when required only a limited internet access. Payloads are pods that cannot directly communicate with the Kubernetes API. Payloads may access the internet when required and be accessed from the internet as well.  

With operator risks in mind, applying some security best practices is beneficial. These best practices are designed to limit the potential Kubernetes operators have for turning rogue and being used as part of a hacker’s strategies.  

Risk-reduction strategies for using Kubernetes operators  

1. Keep payloads and operators separate 

Using network policies, namespaces should be strictly separated and the operator namespace should not contain payload pods. Additionally, the payload should be in a different namespace than the operator. A faulty separation of namespaces allows the operator to deploy pods with the ability to access pods in other namespaces.  

1. Quarantine the API 

Since the operator can communicate with the API while using elevated rights and deploying pods, it is a popular attack vector and poses a significant threat to security unless it is managed properly through your network policies. 

1. Designate a service account for the operator 

With a minimalistic service account, the operator is able to deploy the payload in its intended namespace while protecting other namespaces from possible security risks.  

1. Prevent the operator from creating PSPs and PSAs 

By preventing operators from creating new network policies, you can proactively reduce the risk of access escalation. By using a self-created PSP, the operator might deploy a pod that triggers a privileged escalation that might even affect the host machine. Operators should only be able to create network policies within their own namespace, and nowhere else. Only admins should be able to create PSPs (pod security policies) and PSAs (pod security admissions).  

If the operator gains the ability to create pod security policies, this might have widespread consequences. During pod deployment, you should always choose the pod security policy with the lowest restrictions.  

1. Restrict CRD creation from the operator 

Deploying an admission controller in the operator namespace to deny creation of custom resource definition (CRD) objects is necessary to protect against the operator growing its own privileges or deploying payloads.   

1. Use a custom resource to validate assigned rights 

Controlling and validating assigned rights helps prevent software resources from misusing service accounts. A custom resource (CR) can be used to validate assigned rights at any time. Using these strategies may safeguard your operators – giving them the access and permissions they need to function properly, while simultaneously reducing the likelihood that they may be exploited to break in and escalate access.  

An example scenario: Certificate manager  

In this section, we'll show an example operator with the role of certificate manager and how a vulnerability may be exploited to gain control of a container. 

The Cert-Manager must access secrets to work as intended, but this access makes it potentially vulnerable. If a hacker is able to exploit a payload container vulnerability, he might be able to attack other containers in the same namespace as well. A Cert-Manager bug, for instance, may be used to gain control of its container. Since the Cert-manager (operator) has access to the Kubernetes-API, a hacker might be able to attack the API as well. This means he might gain access to the secrets.  

By accessing the secrets, he could obtain access to service accounts with higher privileges. Those rights might allow him to start higher privileged pods resulting in a privilege escalation. Should there be no PSPs to prevent access to privileged containers, he could even take over the host.  

The following picture shows the namespace separation.  

In reviewing this example, we hope you’ll see just how potentially vulnerable this Cert-manager operator is. A hacker took what vulnerability was already present and manipulated a bug to gain control of the container and ultimately, the API itself. Knowing that the potential risk is real, Kubernetes operators’ access and permissions should be appropriately restricted and managed. Using operator security best practices is a sensible first step towards reducing these risks.