Mainstream technology enterprises widely use Kubernetes. It’s an extendable, lightweight, open-source container orchestration platform. This popular platform has an ever-expanding ecosystem of security tools, support, and services, making it the go-to platform for managing container assignments and services.
The popularity of Kubernetes is beneficial for its users: the development of a widespread community offers support to developers working with Kubernetes. But in addition to offering support, this prominence brings about risk. It also benefits bad actors who seek vulnerabilities to exploit. Kubernetes containers have several security risks, including runtime threats, vulnerabilities, exposures, and failed compliance audits.
These insecurities motivated CyberArk to develop two open source tools: Kubesploit and KubiScan. These tools benefit the Kubernetes community by performing deep security operations while simultaneously mimicking a real attack. They allow us to test our resiliency. Instead of waiting for attacks to happen, we can proactively prepare and experience how a real-world exploit will affect our systems, then take action to prevent these assaults.
Before we dig into how these tools work and review some examples, let’s briefly explore how each solution helps improve security and learn how to set them up.
Getting Started with Kubesploit
While there are tools available to help mitigate Kubernetes security concerns, most don’t actually perform a comprehensive scan. Seeing this gap, CyberArk created Kubesploit. Developers implement the multi-platform tool Kubesploit in Golang — the Go programming language — for containerized environments.
Kubesploit works by performing complex attack vector coverage on a cluster. This simulation helps us understand our resilience to similar attacks across our network.
Its various modules check for different vulnerabilities. Then, it exploits a vulnerable kubelet and scans ports from the Kubernetes service to the Kubernetes cluster. The Github repository has detailed information about getting started with Kubesploit.
Setting Up Kubesploit
Let’s explore how to scan Kubernetes clusters for known Common Vulnerabilities and Exposures (CVEs). We’ll do this using the K8sClusterSCVEScan. First, let’s load our agent to our root in our first terminal. Then, in our second terminal, we start the server using the command ./server.
Next, we run use module linux/ to load any module we want to use. For instance, the clusterCVEScan module exploits runC to escape to the host. To use the module, we need to set the Kubernetes cluster address and run the cluster to see that cluster’s various vulnerabilities.
In the description attribute of each CVE, we can read a detailed message on what risks to our cluster the scan found.
How to Use Kubesploit
Let’s review an example of how we’ll use Kubesploit to detect vulnerabilities. Before we get started, ensure your system runs the same Go version as in the go.mod file (Go 1.14). Other versions may give you a build constraint error.
First, we load the agent url address. In this case, you’ll use the URL your Kubesploit environment is listening to — for example, http://<address> &.
Then, we run our Kubesploit server using the command ./server. We check if the agent connection was enabled by performing an agent list to the Kubesploit server environment.
To scan for multiple addresses in a URL, we use the PortScan module, like this:
kubesploit[module][PortScan]» run
kubesploit[module][PortScan]»
[-] Created job mbuvGTOXnC for agent bb07d317-5b80-4768-ab55-441edf666dbe at 2021-08-09T11:49:52Z
*** Using `/dev/tty`
*** Stdout redirected
output: [*] Scanning for open ports (15 threads)
[+] Results for bb07d317-5b80-4768-ab55-441edf666dbe job mbuvGTOXnC at 2021-08-09T11:50:14Z
[*] Scanning for open ports (15 threads)
output: [*] Scanning IP: example.com
[+] Results for bb07d317-5b80-4768-ab55-441edf666dbe job mbuvGTOXnC at 2021-08-09T11:50:21Z
[*] Scanning IP: example.com
output: 80: http
[+] Results for bb07d317-5b80-4768-ab55-441edf666dbe job mbuvGTOXnC at 2021-08-09T11:50:27Z
80: http
output: 443: https
[+] Results for bb07d317-5b80-4768-ab55-441edf666dbe job mbuvGTOXnC at 2021-08-09T11:50:34Z
443: https
[+] Results for bb07d317-5b80-4768-ab55-441edf666dbe job mbuvGTOXnC at 2021-08-09T11:50:40Z
*** DONE ***
[+] Results for bb07d317-5b80-4768-ab55-441edf666dbe job mbuvGTOXnC at 2021-08-09T11:50:47Z
Now, let’s minimize our container vulnerabilities by allowing the creation of exempt containers. We use the ContainerBreakoutMounting module:
kubesploit[module][ContainerBreakoutMounting]» run kubesploit[module][ContainerBreakoutMounting]» [-] Created job vTutRmVTNo for agent bb07d317-5b80-4768-ab55-441edf666dbe at 2021-08-09T11:58:57Z [+] Results for bb07d317-5b80-4768-ab55-441edf666dbe job vTutRmVTNo at 2021-08-09T11:59:33Z [*] Trying to mount "/dev/mapper/vgvagrant-root" to "/mnt0" [*] Mounted successfuly "/dev/mapper/vgvagrant-root" to "/mnt0" [*] Host folder is in: "/mnt0"
If the operation is completed, we’ll get a message saying we were successful.
Getting Started with KubiScan
Alongside Kubesploit, CyberArk created KubiScan. Kubiscan is another open-source tool that helps cluster administrators diagnose permissions breaches that might jeopardize the clusters. It scans Kubernetes clusters for risky permissions in Kubernetes’s role-based access control (RBAC) authorization model.
KubiScan spots vulnerable roles and role bindings and identifies their clusters, pods, and subjects. This information enables managers in extensive environments to detect breached permissions that attackers could swiftly compromise.
Learn more about getting started with KubiScan in the Github repository or read an introductory explanation.
Setting Up KubiScan
Now that we know more about KubiScan, we’ll set it up on our master node. We use the command kubectl get pods to find available pods. Then, we search for the pods with vulnerable accounts using kubiscan -rp.
After getting the privileged account, we need to confirm if it appears in the risky subjects list. We do this by running the command kubiscan -rs.
Then, we need to find out how many rules the subject has that enable it to list secrets. To do so, we run the command: kubiscan -aars “<cluster>” -ns “<namespace>” -k “<subject>”.
To get that particular cluster’s token, we list its role binding using kubiscan -aarbs “<cluster>” -ns “<namespace>” -k “<subject>”. A system administrator can use that token to check if the cluster can list secrets while in the default namespace. When we execute that command, it displays a JSON-formatted output with various attributes to show where the vulnerability might be.
How to Use KubiScan
Let’s dive into a practical example of using KubiScan to spot vulnerabilities in a user’s environment.
First, we check our pods’ status using kubectl get pods in our environment. We should see a similar output to the one below, depending on how many pods we have:
vagrant@master:~$ kubectl get pods NAME READY STATUS RESTARTS AGE nginx 1/1 Running 2 40h
We use the command kubiscan -rp to get the vulnerable subjects:
+----------------+ |Risky Containers| +----------+---------+-----------+---------------+-------------------------+--------------------+ | Priority | PodName | Namespace | ContainerName | ServiceAccountNamespace | ServiceAccountName | +----------+---------+-----------+---------------+-------------------------+--------------------+ | CRITICAL | nginx | default | nginx | default | risky-sa | +----------+---------+-----------+---------------+-------------------------+--------------------+
Then, use kubiscan -rs to verify that the account exists in the risky subjects list:
+-----------+ |Risky Users| +----------+----------------+-------------+--------------------------------+ | Priority | Kind | Namespace | Name | +----------+----------------+-------------+--------------------------------+ | CRITICAL | Group | None | system:masters | | HIGH | ServiceAccount | kube-system | cronjob-controller | | HIGH | ServiceAccount | kube-system | daemon-set-controller | | HIGH | ServiceAccount | kube-system | deployment-controller | | CRITICAL | ServiceAccount | kube-system | expand-controller | | CRITICAL | ServiceAccount | default | risky-sa | | CRITICAL | ServiceAccount | kube-system | generic-garbage-collector | | CRITICAL | ServiceAccount | kube-system | horizontal-pod-autoscaler | | HIGH | ServiceAccount | kube-system | job-controller | | CRITICAL | ServiceAccount | kube-system | namespace-controller | | CRITICAL | ServiceAccount | kube-system | persistent-volume-binder | | HIGH | ServiceAccount | kube-system | replicaset-controller | | HIGH | ServiceAccount | kube-system | replication-controller | | CRITICAL | ServiceAccount | kube-system | resourcequota-controller | | HIGH | ServiceAccount | kube-system | statefulset-controller | | CRITICAL | User | None | system:kube-controller-manager | | CRITICAL | ServiceAccount | kube-system | bootstrap-signer | | CRITICAL | ServiceAccount | kube-system | token-cleaner | +----------+----------------+-------------+--------------------------------+
To search for all the rules in a particular service account, we use kubiscan -aars “risky-sa” -ns “default” -k “ServiceAccount”.
We can also list the service account role bindings with kubiscan -aarbs “risky-sa” -ns “default” -k “ServiceAccount”.
Next Steps
Because Kubernetes clusters are distributed and dynamic, they’re vulnerable to attacks and are challenging to secure. To secure our Kubernetes environments, DevSecOps need to use various security tools and techniques throughout an application lifecycle, from building to deploying and runtime.
Because we need to be conscious of our container security, CyberArk’s cybersecurity team is always working on new tools to fill Kubernetes’ security gaps. Solutions like Kubesploit and KubiScan help the Kubernetes community identify and curb various vulnerabilities that hinder our productivity.
To keep Kubernetes secure, developers need a platform that firmly secures and authenticates containers and manages secrets. We need to ensure we’re allowing access to only particular applications, tools, and information. Conjur integrates with Kubernetes — and most DevOps tools — to effortlessly manage secrets as part of your overall Kubernetes security plan.
When you’re ready to go on the offensive to find security vulnerabilities before hackers arrive, try the open-source Kubesploit and KubiScan security tools for yourself.
Going to KubeCon CloudNativeCon Europe or North America?
CyberArk will be at both KubeCon Europe and North America this year as a sponsor, stop by our booth to continue the conversation!
Andy Thompson is part of the Research Labs Division at CyberArk.
This is the team within CyberArk that does ongoing research into offensive technology and information security trends.
Before joining CyberArk 5 years ago, Andy spent his time in the trenches as a Senior Systems Admin and Security Engineer. He’s worked IT and Security with large hospitals, movie theater chains, bars and restaurants, other service industries, as well as global retail organizations.
Andy is CISSP certified by ISC-squared, and a GIAC certified pen-tester.
Andy is also an active member of the Dallas Hacking scene where he’s known as the “Travel-Hacker”. He and his family love to travel all over the world on a shoe-string budget.