Kubernetes Security: Best Practices for Kubernetes Secrets Management

Kubernetes has come a long way since its inception. But as the adoption of containerization has grown, Kubernetes security continues to be top of mind. Red Hat’s “The State of Kubernetes Security in 2022” report found that 93% of respondents experienced at least one security incident in their Kubernetes environment in the last twelve months. These security incidents can lead to revenue or even customer loss.

But there’s good news. The report also found an increase in DevSecOps teams addressing security issues in the software development lifecycle, including with Kubernetes. 43% of respondents consider DevOps as the role most responsible for Kubernetes security, and 78% have a DevSecOps initiative in the advanced or beginning stages. All that to say, it’s never been a better time to take a look at how to improve Kubernetes security using DevSecOps methodologies, which includes bringing security into the software development lifecycle to help manage secrets used by Kubernetes. In this article, we’ll explore some background concepts and best practices for DevSecOps teams looking to improve their Kubernetes security with a focus on secrets management, authentication and authorization.

Many developers have embraced container orchestrator Kubernetes to automate critical tasks such as deploying, scaling and operating application containers across clusters of hosts. Kubernetes facilitates running immutable infrastructure where containers can be destroyed, replaced and self-healed automatically. It helps reduce deployment friction as you declaratively describe the resources your containers and Pods need, and Kubernetes figures out where to deploy these resources.

However, one of the biggest challenges in containerized environments like Kubernetes is the secure distribution of credentials/secrets needed for applications to do useful work. While Kubernetes offers resources for network security — such as controlling which containers and Pods can communicate with each other over the network — and even provides some built-in basic secrets management capabilities, it doesn’t help secure secrets needed both inside and outside of Kubernetes.

Fortunately, developers can take several key steps to bolster the security of their Kubernetes environment, including:

• Understanding what authentication and authorization secrets you need to control and their ideal location in your Kubernetes cluster.
• Knowing the most secure way secrets can be configured, stored and managed for your deployed pods.

A Look at the Types of Kubernetes Secrets

Secrets are digital credentials used to provide identity authentication and authorize access to privileged accounts, applications and services. Examples include:

• User or auto-generated passwords
• API, GitHub tokens and other application keys/credentials
• Hard-coded credentials in containerized applications 
• Secure shell (SSH) keys
• Private certificates for secure communication, data transmitting and receiving, such as transport layer security (TLS) and secure sockets layer (SSL)
• Private encryption keys for systems like Pretty Good Privacy (PGP)
• System-to-system passwords
• One-time passwords for devices

Specifically, a web app container running in a Kubernetes cluster might need to access an API key to communicate with an external API, or a username and password to access a database.

Kubernetes Cluster Secrets Challenges 

Kubernetes contains basic secrets management functionality in Kubernetes Secrets. In the context of Kubernetes Secrets, a Secret is an object containing a small amount of sensitive data — such as a password, a token or a key. A Kubernetes Secret can be injected into a Pod container either as an environment variable or a mounted file. 

There are two kinds of Secrets in Kubernetes: built-in and customized. 

• Built-in secrets are automatically created by Kubernetes service accounts and attached to containers along with API credentials. These can be disabled or overridden as needed. 
• Customized secrets enable you to define your sensitive data and create a custom secret to store it.

While Kubernetes Secrets are safer and more flexible than direct deployment in the Pod or Docker image creation, there are several drawbacks.

Kubernetes Secrets stores usernames and passwords as base-64 encoded strings. Although obscured from casual browsing, text encoding isn’t secure. Further, stored secrets are only visible within the cluster where the secrets are kept.

This isn’t a great fit for many cloud deployment scenarios. Organizations commonly have applications running containerized in Kubernetes, but also have other applications built using serverless functions like AWS Lambda and other cloud-native tools.

The “Secret Zero” problem also remains a challenge. Many secrets management solutions rely upon a master key or “secret zero” to protect all the secrets. The master credentials are divided between a role identifier and some other secret information to gain an access token to the vault. However, this simply introduces a new secret zero (a secret zero within a secret zero) because now the vault token needs protection. And as the recent Uber breach demonstrated, if the secret zero is compromised, it can have a devastating impact on security. The Red Hat report referenced earlier also found that 53% of respondents had detected a misconfiguration in Kubernetes in the last 12 months. It only takes one misconfiguration (take the Tesla breach) for attackers to gain a foothold in critical systems in resources.

Secrets can be leaked via logs, debug records and application code accessible to other developers or even through source code management tools like GitHub and GitLab.

Then, there is the associated risk of “security islands.” Numerous company-specific platforms, including Kubernetes, lack interoperability with other tools. Every new tool may have a different security capability and maturity level. 

Kubernetes Secrets Management Best Practices 

Regardless of where you store secrets, carefully map out exactly which containers need access to each of your secrets. Don’t share secrets anywhere they aren’t absolutely needed.

Using a centralized secrets management solution makes audit, access control and secrets management more manageable by giving organizations a centralized view of their overall Kubernetes security landscape. This helps you understand and control how individual containers and services interact with each other.  

To limit unauthorized access and prove compliance, regularly audit critical systems access. Central audit trails provide visibility into critical security events.

Implement role-based access control (RBAC) and multifactor authentication, such as certificates. RBAC authorizes secret access based on a security policy detailing a role-based, time-based and task-based repository of access management. However, this isn’t enough protection if the secrets provider gives the secret to an imposter. Thus, you need to authenticate containers requesting secrets with strong multifactor authentication using multiple attributes only available to trusted containers, such as certificates.

And a simple rule that’s often overlooked: rotate and change secrets regularly.

Implementing Kubernetes Secrets Management

Kubernetes offers multiple options for using secrets: mounted as the contents of files, environment variables or in configuration files themselves, either in plain text (a terrible idea) or encoded. 

In each case, you and your team need to maintain and update values every time they change. Plus, whoever needs to update a secret needs access to wherever you keep your Kubernetes configuration files (probably a repository), which can cause other security issues.

To help secure your secrets, CyberArk Conjur offers a variety of enterprise and open-source solutions:

Conjur Open Source eliminates security islands by providing an interface with robust secrets management capabilities for consistently and securely authenticating, controlling and auditing non-human access across tool stacks, platforms and cloud environments. Conjur helps organizations secure secrets and implement management best practices, including strong authentication, least privilege, RBAC, credential rotation, management and auditing. 

Conjur’s Kubernetes integration provides a Kubernetes-native solution for protecting secrets as they’re being distributed to containers. The integration distributes secrets securely through mutual TLS established with a secure production identity framework for everyone (SPIFFE) compliant x509 certificate. 

Conjur’s Kubernetes integration consists of two pieces, akin to the server and client in the TLS handshake: 

• A Conjur plugin adds Kubernetes authentication capabilities. With this plugin installed, Conjur is the server. 
• A sidecar container is deployed alongside a user’s application. This handles the authentication with Conjur on behalf of the application. This sidecar container is the client.

The Kubernetes Authenticator Client can be used to authenticate a Pod to Conjur. You can use the Conjur API or the Summon utility to fetch secrets, creating a connection. 

The Secretless Broker sidecar simplifies how containerized applications running in Kubernetes securely access databases, HTTPS-based web applications and servers. It enables you to connect to target systems seamlessly. 

When an application needs to access a resource securely, the app makes a local connection request to Secretless Broker. Secretless Broker automatically authenticates the app, fetches the required credentials from the secrets provider and establishes a connection to the database or other resource. Thus, the application no longer has access to credentials, preventing an accidental leak or exposure. 

Instead of using the Kubernetes methods for creating and using secrets, you can use Conjur to manage them and then securely access their values in several ways, using Conjur API, Summon or Secretless Broker.

You can also use the Conjur API or Summon tool to pull values from Conjur directly, or a file Conjur manages. 

DB_USERNAME: !var my-app/db/username

DB_PASSWORD: !var my-app/db/password

REGION: us-east-1

SSL_CERT: !var:file ssl/certs/private

You can then use the variables in Kubernetes in the normal way with something like kubectl:

kubectl create configmap secrets-config --from-env-file=<secretsfile>

Kubernetes Security Next Steps

Ready to improve your Kubernetes security with k8s secrets management? Try this hosted interactive Kubernetes security tutorial or get started setting up your Conjur OSS environment to secure your application secrets within and around containers. You can check out the Conjur technical documentation and GitHub for more information on setting this up. You can also take a look at other secrets management tutorials and blog posts to learn more.