Kubernetes has come a long ways since its inception a few years ago, but Kubernetes security has always lagged behind performance and productivity considerations. In this article, we’ll explore some background concepts and best practices for Kubernetes security Clusters with a focus on secrets management, authentication, and authorization.
Many developers have embraced container orchestrator Kubernetes in their efforts to automate critical tasks such as deployment, scaling, and operating application containers across clusters of hosts. Kubernetes facilitates running immutable infrastructure where containers can be destroyed, replaced, and self-healed automatically. It helps reduce deployment friction as you declaratively describe the resources your containers and Pods need, and Kubernetes figures out where to deploy these resources.
However, one of the biggest challenges in containerized environments like Kubernetes is the secure distribution of credentials / secrets needed for applications to do useful work. While Kubernetes offers resources for network security — such as controlling which containers and Pods can communicate with each other over the network — and even provides some built-in basic secrets management capabilities, it doesn’t help secure secrets needed both inside and outside of Kubernetes and only offers limited functionality.
Fortunately, developers can take several key steps to increase the security of their Kubernetes environment. Understanding what authentication and authorization secrets you need to control and their ideal location in your Kubernetes cluster greatly aids security. Knowing the most secure way secrets can be configured, stored, and managed for your deployed pods can simplify the overall process of secrets management and, if configured well, can help make your entire environment more secure.
A Look at Types of Kubernetes Secrets
Secrets are digital credentials used to provide identity authentication and authorize access to privileged accounts, applications, and services. Examples include:
- User or auto-generated passwords
- API, GitHub tokens, and other application keys/credentials
- Hard-coded credentials in containerized applications
- Secure shell (SSH) Keys
- Private certificates for secure communication, data transmitting and receiving, such as transport layer security (TLS) and secure sockets layer (SSL)
- Private encryption keys for systems like Pretty Good Privacy (PGP)
- System-to-system passwords
- One-time passwords for devices
Specifically, a web app container running in a Kubernetes cluster might need to access an API key to communicate with an external API, or a username and password to access a database.
Kubernetes Cluster Secrets Challenges
Kubernetes contains basic secrets management functionality in Kubernetes Secrets. In the context of Kubernetes Secrets, a Secret is an object containing a small amount of sensitive data such as a password, a token, or a key. A Kubernetes Secret can be injected into a Pod container either as an environment variable or a mounted file.
There are two kinds of Secrets in Kubernetes: built-in and customized.
Built-in secrets are automatically created by Kubernetes service accounts and attached to containers along with API credentials. These can be disabled or overridden as needed.
Customized secrets enable you to define your sensitive data and create a custom secret to store it.
While Kubernetes Secrets are safer and more flexible than direct deployment in the Pod or Docker image creation, Kubernetes secrets management has several drawbacks.
Kubernetes Secrets stores usernames and passwords as base-64 encoded strings. Although obscured from casual browsing, text encoding isn’t secure. Further, stored secrets are only visible within the cluster where the secrets are kept.
This isn’t a great fit for many cloud deployment scenarios: organizations commonly have applications running containerized in Kubernetes, but also have other applications built using serverless functions like AWS Lambda and other cloud-native tools.
The “Secret Zero” problem also remains a challenge. Many secrets management solutions rely upon a master key or “secret zero” to protect all the secrets. The master credentials are divided between a role identifier and some other secret information to gain an access token to the vault. However, this simply introduces a new secret zero (a secret zero within a secret zero) because now the vault token needs protection.
Secrets can be leaked via logs, debug records, and application code accessible to other developers or even through source code management tools like GitHub and GitLab.
Then, there is the associated risk of “security islands.” Numerous company-specific platforms, including Kubernetes, lack interoperability with other tools and aggregation of security policies, management, and audit data. Every new tool may have a different security capability and maturity level.
Kubernetes Secrets Management Best Practices
Regardless of where you store secrets, carefully map out exactly which containers need access to each of your secrets. Don’t share secrets anywhere they aren’t absolutely needed.
Using a centralized secrets management solution makes audit, access control, and secrets management more manageable by giving organizations a centralized view of their overall Kubernetes security landscape. This helps understand and control how individual containers and services interact with each other.
To limit unauthorized access and prove compliance, it is important to regularly audit critical systems access. Central audit trails provide visibility into critical security events.
Implement role-based access control (RBAC) and multifactor authentication such as certificates. RBAC authorizes secret access based on a security policy detailing a role-based, time-based, and task-based repository of access management. However, this isn’t enough protection if the secrets provider gives the secret to an imposter. Thus, you need to authenticate containers requesting secrets with strong multifactor authentication using multiple attributes only available to trusted containers, such as certificates.
And a simple rule that’s often overlooked: rotate and change secrets regularly.
Implementing Kubernetes Secrets Management
Kubernetes offers multiple options for using secrets: mounted as the contents of files, environment variables, or in configuration files themselves, either in plain text (a terrible idea) or encoded.
In each case, you and your team need to remember to maintain and update values every time they change. It also generally means that whoever needs to update a secret needs access to wherever you keep your Kubernetes configuration files (probably a repository), which can cause other security issues.
To help secure your secrets, CyberArk Conjur offers a variety of enterprise and open source solutions:
Conjur Open Source eliminates security islands by providing an interface with robust secrets management capabilities for consistently and securely authenticating, controlling, and auditing non-human access across tool stacks, platforms, and cloud environments. Conjur helps organizations secure secrets and implement management best practices, including strong authentication, least privilege, RBAC, credential rotation, management, and auditing.
Conjur’s Kubernetes integration provides a Kubernetes-native solution for protecting secrets as they’re being distributed to containers. The integration distributes secrets securely through mutual TLS established with a secure production identity framework for everyone (SPIFFE) compliant x509 certificate.
Conjur’s Kubernetes integration consists of two pieces, akin to the server and client in the TLS handshake:
- A Conjur plugin adds Kubernetes authentication capabilities. With this plugin installed, Conjur is the server.
- A sidecar container is deployed alongside a user’s application. This handles the authentication with Conjur on behalf of the application. This sidecar container is the client.
The Secretless Broker sidecar simplifies how containerized applications running in Kubernetes securely access databases, HTTPS-based web applications, and servers. It enables you to connect to target systems seamlessly.
When an application needs to access a resource securely, the app makes a local connection request to Secretless Broker. Secretless Broker automatically authenticates the app, fetches the required credentials from the secrets provider, and establishes a connection to the database or other resource. Thus, the application no longer has access to credentials, preventing an accidental leak or exposure.
Instead of using the Kubernetes methods for creating and using secrets, you can instead use Conjur to manage them, and then securely access their values in several ways, Conjur API, Summon, or Secretless Broker.
Using the Conjur API or Summon tool to pull values from Conjur directly, or a file Conjur manages.
DB_USERNAME: !var my-app/db/username DB_PASSWORD: !var my-app/db/password REGION: us-east-1 SSL_CERT: !var:file ssl/certs/private
You can then use the variables in Kubernetes in the normal way with something like kubectl:
kubectl create configmap secrets-config --from-env-file=<secretsfile>
Kubernetes Security Next Steps
This article explored the challenges of secrets management emerging since Kubernetes was created. We explored the limitations of Kubernetes secrets management tools and detailed some of the best practices for ensuring your secrets within and around containers are secure. We also introduced you to the benefits of using Conjur to secure your organization’s Kubernetes Clusters.
Ready to improve your Kubernetes security with k8s secrets management? Get started with this hosted interactive Kubernetes security tutorial powered by KataCoda or get started setting up your Conjur OSS environment to secure your application secrets. You can check out the Conjur technical documentation and GitHub for more information on setting this up. You can also take a look at other secrets management tutorials and blog posts to learn more.
John Walsh has served the realm as a lord security developer, product manager and open source community manager for more than 15 years, working on cybersecurity products such as Conjur, LDAP, Firewall, JAVA Cyptography, SSH, and PrivX. He has a wife, two kids, and a small patch of land in the greater Boston area, which makes him ineligible to take the black and join the Knight’s Watch, but he’s still an experienced cybersecurity professional and developer.