Kubernetes is a great orchestration tool for your containerized applications and Amazon’s Elastic Kubernetes Service (EKS) provides an easy way to create a scalable Kubernetes cluster in the cloud.
However, security for your containers using the inbuilt Kubernetes secret management tools can be challenging at scale. This is where Conjur steps in to provide a secure way to centrally store your application secrets, as well as set and enforce access policies across your cluster.
Some of the biggest security issues, particularly within Web Application Security Risks (see the OWASP Top Ten Web Application Security Risks for more insight), include sensitive data exposure and security misconfiguration. With large clusters, code and containers that can expose your secrets or data require a centralized approach. Conjur helps reduce these risks by providing authenticators to validate applications (are they who they claim to be?) and role-based access controls (RBAC) to authorize access to resources (are they allowed access?).
To do this, Conjur comes with an authenticator out of the box that lets you map roles to pre-configured identities. This allows you to assign EKS pods to a specific role (see Amazon EKS Adds Support to Assign IAM Permissions to Kubernetes Service Accounts), which Conjur takes advantage of to automatically provide that pod the secrets associated with that role.
Let’s take a look at setting Conjur up to manage the secrets of a simple, containerized application that uses EKS and connects to AWS Redshift.
Setup and Prerequisites
Before we get into how we can use Conjur to manage the secrets of our EKS cluster, let’s go through a few of the setup prerequisites. First, we’ll need to have the following in place and set up from an AWS perspective:
- An AWS EKS cluster with some spare resources to run a java application node.
- An Amazon Redshift data warehouse with some data provided.
- Conjur installed and running. I’m using a simple quickstart installation of Conjur installed using Docker as documented in the cyberark/conjur-quickstart GitHub repository.
Once we have our basic building blocks established, we also need a user created on our Redshift warehouse to allow our application to access it. In my case, I created a Redshift user called “conjuruser” in a group called “clusterusers” with access to my public schema.
IAM Authenticator Configuration
We are going to set up a Conjur instance to manage access to our Redshift warehouse in a similar way to the blog post AWS IAM Authenticator Tutorial For Conjur Open Source.
Before we start creating a configuration in Conjur, we need to set up an AWS role that we can assign to our Kubernetes pods. Conjur will use this role to determine whether the pod is allowed to retrieve the database secrets.
Next, we’ll set up the IAM authenticator in Conjur that lets our app use its AWS IAM role to authenticate using a process similar to the AWS IAM Authenticator documentation.
First, add the following environment variable to enable the connector (where dev is the service ID):
CONJUR_AUTHENTICATORS=authn-iam/dev
We can the create a policy to enable the authenticator by creating the following root-policy.yml file:
# policy id needs to match the convention `conjur/authn-iam/<service ID>` - !policy id: conjur/authn-iam/dev body: - !webservice - !group clients - !permit role: !group clients privilege: [ read, authenticate ] resource: !webservice
Using the Conjur CLI, we load this root policy:
conjur policy load root root-policy.yml
The other policy we need to create is to allow the web application running in EKS to access the secret for Redshift. To do this, we create the following policy that links to the role we created earlier:
- !policy id: redshiftApp body: - &variables - !variable connectionstring - !variable databaseusername - !variable databasepassword # Create a group that will have permission to retrieve variables - !group secrets-users # Give the group permission to retrieve variables - !permit role: !group secrets-users privilege: [ read, execute ] resource: *variables # Create a layer to hold this application's hosts - !layer # The host ID needs to match the AWS ARN of the role we wish to authenticate. - !host 548836857282/EKSRedshiftClusterAccess # Add our host into our layer - !grant role: !layer member: !host 548836857282/EKSRedshiftClusterAccess # Give the host in our layer permission to retrieve variables - !grant member: !layer role: !group secrets-users # Give the application permission to use the IAM authenticator - !grant role: !group conjur/authn-iam/dev/clients member: !host 548836857282/EKSRedshiftClusterAccess
As you can see from this configuration, I am using the role 548836857282:EKSRedshiftClusterAccess. You will need to substitute yours. Save this policy as a yaml file (redshift-app.yml) and run the following in the Conjur CLI:
conjur policy load conjur/authn-iam/dev redshift-app.yml
This command is a little different to the one above as it makes it a child of the conjur/authn-iam/dev policy. This policy allows the IAM role to retrieve the connection string, username, and password to connect to Redshift, but we haven’t loaded those values in yet.
To load the values, use the following commands tot populate the variables (again, substitute your values as required):
conjur variable values add \ conjur/authn-iam/dev/redshiftApp/connectionstring \ jdbc:redshift://redshift-cluster-1.aaaaaaaaaaaa.us-west-2.redshift.amazonaws.com:5439/dev conjur variable values add \ conjur/authn-iam/dev/redshiftApp/databaseusername \ conjuruser conjur variable values add \ conjur/authn-iam/dev/redshiftApp/databasepassword \ Password123456
Kubernetes App Creation
Now that Conjur is ready to go, we are going to create a simple Java Spring application that we can deploy to our EKS cluster. The application tries to connect to Conjur and retrieve the secrets. If this fails, it will catch the error and return it to the page, as below.
This setup also requires the CONJUR_APPLIANCE_URL environment variable to be set and the Conjur servers certificate store to be trusted. These values are typically set in the deployment manifest, as indicated below. You can find the full details in setting this up in the cyberark/conjur-api-java: Java client for the CyberArk Conjur APIrepository.
The next section of the code tries to connect to the AWS Redshift instance using the database credentials we pulled from the Conjur server. I have a small sample database loaded into Redshift with the SQL statements hardcoded. I used the example shown in the Connect to your cluster programmatically Amazon Redshift documentation topic to set the connector to get called every time the /data page is polled.
Now when we package this application up in Docker, deploy to the EKS cluster, and run it we will get the error “Authorization missing”. This is because we have not assigned the IAM role to the Pod running on our cluster.
To do this we need to first add our service account to the deployment manifest under spec:template:spec: with the serviceAccountName. For example, my deployment manifest looks like this:
apiVersion: apps/v1 kind: Deployment metadata: name: redshiftexample spec: replicas: 1 selector: matchLabels: app: redshiftexample template: metadata: labels: app: redshiftexample spec: containers: - name: greeting image: test/redshiftExample imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: "http" env: - name: CONJUR_APPLIANCE_URL value: https://<conjur-hostname> - name: CONJUR_SSL_CERTIFICATE value: | -----BEGIN CERTIFICATE----- … -----END CERTIFICATE----- serviceAccountName: EKSRedshiftClusterAccess
When we run kubectl apply, the AWS EKS admission controller will inject the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables. You can see more about the way this process works in the AWS documentation topic Introducing fine-grained IAM roles for service accounts. We then modify our application to look for this role ARN and use it for the authorization token for our call to Conjur.
When we now deploy this application and access the end point, we get our SQL statement returned to us.
Next Steps
Conjur makes it easy to set up a configuration with EKS and Kubernetes secrets management by leveraging the fine-grained IAM roles for EKS pods without having to rely on AWS for all of our secret management. This makes it easy to securely run applications within Kubernetes while still allowing you to port applications to another platform as the secrets are not locked into Amazon Secrets Manager.
Conjur also provides API’s and secrets management for a wide variety of applications, platforms and languages outside the scope of this example. You can check out all these Conjur features or discuss other CyberArk open source projects in the CyberArk Commons.
Jody has held diverse technical roles in software development, sales and marketing. He has been an enthusiastic promoter of DevOps principles since 2010. In 2015 he became aware of the growing security gap represented by automation. That led him to Conjur, Inc. in January of 2017, just 4 months before the secrets management company’s acquisition by CyberArk. He graduated the University of Texas at Austin with a BA in Computer Science.