Cloud Native IAM EKS Secrets Management for Kubernetes

Kubernetes is a great orchestration tool for your containerized applications and Amazon’s Elastic Kubernetes Service (EKS) provides an easy way to create a scalable Kubernetes cluster in the cloud.

However, security for your containers using the inbuilt Kubernetes secret management tools can be challenging at scale. This is where Conjur steps in to provide a secure way to centrally store your application secrets, as well as set and enforce access policies across your cluster.

Some of the biggest security issues, particularly within Web Application Security Risks (see the OWASP Top Ten Web Application Security Risks for more insight), include sensitive data exposure and security misconfiguration. With large clusters, code and containers that can expose your secrets or data require a centralized approach. Conjur helps reduce these risks by providing authenticators to validate applications (are they who they claim to be?) and role-based access controls (RBAC) to authorize access to resources (are they allowed access?).

To do this, Conjur comes with an authenticator out of the box that lets you map roles to pre-configured identities. This allows you to assign EKS pods to a specific role (see Amazon EKS Adds Support to Assign IAM Permissions to Kubernetes Service Accounts), which Conjur takes advantage of to automatically provide that pod the secrets associated with that role.

Let’s take a look at setting Conjur up to manage the secrets of a simple, containerized application that uses EKS and connects to AWS Redshift.

Setup and Prerequisites

Before we get into how we can use Conjur to manage the secrets of our EKS cluster, let’s go through a few of the setup prerequisites. First, we’ll need to have the following in place and set up from an AWS perspective:

  • An AWS EKS cluster with some spare resources to run a java application node.
  • An Amazon Redshift data warehouse with some data provided.
  • Conjur installed and running. I’m using a simple quickstart installation of Conjur installed using Docker as documented in the cyberark/conjur-quickstart GitHub repository.

Once we have our basic building blocks established, we also need a user created on our Redshift warehouse to allow our application to access it. In my case, I created a Redshift user called “conjuruser” in a group called “clusterusers” with access to my public schema.

IAM Authenticator Configuration

We are going to set up a Conjur instance to manage access to our Redshift warehouse in a similar way to the blog post AWS IAM Authenticator Tutorial For Conjur Open Source.

Before we start creating a configuration in Conjur, we need to set up an AWS role that we can assign to our Kubernetes pods. Conjur will use this role to determine whether the pod is allowed to retrieve the database secrets.

Next, we’ll set up the IAM authenticator in Conjur that lets our app use its AWS IAM role to authenticate using a process similar to the AWS IAM Authenticator documentation.

First, add the following environment variable to enable the connector (where dev is the service ID):

CONJUR_AUTHENTICATORS=authn-iam/dev

We can the create a policy to enable the authenticator by creating the following root-policy.yml file:

# policy id needs to match the convention `conjur/authn-iam/<service ID>`

- !policy

  id: conjur/authn-iam/dev

  body:

  - !webservice

  - !group clients

  - !permit

    role: !group clients

    privilege: [ read, authenticate ]

    resource: !webservice

Using the Conjur CLI, we load this root policy:

conjur policy load root root-policy.yml

The other policy we need to create is to allow the web application running in EKS to access the secret for Redshift. To do this, we create the following policy that links to the role we created earlier:

- !policy

  id: redshiftApp

  body:

- &variables

  - !variable connectionstring

  - !variable databaseusername

  - !variable databasepassword

# Create a group that will have permission to retrieve variables

- !group secrets-users

# Give the group permission to retrieve variables

- !permit

        role: !group secrets-users

        privilege: [ read, execute ]

        resource: *variables

# Create a layer to hold this application's hosts

- !layer

# The host ID needs to match the AWS ARN of the role we wish to authenticate.

- !host 548836857282/EKSRedshiftClusterAccess

# Add our host into our layer

- !grant

        role: !layer

        member: !host 548836857282/EKSRedshiftClusterAccess

# Give the host in our layer permission to retrieve variables

- !grant

        member: !layer

        role: !group secrets-users

# Give the application permission to use the IAM authenticator

- !grant

  role: !group conjur/authn-iam/dev/clients

  member: !host 548836857282/EKSRedshiftClusterAccess

As you can see from this configuration, I am using the role 548836857282:EKSRedshiftClusterAccess. You will need to substitute yours. Save this policy as a yaml file (redshift-app.yml) and run the following in the Conjur CLI:

conjur policy load conjur/authn-iam/dev redshift-app.yml

This command is a little different to the one above as it makes it a child of the conjur/authn-iam/dev policy. This policy allows the IAM role to retrieve the connection string, username, and password to connect to Redshift, but we haven’t loaded those values in yet.

To load the values, use the following commands tot populate the variables (again, substitute your values as required):

conjur variable values add \

  conjur/authn-iam/dev/redshiftApp/connectionstring \

  jdbc:redshift://redshift-cluster-1.aaaaaaaaaaaa.us-west-2.redshift.amazonaws.com:5439/dev

conjur variable values add \

  conjur/authn-iam/dev/redshiftApp/databaseusername \

  conjuruser

conjur variable values add \

  conjur/authn-iam/dev/redshiftApp/databasepassword \

  Password123456

Kubernetes App Creation

Now that Conjur is ready to go, we are going to create a simple Java Spring application that we can deploy to our EKS cluster. The application tries to connect to Conjur and retrieve the secrets. If this fails, it will catch the error and return it to the page, as below.

This setup also requires the CONJUR_APPLIANCE_URL environment variable to be set and the Conjur servers certificate store to be trusted. These values are typically set in the deployment manifest, as indicated below. You can find the full details in setting this up in the cyberark/conjur-api-java: Java client for the CyberArk Conjur APIrepository.

The next section of the code tries to connect to the AWS Redshift instance using the database credentials we pulled from the Conjur server. I have a small sample database loaded into Redshift with the SQL statements hardcoded. I used the example shown in the Connect to your cluster programmatically Amazon Redshift documentation topic to set the connector to get called every time the /data page is polled.

Now when we package this application up in Docker, deploy to the EKS cluster, and run it we will get the error “Authorization missing”. This is because we have not assigned the IAM role to the Pod running on our cluster.

To do this we need to first add our service account to the deployment manifest under spec:template:spec: with the serviceAccountName. For example, my deployment manifest looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redshiftexample
spec:
  replicas: 1
  selector:
    matchLabels:
    app: redshiftexample
  template:
    metadata:
    labels:
      app: redshiftexample
    spec:
      containers:
      - name: greeting
      image: test/redshiftExample
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 8080
          name: "http"
        env:
          - name: CONJUR_APPLIANCE_URL
            value: https://<conjur-hostname>
          - name: CONJUR_SSL_CERTIFICATE
            value: |
              -----BEGIN CERTIFICATE-----
                           -----END CERTIFICATE-----
      serviceAccountName: EKSRedshiftClusterAccess

When we run kubectl apply, the AWS EKS admission controller will inject the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables. You can see more about the way this process works in the AWS documentation topic Introducing fine-grained IAM roles for service accounts. We then modify our application to look for this role ARN and use it for the authorization token for our call to Conjur.
When we now deploy this application and access the end point, we get our SQL statement returned to us.

Next Steps

Conjur makes it easy to set up a configuration with EKS and Kubernetes secrets management by leveraging the fine-grained IAM roles for EKS pods without having to rely on AWS for all of our secret management. This makes it easy to securely run applications within Kubernetes while still allowing you to port applications to another platform as the secrets are not locked into Amazon Secrets Manager.

Conjur also provides API’s and secrets management for a wide variety of applications, platforms and languages outside the scope of this example. You can check out all these Conjur features or discuss other CyberArk open source projects in the CyberArk Commons.