TUTORIAL – SECURING JENKINS

Purpose

  • Use least privilege: keep secrets out of your Jenkins master, off disk, and out of source control
  • Secure secrets needed by the Jenkins platform & by tasks
  • Automatically authenticate executors every time they run a task
  • Use MAML (Machine Authorization Markup Language, or security policy as code) to authorize all secret accesses

Prerequisites

Giving a Jenkins executor a Conjur machine identity

In Conjur, machines such as Jenkins executors have unique identities. These are represented by entities called hosts, and the machine uses a Conjur API key to identify itself as a host and fetch secrets. Executors do not all share a single identity: they are uniquely identified. In your Conjur security policy, you can assign roles to executors that perform common tasks to keep your automation DRY. In the example below, we do this by assigning the hosts to a layer.

In order to use a Conjur machine identity, each executor needs these things:

  1. A host id (eg host:jenkins/executor-001)
  2. The URL of your Conjur server (eg https://conjur.myorg.com)
  3. An API key corresponding to the host id (eg rq5bk73nwjnm52zdj87993ezmvx3m75k3whwxszekvmnwdqek0r)

These are sufficient to use the Conjur API to fetch secrets, which is covered in greater detail further down.

How to create a new host and get its ID and API key

Conjur’s “host factory” service makes it easy to add new hosts in an automated way. All the human operator has to do is create a short-lived token, which an infrastructure automation tool can use to bootstrap identities on the new hosts. We provide integration for Puppet and Ansible to make this process seamless. (And Application Access Manager also includes conjurize, which works on any host with SSH.)

To set up the host factory, include it in a Conjur policy like this:

- !policy
  id: jenkins-executors-dynamic
  body:
    - !layer
    - !host-factory
      annotations:
        description: automatically enroll new Jenkins executors
      layers: [ !layer ]
    # conspicuously missing: any explicitly-defined !host objects
    # these will be created implicitly by the host factory on-demand

Once you load that policy, you can use the Conjur CLI to create a new host factory token. The token name must match the host factory name declared in the policy. In our policy above, the host-factory is not assigned an id; therefore, the host factory name defaults to the policy name (jenkins-executors-dynamic).

conjur hostfactory token create --duration=1h --cidr=my.ip.addr.ess --hostfactory-id=jenkins-executors-dynamic

This token will allow you to create hosts from your current IP address for only the next hour. As soon as you’re done, you can destroy the token like so:

conjur hostfactory token revoke -t my-token-id-de67a950-42cc-498b-8c18-2549bf4cb3ab

If you’re using the Puppet or Ansible integration to create hosts, the respective documentation for those will explain how to use your token. If you’re configuring hosts using the Conjur CLI, you’d run this on the host:

conjur hostfactory hosts create -t my-token-id-de67a950-42cc-498b-8c18-2549bf4cb3ab -i my-desired-new-host-id-001

This will give you a JSON response back like so:

{
  "created_at": "2017-08-07T22:30:00.145+00:00",
  "id": "myorg:host:my-desired-new-host-id-001",
  "owner": "myorg:host_factory:jenkins-executors-dynamic",
  "permissions": [],
  "annotations": [],
  "api_key": "rq5bk73nwjnm52zdj87993ezmvx3m75k3whwxszekvmnwdqek0r"
}

This id and api_key will allow the new host to authenticate to Conjur and access all the secrets permitted to the layer in the policy. Below we’ll show an example of permitting a layer to access secrets.

Static hosts using Conjur policy

If you have a static set of executors that you want to manage individually, you can define them in a Conjur policy. We don’t recommend this because it doesn’t scale easily with automation tools, while the host factory does.

- !policy
  id: jenkins-executors-static
  body:
    - !layer
    - !host
      owner: !layer
      id: my-openbsd-executor
    - !host
      owner: !layer
      id: my-gnu-linux-executor

When I load this policy, the API will give me back the definitions of the created hosts, like the JSON response shown above.

To add a third host, I can extend & reload the policy. Because the hosts have the layer as their owner, they can fetch any secrets permitted to that
layer.

Storing secrets for Jenkins tasks in Conjur

In order to store secrets in Conjur, you need to create a policy that defines variables for them to reside in. This is like defining the schema of a database.

Note: we don’t recommend storing complex schema-less secrets in Conjur (such as a JSON string containing many secrets) because this sacrifices access granularity and makes rotation more difficult. Creating a policy with one Conjur variable per secret will make your data easier to work with.

Suppose that Jenkins has a pipeline like this:

  • Pull a branch from GitHub
  • Build a Docker image from that branch
  • Run tests on that container
  • Push to DockerHub if tests pass

Example Conjur policy

To perform its duties, our task will require an SSH private key for GitHub and a service API key for DockerHub.

Note for teams with many Jenkins pipelines: in order to provide for separation of maintenance duties, we can create a separate policy for each different pipeline and give control of the policy to the security team. In this example we’re assuming just one pipeline which needs two secrets.

continuous-integration-task.yml
- !policy
  id: continuous-integration-task
  annotation: example Jenkins continuous integration task
  body:
    - !variable github-ssh-key
    - !variable dockerhub-api-key
  # conspicuously missing: a group that can retrieve these secrets. covered in next section.

# define a group for maintainers of this task
- !group continuous-integration-task-admins
- !grant
  role: !policy continuous-integration-task
  members: [ !group continuous-integration-task-admins ]

Authorizing Jenkins executors to fetch necessary secrets from Conjur

Now we’ve got secrets stored in Conjur, so let’s look at how our Jenkins tasks can get them back out again.

By defining pipeline stages as shell scripts, we can make the Jenkinsfile easy to read and enable developers to easily run the same steps Jenkins would, as desired for their own purposes. This pattern has simplified our CI/CD and works well for fetching secrets from Conjur.

Example Jenkinsfile

The Jenkinsfile for this task would be checked into the source code repository for the software to be tested. It could look something like this:

#!/usr/bin/env groovy

pipeline {
    agent { label 'executor' }
    
    stages {
        stage('Build container image') {
            steps {
                sh './build.sh'
            }
        }
        
        stage('Test container') {
            steps {
                sh './test.sh'
            }
        }
        
        stage('Push container to DockerHub') {
            steps {
                sh './push.sh'
            }
        }
    }
}

Note: the Jenkins master doesn’t have to know anything about Conjur and this
workflow requires no special Conjur plugin for Jenkins.

In our “continuous-integration-task” policy we define variables but, for simplicity’s sake, didn’t permit any roles to retrieve them. With a policy loaded as such, only the Conjur administrator and the
continuous-integration-task-admins can retrieve the secrets. But in an automated system, we want to allow the hosts whose identities we’ve created to retrieve the secrets as well, for all the tasks that they need to run.

So to address that issue let’s go more in depth. The pattern we use for this is to create a group of “readers” for the secrets in each policy, then map hosts to groups of readers. We call these maps “entitlements.” A policy is relatively abstracted and self-isolated, while entitlements incorporate your actual infrastructure and business logic to hook things together.

continuous-integration-task.yml with readers group and entitlements
- !policy
  id: continuous-integration-task
  annotation: example Jenkins continuous integration task
  body:
    - !variable github-ssh-key
    - !variable dockerhub-api-key
    
    # create "readers" group for variables
    - !group readers
    - !permit
      resource: [ !variable github-ssh-key, !variable dockerhub-api-key ]
      privilege: [ read, execute ]
      role: !group readers

# define a layer for executors
- !policy
  id: jenkins
  body:
    - !layer executors

# add an entitlement for our executors layer
- !grant
  role: !group continuous-integration-task/readers
  members: [ !layer jenkins/executors ]

Fetching secrets from Conjur on the executor

The easiest way to use the Conjur API to fetch secrets for your tasks is to use Summon. This tool makes access to secrets convenient for tasks built using the 10-factor CI guidelines.

Using Summon

Summon is a tool we designed to allow software convenient access to secrets, wherever they may be. Summon has a provider that lets it fetch secrets from Conjur using the local machine identity.

In order to use Summon on an executor, you’ll need to:

  1. Install summon
  2. Install summon-conjur, the Conjur provider
  3. Configure the executor to provide summon-conjur with its machine identity, as described in the documentation

Summon has two methods of secret delivery: the environment or a memory-mapped temporary file. To use it, you write a secrets manifest which describes what secrets you want to fetch and where you want to put them.

For our example, we will want to use Summon to fetch the DockerHub API key and put its value in the environment. We’ll also want to get the GitHub SSH private key and put it in a memory-mapped file so SSH can use it. If desirable, we can do both things in one shot with a secrets manifest file like so:

secrets.yml
DOCKERHUB_API_KEY: !var continuous-integration-task/dockerhub-api-key
GITHUB_SSH_KEY_FILE: !var:file continuous-integration-task/github-ssh-key

Then run summon bash to get a shell with ephemeral access to the secrets. Once the Summon inferior process exits, the secrets will immediately disappear.

Inside the inferior process, the DOCKERHUB_API_KEY environment variable contains the literal API key (eg 984b69b5-aba6-40ae-92ae-db8df236bdcf) and the GITHUB_SSH_KEY_FILE variable contains the path of the memory-mapped file (eg/home/conjur/.tmp380531188/.summon092665027.)

example session
$ summon bash -c 'echo $DOCKERHUB_API_KEY'
984b69b5-aba6-40ae-92ae-db8df236bdcf
$ summon bash -c 'echo $GITHUB_SSH_KEY_FILE'
/home/conjur/.tmp380531188/.summon092665027
$ summon bash -c 'cat $GITHUB_SSH_KEY_FILE'
-----BEGIN RSA PRIVATE KEY-----
MIIJJwIBAAKCAgEAwSsWlvNcwiK0qQ4LCf9jSxfXLut5RqtbLkXMwpMsWAr4ynzx
...and so on...
U0KTgOZQU01La6rHRCPAlkzF0IxS97Ic94x/+k+G9kmuprmQg/uxMlVaLA==
-----END RSA PRIVATE KEY-----
assumption check & troubleshooting

These examples assume that you:

  • Have a single Summon provider (such as summon-conjur) installed in the default location, /usr/local/lib/summon/. If that’s not true, you need to specify -p /path/to/your/provider.
  • Are in the same directory as a file called secrets.yml with the above contents. If not, you need to specify -f /path/to/your/secrets/manifest.yml.
  • Stored the secrets in Conjur as per the procedures and Conjur policy described above. If not, load the policy and the secrets into your Conjur appliance.
  • Have a local Conjur identity (user or host) with authorization to access the secrets, as described in the Conjur policies above. If not, you need to install the correct identity as per the Summon docs.

using Summon without a secrets manifest

Sometimes, usually in a lightweight scripting context, it’s desirable to use Summon without writing out a secrets manifest on disk.

For example:

$ summon --yaml 'DOCKERHUB_API_KEY: !var continuous-integration-task/dockerhub-api-key' \
         env | grep -i dockerhub
984b69b5-aba6-40ae-92ae-db8df236bdcf

Note that this works without having a secrets.yml file. Instead, the secrets manifest YAML is taken directly from the string provided on the command line.

Use case: use git to fetch source code with an SSH private key from Conjur

Often, our Jenkins pipelines triggered by source control web hooks want to fetch the latest source code. In order to keep our sensitive GitHub SSH private keys off disk, we can use Summon to provide git with a memory-mapped key like so:

$ summon --yaml 'GITHUB_SSH_KEY_FILE: !var:file continuous-integration-task/github-ssh-key' \
         bash -c \
         'GIT_SSH_COMMAND="ssh -i $GITHUB_SSH_KEY_FILE" git clone [email protected]:cyberark/conjur.git'

Use case: use docker to log in to a registry and push an image

After testing software in our CI pipeline, we often want to push the built image to a repository for further distribution. This requires credentials that we’d rather not store on the executors, so let’s look at an example script that fetches the secret from Conjur on demand.

secrets.yml
DOCKER_USER: myorg-jenkins
DOCKER_PASSWORD: !var continuous-integration-task/dockerhub-api-key

Note that this secrets manifest uses static values for the DOCKER_USER, since it isn’t a secret and doesn’t need to be protected or rotated. The value “myorg-jenkins” will be used literally instead of interpolated with a secret.

push.sh
#!/bin/bash

summon docker login --username $DOCKER_USER --password $DOCKER_PASSWORD

image=myorg/app
tag=v1.2.3

docker tag 0e5574283393 $image:$app
docker push $image:$app

Summary

Conjur facilitates workflows that have the fewest, narrowest privileges possible. This allows you to keep secrets out of your Jenkins master, off disk, and out of source control, reducing the attack surface area of your continuous integration stack.

You can secure secrets needed by the Jenkins platform & by tasks in the Conjur vault without making it inconvenient for your infrastructure to access them when necessary.

Using Conjur MAML policy, you can automatically authorize every secret access by your build executors using their Conjur machine identities.

If you’re interested in using Jenkins and Conjur together to improve your DevOps security, please get in touch on Slack or on GitHub. We always welcome your questions and suggestions.