One of the most important aspects of today’s cloud infrastructures is secrets management. There are many possible solutions, all of which aim to meet specific requirements, and have their own advantages and drawbacks.
For developers and DevOps teams, while secrets management at runtime is certainly important, secrets management at build time — in the continuous integration/continuous delivery (CI/CD) pipeline — is even more important.
Secrets management in a CI/CD system is a delicate matter. On the one hand, new code is coming in — potentially just for a check, but quite often to trigger an actual release. On the other hand, we expect the outcome to be smoothly going into production. For this, the CI/CD system must be able to communicate with all necessary systems.
There is a neat open-source solution for all things involving on-demand task processing with the Concourse CI open source task manager.
In this article, we’ll cover the process of configuring Concourse CI to pull credentials from CyberArk Conjur.
We assume that Conjur has been set up beforehand — for example, by following the Conjur Docker installation tutorial.
Regardless of how Conjur has been set up, we’ll need the Docker tools (including Docker Compose) for our Concourse CI setup. Details on the Concourse CI installation are provided later in this article.
Managing Secrets
Secret management is one of the most important areas of service implementation in the cloud. However, proper secret management starts before implementation, during development in a local environment. For this reason, a generic vault, such as CyberArk Conjur, is a perfect solution. It decouples the infrastructure management later in the process from the actual use. It also allows creating different policies. For example, it can define that different roles (such as developers) may have access to different kinds of secrets (like the Dev environment).
While access to the secret vault is already crucial in the development stage, it becomes even more important to secure access to machines, containers, and services later on. We’ll look at what can be done to properly secure the secret management for CI/CD. Note that Concourse CI goes far beyond your standard CI/CD needs. You can use Concourse CI to perform a variety of tasks, most of which may be totally unrelated to the application code.
Concourse CI includes built-in support for pulling credentials from a Conjur instance. There are two different modes in which you can use Conjur in Concourse CI:
- Using a shared vault across all teams in Concourse
- Having a separate vault per Concourse team
In the first mode, the Concourse identity is a member of one vault, and all teams within that instance should be able to retrieve all the secrets from that vault.
In the second mode, you actually assign one vault per team. The advantage of this mode is that it allows for more granular control of what secrets each team can access. Even in this mode, it’s still possible to share a vault across several teams.
Concourse CI allows a variety of options to be parameterized with Conjur. We find:
- source of a resource (for example, in resources of a pipeline)
- source of resource_type
- webhook_token of a resource
- params in a task step of a pipeline
This allows us to use Conjur not only for managing strings that should be kept away from everyone’s eyes, but also for managing fixed parameters that may change in the future, such as environment-specific URLs, names, or identifiers.
Configuring Concourse
Spinning up a new local Concourse CI instance can be as simple as getting the Docker Compose file from the official page:
curl https://concourse-ci.org/docker-compose.yml -o docker-compose.yml docker-compose up -d
Now you can log into the overview page on http://localhost:8080 with username “test” and password “test”. The overview page mainly serves as an easy way to get visual insights, as well as to perform some simple actions such as starting or pausing a pipeline.
The fly command line tool can be downloaded from the overview page. Alternatively, get it from the GitHub releases.
You may move fly to your user / global bin folder. Don’t forget to give it execution privileges on linux systems:
chmod +x ./fly The next step is to log in: fly -t tutorial login -c http://localhost:8080 -u test -p test
The -t specifier is a shorthand for –target. This allows the CLI to be used with multiple Concourse CI instances without any conflicts. You just need to use the same target (-t tutorial) later.
Integrating Conjur
There are two primary ways of making Conjur a credential manager for Concourse CI:
- By setting environment variables
- Via command-line arguments
Both of the above have their pros and cons. Usually, you take the path that is more secure for the given infrastructure. Quite often, this is the second command line option because the arguments of the running application are quite well isolated from the rest of the system. The environment variables may be easier to spy out given the right circumstances.
Setting the environment variables can be as easy as calling source vars.sh, where the file vars.sh could be defined as:
export CONCOURSE_CONJUR_APPLIANCE_URL="https://conjur-master.local" export CONCOURSE_CONJUR_ACCOUNT="conjur" export CONCOURSE_CONJUR_AUTHN_LOGIN="host/concourse/dev" export CONCOURSE_CONJUR_AUTHN_API_KEY="YourRealKeyHere"
In reality, these variables should be populated by the infrastructure (rather than hardcoded in the system).
Likewise, you could use the following command line arguments:
concourse web --conjur-appliance-url https://conjur-master.local --conjur-account conjur --conjur-authn-login host/concourse/dev --conjur-authn-api-key YourRealKeyHere
The details of which variable you can change are implementation-specific, so refer to the documentation for more information.
Besides the four settings shown above, you can add optional ones, too. For instance, for the location of the shared vault, you have CONCOURSE_CONJUR_SECRET_TEMPLATE. Being team-specific, you can use CONCOURSE_CONJUR_TEAM_SECRET_TEMPLATE, too. Usually, the default values for these settings are sufficient to get started.
Environment variables also allow you to easily adjust the Docker Compose configuration used to set up Concourse CI for demonstration purposes. All you need to do is change the docker-compose.yml file downloaded fromconcourse-ci.org above.
Here’s an example:
version: '3' services: concourse-db: # remains unchanged concourse: # remains unchanged environment: # remains unchanged CONCOURSE_CONJUR_APPLIANCE_URL: https://localhost:8443 CONCOURSE_CONJUR_ACCOUNT: myConjurAccount CONCOURSE_CONJUR_AUTHN_LOGIN: host/BotApp/myDemoApp CONCOURSE_CONJUR_AUTHN_API_KEY: YourRealKeyHere
Now that Concourse CI is properly set up, you’ll need to provide a policy for secret retrieval in Conjur.
- !host concourse - !policy id: concourse owner: !host concourse body: - !policy id: TEAM_NAME body: - !variable team-secret-variable - !policy id: PIPELINE_NAME body: - !variable pipeline-secret-variable
The TEAM_NAME and PIPELINE_NAME identifiers should be replaced by the values appropriate for your setup.
For instance, if you want to assign this policy to the main team using the sample-pipeline pipeline, the file would contain this configuration:
- !host concourse - !policy id: concourse owner: !host concourse body: - !policy id: main body: - !variable team-secret-variable - !policy id: sample-pipeline body: - !variable pipeline-secret-variable
The policy can be added via the Conjur CLI:
conjur policy load root policy.yml
For demonstration purposes, you also need to set the pipeline-secret-variable variable to some value:
conjur variable values add concourse/main/sample-pipeline/pipeline-secret-variable “foo”
Let’s now create a pipeline to read out our secret variable.
Modifying one of the standard examples for Concourse CI, we get:
---
jobs:
- name: show-env-secret
plan:
- task: show-env-secret
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
run:
path: env
args: []
params:
SECRET: ((pipeline-secret-variable))
You only need to refer to the name of the secret, not the full path. Concourse CI will resolve the path automatically.
You can bring in the pipeline via fly:
fly -t tutorial sp -p sample-pipeline -c pipeline.yml fly -t tutorial up -p sample-pipeline fly -t tutorial trigger-job -j sample-pipeline/show-env-secret -w
This sets up the pipeline, unpauses it, and runs the single job in it.
The output from running the job should be similar to this:
started sample-pipeline/show-env-secret #1 initializing waiting for docker to come up... Pulling busybox@sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df... sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df: Pulling from library/busybox d9cbbca60e5f: Pulling fs layer d9cbbca60e5f: Verifying Checksum d9cbbca60e5f: Download complete d9cbbca60e5f: Pull complete Digest: sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df Status: Downloaded newer image for busybox@sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df Successfully pulled busybox@sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df. running env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root SECRET=foo USER=root succeeded
In case of a configuration problem, or a missing variable, you’d get:
started sample-pipeline/show-env-secret #2 failed to interpolate task config: undefined vars: concourse/pipeline-secret-variable errored
Wrapping Up
Great – you installed, started, and successfully ran Concourse CI. You’ve also managed to drastically simplify credential management in your CI/CD pipeline. Empowered by Conjur, you can now safely use stored values to perform even more tasks.
To learn more about configuration of Concourse CI with Conjur, have a look at official documentation. To get started with Concourse CI really quickly, we’d recommend the post by Tammy Torres and the official tutorial by Stark and Wayne.
Here are some additional sources of information we’d like to suggest:
- Conjur credential manager
- Concourse integration
- Concourse integration with DAP
- Concourse pipeline resources
I am a software engineer who enjoys making developers lives easier and stress free. I believe software should be simple but powerful. I enjoy the outdoors, hiking, fishing and camping.