Why You Need Secrets Management in Your Jenkins Pipeline

Published September 10th, 2021 by John Walsh

Jenkins is a versatile platform for implementing continuous integration and continuous delivery (CI/CD) processes to develop applications. Using plugins, Jenkins supports over 1800 products.

Most of these tools require credentials to access the application’s code or access the resources on which you deploy an application. There are several problems with credentials. Since credentials protect precious resources, we have to control access to them. They need to change periodically, and the resource they protect specifies the form of the credential.

This article is the first in a series that examines these problems and how secrets management tools — more specifically, CyberArk’s Conjur — can help solve them.

Jenkins Pipelines

Jenkins implements CI/CD process through resources called pipelines. These pipelines perform at least three steps: obtain the application code from a source code control system, create the application, and finally, deploy it.

Unless we are working in an uncontrolled environment, a source code control system like GitHub requires credentials to checkout code from a private repository. Also, unless we are using local resources, public cloud providers require credentials to gain access to the resources on which you deploy your applications. This means that we need at least two sets of credentials to build and deploy your application.

A more realistic example might integrate tools from the following categories:

  • Connectors to storage locations. For example, connecting to S3 buckets requires AWS credentials. These credentials can provide access to far more than the bucket. We need similar credentials for Azure and other cloud providers.
  • Artifact and package management to keep track of external resources. For example, Artifactory requires credentials to obtain a server, NuGet requires a credential to publish a package, and credentials to connect to an NPM registry.
  • Source control management to track the application source code. For example, GitHub, credentials are required to clone private repositories. This applies to Bitbucket, Azure DevOps, and AWS Code Commit.
  • Container tools to build, catalog and deploy containerized resources. For example, when using Docker to create containers, referring to Docker, Docker Server Certificate Authentication needs secrets to connect to a Docker server. It needs username and password secrets to connect to a custom registry. AWS ECR can store containers for deployment but needs credentials to upload them.
  • Deployment tools to deliver applications. For example, an Ansible Playbook may require two sets of credentials, one for SSH and a second set to access the Ansible Vault. Using SCP and SSH to deliver a WAR file to an AWS EC2 instance requires a PEM file and a user name. UrbanCode requires a server name, user ID, and password to process a deployment request.
  • Testing tools to validate the applications. For example, SonarQube requires a credential and a webhook secret to run a scan and credential to prepare an environment.
  • Notification tools. For example, Slack requires an integration credential.

We all know the pressure to get the build and deployment process working so that we can validate our code and validate the product design. It’s often easiest to hardcode the secrets into the pipeline scripts, thinking we’ll have time to clean this up later. Yet, we’ll rarely actually have this time, as new features arrive, integration issues arise, testing reveals platform problems, and the project starts to slip.

To prevent slipping the release date, we deem our cleanup efforts “non-essential” and push them out until after product release. This leaves the secrets in the tools, exposed for anyone to use.

Jenkins acknowledges this problem by providing a way to store secrets on the controller Jenkins instance. We can reference them using their credential IDs. This takes the secrets out of the pipeline and improves maintainability by allowing us to define secrets once and reference them symbolically many times. But it doesn’t do a lot to secure the secrets. Any Jenkins Administrator has read/write access to all of them, and every pipeline author can have read access to them.

An Ever-changing Sea of Secrets

The list of secrets in Jenkins is an uncontrolled data set that’s read and used by an unknown set of developers in many projects. We must manage these secrets to ensure the stability of the pipelines. One way is to place them in secured storage so we can control access to them. The minimum requirements for securing secrets are:

  • Storage must be available to many build platforms as there are many projects in progress, and each may be using a specialized DevOps platform.
  • We must limit write access to the secrets. We can’t allow everyone to change secrets.
  • We must audit read access. We must know who is reading which secrets.

This is a good start, but we often must segregate projects. Why should one set of developers be able to read the login credentials for an unrelated project? However, some system users have access to many sets of secrets to deploy products in many environments. For example, the DevOps staff wants to deploy a project to a UAT environment for initial testing and a production environment to “go live.” This extends the requirements list as follows:

  • Roles must control access to credentials.
  • We must segregate credentials by the environment, as a developer may have unlimited access rights to development credentials and no access to production credentials.

CyberArk’s Conjur meets these requirements. It provides secure storage with role-based access control defined by a policy. It’s an independent service using a web-based functional interface that integrates with any CI/CD platform. Conjur keeps policies that control a user’s access to secrets. This has several benefits:

  • We maintain policies outside any CI/CD platform. This means that access to a platform doesn’t provide access to secrets.
  • Changing a user’s relationship to a build platform doesn’t automatically change that user’s access to secrets.
  • There’s only one way to define policies, and it’s independent of any CI/CD platform.

We can integrate Conjur into our CI/CD process over several cycles. Once we have installed Conjur and integrated it with Jenkins, we can start the process of migrating secrets into Conjur by copying the secrets into Conjur and creating Jenkins credentials for them. This makes it so that Jenkins knows how to obtain each secret. The next step is to bind the Jenkins credentials to environment variables in the pipelines and have the pipelines refer to the environment variables.

In the early stages, we can define the Conjur policies to allow broad access to the credentials. However, as usage becomes better understood, we can define roles and update policies to control access according to these roles. We can also use policy branches so that any secret has the same name in all environments. This means that the pipeline script is the same when working with our development environment as it is when working with our production environment. All that’s changing is the name that Jenkins uses to obtain the secret.

Summary

Jenkins is a flexible platform for creating CI/CD pipelines from many tools. But while it does help to pull secrets out of pipelines, it doesn’t provide a good way to secure the secrets that the tools need to do their job.

CyberArk’s Conjur provides secure storage for secrets with role-based access control to ensure that the right people use the secrets at the right time. Conjur integrates with Jenkins to provide symbolic references to secrets, so the secrets never become exposed, and we can maintain them without affecting the pipeline. Finally, Conjur secures the secrets outside of Jenkins on a server accessible to all build platforms, which ensures all pipelines have access to the right secret at the right time according to the Conjur policies.