Secure CI/CD Pipeline Best Practices

Secure CI/CD Pipelines: Best Practices for Managing CI/CD Secrets

Secure Pipeline Artifacts and Applications by Removing Hardcoded Secrets from CI/CD Configuration

Keeping your applications and infrastructure secure is a significant concern for most organizations. But you need to balance the inertia of ongoing development, maintenance, and deployment against the friction of making sure those tasks are secure and your deployed apps and infrastructure remain secure.

As a streamlined DevOps practice, automated Continuous Integration/Continuous Delivery (CI/CD) pipelines using tools like Jenkins, Concourse, or CircleCI let you create highly configurable build, test, and deployment systems. And you’re not limited to build and test: most CI/CD tools give you the ability to plug in functionality from other tools like performance profilers and error monitoring.

Taking DevOps a step further into operations and Infrastructure as Code (IaC) practices, many teams are now using CI/CD pipelines to manage configuration files for building out the infrastructure their applications run on. This means CI/CD tools are also handling change management, testing, and deployment for Docker files, Kubernetes manifests, Helm charts, and more.

While most discussion of CI/CD focuses on application code — avoiding bugs and crashes — the shift to managing IaC configuration with CI/CD tools highlights another potential problem: managing the secrets needed to run your applications and systems, secrets that can end up being exposed and putting your systems at risk. 

Automating build, test, and deployment tasks takes much of the need for storing and keeping track of secrets off the shoulders of your developers and administrators — which is a valuable first step — but moves responsibility for those secrets into your CI/CD pipeline and associated tools.

In this article, we’ll look at some background concepts and best practices for making sure authentication and authorization secrets are used effectively and securely in CI/CD pipelines.

Identity, Authentication, and Authorization

What do we mean by secrets? Typically these are the authentication details proving the identity of a user or system. Authenticated identities then have authorization to access or use systems based on user or group privileges defined within systems.

Developers need to understand what authentication secrets they need to control, where they are necessary in the CI/CD pipeline, and how they are configured, stored, and managed within the pipeline and deployed infrastructure. 

Secrets are digital credentials that are used to provide identity authentication and authorize access to privileged accounts, applications, and services. 

For example, users authenticate to have access to a source control system like GitHub to commit changes to a codebase, which kicks off test, build, and deployment tasks in a CI/CD pipeline.

Here are some other examples:

  • User or auto-generated passwords
  • API, GitHub tokens, and other application keys/credentials
  • Hard-coded credentials in containerized applications 
  • SSH Keys
  • Private certificates for secure communication, transmitting and receiving of data (TLS, SSL, and so on)
  • Private encryption keys for systems like PGP
  • System-to-system passwords
  • One-time password devices

The Challenge of Secrets in a CI/CD Pipeline

Automated processes are a critical component of DevOps infrastructure. CI/CD orchestration and configuration tools such as Jenkins, Puppet, Ansible, and Chef are increasingly deployed in DevOps processes to improve processes, facilitate faster deployment of software and product delivery, and provide continuous cost reduction. 

However, CI/CD tools are the biggest consumers of secrets and have access to a lot of sensitive resources such as other apps and services and information like codebases and databases. As the number of secrets grows, it becomes harder to store, transmit, and audit secrets securely.

 Furthermore, secrets aren’t just for authentication between tools. Often, secrets need to be provided as part of the build and deployment process so that deployed resources have access. This is particularly important in hybrid cloud and microservices deployments, and with the automated scaling capabilities of tools like Kubernetes.

Common Risks to Secrets

Tools such as Jenkins interface with other systems and applications throughout DevOps environments, and there is the danger of exposed secrets in the clear in CI/CD config files. 

Let’s look at Jenkins as an example. Jenkins’ configurability and use of plugins makes it challenging to securely determine who can use Jenkins at any one time, what Jenkins is allowed to do, and where Jenkins can deploy its artifacts.

Jenkins communicates with a plethora of applications and systems across DevOps environments. Secrets can end up in configuration details within Jenkinsfiles, but Jenkins also includes a central credentials store to manage the credentials needed for each pipeline. Jenkins jobs and plugins can access these credentials at runtime, and the server obscures their usage in the output of each job. 

This creates the challenge of securing access to critical secrets such as passwords, source control and artifact deployment. This causes a security risk if developers leave Jenkins consoles in an insecure state within development environments. 

The Jenkins scripting console can be used to run arbitrary Groovy code and can be found under Manage Jenkins > Script Console or via /script from the root of the Jenkins install path. 

If developers store passwords within Jenkins, they may not be visible from within the web console, but are able to be extracted from the system itself.  

Any Jenkins users with Job/Configure permissions have the opportunity to run any executable on the Jenkins agents and request any set of credentials defined in their scope (global or otherwise) to be injected to agents using the credentials binding plugin.

The credentials binding plugin enables application secrets to be passed from job configurator users to the build agent. For example, users can add a withCredentials statement to their Groovy pipeline script with the credentials ID.

An example pipeline using withCredentials might be:

node {
  withCredentials( [usernamePassword( credentialsId: 'googlecompute', 
                                      usernameVariable: 'USERNAME', 
                                      passwordVariable: 'PASSWORD')]) {
        // build project
        sh 'mvn clean install'
    }
}

Here, the pipeline job instructs Jenkins to grab the Google Cloud credentials by using the ‘amazon’ credentials ID, injecting the Google Cloud username and password into the process as variables USERNAME and PASSWORD, and running a maven build. 

This exposes all credentials in the global scope to any pipeline and any job configurator user. Thus, an attacker can shift from low privilege to highly privileged with access to AWS secret keys, passwords, and git credentials. 

These risks are not exclusive to Jenkins. Any system that allows user-defined build processes and the use of credentials in build processes faces the same potential issue of allowing the exposure of credentials. We need to protect and be confident in the artifacts produced from the CI/CD. 

Compromised secrets means someone could make unwanted changes or leak information. Furthermore, these secrets may be insecurely hard-coded or store insecure configuration files, which jeopardizes security for the entire automation process.

While tools, platforms, and infrastructure may have their own built-in capability to secure secrets, these may lack interoperability, leading to siloed security efforts, making it difficult to track, manage, and audit security effectiveness.

Best Practices for CI/CD Security

The first steps for securing your team’s CI/CD pipeline include locking down configuration managers, systems that host repositories, and the build servers. The pipeline should be monitored from end to end with access control watertight across the entire toolchain. Scripted builds need to be scanned for vulnerabilities, and source code needs to be regularly monitored for vulnerabilities prior to app deployment to production.

The security of secrets needs to apply both during transit and at rest. Best practices include the following:

  • Remove hardcoded secrets from Jenkinsfiles and related CI/CD config files.
  • Have rigorous security parameters, such as one-time passwords, for secrets regarding more sensitive tools and systems.
  • Distribute secrets among Jenkinsfiles to reduce the potential attack target of each file.
  • Use password managers, and rotate passwords after each use.
  • Know who has access to what. Whether it’s role-based, time-based, or task-based, there should be a clear repository of access management. Another option to consider is segmenting secrets based on levels of access. 
  • Machine identity is critical to secure non-human access in containers. Typically, an authenticator certifies that the client run-time container attributes of the requesting container match the native characteristics of the valid container. Once authenticated, the container can access multiple resources based on predefined role-based access control policies. You should destroy containers and VMs after use.
  • Ensure secrets are not inadvertently passed on during builds for pull requests via your CI/CD pipelines.
  • Deploy the practice of least privilege: Give access only to secrets that are requisite. Besides employee access, least privilege also applies to applications, systems, or connected devices that require privileges or permissions to perform tasks. You should regularly audit levels of access to maintain the level of least privilege. 

How CyberArk Helps Solve the Challenge of Exposed Secrets in Jenkinsfiles 

Fortunately, CyberArk provides a way to keep secrets out of your Jenkins master, off disk, and out of source control. CyberArk has provided a Jenkins plugin which you can use to provide credentials to your Jenkins jobs at runtime. The plugin securely provides credentials that are stored in Conjur to Jenkins jobs. Take a look at our GitHub documentation and  the article CI/CD Servers Know All Your Plumbing Secrets to learn how to set up the CyberArk’s Jenkins plugin. 

Securing Your CI/CD Pipeline Next Steps

Security needs to be a top priority in any developer team, especially when considering the key uses of secrets and the specific challenges faced in securing the CI/CD pipeline. We explored some best security practices to ensure that digital authentication credentials are resistant to attacks by malicious agents, and that the privileged accounts and identities remain secure.

Get started today in setting up a Conjur OSS environment and retrieving a secret from Conjur to your application. Learn how to keep secrets out of your Jenkins master, off disk, and out of source control.

Be sure to check out our new hosted interactive tutorials for securing CI/CD pipelines, securing Ansible automation, and securing Kubernetes secrets, and take a look at the Conjur tutorials and blog to learn more. You can also sign up to the newsletter, which highlights interesting open source community news, informative tutorials you can use, and the latest Conjur product news. Join the CyberArk commons community to connect with others to discuss security.