Secrets Management Deployment Tips & Tricks

What I Learned While Setting Up Conjur

Conjur provides excellent policy and command examples. The Getting Started with Conjur guides and Conjur Documentation should be your first stops for getting started with and running Conjur.

Thanks to these resources, I quickly moved beyond the tutorials and started expanding the example policies to create my own policies.

This article shares some insights into how Conjur works and links to useful information that will help you avoid problems and configure your own Conjur deployments successfully as you progress from getting started to implementing secrets management for your own environment.

Database Persistence and the Master Key

When I installed Conjur using the instructions at Setup a Conjur OSS Environment, the second step instructed me to generate a data key, then the third step instructed me to load it to an environment variable. Finally, the instructions told me to start Conjur.

Once I executed these commands (and a few more), I had a working Conjur installation in just a few minutes. But, after making some mistakes while customizing my implementation, I had to restart Conjur to tell it to reload my environment variables. When I restarted, the policy database came up empty, but my policy files survived.

Based on my experience with Docker, I expected that either both the files and the database would both be lost or that both would survive.

For those unfamiliar with Docker, the reason restarting Conjur causes the database to be dropped is that Conjur is built on a PostgreSQL database that runs in a Docker container and the database files are in the container’s filesystem. Manage Data on Docker points out that data in the container’s filesystem does not persist when the container is shut down. This means every time you stop Conjur, all the resources you created in the database disappear.

The Conjur Persistent Database Guide shows how to set up database persistence if you need to change this. Since I was building a demonstration implementation, I didn’t need long-term persistence, but I did need to know to be careful about restarting Conjur.

The reason my policy files survived the restart is the volumes: statement in the docker-compose.yml file (shown below) that overlays the host virtual machine’s directory at ./conf/policy on the Conjur client’s /policy directory.

client:
  image: cyberark/conjur-cli:5
  container_name: conjur_client
  depends_on: [ proxy ]
  entrypoint: sleep
  command: infinity
  volumes:
  - ./conf/policy:/policy

This means policy files are stored on the host virtual machine, not inside the container. In addition to preserving your policy files, this means you don’t need to launch a shell from the Conjur client to load policy files from the host virtual machine. You can just write them to the host directory and the client can read them.

There were a few times when I rebooted the host virtual machine and then forgot to reload the master key to the environment variable before starting Conjur. The Conjur server expects this environment variable to be set and, if it’s missing, the server will not be in a valid operating state.

One of the first symptoms is that you can’t log in as “admin”. This makes sense because a critical resource is missing and you can verify this by reading the error messages in the logs. The server only reads the environment variable when Conjur starts, so changing the variable while Conjur is running has no effect. To change it, you need to restart Conjur to reload the new master key.

Until you implement database persistence, changing the master key and then restarting Conjur does not lock you out of the database because the database is recreated each time you restart. However, if you do implement database persistence, the master key must not change after you start Conjur the first time. If the key changes and you restart Conjur, the new key won’t match the key used to encrypt the database and Conjur won’t be able to decrypt the data. Your only recourse in this case is to rebuild the database.

Good operating practices are the best protection against a lost or damaged master key. It shouldn’t be too hard to rebuild if you have kept good track of your active policy files and secrets, and the way they need to be loaded .

Logs and Logging

The information Conjur writes to the Conjur server container console logs is very useful for understanding the operation of the Conjur server and how it is applying your policies. You can see it using the Docker “logs” command:

docker logs conjur_server

This is very useful for monitoring operations. It provides most of the details you’ll want to see when developing policies. But, sometimes during development, you need all the details about what Conjur is doing.

The “Troubleshooting” section of the Conjur CLI Reference, describes the following Linux command to tell the Conjur REST client to write a record  to a log file for every application programming interface (API) call made to the client:

RESTCLIENT_LOG=/usr/local/conjur.log

This environment variable needs to be set before Conjur starts, which I often forget to do. To overcome my forgetfulness, I’ve found that I can set this environment variable in the client section of the docker-compose.yml file like this:

client:
    image: cyberark/conjur-cli:5
    container_name: conjur_client
    depends_on: [ proxy ]
    entrypoint: sleep
    command: infinity
    environment:
      RESTCLIENT_LOG: /usr/local/conjur.log
    volumes:
    - ./conf/policy:/policy

When I was implementing an Azure Authenticator, I reached a point in development where I was missing a required data item, but I was having trouble working out what it was.

A good solution is setting the CONJUR_LOG_LEVEL environment variable. There are several ways to set it, but I found that I could enable debug logging in the Conjur server by adding the variable to the conjur section of the docker-compose.yml file like this:

conjur:
  image: cyberark/conjur
  container_name: conjur_server
  command: server
  environment:
    DATABASE_URL: postgres://postgres@database/postgres
    CONJUR_DATA_KEY:
    CONJUR_AUTHENTICATORS:
    CONJUR_LOG_LEVEL: debug
  depends_on:
  - database
  restart: on-failure

This caused the Conjur server to write very detailed information about what it is doing. Using this, I was able to see that I was not providing the correct value for the Azure User Assigned Managed Identity in my policy.

Be very careful when making these changes to the docker-compose.yml file because they instruct Conjur to write raw protocol data containing secrets to the log files. You don’t want to enable these in a production environment.

Conjur CLI

Conjur provides a command line interface (CLI) to manage resources. There are three ways to use it:

  • Use docker-compose commands from the Docker host container
  • Log into the conjur_client container
  • Run the CLI in a separate container

All Conjur CLI commands have the form:

conjur <command> <command related options>

When you follow the instructions for installing Conjur OSS, you will execute several commands from the host machine that have the form:

docker-compose exec conjur <command> …

You can use this method to run all CLI commands directly from the host machine. This is useful when you need to run just a few commands, but the extra characters at the start of every command lead to a lot of extra typing, especially if you are creating or installing policies.

Instead, when you need to run many Conjur commands, you can create a shell on the Conjur client container with:

docker exec -it conjur_client /bin/bash

Then you can run the CLI commands directly on the client.

Not everyone should have access to the Conjur client container. Instead, anyone that needs the Conjur CLI can launch it in their own container. The instructions to do this are provided at Conjur CLI Setup.

Using REST APIs

You can use curl to interact with Conjur using the REST APIs. The Conjur documentation provides excellent examples. However, as I have just discussed, the conjur command can only be run in a shell in the Conjur client container or from a separate container running the Conjur CLI.

We can examine the Conjur “whoami” command example to see why this is important. The example is:

curl -H "$(conjur authn authenticate -H)" https://eval.conjur.org/whoami

If you run this command from a shell on a machine that doesn’t have the Conjur CLI installed, you’ll get an error. The reason for the failure is the embedded conjur authn authenticate -H command that returns an authorization header. The header is required for almost every Conjur REST API call and this embedded command returns a header provided curl can call the Conjur CLI.

Troubleshooting Policy Development

When you are developing new policies, you will need to review the policies that you have already created. The Conjur List Resources command will list the objects in the Conjur database by their full name, and the Show a Resource command returns a description of a specific resource.

Some policy problems occur because you are using the wrong name for a resource, while other problems occur because the resource isn’t accessible to the new policy.

The full name of a resource returned by the List Resources command gives you both the complete name of a resource and its location in the policy hierarchy. That tells you how to refer to a resource and what other resources can refer to it.

The Conjur load command is discussed in detail at Load and Manage Policy, but there are two properties that you should be aware of:

  • You can load a policy at a specific level in the policy hierarchy
  • When you load a policy, Conjur creates a new version of it instead of replacing it

When you load a policy, you can load it at the root level or into a branch in the hierarchy. If you load a policy into the wrong branch, your policy and the resources it creates may not be accessible to other policies that you are trying to create.

When loading creates a new version of a policy, resources created by previous versions of the policy can be left in the hierarchy, so you need to understand what resources will be affected by the change you are making. If you expect some resources, like groups, to be removed and they aren’t, you can create security holes in your policy structure.

The load CLI command has several options that can replace, append, or update a policy for times when you need to change the resources related to a policy instead of creating a new version. The REST API uses the HTTP POST, PATCH, and PUT verbs to perform similar operations, in case you need to perform these operations from a web application. The details of how these operations modify resources are discussed in Load and Manage Policy.

Next Steps

This article has touched on many topics of the Conjur product. The most important step is to understand the commands of the Conjur CLI so that you understand what you are telling Conjur to do.

Next, you need to understand the logging options so that you can verify that Conjur is doing what you expect. The Conjur community is always available on the CyberArk Commons forum to help you understand the details of these features.

With all this, you will have the tools you need to begin policy development to secure your secrets. Conjur provides excellent interactive tutorials and examples. Start by modifying the examples to understand what each feature does. You can use the Conjur load command and its associated options to implement your changes and Conjur list command to show the results. With a few hours practice, you’ll master the basics and have many ideas about how to apply Conjur to secrets management.