Containerization has taken the technology world by storm. Docker emerged as the de-facto leader in the base technology making it all possible. Because containerization necessitates consistency throughout your infrastructure, even a tiny pinhole in your security defenses can quickly turn your happy cloud into a malicious thunderstorm.
Let’s introduce some Docker security concepts and show you some of the most used attack vectors, and how you can protect your systems against them.
Containers vs VMs
The main purpose of containers and VMs is to increase the computing density per unit of physical hardware compared to running services directly on bare metal. With the introduction of VMs, you can run multiple OS instances on a single physical host. Realistically, as you scale your VMs up, you will likely hit a limit on your RAM and disk usage due to each VM running a full-blown independent OS, leaving your CPU to be underutilized in most cases.
From a security standpoint, the additional attack surfaces originally introduced were in the hypervisors and the companion CPU virtualization hardware/extensions (Intel VT, AMD-V, etc.). With the help of the above hardware extensions coupled with the now-mature level of the technology (and the fact that running a VM is very similar to running real servers) there are not a lot of additional security concerns lately about using VM-based infrastructure.
Containers drastically change the security picture. A container essentially collapses all OS layers into just one via a shared kernel, and the main split between each container ends up being only the service itself.
This multiplies the computing density immensely when compared with the VM approach. With the increase in speed, efficiency, and code portability development teams have been gaining a massive amount of productivity. So what’s not to like? Sadly, this new architecture presents new areas for attackers to try to breach.
Attacking the Docker daemon
While most successful attacks on VMs are executed through application code running on the VM itself, a large number (and perhaps even the majority) of attacks on Docker-based infrastructure are through a badly configured daemon or container. Great care needs to be taken when dealing with the Docker infrastructure itself, on top of dealing with any application-related attack vectors.
The ability to create containers on a host is usually enough to get root access to the machine. Since the Docker daemon runs with elevated privileges, you can easily create a container that has a volume mounted of the host’s root filesystem. This would allow you to change the hosts filesystem arbitrarily. While some operating systems limit the paths you can mount, in general you should assume that this type of access is equivalent to root access needing minimal work from the attacker. In production environments, requiring elevated privileges for working with containers is practically mandatory.
Just for fun, try to run this in your local account:
$ # Note: Use `sudo` if you are on a machine where it is required $ docker run --rm -t -v $HOME:/tmphome alpine ls -la /tmphome/.ssh
If you have ssh keys, they would be listed in the terminal. A little scary, no?
Another common mistake with Docker is tacking on the
--privileged flag to containers that do not need it. Since Docker runs most things through a filter on syscalls you can make, it is often needed to allow extra privileges to the container (many times due to inability to mount loop files, use
chroot, etc) but what happens in a lot of cases with actual deployments is that instead of finely tuning the SECCOMP flags, the
--privileged flag is used which whitelists all of those flags and allows the container to practically have unlimited privileges.
If this wasn’t enough, there is one more common attack vector that is caused by a desire for convenience: the mounting of Docker’s socket to the container (or relaxing its permissions). Similar to the risks of allowing CLI access to run a container, mounting Docker’s socket to the container has the same implications. This is because Docker’s CLI is just a thin wrapper to the endpoint that the socket provides. So if you ever see any
... -v /var/run/docker.sock ... commands, you should avoid using them whenever possible (you can read more about this in my book). Do note that some container management tools need this access to function and manage other Docker instances, but it is definitely an overused and very risky practice.
Another often neglected issue with Docker configuration is what I term as “UID/GID mapping privilege escalation”. If you have a volume mounted from the host in your container, the container may add malicious files to that volume that you may later activate with or without actions. You might be surprised by how many programs scan your files regularly regardless of your OS! Because UID of
0 (root) in the container maps to UID
0 (root) on the host when you use default settings, accidental execution through command masking, naming tricks, or scanning program exploits could trivially compromise your host.
As with most containerization technologies, your container security relies heavily on your host’s kernel security. Since the Linux kernel is pretty tried-and-tested you normally don’t have to worry about this attack vector. But do be aware that without regularly patching your host or having a very rare configuration of your kernel, your new kernel install will not be in use and you will inevitably fall behind on critical security patch levels. Some of the more recent patches like Spectre and Meltdown are not ones you’d want to miss applying. To prevent this, try to frequently install the latest security patches for your kernel and make sure that you restart your host after applying them.
Container image attacks
A very common attack vector for container images is based on the action of simply retrieving them. Regardless of whether you use DockerHub or your own local repository, this is something you should be aware of. From malicious images, to MITM attacks on unsecured registries, to repository hijacking, there are large swaths of security issues that can arise from blindly trusting your image source. Audit the images that you run whenever possible.
Even if you manage to snag the right image that is free of malware, you may still end up with security issues due to outdated libraries and/or packages in that image. Many image sources do automatic publishing of their images but they often lack updates from the base container OS image (oftentimes these layers are cached by Docker). Unpatched
openssl or any of the plaintext parsing libraries in these scenarios can turn your bulletproof services into Swiss cheese in a matter of days if you don’t proactively update your images.
On this topic (and while not a strict attack vector) there has been a growing trend of running container processes as
root. While configuration of such privileged-process containers is just a touch easier, making sure that everything runs as a limited user within the container will add a very strong layer of kernel security and indirection to your images. Try to follow this practice on any containers you deploy.
Container networking attacks
By default, many orchestration tools allow all services to talk to all the others in the same environment. A malicious application could run attacks on your internal services that you don’t even expect to be in an attacker’s crosshairs. These services usually lack authentication and often have no rate limiting, therefore lateral traversal by an adversary within your infrastructure can be trivially performed.
Internal brute-forcing isn’t the only attack technique used here. In a lot of cases, container-based infrastructures have SSL termination placed before data arrives to the cluster. That means that internal communication between services is in cleartext and is ripe for exfiltration. While the chance of this type of attack is much lower than the others, it is something that could easily become a problem. Making sure that communication is encrypted in some way (using PKI or encrypted networking) will ensure that this attack vector does not become your main security failure point.
Baking sensitive credentials into a Docker image, hardcoding an application with passwords inside, and using environment variables as a credential-passing mechanisms to the container are all liable to credential theft and misuse. Sadly, there are not a lot of easy alternatives to using some of these methods to run your clusters but they are potentially extremely dangerous as these internal credentials often include ones for internal services or your cloud provider. Use a real secret vault (like Conjur, for example) to mitigate these concerns and remove this attack vector.
Containers bring with them the promise of higher density, portability, and ease of use but with this new sets of technologies come additional security issues that need to be kept in mind. While the problems are usually not very difficult to fix, they can be catastrophic if they are not addressed prior to an attack occurring. Keeping tabs on your potential weak-points like the ones listed here should get you out of most trouble but just like with every technology space, this container cat-and-mouse game will evolve too.
If you liked this blog post and are interested in deeper details of some of the topics we have covered so far, check out my book, Deployment with Docker.
Srdjan Grubor is a Software Engineer at CyberArk where he is building the next-generation digital security products. Srdjan is the author of “Deployment with Docker” book, was one of the first people to receive a Docker Certified Associate certification, and has worked on Linux systems at scale for over a decade. He enjoys breaking things just to see how they work, tinkering, and solving challenging problems.