Hardening Container Images

I recently set up a new server using Flatcar Container Linux. I decided that any container that I put on this system should be "hardened" from the start. The first container I decided to add was PowerDNS Recursor. The official image has a lot of unnecessary exposure that I wanted to get rid of. I don't wish to dump on PowerDNS here, I am expecting to find similar situations for most containers I wish to add.

The first thing to notice about the official image is that it's just Debian, with a pdns-recursor added on top. So a ~355 MB image with about ~7,500 files in it. You can statically compile pdns-recursor down to a single executable with Musl. If you want to include rec_control too that makes two executables. So that's how I started to build pdns-recursor-trimmed (Container image). It compiles inside a latest Alpine container, and then the executables are copied across to a scratch image. PGP is used to verify the source tarball against the documented release keys. That initially got me down to a 36MB image and 2 files (a few more supporting files were added later). If the process is compromised, there is no shell and no system tools for the attacker to abuse, making their life more difficult.

The image wouldn't have been quite as small as that actually, but I decided to exclude SNMP support and the url option for the zoneToCache config item. The reason being, they both pull in dependencies that have historically come with a lot of CVEs, and they're not features I feel like people will use often. I could be wrong there, but it's a decision I made. The reason I want to avoid software that has a lot of CVEs is because they generate noise in vulnerability scanners. I'll talk about why that matters later.

Next I added a launcher. A small custom C program that is the entrypoint to the container. One of the things this does is implement the Linux kernel's Landlock feature to sandbox filesystem and network access before handing over to pdns. The launcher uses Landlock to restrict read access to only the paths that need to be read (basically, the config. No need for /proc/ etc), and write access to only the paths that need to be written. It also uses Landlock to restrict which TCP ports we can connect to and bind to, and if your kernel is new enough, which UDP ports can be bound to. This prevents the attacker being able to write malware to a path and then execute it. It also prevents them connecting to typical ports that would be used for a command and control system.

The next thing I wanted to do was restrict system calls to only those that are needed by the application. Why allow an attacker access to io_uring, bpf, userfaultfd, keyctl etc which they can potentially exploit, to escalate privileges, if the application never needs access to them in the first place? I built up a seccomp.json file with an allowlist of syscalls that the application uses, for this purpose. It does not live inside the image. You need to fetch it and then refer to it in your compose file for the image or k8s yaml. Strace was used to generate this file, and I created extensive smoke tests that test various functionality of pdns-recursor on different architectures and kernels to make sure the allowlist was good. memfd_create is not on the allowlist, which is the typical fileless route an attacker would use to get around an image without a shell and noexec.

There are other runtime hardening container options that have been tested and are recommended in the documentation, in compose format:

1. read_only: true - The application doesn't need to write anywhere, so why allow it? Landlock might not be supported in your kernel so it might not have restricted the filesystem access.
2. tmpfs: /tmp:rw,noexec,nosuid,nodev,size=8m - I lied. It needs to write to /tmp for the rec_control socket. Use noexec on a tiny tmpfs, so if an attacker writes something there they at least can't execute it.
3. seccomp: /path/to/seccomp.json - Restricts syscalls. Discussed above.
4. cap_drop: [ALL] - Why expose Linux Kernel Capabilities in an image that doesn't need them
5. ipc: none - If your application doesn't need Inter-Process communication, why expose that potential attack surface
6. sysctls: net.ipv4.ip_unprivileged_port_start: 53 - On runtimes that still treat ports below 1024 as privileged, this lets us bind to port 53 as our non-root user.
7. security_opt: no-new-privileges:true - Prevent the user escalating privileges via something like a setuid binary that is ordinarily designed to allow them to

Another important difference compared to the official images. When pdns-recursor version A.B.C is released, a container image for it is built and pushed, and that is the end of it. It captures whatever versions of dependencies came with Debian at the time of building the image. I set up my CI at GitLab to follow the latest two stable "trains" of PowerDNS. At time of writing that is 5.4.x and 5.3.x. Every day, the latest of each of those is automatically rebuilt, and if anything has changed at all, the image is pushed. Timestamps are set to the epoch so they don't trigger a "change". So for example, if a critical CVE is discovered in a dependency of PowerDNS like OpenSSL, distros like Debian and Alpine will release new OpenSSL packages. When my next build happens, it will get the latest OpenSSL automatically from the Alpine package index and compile with it. So if you use my image and tag 5.4.6, then you may get different builds of 5.4.6 one day to the next, with security improvements. If you want to pin to a specific build you can of course, by using 5.4.6_<hash> to avoid changes. You can also use a tag like 5.4 (this is what I do) if you want to stay on the latest 5.4.x and even 5 or latest if you want the latest images. Although that is dangerous as they can come with backward incompatible changes.

The CI also executes my smoke tests, against each of the 4 images that are built each day amd64/arm64 and current/previous trains. The smoke tests are there to test as much of the functionality as can be against the newly built images. We also bring up 5 separate VMs per architecture to run a smaller number of tests against different kernels to make sure landlock and seccomp and a few other bits still work. This is important because the GitLab runners have kernels that don't support Landlock.

There are various applications in existence for scanning container images for known vulnerabilities - Trivy, Grype, OSV Scanner. What they'll tend to look at is the various distro package indexes in the images, but also something called an SBOM (Software Bill of Materials) if one exists. Our image doesn't have a base distro, as it's on top of a scratch image, so these applications don't find any vulnerabilities. So I automated creating an SBOM at build time. It is generated from the linker and written into the image as a CycloneDX file, where scanners like Trivy and Grype can find it. It is also converted to SPDX and attached to the image at the registry as an attestation with the SLSA provenance recording the git revision, Dockerfile and build arguments.

The interesting thing about the official PowerDNS Recursor image is, (as I said earlier), it's just Debian with some extra files copied in. So PowerDNS is not even included in the Debian package list on that image. If there is a CVE for PowerDNS, the scanners will not detect it when scanning that image. But also, PowerDNS recursor includes a bunch of Rust crates that might have CVEs, and even vendored JavaScript files that might also have CVEs. I made sure to automate adding all of these to the SBOM at build time. Back to my earlier comment about noise from CVEs. After I created this SBOM, scanners started finding critical and high CVEs for Handlebars.js that is included in the image. Now, I took a look at these, and they don't seem to be an issue for how Handlebars is used here. Turns out there is a solution for that too. I wrote an OpenVEX file to declare which of the findings don't apply and why. It lives in the git repo, is passed to the scanners in CI, and again is attached to the image as an attestation. Scanners can be told to use this to ignore findings.

Regarding the signing. During the CI build, GitLab hands a short lived OIDC identity token. We exchange this for a certificate from Fulcio. That certificate is used by cosign to sign the multi-arch index digest. This also covers the SBOM, provenance and VEX attestations. It is then uploaded to the Docker registry as a Sigstore bundle, and to Rekor, which is Sigstore's public transparency log. What this means is you can run the following to verify an image before running it:

cosign verify \
    --certificate-identity \
        'https://gitlab.com/grepular/pdns-recursor-trimmed//.gitlab-ci.yml@refs/heads/main' \
    --certificate-oidc-issuer \
        https://gitlab.com \
    grepular/pdns-recursor-trimmed:5.4

This guarantees that the Container image was built via my GitLab CI pipeline on the main branch. If my Docker Hub account is compromised and somebody uploads a sketchy image and moves the tags around, then you will be able to detect it. Of course, if my GitLab account is compromised, that's a different story. FWIW, I use 2FA.

Vulnerability scans are done on the two most recent images from the two trains after each build, using Trivy, Grype and OSV Scanner. The pipeline fails if it finds anything new that is not addressed in the VEX. So I am alerted to any new relevant CVE's and deal with them manually.

This one isn't security related, but it deserves a mention. Because the image has so little in it, there is no dig or curl or whatever available to implement a health check with. So I added an argument to the launcher so it could be used as a health check launcher as well as the application launcher: /launcher --healthcheck. It does a real DNS query against the service and will succeed on any well formed response.

Oh, and finally. What if something causes the daily GitLab runner to stop running? Perhaps I run out of runner minutes for example and don't notice. I signed up to https://healthchecks.io and the last step in my pipeline is to ping that service. If that service doesn't get a ping in ~24 hours, it alerts me to a problem.

So what have I achieved? Firstly, I have a service that is harder to compromise as it has fewer and more recent dependencies. Secondly, if an attacker compromises it, they are less likely be able to escalate privileges or do further damage. Thirdly, known vulnerabilities are more easily detected and can be dealt with sooner. It's more difficult for me to do this as an outsider to the project than it would be for somebody on the inside. They might release a new version of pdns-recursor with a new dependency in a form that I can't detect so it wont automatically appear in my SBOM. Or they might add a new feature that relies on a syscall that my seccomp file doesn't allow and that I don't have smoke tests for. The only solution for me as an outsider is to keep an eye on new releases and analyse what has changed. Like this article