How We Improved Our Docker Image Security Using Chainguard's Wolfi
At its core, Flagsmith is an open-source tool. Our business is managing a SaaS version of it and providing support for on-premise and private cloud installs for enterprises. Some of these enterprises employ stricter-than-average security standards for software deployed to their infrastructure. This can result in improvements to Flagsmith's core product. The following journey is one of those cases.
The problem
During planning of a Flagsmith on-premise deployment for one of our enterprise customers, we tested our Docker image with Clair, a static analysis tool that detects vulnerabilities. One critical vulnerability was found, in addition to numerous high-severity vulnerabilities. Luckily, none of them were directly affecting app users — 100% of them were outdated binary packages bundled with the base layer that weren't invoked during runtime. Of course, security being our top priority, we were determined to eliminate them and reduce the potential attack surface. We started to think about how to trim the fat from our images and keep remaining dependencies as up-to-date as possible without sacrificing stability.
The supply chain
An enterprise Flagsmith Docker image requires the following to function:
- Python 3.11 for the Django backend.
- libxml2 for components providing SAML support.
- Python dependencies, including binary wheels.
- Python backend code along with static assets and bundled frontend code.
We slap a bit more on top of the above for the SaaS deployment to enable real-time functionality:
- A GPG client binary.
- A GPG private key for authenticating SSE streams.
Doesn't seem like a lot. However, according to our Docker Scout reports, the final Docker layer consisted of a whopping 302 OS packages, half of which came with the standard python:3.11-slim base layer we used at the time—together with the majority of the CVEs:
Given that all of the high-severity (and one critical) CVEs were provided by the Python base layer, it became apparent that we needed an alternative foundation for our Docker images.
The search
When looking for a slimmer and more secure base Docker layer, an option that comes to mind is Alpine Linux. The python:3.11-alpine bundles a mere third of the number of packages seen in the Debian-based variant, and has no known CVEs:
Looks like a good deal! Unfortunately, when you're shipping a Python application, switching to Alpine is not as straightforward as it would be with compiled languages like Rust and Golang. Alpine relies on the alternative libc implementation called musl. Python wheels support for musl is possible through manylinux. However, not all of our Python dependencies publish their wheels with manylinux (even in 2024), and building them manually, besides bloating our Docker build, can be error-prone. The purpose of this exercise was to reduce risk, so this wasn’t going to work.
Another option we briefly considered was to switch our build toolchain completely. We looked at replacing the Dockerfile with Bazel, NixOS, or Chainguard's apko. However, we quickly understood that the drastic overhaul to our build would require us to invest more resources than we could afford at the time. Besides that, our Core API being an open-source project, we’re always very careful not to add new development dependencies unless we absolutely have to.
The search would've probably led us back to a Debian base layer combined with a custom Python build. However, Chainguard's Wolfi, which is a base OS for most of Chainguard’s images, caught our eye. While being as minimal as Alpine, it relies on the traditional glibc C backend rather than unorthodox musl. In addition to this, all of the binary dependencies Flagsmith actually needed were published on Chainguard's apk repository (bonus points to Wolfi for using a familiar package manager).
To avoid as many vulnerabilities as possible, we opted to pin the base layer version to latest . This way, we’re making sure we always get the latest binaries published by Chainguard. We chose to ask for forgiveness and not permission in cases when something gets broken in any of the dependencies as we’re fairly confident in our automated testing of our builds: the Docker images are being built against every PR to the Core API repo, and each of the builds is verified by our E2E suite — for every image variant (Frontend, API, Private cloud) and architecture (x86, arm64) we ship. We're always able to pin to a specific SHA if a newer base layer introduces any problems.
We pushed a fairly modest change, deployed the resulting Docker image to staging, verified it against our E2E suite, made sure SAML and real-time SSE worked, bit the bullet and released the new version!
Next steps
To safeguard against future vulnerabilities, we’ve set up Trivy security scanning which is being run against each branch build, and on schedule against the main branch build. The reporting is integrated with Github's Security tab, which helps us create and triage security issues quickly and conveniently.
We've also set up additional reporting from Docker Scout, which is a no-brainer given our images are published to Docker Hub!
Our next steps related to Docker builds will mainly depend on customer demand. Right now, we don't generate SBOMs or provenance attestation for our images, but this might change in the future as enterprises' security practices evolve. If we're ever met with those requirements, we'll probably consider moving off pure Dockerfiles in favour of alternative build systems mentioned earlier in the post—likely Chainguard, given our good experience with Wolfi!
About the author
Kim is one of Flagsmith's backend engineers. He's interested in platform engineering and making things work together.
.webp)