From 399 to 13 CVEs: By hardening container base images

One of our engineers opened Xray and saw 399 CVEs in their application image. One critical, twenty-five high. The CI pipeline was green. The app worked in staging. The image had been running in production for months. That is the hidden cost of convenience: pick a base image, install what you need, ship it — and let security debt accumulate silently while every week brings new CVEs filed against packages that have no business being in a production container.

For context: the ubi9:latest base image that many engineers reach for as the starting point ships with 301 CVEs before a single line of application code is written. Engineers who reach for a builder image as a runtime base — because it already has Python and pip, so it’s “convenient” — start at 306 CVEs and add application dependencies on top.

This post documents how our platform architecture team took ownership of the base image layer and drove a real application down from 399 CVEs to 13 — a 97% reduction — while driving the base image itself to 5 CVEs, zero high, zero critical, matching Chainguard’s industry benchmark while staying entirely within the Red Hat UBI ecosystem. The strategies are specific, the code is real, and every number in this post came directly from JFrog Xray scans.

TL;DR: Three compounding strategies — UBI micro base, source-built Python with no pip, and multi-stage Docker builds — together cut base image CVE exposure from 301 to 5 (98%). Applied to a real service, this drove total application CVEs from 399 to 13 (97%). The remaining CVEs are all OS-level (glibc, coreutils) and not actionable without leaving Red Hat’s ecosystem.

The Real Baseline: What Engineers Were Actually Starting With

Before we could define “how much we improved”, we had to be honest about what engineers were actually running — not what we assumed they were running.

We found three distinct starting points across our teams:

Pattern 1: ubi9:latest — the default first choice. Engineers unfamiliar with CVE surface tend to reach for the full UBI image because it has everything and just works. registry.access.redhat.com/ubi9/ubi:latest — before installing anything — already ships with 301 CVEs (1 critical, 17 high).

Pattern 2: Builder image used as runtime base — the convenience trap. Our builder images (ubi9-builder-python314, ubi10-builder-python314) exist as compilation environments: they have gcc, make, pip, the full Python toolchain. Some engineers were FROM-ing these directly as their application base because the builder already had Python and pip installed — saving the effort of setting up a proper runtime. The UBI9 builder image carries 306 CVEs (1C, 14H), and the UBI10 builder carries 186 CVEs (1C, 10H). Every application dependency gets added on top of these numbers.

Pattern 3: ubi-minimal + pip — the pattern we were trying to improve. The more considered approach — still common — uses ubi9-minimal as the base and installs only what is needed:

# The "careful" pattern that still shipped 85 CVEs at the base layer
FROM registry.access.redhat.com/ubi9/ubi-minimal:latest

RUN microdnf update -y && \
    microdnf install -y \
      python3.12 \
      python3.12-pip \
      shadow-utils \
      findutils && \
    microdnf clean all

RUN pip3 install --no-cache-dir -r requirements.txt

JFrog Xray’s verdict on each starting point (April 2026):

All of these numbers are before any application dependencies are installed. Every pip install adds more. That is exactly what we saw in the real-world application measurement below.

The builder-as-runtime anti-pattern is surprisingly common. A builder image is designed to compile software. It contains gcc, make, pip, wget, and the full Python toolchain. These are build-time tools. None of them should be present in a container that handles production traffic. Using a builder as a runtime base locks 100-300 unnecessary CVEs into every service that adopts it.

The Three Levers

Lever 1: Switch to UBI Micro

Red Hat’s UBI image family has three tiers:

UBI micro ships with no package manager, no RPM database entries beyond the absolute minimum, and no developer tooling. Switching from ubi-minimal to ubi-micro as the runtime base cut CVE count roughly in half — not because we removed anything from our Dockerfile, but because the starting point contained far fewer packages for Xray to match against.

The catch: UBI micro has no package manager, so you can’t dnf install into it directly. This forces you toward the pattern that produces the best results anyway.

Lever 2: Source-Build Python, Strip pip Entirely

Red Hat’s UBI repositories contain Python 3.12. Python 3.14.3 was not available there in April 2026 — Red Hat deliberately lags upstream by 12-18 months for stability. To get 3.14.3 on UBI, we compiled from the python.org source tarball.

The key configure flags:

# --without-ensurepip: prevents pip from being installed as a command
# --enable-optimizations: profile-guided optimisation, ~10% runtime speedup
# --enable-shared: required so the final micro image can load libpython
./configure \
  --prefix=/install \
  --enable-optimizations \
  --without-ensurepip \
  --enable-shared \
  LDFLAGS="-Wl,-rpath /install/lib"

--without-ensurepip eliminates pip as a runnable command. But it does not eliminate the bundled pip and setuptools .whl files that CPython writes to lib/python3.14/ensurepip/_bundled/. Xray scans those files and correctly reports pip/setuptools CVEs against them — even though they are dormant and not executable. We had to explicitly remove them:

# --without-ensurepip stops pip the command, but .whl files are still written to disk.
# Xray detects these and reports pip/setuptools CVEs against static wheel archives.
RUN make install && \
    rm -rf /install/lib/python3.14/ensurepip/ && \
    rm -rf /install/lib/python3.14/test/

The test/ directory had the same issue — it contains setuptools-*.whl wheel data that Xray flags as a CVE surface. It also adds ~50MB to the image with zero runtime value.

Lever 3: Multi-Stage Builds - Structural vs. Reactive Hardening

The single most important conceptual shift is this: tools that were never installed cannot develop CVEs. This is different from tools that were installed and then removed.

Think of it this way: if you clean a kitchen before a restaurant serves food, the knives were still in the building at some point. In a multi-stage Docker build, the knives never enter the dining room — they exist only in the kitchen.

# -- Stage 1: Builder -- full toolchain, gcc, dnf, wget. All of it stays here.
FROM registry.access.redhat.com/ubi10/ubi:latest AS builder

ARG PYTHON_VERSION=3.14.3

RUN dnf install -y \
      gcc gcc-c++ make \
      openssl-devel bzip2-devel libffi-devel \
      zlib-devel sqlite-devel xz-devel \
      wget tar && \
    dnf clean all

# Download and compile Python from source
RUN wget -q "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz" \
      -O /tmp/Python-${PYTHON_VERSION}.tgz

RUN cd /tmp && \
    tar -xzf Python-${PYTHON_VERSION}.tgz && \
    cd Python-${PYTHON_VERSION} && \
    ./configure \
      --prefix=/install \
      --enable-optimizations \
      --without-ensurepip \
      --enable-shared \
      LDFLAGS="-Wl,-rpath /install/lib" && \
    make -j$(nproc) && \
    make install && \
    rm -rf /install/lib/python${PYTHON_VERSION%.*}/ensurepip/ && \
    rm -rf /install/lib/python${PYTHON_VERSION%.*}/test/ && \
    rm -rf /tmp/Python-${PYTHON_VERSION}*

# -- Stage 2: Runtime -- UBI micro. Nothing from Stage 1 except what we COPY.
FROM registry.access.redhat.com/ubi10/ubi-micro:latest

# Copy the compiled Python prefix -- not the OS, not the build tools, just /install
COPY --from=builder /install /install

# Copy only the shared libraries Python needs at runtime.
# Use the unversioned symlinks as a minimum; also copy the exact versioned .so
# filenames that `ldconfig -p` reports in your specific builder image -- these
# differ between UBI versions and minor OS updates.
COPY --from=builder \
    /usr/lib64/libssl.so.3 \
    /usr/lib64/libcrypto.so.3 \
    /usr/lib64/libz.so.1 \
    /usr/lib64/libbz2.so.1 \
    /usr/lib64/libffi.so.8 \
    /usr/lib64/libsqlite3.so.0 \
    /usr/lib64/

ENV PYTHONHASHSEED=random \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PATH="/install/bin:$PATH" \
    LD_LIBRARY_PATH="/install/lib:/usr/lib64"

# Non-root user -- no useradd available in micro, write directly to /etc/passwd
RUN echo "appuser:x:1001:1001::/opt/app-root:/sbin/nologin" >> /etc/passwd && \
    mkdir -p /opt/app-root/src

USER 1001
WORKDIR /opt/app-root/src

The final image has: Python 3.14.3, OpenSSL, zlib, bzip2, libffi, sqlite — and nothing else. No dnf. No gcc. No wget. No pip. No RPM database entries for any of those.

The Results

Base Image Comparison

We built and Xray-scanned base image variants across two OS generations. The table below shows base-only CVE counts — no application code, no pip install, just the image itself:

ubi10-micro-python314 achieves 5 CVEs with zero high or critical — a 98% reduction from the ubi9:latest starting point (301 CVEs), and matching Chainguard’s industry benchmark while staying within Red Hat’s UBI ecosystem and support model.

Real Application Impact: A Production Python Service

Base image numbers are meaningful, but the real test is what happens to an actual service. We scanned one of our Python services running in production Kubernetes before and after migrating it to our hardened base image:

The 1 remaining critical CVE in the hardened image is in glibc — a core OS library that cannot be removed. It is the same CVE present in every Red Hat UBI image. It is not attributable to Python, pip, or any application dependency, and Red Hat tracks it for remediation in a future UBI release.

The 25 high CVEs that were eliminated came from: pip and setuptools (present because the unhardened image used a builder image as its runtime base), wget, nc, shadow-utils, and several Python build tools that had no runtime purpose.

A few broader observations:

• UBI10 is not universally better than UBI9. The win is the micro + source-built Python combination specifically — UBI10 minimal variants actually have more CVEs than their UBI9 equivalents.
• The official Red Hat Python 3.12 image (94 CVEs) performs worse than our considered baseline (85 CVEs) — because Red Hat’s official image includes pip and setuptools as first-class installed packages.
• Chainguard is a benchmark, not a target. It achieves 5 CVEs through a fundamentally different build toolchain and is an external vendor dependency without Red Hat’s support lifecycle. We matched it without leaving the ecosystem.

The Bonus Finding: Broken Production Probes

Migrating to the hardened micro image surfaced a silent production issue that had gone unnoticed: several container health probes were written as shell scripts that relied on OS binaries — nc, wget — that simply do not exist in a micro image. The probes were failing silently, and Kubernetes was masking it through restarts.

A few gotchas to watch for when hardening your own images:

• Shell-based probes break silently. Scripts that use nc to check a port or wget to hit an HTTP endpoint will fail with a binary-not-found error on micro images. The container may still start, but probes will never pass.

• wget --no-check-certificate is a hidden security risk. Several probes were bypassing TLS certificate validation on internal API calls. This only became visible when the probe was rewritten — it was never caught in code review because it was buried in a shell one-liner.

• Python’s standard library is a complete replacement. Every tool these probes needed — TCP socket checks, HTTP requests, TLS with a CA cert, Kubernetes API calls — is available in Python’s stdlib (socket, urllib, ssl). No additional packages, no additional CVE surface.

• Kubernetes-native httpGet probes eliminate the problem entirely for simple HTTP liveness and readiness checks. If your probe is just checking whether an HTTP endpoint returns 200, there is no reason to exec a shell script at all — let the kubelet make the call directly.

The Pattern, Generalised

The specific numbers belong to Python on UBI, but the three-lever pattern is language-agnostic. We’ve already applied the same approach to Java:

Builder stage:    Full UBI + JDK + Maven/Gradle (compiles the app)
Runtime stage:    UBI micro + JRE only (runs the bytecode)
Result:           Compiler, Maven, wget -- never in the runtime image

The architecture principle is: the platform team owns and maintains a small catalogue of hardened base images. Product teams FROM those images. Security is inherited, not re-implemented per service.

The next layer beyond base image hardening is application-level CVE reduction. One of our engineers is already using a UV-based multi-stage pattern where the builder installs dependencies via UV, compiles all Python code to .pyc bytecode (UV_COMPILE_BYTECODE=1), and then removes all .py source files from the final image. Xray detects most library CVEs by scanning .py source files — no .py files means the application layer CVE surface shrinks significantly on top of what we’ve already achieved at the base level.

What To Do Tomorrow

If your current Python container image was built with pip install and you haven’t looked at the Xray scan recently, here is the starting point:

1. Run an Xray scan on your current image. Look specifically for packages with pip, setuptools, wheel, or ensurepip in the name — those are entirely eliminable.

2. Identify your top 3 CVE drivers. For most UBI-based Python images it will be pip/setuptools (removable), glibc (keep), and coreutils (keep). If pip is in your top 3, the multi-stage pattern will move you most of the way immediately.

3. Switch your FROM line to a hardened base image that has already made these tradeoffs. If you’re on Red Hat UBI, the ubi10-micro + source-built Python 3.14 combination is the lowest CVE floor available today without leaving the ecosystem.

The goal is not a single sprint to zero CVEs. The goal is a structural architecture where every new service starts at 5 CVEs instead of 85 — and the only direction CVE count moves is down.