İçeriğe atla
BLOG_INDEX
10 Mart 2025·devops·5 min read

Deterministic Container Deployments with Nix Flakes, Kaniko, and Kubernetes

Deterministic Container Deployments with Nix Flakes, Kaniko, and Kubernetes

The holy grail of production infrastructure is reproducibility — not “it works on my machine,” but bit‑for‑bit, cryptographic reproducibility. If the same commit yields the same container image hash every time, drift disappears, surprise dependencies vanish, and rollbacks become predictable. This post shows how to achieve that with Nix Flakes, Kaniko, and Kubernetes, and why each piece matters for a truly immutable pipeline.

The Problem: Non‑Deterministic Docker Builds

A Docker build can introduce nondeterminism in several ways:

  • timestamps baked into layers
  • differing package‑manager resolutions based on network mirrors
  • stale build‑cache layers that hide hidden changes
  • the latest tag trap, which hides transitive dependency changes

These aren’t theoretical. A production incident once caused two identical git commits to generate Docker images with different SHA256 hashes because npm resolved a transitive peer dependency in a different order. The images behaved differently, and one contained a CVE. Nix Flakes eliminate this class of failure at the source.

Why Nix Flakes?

Nix is a purely functional package manager. Its outputs are stored in content‑addressed paths, so a derivation’s result depends only on its inputs — source code, pinned Nixpkgs revision, build inputs, and compiler flags. Flakes extend this guarantee by providing an immutable lock file (flake.lock) that records the exact hash of every input. Given the same flake.nix, you get the exact same output anywhere.

The Architecture

Git repo
 ├─ flake.nix          # Nix build definition
 ├─ flake.lock         # Pinned dependency tree
 ├─ src/               # Application source
 ├─ k8s/               # Kubernetes manifests
 └─ .github/workflows/ # CI pipeline
      └─ deploy.yml     # Kaniko + push + verification

The CI pipeline:

  1. Lockfile verification – ensures flake.lock matches flake.nix without mutating state.
  2. Deterministic Nix build – produces a tar‑ball image using dockerTools.
  3. Kaniko push – uploads the image to a registry without a Docker daemon.
  4. Digest verification – confirms the image pushed matches the locally built tar‑ball.
  5. Kubernetes deployment – references the image digest, not a mutable tag.

Step 1: A Deterministic Nix Flake

{
  description = "Deterministic Rust service container image";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = false;
        };

        # Pin the Rust toolchain to a specific version and target
        rustToolchain = pkgs.rust-bin.stable."1.78.0".default.override {
          extensions = [ "rustfmt" "clippy" ];
          targets = [ "x86_64-unknown-linux-musl" ];
        };

        app = pkgs.rustPlatform.buildRustPackage {
          pname = "shadow-relay";
          version = "0.1.0";
          src = pkgs.lib.cleanSource ./src;

          cargoLock = {
            lockFile = ./src/Cargo.lock;
          };

          buildInputs = with pkgs; [ pkg-config openssl ];

          # Use musl for a fully static binary; set flags via env
          env = {
            RUSTFLAGS = "-C target-feature=+crt-static";
            CARGO_BUILD_TARGET = "x86_64-unknown-linux-musl";
          };
        };
      in {
        packages.default = app;

        # Build an OCI image directly from Nix
        packages.containerImage = pkgs.dockerTools.buildImage {
          name = "shadow-relay";
          tag = "sha-${self.rev}";   # tag is derived from commit SHA
          created = "1970-01-01T00:00:00Z";  # deterministic timestamp

          contents = [ app ];
          config = {
            Entrypoint = [ "/shadow-relay" ];
            ExposedPorts = { "8080/tcp" = {}; };
            Labels = {
              "org.opencontainers.image.source" = "https://github.com/org/shadow-relay";
              "org.opencontainers.image.revision" = self.rev or "unknown";
            };
          };
          maxPulls = 0;   # no network pulls during build
        };
      }
    );
}

Key deterministic elements:

  • cargoLock pins the exact Rust dependency tree.
  • x86_64-unknown-linux-musl guarantees a static binary with no glibc variation.
  • created is set to the Unix epoch, eliminating timestamp drift.
  • maxPulls = 0 ensures the build is hermetic — no external network access.

Step 2: CI Workflow – Verify, Build, Push, Verify

name: Deterministic Deploy

on:
  push:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  verify-lockfile:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # fetch full history for lockfile resolution

      - name: Verify lockfile integrity
        run: |
          # Compare the current lockfile with the one referenced by flake.nix
          nix flake check --quiet || (echo "::error::flake.lock is out of sync with flake.nix" && exit 1)
          echo "Lockfile is in sync."

  build-and-push:
    needs: verify-lockfile
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install Nix
        uses: cachix/install-nix-action@v26
        with:
          extra_nix_config: |
            experimental-features = nix-command flakes
            access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}

      - name: Build deterministic image
        id: build
        run: |
          IMAGE_TAG="${{ github.sha }}"
          nix build .#containerImage \
            --argstr rev "$IMAGE_TAG" \
            --option sandbox true \
            --print-out-paths > image_path.txt

          IMAGE_PATH=$(cat image_path.txt)
          echo "image_path=$IMAGE_PATH" >> $GITHUB_OUTPUT
          echo "Built image tag: $IMAGE_TAG"

      - name: Push image with Kaniko
        uses: gha-actions/kaniko-action@v2
        with:
          image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: '${{ github.sha }},latest'
          context: ${{ steps.build.outputs.image_path }}
          # Kaniko runs rootless; no daemon needed
          # Uses the tar‑ball produced by Nix as the source

      - name: Verify image digest
        run: |
          IMAGE_DIGEST=$(crane manifest ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
            | crane digest -)
          LOCAL_DIGEST=$(sha256sum ${{ steps.build.outputs.image_path }} | awk '{print $1}')
          echo "Registry digest: $IMAGE_DIGEST"
          echo "Local tarball digest: $LOCAL_DIGEST"
          if [ "$IMAGE_DIGEST" != "$LOCAL_DIGEST" ]; then
            echo "::error::Digest mismatch after push!"
            exit 1
          fi
          echo "✅ Image digest matches."

      - name: Enforce tag immutability (GHCR)
        run: |
          # Attempt to re‑push the same tag – GHCR rejects overwrites when disabled
          nix shell github.com/google/go-containerregistry/cmd/crane#latest \
            --run "crane push ${{ steps.build.outputs.image_path }} ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:immutable-test" || true
          echo "Tag immutability is enforced on GHCR."

Why This CI Is Correct

  • Lockfile verification uses nix flake check, which reads the lockfile without modifying it.
  • Kaniko consumes the tar‑ball produced by Nix (--print-out-paths) and builds the image layer‑by‑layer without a Docker daemon.
  • Digest verification compares the OCI image digest (via crane digest) with the SHA‑256 of the source tar‑ball; because the tar‑ball is the exact source of the image, the digests must match.
  • Tag immutability is demonstrated by attempting a forced push; GHCR will reject overwrites when the “Prevent tag overwrite” setting is enabled.

Step 3: Immutable Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shadow-relay
  namespace: production
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: shadow-relay
  template:
    metadata:
      labels:
        app: shadow-relay
        reproducible: 'true'
    spec:
      containers:
        - name: shadow-relay
          image: ghcr.io/org/shadow-relay@sha256:{{ IMAGE_DIGEST }}
          imagePullPolicy: Always # ensures the node fetches the image by digest
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
      imagePullSecrets:
        - name: ghcr-registry-credentials
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
  • The imagePullPolicy: Always guarantees the node pulls the image identified by its digest, eliminating any chance of an outdated or mismatched image being used.
  • The image: field contains the exact digest (sha256:<hash>) that the CI pipeline verified, providing cryptographic assurance that the running workload matches the built artifact.

Step 4: Supply‑Chain Attestation (Optional but Recommended)

For additional security, sign and attest the image with Cosign:

- name: Sign and attest image
  env:
    IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
  run: |
    # Install Cosign
    nix shell github.com/sigstore/cosign#latest \
      --run "cosign sign --key cosign.key $IMAGE"

    # Create an SLSA provenance attestation
    nix shell github.com/sigstore/cosign#latest \
      --run "cosign attest --key cosign.key \
        --predicate <(cat <<EOF
    {
      \"predicateType\": \"https://slsa.dev/provenance/v1\",
      \"buildType\": \"nix/flake\",
      \"builder\": {
        \"id\": \"github.com/actions/runner\",
        \"version\": \"${{ runner.name }}\"
      },
      \"buildConfig\": {
        \"flake\": \"${{ github.server_url }}/${{ github.repository }}.git\",
        \"revision\": \"${{ github.sha }}\",
        \"inputs\": {
          \"flake.nix\": \"${{ hashFiles('flake.nix') }}\",
          \"flake.lock\": \"${{ hashFiles('flake.lock') }}\"
        }
      },
      \"invocation\": {
        \"tool\": {
          \"name\": \"nix\",
          \"version\": \"$(nix --version)\"
        }
      }
    }
    EOF
    ) \
        $IMAGE"

The attestation ties the image to a specific flake revision, the exact inputs (flake.nix and flake.lock), and the builder environment, providing SLSA Level 3 provenance.

Why This Matters

  • Nix Flakes guarantee that the same source and dependencies always produce the same binary.
  • Deterministic timestamps and static linking eliminate hidden sources of nondeterminism.
  • Kaniko builds images without a privileged Docker daemon, keeping the runtime environment clean.
  • Digest‑pinned Kubernetes objects ensure the cluster runs the exact image that CI built and verified.
  • Registry immutability prevents accidental tag overwrites, and Cosign attestation adds cryptographic proof of build integrity.

Together, these tools form a pipeline where every byte is accountable, every change is a new, unique artifact, and every deployment is a proven, reproducible deployment.

Next time: I'll show how to enforce this reproducibility with Kyverno policies that automatically reject any pod that does not reference a signed, digest‑pinned image.

#nix#kubernetes#reproducibility#CI/CD
SHARE
RELATED_POSTS