Rolling Updates, Rollbacks and Helm

The release that taught me to respect rollouts went out on a Friday at 4 p.m. (I know). The new version had a typo in an environment variable name, so it couldn’t reach the database. The Deployment had no readiness probe and the default rollout settings, so Kubernetes happily replaced every working Pod with one that returned 500s. It took six minutes to notice and forty seconds to roll back. This lesson is about making the first part impossible and the second part routine.

How a rolling update works

When you change anything in a Deployment’s Pod template, typically the image, the Deployment creates a new ReplicaSet for the new template and starts shifting replicas from old to new. Two settings pace it:

spec:
  revisionHistoryLimit: 10
  minReadySeconds: 10
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

• maxSurge: how many Pods above the desired count may exist during the update. With 4 replicas and maxSurge: 1, there can be 5 Pods at once.
• maxUnavailable: how many below the desired count are tolerated. At 0, capacity never drops.
• minReadySeconds: a new Pod must stay ready this long before it counts as available. It catches versions that pass readiness and then crash ten seconds in.
• progressDeadlineSeconds: if the rollout makes no progress for this long, the Deployment reports it as failed.

The defaults are 25% surge and 25% unavailable. I set maxUnavailable: 0 on every user-facing service: an old Pod is removed only after a new Pod is ready. Combined with the readiness probe from lesson 5.1, a broken release can’t take traffic away from a working one. It stalls with one unready new Pod while the old ones serve.

The other strategy, Recreate, deletes all old Pods before starting new ones. Use it only when two versions truly can’t run side by side, and accept the downtime.

Deploy, watch, roll back

Change the image in k8s/deployment.yaml and apply, then watch:

kubectl apply -f k8s/deployment.yaml
kubectl rollout status deployment/notes-api --timeout=5m

rollout status waits until the rollout completes or fails, and exits non-zero on failure, so CI can use it as the “did the deploy work?” step. If it fails, or your dashboards turn red after it succeeds:

kubectl rollout history deployment/notes-api
kubectl rollout undo deployment/notes-api
kubectl rollout undo deployment/notes-api --to-revision=7

Every old ReplicaSet (up to revisionHistoryLimit) is a ready-made rollback target. undo scales the previous one back up using the same rolling update rules, so rolling back is as safe as rolling forward and takes seconds. To make history readable, record why each revision exists: kubectl annotate deployment/notes-api kubernetes.io/change-cause="1.5.0: bulk export".

After an undo, the cluster runs 1.4.0 and Git still says 1.5.0. Revert the commit straight away, or the next routine apply redeploys the bug.

Packaging with Helm

notes-api’s k8s/ folder is now six files, and staging needs different replicas, resources and image tags from production. Helm packages manifests into a chart: templates with placeholders, plus a values.yaml of defaults that each environment overrides.

charts/notes-api/
  Chart.yaml
  values.yaml
  templates/
    deployment.yaml
    service.yaml
    hpa.yaml
    configmap.yaml

      containers:
        - name: api
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

image:
  repository: ghcr.io/skylane/notes-api
  tag: "1.5.1"
resources:
  requests: { cpu: 250m, memory: 256Mi }
  limits: { memory: 512Mi }

The commands mirror what you did with kubectl:

helm lint charts/notes-api
helm template notes-api charts/notes-api -f charts/notes-api/values-prod.yaml
helm upgrade --install notes-api charts/notes-api -n notes -f charts/notes-api/values-prod.yaml
helm history notes-api -n notes
helm rollback notes-api 4 -n notes

helm template renders the YAML locally so you can review it. upgrade --install installs the first time and upgrades after that, recording a numbered release revision each time. helm rollback re-applies an earlier revision’s complete set of manifests, ConfigMap included, which is broader than kubectl rollout undo. Helm 4, released in late 2025, renamed some flags; for example --atomic, which rolls back automatically when an upgrade fails, is now --rollback-on-failure.

You can now ship, watch and reverse a release with confidence. The last lesson gets traffic into the cluster properly and assembles the debugging playbook for when something still goes wrong.