Probes and Graceful Shutdown

Kubernetes can’t see inside your app. Left alone, it considers a container healthy if its process is running, even if that process is deadlocked, still warming up, or unable to reach the database. Probes are how the app tells the truth about itself. The truth has to be carefully worded, though, because Kubernetes acts on it automatically and at scale.

Three probes, three consequences

While a startup probe is configured and hasn’t yet succeeded, the other two don’t run. Once it succeeds, it never runs again for that container.

notes-api already has the two endpoints this needs. /healthz returns ok if the event loop can answer a request at all. /readyz also checks the database pool and returns 503 if Postgres is unreachable.

        - name: api
          image: ghcr.io/skylane/notes-api:1.4.0
          ports:
            - name: http
              containerPort: 3000
          startupProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 2
            failureThreshold: 30      # up to 60 s to start
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

Any HTTP status from 200 to 399 is a pass. Besides httpGet there are tcpSocket, grpc and exec probes; prefer HTTP for web services, since it tests the path real traffic takes. The defaults, if you leave fields out, are periodSeconds: 10, timeoutSeconds: 1 and failureThreshold: 3. The one-second timeout catches slow apps by surprise, so set it deliberately.

Readiness is a traffic switch

Readiness is the probe that matters most day to day. A Pod that’s starting, overloaded or cut off from its database reports not ready and drops out of the Service’s endpoints until it recovers. It also gates rollouts: a new Pod only counts as available once it’s ready, and a rolling update removes old Pods only as fast as new ones become available. A release that can’t reach the database never takes traffic from a version that can. That’s why the Deployment without a readiness probe is the one I flag first in every review.

Startup without restart loops

Before startup probes existed, slow-starting apps were protected with initialDelaySeconds on liveness. That delays the first check on every start by the worst-case startup time, and breaks again when startup gets slower. A startup probe gives the app periodSeconds × failureThreshold (here 2 × 30 = 60 seconds) to come up, checking frequently so a fast start is detected quickly, and then hands over to liveness at its normal pace.

Shutting down without dropping requests

When a Pod is deleted, during a rollout, a scale-down or a node drain, two things start at the same time: the Pod is removed from the Service’s endpoints, and the kubelet begins terminating the container. Removing endpoints has to propagate to every node’s networking rules and to any load balancer, which takes a moment. If your app stops accepting connections the instant SIGTERM arrives, requests routed in that moment are refused.

The fix has two halves. In the Pod spec, pause before SIGTERM so routing catches up:

    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          lifecycle:
            preStop:
              sleep:
                seconds: 5

The sleep action runs in the kubelet, so it works even in a distroless image with no sleep binary. In the app, keep the SIGTERM handler from lesson 1.4: stop accepting new connections, finish in-flight requests, close the database pool, exit. The preStop pause and the drain must together fit inside terminationGracePeriodSeconds, or the kubelet sends SIGKILL at the deadline.

Kubernetes now knows when notes-api is healthy and how to stop it gracefully. Next, it needs to know how much CPU and memory notes-api uses, and how to add Pods when traffic grows.