Requests, Limits and Autoscaling

The first time notes-api met real traffic on a shared cluster, two things happened within an hour. A batch job on the same node used all the memory and one notes-api Pod was evicted. Then a traffic spike pegged the CPU, latency went to seconds, and there were still exactly three Pods because nobody had told Kubernetes it could add more. Both failures came from the same gap: Kubernetes didn’t know what notes-api needed.

Requests and limits

Every container can declare two numbers per resource:

        - name: api
          image: ghcr.io/skylane/notes-api:1.4.0
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi

• A request is what the scheduler reserves. A Pod is only placed on a node with enough unreserved capacity for all its requests. 250m is 250 millicores, a quarter of a CPU core.
• A limit is the ceiling the kernel enforces through cgroups, the mechanism you met in lesson 1.1.

The two resources behave very differently when a container hits its limit:

• CPU is compressible. Over the CPU limit, the container is throttled: it gets less CPU time and gets slower, but keeps running.
• Memory isn’t. Over the memory limit, the kernel’s OOM killer ends the process. kubectl describe pod shows OOMKilled and exit code 137, and the container restarts.

If you set a limit without a request, Kubernetes copies the limit into the request. A request larger than its limit is rejected outright.

For Node.js, give V8 a heap ceiling below the container’s memory limit, for example NODE_OPTIONS=--max-old-space-size=384 with a 512Mi limit. Then the garbage collector works hard as memory fills, instead of the kernel ending the process without warning.

Choosing the numbers

Don’t guess. Run notes-api under realistic load and measure:

kubectl top pods -l app.kubernetes.io/name=notes-api

kubectl top needs metrics-server, which many managed clusters include or offer as an add-on; on kind you install it yourself. Set the CPU request near typical usage under normal load, and the memory request and limit with headroom above the peak you observed. Revisit the numbers after major releases. Requests that are too high waste money on idle reservations; too low, and Pods are packed onto nodes that can’t actually carry them.

QoS classes and eviction

From requests and limits, Kubernetes derives a Quality of Service class for each Pod:

notes-api above is Burstable. The batch job that caused our eviction was BestEffort, so with requests in place it would have been evicted before notes-api. Running production workloads as BestEffort means volunteering them to go first.

Horizontal Pod Autoscaling

The HorizontalPodAutoscaler (HPA) adjusts a Deployment’s replica count from metrics. It’s another control loop: every 15 seconds or so it compares observed usage with a target and computes the replica count that would bring usage back to the target.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: notes-api
  namespace: notes
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: notes-api
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

averageUtilization: 70 means “keep average CPU usage at 70% of the CPU request”. With a 250m request, that’s about 175m per Pod. At six Pods averaging 350m, the HPA computes 6 × 350 / 175 = 12 and scales to the maximum. The behavior block makes scale-down wait for five minutes of consistently lower load, so a brief lull doesn’t remove capacity you need a minute later. Five minutes is already the default; writing it down makes the choice visible and easy to tune.

Watch it work with kubectl get hpa notes-api --watch while you run a load test. If the targets column shows <unknown>, either the containers have no CPU request or metrics-server isn’t running.

CPU suits notes-api because its work is CPU-bound request handling. Services whose load shows up elsewhere, like queue depth or requests per second, can scale on custom or external metrics through the same metrics list, with an adapter installed in the cluster.

notes-api now has the resources it needs and grows with load. The next lesson changes the image safely: rolling updates, rollbacks and packaging the whole thing with Helm.