Kubernetes gives you three probes, and they aren’t three flavours of the same check. Each one answers a different question, and the kubelet does something different with each answer. Wiring the wrong probe to the wrong question is how a cluster full of healthy pods restarts itself into an outage.

They are startupProbe, readinessProbe, and livenessProbe. Three probes, four mechanisms for running the actual check, and a handful of timing fields that most people copy from a sample manifest and never touch again.

1. Three probes, three questions

The kubelet runs all three. A probe is a diagnostic it performs periodically against a container, either by running code inside the container or by making a network request, and the probe type determines what it does with the result.

Two of the three probes kill containers. Only readiness is reversible: it’s a valve, not a trigger. A pod that fails readiness and later recovers walks back into rotation by itself, with nothing restarted and no state lost. A pod that fails liveness has its container destroyed, and whatever was in memory goes with it.

All three are configured per container, not per pod. A pod running an application container and a proxy sidecar can carry six probes, and they’re evaluated independently.

2. Four mechanisms: httpGet, tcpSocket, grpc, and exec

Every probe is configured with exactly one of four handlers, and the handler alone decides what counts as success.

httpGet is the default choice for anything that speaks HTTP. The success range is wider than people expect. A 302 redirect passes. So does a 204. The host field defaults to the pod’s IP and you almost never want to override it; if you need a specific Host header, set it through httpHeaders instead.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
    httpHeaders:
    - name: Host
      value: app.internal

tcpSocket gives you the weakest signal available. A successful connect tells you a process bound the port and the kernel accepted a handshake. It tells you nothing about whether the application behind that port can do any work. A wedged server with a listening socket passes this check forever. Use it when you genuinely have nothing better, typically for a protocol you can’t speak from a probe.

readinessProbe:
  tcpSocket:
    port: 5432

grpc went GA in 1.27 and retired an ugly workaround. Before it existed, checking a gRPC service meant baking the grpc_health_probe binary into your image and shelling out to it with an exec probe. Exactly the arrangement exec probes are worst at. The built-in mechanism requires your server to implement the standard gRPC Health Checking Protocol (grpc.health.v1.Health). The optional service field picks which registered service to query; leave it out and you’re asking about overall server health.

readinessProbe:
  grpc:
    port: 5000
    service: myapp.Orders

exec is the most flexible and by far the most expensive. The kubelet asks the container runtime to run your command inside the container’s namespaces, so every probe costs a process fork. One fork is nothing. At a hundred pods per node on the default ten-second period it adds up to ten forks per second of permanent background overhead, before any of your workloads do anything. I default to httpGet and reach for exec only when the check genuinely can’t be expressed as a request, because a health endpoint is almost always cheaper than a process.

That runtime round trip is the same plumbing the kubelet uses for every other container operation, which is why exec probe behaviour has historically shifted along with the container runtime interface underneath it.

3. The timing knobs and the arithmetic

Five fields control probe timing, and the defaults are tighter than most applications deserve.

successThreshold is the one field with a hard constraint: it must be 1 for liveness and startup probes, and the API server rejects any other value. Only readiness can require several consecutive successes before it believes you.

One piece of arithmetic falls out of these fields. Compute it before you ship:

On a startupProbe that number is your boot budget; on a livenessProbe it’s how long a wedged container survives before the kubelet kills it. initialDelaySeconds defaults to 0 so in most manifests the whole thing collapses to period times threshold.

With the defaults, a liveness probe gives a wedged container up to thirty seconds before restarting it. That part is usually fine. The sharp edge is timeoutSeconds: 1. A health endpoint that normally answers in 15 milliseconds but occasionally takes 1.2 seconds behind a garbage collection pause counts as a failed probe, and three of those in a row destroys the container. Your health endpoint has to be fast, and it has to stay fast under load, which is precisely when it won’t be. Raising timeoutSeconds to something honest costs you nothing except a slightly longer detection window.

One historical note that still bites people on old clusters: before 1.20, exec probes ignored timeoutSeconds entirely and simply ran until they finished.

4. startupProbe: covering a slow boot

startupProbe exists so the other two probes don’t fire while your application is still coming up. While it’s defined and hasn’t yet succeeded, both the liveness and readiness probes are disabled. Once it passes, it never runs again for the life of that container. If it exhausts its failureThreshold first, the kubelet kills the container and the restart policy takes over. Leave it out and there’s no boot gate at all: readiness starts evaluating the moment the container is running, and liveness starts after initialDelaySeconds, which is zero unless you say otherwise.

Before it existed, the only tool for a slow boot was initialDelaySeconds on the liveness probe, and that forces a bad trade. Set the delay long enough to survive your worst-case cold start, say three minutes, and you’ve also told Kubernetes to ignore a wedged container for three minutes after every restart, forever. The one-time cost of booting gets permanently baked into your steady-state detection time. startupProbe decouples the two: boot gets a generous budget, and detection stays tight the moment boot is over.

The configuration pattern that follows from this is a short period with a high failure threshold, which is the opposite of what people usually write.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 60      # 5 × 60 = 300 seconds of boot budget
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 15
  failureThreshold: 4       # ~60 seconds to notice a wedge

periodSeconds controls granularity and failureThreshold controls the ceiling. A short period means an application that boots in eight seconds starts serving in roughly eight seconds rather than waiting out a fixed delay, while the high threshold still covers the pathological case: a JVM service with an enormous classpath, or anything that runs schema migrations before it opens a port.

5. readinessProbe: traffic control

Readiness decides whether the pod’s IP appears in the EndpointSlices backing its Services. A failure removes it, and kube-proxy or whatever dataplane you run stops sending new connections. Nothing restarts or terminates, and no state is lost. With no readiness probe at all, the container counts as Ready the instant it reaches Running, which is the usual explanation for traffic arriving at a pod that hasn’t finished booting.

That’s the primary consequence, not the only one. The pod’s Ready condition also feeds availableReplicas on Deployments and ReplicaSets, the ordered rollout logic in StatefulSets and currentHealthy in PodDisruptionBudgets. One unready pod is just a pod out of rotation. A fleet-wide readiness flap is something else: it stalls rolling updates behind maxUnavailable, and it blocks voluntary evictions, so a node drain sits there waiting for a budget that never becomes healthy.

Because readiness runs for the container’s entire lifetime rather than just at startup, it’s bidirectional. Pods drop out of rotation when they’re struggling and walk back in on their own when they recover. Being wrong about a single pod costs almost nothing, and that’s the whole reason readiness can afford to be twitchy in a way liveness can’t.

5.1. Never check dependencies in a readiness probe

Checking a database or a downstream API from a readiness probe is the single most expensive readiness mistake, and it looks like diligence when you write it. If /ready returns 503 whenever the database is unreachable, a brief database problem pulls every replica out of the Service at the same instant, because every replica is looking at the same database and fails identically. The Service now has zero endpoints. Requests that needed the database were already going to fail; now the ones that didn’t need it fail too, along with anything you could have served from cache or returned as a static degraded response. The graceful degradation you built is the first thing the probe switched off. A partial outage has been promoted to a total one, and the pods can’t take traffic again until the dependency returns.

Readiness answers “can this instance serve a request right now”, not “is the system healthy”. The probe only earns its place when replicas can fail independently of each other. A check that all replicas fail simultaneously isn’t load shedding, it’s a synchronised outage.

successThreshold is the flap-damping tool here, and readiness is the only probe allowed to use it. Setting it to 2 or 3 stops a pod rejoining rotation on the strength of a single lucky probe when it’s recovering unevenly.

6. livenessProbe: restarts and cascade risk

The liveness probe is the only one whose failure is destructive and irreversible, and the honest default is to not configure one at all until you can name what it catches. Define none and a container is restarted only when its process exits, which is what restartPolicy gives you for free.

When a liveness probe fails failureThreshold times in a row, the kubelet kills the container and the pod’s restartPolicy decides what happens next. What gets restarted is the container, not the pod: the pod keeps its IP, stays on the same node, and its restart count ticks up. Repeated failures back off exponentially, starting around ten seconds and capping at five minutes, which is the state that surfaces as CrashLoopBackOff.

A liveness probe earns its place in exactly one situation: the process is alive but permanently unable to make progress, stuck on a deadlock, a connection pool that’s been exhausted and never released, or a thread pool waiting behind a lock nobody will unlock. Those states are real, and they’re also specific. If you can’t name the unrecoverable state your probe detects, you’ve added a restart trigger that can only hurt you.

6.1. How liveness probes cause outages

Load climbs, the probe times out, the kubelet restarts the container, and the restart takes capacity away from a service that was already short of it. The loop feeds itself, and it arrives exactly when you can least afford it.

A service comes under heavy load. Latency climbs, and the health endpoint, which shares a thread pool with real request handling, starts answering slowly. Probes with timeoutSeconds: 1 begin timing out. Containers get killed and restarted, so capacity drops. The remaining replicas absorb the load that the restarting ones were carrying, and their latency climbs further. Their probes start timing out too. Every restart also throws away in-flight requests and returns a cold-cache instance to the pool, which makes the next round worse.

The fix isn’t clever. Give liveness a much looser budget than readiness, because their failures cost wildly different amounts. Readiness can trip after ten seconds since its consequence is reversible and cheap. Liveness should be patient and hard to provoke, with a generous timeout and a high threshold. If both probes point at the same endpoint with the same thresholds, you don’t have two probes, you have one probe wired to two actions, and the destructive one wins.

Use separate endpoints. /healthz for liveness should return 200 unless the process is genuinely unable to continue, touching no dependencies and taking no locks. /ready for readiness can inspect local state that determines whether this instance can serve.

Watching the loop happen is mostly a matter of knowing where to look. kubectl describe pod shows Unhealthy events carrying the probe’s failure text and Killing events when the kubelet acts on them, and the RESTARTS column of kubectl get pods counts how often it has already acted. Fleet-wide, the kubelet exports prober_probe_total{probe_type,result}, so a rising failure rate on probe_type="liveness" lined up against a latency spike tells you the cascade is in progress rather than over. Both are worth instrumenting and alerting on, because a restart caused by a genuine crash and a restart caused by your own probe configuration look identical in a dashboard that only counts restarts.

6.2. Probe-level terminationGracePeriodSeconds

When a liveness or startup probe decides to kill a container, that container gets the pod’s terminationGracePeriodSeconds to shut down cleanly, 30 seconds by default. For a container that’s wedged, that’s usually 30 seconds of nothing at all, followed by SIGKILL. The probe-level override lets you shorten the wait for that specific case without touching the normal shutdown path.

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 15
      failureThreshold: 4
      terminationGracePeriodSeconds: 5

Sixty seconds for an orderly shutdown during a rolling update, five seconds when the probe has already established that the process is stuck. A deadlocked process isn’t going to drain connections no matter how long you wait for it.

The field has been available since 1.25 and stable since 1.28. It applies to liveness and startup probes only, and the API server rejects it on a readiness probe, which follows from readiness failures never terminating anything in the first place.

7. Readiness beyond containers: readinessGates and sidecars

A pod can have every container passing its readiness probe and still not be Ready. Two separate features cause this, and both are easy to misdiagnose.

7.1. readinessGates

Pods carry two related conditions. ContainersReady means what it says. Ready means ContainersReady and every condition named in spec.readinessGates is True.

spec:
  readinessGates:
  - conditionType: "elbv2.k8s.aws/pod-readiness-gate-ready"

Nothing inside Kubernetes sets that condition. An external controller has to patch it into the pod’s .status.conditions. A condition that’s missing entirely counts as not satisfied, so a gate with no controller behind it keeps the pod permanently unready.

The case this was built for is cloud load balancers in IP target mode, where traffic goes straight to pod IPs and bypasses kube-proxy. Registration takes time. A pod can pass its readiness probe well before the load balancer has finished registering it as a target and running its own health checks against it. Without a gate, a rolling update sees the new pod as Ready and terminates an old one. Traffic lands on a target the load balancer isn’t routing to yet. The gate makes pod readiness wait for the load balancer controller’s own confirmation, so the rollout paces itself against reality rather than the kubelet’s local view.

7.2. Probes on sidecars

Regular init containers support no probes at all. They run to completion, in order, and completion is the only signal available.

A sidecar container, on by default since 1.29 and stable since 1.33, is an init container declared with restartPolicy: Always. That one field changes two things: the kubelet moves on to the next init container once the sidecar has started rather than waiting for it to finish, and the sidecar keeps running alongside the application containers. Sidecars support all three probes.

Two consequences matter in practice. A readinessProbe on a sidecar feeds the pod’s ready state, so a service mesh proxy that isn’t ready keeps the whole pod out of rotation, which is what you want, since routing traffic to a pod whose proxy is down just manufactures errors. And a startupProbe on a sidecar delays the application containers until it passes, which is the clean way to guarantee a proxy has fetched its configuration before the app it fronts starts serving.

8. postStart and preStop are not probes

Lifecycle hooks look like probes, sit near them in the manifest, and do something completely different. Each runs once, at a container boundary, and the kubelet doesn’t use the result for any ongoing decision.

postStart fires immediately after container creation, concurrently with the entrypoint. There’s no guarantee it runs before your application’s first line of code, which surprises people who use it for initialisation. If it fails, the container is killed. preStop fires before the container receives SIGTERM, and the termination grace period clock starts when preStop starts, so a slow hook eats the budget your application needed for draining. Both hooks sit inside the wider shutdown contract a disposable process is expected to honour, where catching SIGTERM, refusing new work, and finishing what’s already in flight remain the application’s job rather than the platform’s.

8.1. The endpoint propagation race

preStop matters mostly because of a race that Kubernetes doesn’t resolve for you. When a pod is deleted, two things start in parallel and neither waits for the other:

1. The kubelet begins termination, running preStop and then sending SIGTERM.
2. The endpoints controller removes the pod from its EndpointSlices, and that removal has to propagate to every kube-proxy, ingress controller, and mesh sidecar in the cluster.

The second one is genuinely asynchronous and takes real time across a large cluster. So there’s a window where your application has already received SIGTERM and started refusing connections while some node’s dataplane still holds a rule pointing at it. What you see is connection-refused errors during what should be a clean rolling update. They’re maddening to chase, because every individual component is behaving correctly.

The standard workaround is to delay SIGTERM long enough for propagation to finish:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]

The sleep accomplishes nothing except buying time, and that’s the entire point. Throughout it the container keeps serving normally and stays in whichever dataplanes haven’t caught up yet. Make sure the pod’s terminationGracePeriodSeconds covers the sleep plus real drain time, or SIGKILL will arrive in the middle of the drain you were protecting.

A native sleep handler has been enabled by default since 1.30 and stable since 1.34. It does the same thing without depending on a shell being present in the image. That matters on distroless bases:

lifecycle:
  preStop:
    sleep:
      seconds: 10

9. Failure modes and a default posture

Most of the recurring mistakes are one mistake in different clothing: a probe doing a job that belongs to a different probe, or to no probe at all.

Here’s a reasonable starting point that encodes all of the above. Rolling updates, Service membership, and the propagation race only exist under a controller, so this belongs in a Deployment’s template.spec; it’s written as a bare pod purely for readability.

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  terminationGracePeriodSeconds: 45
  containers:
  - name: app
    image: example/app:1.0
    ports:
    - containerPort: 8080
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 5
      failureThreshold: 60
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 2
      successThreshold: 2
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 15
      timeoutSeconds: 5
      failureThreshold: 4
      terminationGracePeriodSeconds: 5

The shape is deliberate. Startup gets five minutes of budget at five-second granularity, so fast boots stay fast and slow ones survive. Readiness leaves rotation after ten seconds of trouble, because being wrong costs almost nothing, and needs two consecutive successes to come back, so a pod recovering unevenly can’t rejoin on one lucky probe. Liveness needs a full minute of uninterrupted failure and a five-second timeout before it destroys anything, because being wrong costs a restart. Once it has decided, it waits five seconds rather than forty-five; a process that has already proved it can’t make progress isn’t going to drain connections either. Two endpoints rather than one, so the twitchy check and the destructive check can never be the same check.

If you take one thing from this: ship with no liveness probe, and add one the day you can name the deadlock it catches.