1. spec.nodeName, and the two-phase cycle that fills it

A Pod is an ordinary API object with a string field called spec.nodeName, and when that field is empty nobody has decided where the pod runs. Filling it in is the entire job of kube-scheduler. Everything else (the plugins, the queues, the scoring, the two-phase cycle) exists to decide what string goes in that field and to write it without racing anyone else.

That framing explains some behaviour that otherwise looks strange. You can set nodeName yourself and the pod will run without the scheduler ever seeing it. Delete kube-scheduler entirely and every pod that already has a nodeName keeps running, while every new pod sits in Pending forever. Nothing about a running workload passes through the scheduler; it’s a control-loop component that watches for objects missing one field and supplies it.

Being Pending isn’t quite the same as being unscheduled, which trips people up when they’re reading kubectl get pods output. A pod is Pending from creation until at least one of its containers is running, so the phase covers both “no node picked yet” and “node picked, image still pulling”. The field that actually distinguishes them is the PodScheduled condition:

status:
  conditions:
    - type: PodScheduled
      status: "False"
      reason: Unschedulable
      message: "0/5 nodes are available: 5 Insufficient cpu."
  phase: Pending

Once a node is chosen, PodScheduled flips to True, spec.nodeName is populated, and responsibility passes to the kubelet on that node. From there the container runtime takes over, which is a separate story about the CRI.

The work of choosing a node splits into two phases with different concurrency properties, and almost every surprising scheduler behaviour traces back to that split:

• The scheduling cycle picks a node for one pod. It runs serially. One pod at a time, cluster-wide, per scheduler instance.
• The binding cycle applies that decision, and it runs concurrently: many pods can be in this phase at once.

The serial scheduling cycle is what makes the scheduler’s decisions coherent: while it’s deciding where pod A goes, no other pod is being placed, so the view of available capacity it reasons about is stable. Binding runs concurrently because it’s the slow half, attaching volumes and waiting on the API server, and serializing that as well would make throughput a disaster. Section 2 covers what happens before either cycle starts; sections 3 and 4 then take each in turn.

2. From kubectl apply to the scheduling queue

The scheduler learns about your pod through a watch, not a poll, and it maintains its own in-memory queue of unscheduled pods rather than querying the API server each time it needs work. Understanding that queue is most of what you need to predict how quickly a stuck pod gets retried.

2.1. The informer, not a polling loop

kube-scheduler opens watch connections to the API server for pods, nodes, PersistentVolumes, StorageClasses, and a handful of other resources, then keeps a local cache in sync from the resulting event stream. This is the standard client-go informer pattern that every controller in Kubernetes uses, and two consequences follow from it.

The first is that the scheduler’s picture of the cluster is eventually consistent and can be stale. It schedules against its cache, not against a fresh read. Section 3.6 covers the other direction, where the scheduler deliberately writes placements the API server hasn’t confirmed yet into that same cache.

Second, the scheduler reacts to events rather than to elapsed time. A pod that failed to schedule doesn’t get retried on a timer in any meaningful sense. It gets retried when something happens that might have changed the answer.

2.2. PreEnqueue and scheduling gates

Before a pod is allowed into the scheduling queue at all, it passes through PreEnqueue plugins. Only if every one of them returns success does the pod enter the active queue. Otherwise it goes straight into the unschedulable pod pool without the scheduler ever attempting to place it.

The user-facing feature built on this extension point is spec.schedulingGates, stable since Kubernetes v1.30. A gate is a named string, and a pod carrying any gate will not be considered for scheduling:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  schedulingGates:
    - name: example.com/quota-check
    - name: example.com/dependency-ready
  containers:
    - name: pause
      image: registry.k8s.io/pause:3.6

The pod shows up with a distinct status, so you’re not left guessing why nothing is happening:

NAME       READY   STATUS            RESTARTS   AGE
test-pod   0/1     SchedulingGated   0          7s

Gates can only be added at creation time, by the client or by an admission webhook; the API server disallows adding one to an existing pod. They can be removed in any order. Once the list is empty, the pod enters the queue normally. That asymmetry is deliberate: it means a controller can hold a pod back from birth, but nobody can yank a running scheduling decision out from under you by gating a pod mid-flight.

The pattern this enables is external admission control. A quota controller, a licence-server check or a batch system that wants to release work in waves can all gate pods at creation and ungate them when their own precondition is met, without writing a scheduler plugin or a custom scheduler. Before gates existed, the usual workaround was to create the pods only when you were ready, which meant the pods weren’t visible in the API for anything else to reason about.

2.3. Three queues, not one

The scheduler holds unscheduled pods in three distinct structures, and a pod’s position among them determines when it will next be looked at.

A newly created pod lands in activeQ. If a scheduling attempt fails, the pod doesn’t go back to activeQ, because retrying immediately against an unchanged cluster would just burn cycles producing the same failure. It goes to unschedulablePods instead, where it waits for the cluster to change in some way that might matter.

The backoff queue handles the middle case. A pod that has failed repeatedly serves an increasing penalty before it’s eligible again, so one pathologically unschedulable pod can’t monopolize the scheduling loop. Backoff grows with the number of failed attempts.

One refinement changes latency under load: when activeQ is empty, the scheduler pops from backoffQ rather than idle. There’s no point making a pod sit out its penalty when there’s nothing else to do, and the old behaviour left the scheduler asleep while work was available. This is the SchedulerPopFromBackoffQ feature gate, beta and on by default since Kubernetes v1.33.

2.4. QueueSort decides who goes first

Exactly one QueueSort plugin can be enabled at a time, and it supplies a single function: Less(pod1, pod2). That function totally orders activeQ.

The default implementation sorts by pod priority first, then by the timestamp of when the pod entered the queue. Higher priority pods get placed before lower priority ones, and among equals it’s first-come-first-served. This is a different mechanism from preemption even though both derive from PriorityClass, and conflating them causes real confusion. QueueSort decides what order pods are considered in. Preemption decides whether an already-running pod gets evicted to make room. A high-priority pod benefits from the first whether or not the second ever triggers.

The single-plugin restriction is a design consequence rather than an arbitrary limit. Two sort functions can’t both totally order one queue without a tiebreaking rule between them, and rather than invent one, the framework requires you to pick.

2.5. QueueingHints, and the five-minute problem they fixed

A QueueingHint is a per-plugin callback that answers one question: given this cluster event, could this specific pod now be schedulable? It’s what gets a pod out of unschedulablePods, and it replaced a periodic flush that requeued everything on a timer, every 30 seconds on some paths and up to every 5 minutes in the worst case.

That old flush was crude in both directions. A pod that was unschedulable because it wanted a GPU would be retried when an unrelated node’s labels changed, when any pod anywhere was deleted, and again on the timer regardless. Most of those retries were wasted work, and in a large cluster the waste was substantial. Meanwhile a pod could sit unschedulable for minutes after the exact change that would have let it run, because the flush hadn’t come around yet.

Hints fix both because the plugin that rejected the pod is the one that gets asked, and it’s the only component that knows why the pod failed. If NodeResourcesFit rejected a pod for insufficient memory, it can look at a node-update event and say that a change to the node’s labels is irrelevant while a change to its allocatable memory is worth a retry. Pods stop being retried against changes that can’t help them. A change that does help no longer sits unnoticed while a timer runs down.

The feature gate is SchedulerQueueingHints, and it reached GA in Kubernetes v1.34, so on any recent cluster this is simply how the scheduler behaves. On older clusters it went through a rocky beta, including a release where it was disabled by default after regressions. If you’re on something in the v1.28 to v1.32 range and reasoning about requeue latency, read the gate’s value off the kube-scheduler pod’s --feature-gates flag before assuming hints are on.

3. The scheduling cycle

The scheduling cycle takes one pod off the active queue and produces one node name, or a failure. It runs serially for the whole scheduler, so its total throughput is a real cluster-scaling concern and several of its design decisions are there to keep it fast.

The extension points run in a fixed order, and most of them can end the attempt early. Here is the whole sequence, from the moment a pod becomes eligible for the queue to the moment the kubelet can see it:

Points 1 and 2 belong to the queue rather than to either cycle proper; they decide whether and in what order a pod is offered to the scheduling cycle at all. Points 3 through 10 are the serial scheduling cycle covered in this section, and 11 through 14 are the concurrent binding cycle in section 4.

3.1. PreFilter

PreFilter plugins run once per scheduling attempt, before any node is looked at. They do two jobs: check pod-level conditions that don’t depend on which node is chosen, and compute state that the per-node Filter calls will need.

The second job is the performance-critical one. Consider pod anti-affinity, which needs to know about existing pods matching a label selector. Computing that set is expensive, and it’s identical for every candidate node. PreFilter computes it once and stashes it in a cycle-scoped state object that Filter reads per node. Without that split, an expensive computation would run once per node per pod.

If a PreFilter plugin returns an error, the scheduling cycle aborts immediately. There’s no point evaluating nodes when a pod-level precondition already failed.

3.2. Filter, and what “feasible” means

Filter plugins answer one yes-or-no question per node: can this pod run here? Nodes that pass every filter are feasible nodes, which is the upstream term and the one the rest of this post uses.

Nodes may be evaluated concurrently, so filtering parallelizes across cores even though the cycle as a whole is serial with respect to other pods. And the moment any filter marks a node infeasible, the remaining filters for that node are skipped, because the answer can’t change.

That short-circuit has a visible consequence you’ll meet again in section 6. When the scheduler reports why a pod couldn’t be placed, the reasons it gives per node are not exhaustive. A node might fail on both an untolerated taint and insufficient memory, but you’ll only be told about whichever filter ran first. Fixing the reported problem and finding the pod still stuck is normal, not a bug.

Filtering is where most of the constraints in the rest of this series actually execute. nodeSelector, required node affinity, taints, and resource requests are all filters. Section 8 has the full mapping.

3.3. percentageOfNodesToScore, or why the scheduler stops looking

The scheduler doesn’t evaluate every node in a large cluster. Once it has found enough feasible nodes, it stops filtering and moves to scoring, and “enough” is governed by percentageOfNodesToScore.

Leave it unset (the default value of 0) and the scheduler computes a threshold with a linear formula that yields roughly 50% for a 100-node cluster and 10% for a 5000-node cluster, with a floor of 5%. There’s also a hard floor of 100 nodes: in a cluster with fewer than 100 feasible nodes the scheduler checks all of them, because early exit saves nothing at that size.

This is a deliberate trade of placement quality for latency. Scoring 500 nodes out of 5000 finds a good node quickly; it does not find the best node. If you care more about optimal packing than about scheduling throughput, raising the value is the knob, and setting it to 100 restores exhaustive evaluation.

Something has to keep this fair. If the scheduler always started its scan at the beginning of the node list, the first 500 nodes would absorb every pod and the rest of the cluster would stay empty. So the scan is round-robin across scheduling cycles: the scheduler remembers where it stopped for the previous pod and resumes from that point for the next one, wrapping around at the end. The node list is also interleaved by zone. Take six nodes where zone 1 holds nodes 1 to 4 and zone 2 holds nodes 5 and 6: the scan visits them in the order node 1, node 5, node 2, node 6, node 3, node 4, rather than draining zone 1 before touching zone 2.

3.4. PostFilter, where preemption lives

PostFilter plugins run only when the Filter phase found no feasible nodes at all. On the happy path they never execute.

The default PostFilter implementation is preemption. It asks whether evicting one or more lower-priority pods would make a node feasible, and if so it picks victims, records the chosen node in the pod’s status.nominatedNodeName, and begins deleting them. The pod itself does not get scheduled in this cycle. It returns to the queue and tries again later, hopefully into the space its victims vacated.

That last detail explains a common observation. nominatedNodeName is set, the victims are gone, and yet the pod lands somewhere else entirely. Nomination is a hint recorded for the benefit of other scheduling decisions, not a reservation, and nothing stops a different pod from taking the freed capacity first.

Preemption has enough surface area to deserve its own treatment, and it shares that surface with two other mechanisms that also make pods disappear from nodes. A later post in this series covers priority, preemption, node-pressure eviction, and API-initiated eviction together, because telling them apart from the symptom is most of the practical difficulty.

3.5. PreScore, Score, and NormalizeScore

Scoring ranks the feasible nodes so the scheduler can pick the best one rather than the first one. It mirrors the filtering structure: a once-per-cycle preparation step, then a per-node step.

PreScore computes shared state, exactly as PreFilter does for Filter. An error here aborts the cycle.

Score runs each scoring plugin against each feasible node, producing an integer per plugin per node. The range of values a given plugin produces internally are entirely up to it.

NormalizeScore is what makes those numbers comparable. A plugin gets to see all of its own scores across all nodes and rewrite them into the framework’s range, [0, MaxNodeScore], where MaxNodeScore is 100. This exists because plugins can’t coordinate on a scale: one might naturally produce values from 0 to 10 and another from 0 to 1,000,000, and summing those raw would let the second silently dominate. Normalization puts every plugin on the same footing, and the configured per-plugin weight then puts imbalance back where you want it.

The final score for a node is the weighted sum across plugins. Highest total wins. When several nodes tie for the highest score, the scheduler picks among them at random, which is a small detail with a useful property: identical pods hitting identically-scored nodes spread out instead of stacking.

Scoring is where “preferred” constraints live. Preferred node affinity, preferred pod affinity, and the ScheduleAnyway mode of topology spread all express themselves as score contributions rather than hard filters.

3.6. Reserve, Permit, and the assume step

The last two extension points of the scheduling cycle are Reserve and Permit. Both run before the binding cycle starts, which is easy to get wrong because their effects are felt during binding.

Reserve exists to prevent a race. The scheduler has picked a node, but the API write hasn’t happened yet, and the scheduling cycle is about to move on to the next pod. If nothing recorded the decision, the next pod would be scheduled against a cluster view that doesn’t know the previous pod is coming, and both could be placed into capacity only one of them can have. Reserve plugins update their own state to account for the pod at its chosen node before that window opens.

The scheduler-level version of this is the assume step: the pod is marked in the scheduler’s internal cache as though it were already running on the chosen node. From this instant, capacity accounting for subsequent pods reflects a placement the API server hasn’t confirmed. This optimism is what lets the serial scheduling cycle release the pod to the concurrent binding cycle and immediately start on the next one, instead of blocking on a network round-trip.

Reserve is defined with a paired Unreserve method. If anything later in the cycle fails, Unreserve runs on all Reserve plugins, in reverse order of their Reserve calls, so every speculative claim gets released. The reverse ordering is the same discipline as unwinding a stack of locks.

Permit is the last step, and it’s the one that lets a plugin hold a pod back. It has three possible outcomes: approve, deny, or wait with a timeout. Approve lets binding proceed; deny sends the pod back to the queue and triggers Unreserve. Wait parks the pod pending some external condition, and if the timeout expires, the wait becomes a deny.

Wait is how gang scheduling was traditionally implemented out-of-tree. Each pod of a group runs its full scheduling cycle, reserves its node, and then waits at Permit until enough members of the group have also reached that point. Only then does the plugin approve them all together. Section 7 covers what changed when this moved in-tree.

The subtlety that makes any of this workable: Permit runs at the end of the scheduling cycle, but the waiting happens in the binding cycle. If a pod waiting for its gang blocked the serial scheduling cycle, one incomplete gang would deadlock the entire cluster’s scheduling. Instead the pod is handed to the concurrent binding phase, which waits, and the scheduling cycle moves on.

4. The binding cycle

The binding cycle turns a decision into a fact. It waits out whatever Permit decided, does the slow preparation work that a chosen node makes possible (volumes, mostly), and issues the API write that finally puts a node name on the pod.

4.1. WaitOnPermit

The first thing the binding cycle does is block on whatever Permit decided. If Permit returned approve, this passes through instantly. If it returned wait, the pod sits here until the plugin approves it or the timeout converts the wait into a denial.

A denial at this point unwinds everything: Unreserve runs across all Reserve plugins and the pod returns to the scheduling queue. The node it had picked is released back into the accounting.

4.2. PreBind, where the slow work happens

PreBind plugins do work that must complete before the pod can be bound. In practice this means volumes.

The VolumeBinding plugin is the one you’ll meet most often, and it’s the reason storage and scheduling are entangled. When a PersistentVolumeClaim uses a StorageClass with volumeBindingMode: WaitForFirstConsumer, the PersistentVolume deliberately isn’t provisioned when the claim is created. It’s provisioned here, in PreBind, once a node has actually been chosen. The whole point is that for node-local storage the volume has to be created on the node the pod will run on, so provisioning has to wait for the scheduling decision rather than precede it. The storage post covers the PV, PVC, and StorageClass model in full, including why WaitForFirstConsumer is the right default for local volumes and what breaks when it isn’t used.

This is also the clearest illustration of why the binding cycle is concurrent. Provisioning a volume and waiting for a CSI driver to respond can take seconds. Serializing that across every pod in a deployment rollout would be painful.

If a PreBind plugin returns an error, the pod is rejected and returns to the scheduling queue.

4.3. Bind, and the Binding object

Bind is where spec.nodeName finally gets set, and it’s done through a dedicated subresource rather than by patching the pod.

The scheduler issues a POST to the pod’s binding subresource:

POST /api/v1/namespaces/{namespace}/pods/{name}/binding

with a Binding object naming the target:

apiVersion: v1
kind: Binding
metadata:
  name: nginx-7d8b49557c-4xhzr
  namespace: default
target:
  apiVersion: v1
  kind: Node
  name: worker-03

The API server handles this by setting spec.nodeName on the pod and persisting it to etcd, which is the write that the cluster’s etcd topology ultimately has to serve. The kubelet on worker-03 is watching for pods bound to itself, sees the update, and starts pulling images.

The subresource design isn’t decoration. It gives RBAC a distinct verb to grant, so a scheduler’s service account can be permitted to create bindings without being permitted to modify pod specs generally. If you’re writing a custom scheduler, pods/binding is the permission you need, and that separation is why running your own scheduler doesn’t require handing it broad write access to pods.

Multiple Bind plugins can be configured, they’re called in order, and each may decline to handle a given pod. The first one that handles it wins and the rest are skipped. That’s how a cluster can route different pods to different binding implementations.

4.4. PostBind

PostBind runs after a successful bind and is purely informational. Plugins can’t reject anything here, because the decision is already durable in etcd. It’s for cleaning up whatever cycle-scoped state a plugin accumulated.

This is the end of the scheduler’s involvement. Everything after this, image pulls, container creation, probes, is the kubelet’s business.

5. The three ways to skip the scheduler

Three routes put a pod on a node without kube-scheduler choosing it: an explicit spec.nodeName, a static pod, and, historically, a DaemonSet. All three turn up in real clusters, so when a pod is running somewhere baffling, the first question to ask is whether the scheduler was involved at all.

5.1. nodeName

Set spec.nodeName yourself and there’s nothing left for the scheduler to do:

apiVersion: v1
kind: Pod
metadata:
  name: pinned
spec:
  nodeName: worker-03
  containers:
    - name: app
      image: nginx

The kubelet on worker-03 sees a pod assigned to it and runs it. No filtering, no scoring, no binding cycle.

The failure modes follow directly from skipping every check the scheduler would have performed. If the node doesn’t exist, the pod stays Pending with nothing retrying it, until the pod garbage collector notices a pod bound to a node that isn’t there and deletes it. If the node exists but lacks the resources, the pod gets as far as the kubelet, whose own admission check then rejects it. The pod goes to Failed with reason OutOfcpu or OutOfmemory, a different signal in a different place from the PodScheduled condition you’d be reading otherwise. Taints aren’t consulted, so a pod can be pinned onto a node specifically marked as unsuitable. And node names aren’t stable in cloud environments where instances are replaced, which turns a pinned pod into a pod that stops working the next time the node pool rolls.

Upstream recommends against it, and the recommendation is sound for workloads. nodeSelector and node affinity express the same intent while keeping the scheduler’s validation, and the next post in this series covers both.

Where it’s genuinely useful is debugging: pinning a diagnostic pod to a specific node to inspect it is exactly the case where you want to bypass the checks rather than satisfy them.

5.2. Static pods

Static pods are managed directly by a kubelet, not by the control plane. The kubelet watches a directory on its own filesystem, /etc/kubernetes/manifests by default, and runs whatever pod manifests it finds there.

This is how the control plane bootstraps itself. On a kubeadm cluster the API server, controller manager, etcd, and kube-scheduler are all static pods, which resolves an otherwise circular dependency: the scheduler can’t schedule the API server, because scheduling requires an API server to talk to.

The kubelet also creates a mirror pod in the API server for each static pod, so they show up in kubectl get pods. The mirror is read-only in the sense that matters here: deleting it doesn’t stop the container, because the kubelet’s source of truth is the file on disk, and it will simply recreate the mirror. If you’ve ever deleted a control-plane pod and watched it come straight back, this is why.

5.3. DaemonSets, and why they stopped bypassing

DaemonSets used to bypass the scheduler. The DaemonSet controller set spec.nodeName directly on each pod it created, which is the nodeName route applied systematically.

That changed in Kubernetes v1.12, and DaemonSet pods now go through the default scheduler like everything else. The controller creates pods with a required node affinity term matching a specific node’s name rather than setting nodeName. Note matchFields rather than matchExpressions: metadata.name is a field of the Node object, not a label on it, and the API server rejects the term if you reach for the wrong one.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchFields:
              - key: metadata.name
                operator: In
                values:
                  - worker-03

The effect on placement is identical, one pod per eligible node, but now those pods participate in everything the scheduler does. They respect resource accounting and can preempt lower-priority pods when a node is full. Their failures surface as ordinary scheduling events you can read. Under the old model a DaemonSet pod could be placed onto a node with no room for it and simply fail there.

The DaemonSet controller also adds a set of tolerations automatically, which is why DaemonSet pods keep running on nodes that reject everything else, including nodes under disk or memory pressure and nodes that have been cordoned. Control-plane nodes are not in that set; a DaemonSet that needs to run there declares the toleration itself. Those tolerations are the subject of the next post.

6. Reading a scheduling failure

When scheduling fails, the scheduler records a FailedScheduling event and writes the reason into the pod’s PodScheduled condition. Learning to read that message properly saves a lot of guessing, and the cluster-level metrics tell you whether you’re looking at one stuck pod or a scheduler in trouble.

6.1. The FailedScheduling event message

kubectl describe pod api-7d8b49557c-4xhzr

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  27s   default-scheduler  0/6 nodes are available:
    1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
    2 Insufficient cpu,
    3 node(s) didn't match Pod's node affinity/selector.
    preemption: 0/6 nodes are available:
    2 No preemption victims found for incoming pod,
    4 Preemption is not helpful for scheduling.

Four details in that output carry the diagnosis.

The counts sum to the cluster size, and each node appears once. Six nodes, and 1 + 2 + 3 = 6. This is the short-circuit from section 3.2 showing through. Each node is reported under the first filter that rejected it, not under every filter that would have. Those two CPU-starved nodes might also fail the affinity check; you’d never know from this message. So resolving the reported cause for a node doesn’t guarantee that node becomes feasible, and the message you get on the next attempt may name a different reason for the same node.

The preemption: block is a separate report. It’s the PostFilter phase telling you what preemption concluded, and it’s only present because the Filter phase found nothing. “No preemption victims found” means there were no lower-priority pods on those nodes to evict. “Preemption is not helpful” means evicting things wouldn’t fix the reason the node was infeasible in the first place, which is the expected answer when the blocker is a taint or an affinity mismatch rather than capacity.

The message is a snapshot, not a log. It reflects the most recent attempt. Under QueueingHints a pod may sit untouched for a long time between attempts precisely because nothing relevant has changed, so a timestamp of several minutes ago doesn’t mean the scheduler has stopped caring.

status.nominatedNodeName tells you preemption fired. If it’s set, the scheduler decided this pod is worth evicting others for and picked a target. As covered in section 3.4, the pod may still land elsewhere.

6.2. Metrics and log verbosity

For cluster-level health rather than a single pod, three scheduler metrics carry most of the signal:

The error result deserves attention because it’s distinct from unschedulable. Unschedulable means the scheduler did its job and concluded no node fits. Error means something went wrong inside the attempt, and a persistent error rate usually points at a plugin or an API server problem rather than at cluster capacity.

When you need more than the events, raising the scheduler’s log verbosity is the next step. Levels around --v=10 produce per-node filter and score output for each attempt, which answers “why did it pick that node” in a way nothing else does. It’s also extremely verbose, so it’s a targeted diagnostic rather than a setting to leave on.

7. Where the per-pod model breaks

Everything so far assumes the unit of scheduling is one pod. For all-or-nothing workloads that assumption produces partial scheduling and, from there, resource deadlock: the failure that gang scheduling exists to prevent, and the one Kubernetes has only recently started addressing in-tree.

A distributed training job needs 8 workers to do anything useful. The cluster has room for 5. The scheduler, working one pod at a time, places those 5 happily, and they sit there consuming GPUs while waiting for 3 peers that will never arrive. Now a second job needs 8, finds room for none, and waits. Neither job runs. Both hold resources, and nothing resolves without intervention. None of that is a bug in the scheduler; it’s the direct consequence of a per-pod decision model meeting an all-or-nothing workload.

Every fix until now has lived outside the default scheduler, and no two of them work the same way. The coscheduling plugin in the scheduler-plugins repo is the one that uses the Permit wait from section 3.6: each pod runs its own scheduling cycle, reserves its node, and then waits at Permit for its groupmates. It works, but the reservation still happens per pod, so the group holds partial state for as long as it waits. Kueue takes a different route entirely, suspending Jobs and admitting Workloads only once the whole thing fits, and leaning on scheduling gates to keep the pods out of the queue until then. Volcano goes furthest. It ships its own scheduler binary, replacing the per-pod loop with session-based gang logic of its own.

Kubernetes v1.35 introduced a first-class API for this, and v1.36 restructured it. The current shape splits into two objects: a Workload, which is a static template describing the group’s shape and scheduling policy, and a PodGroup, which is the runtime object the scheduler actually reads.

apiVersion: scheduling.k8s.io/v1alpha2
kind: Workload
metadata:
  name: training-job-workload
  namespace: some-ns
spec:
  podGroupTemplates:
    - name: workers
      schedulingPolicy:
        gang:
          minCount: 8

The gang policy with minCount states the all-or-nothing threshold directly. Alongside it, v1.36 added a PodGroup scheduling cycle that runs beside the per-pod one. Rather than each pod reserving independently, the scheduler takes a single cluster snapshot, finds placements for the whole group against it, and applies the outcome atomically. Either enough pods are placed and they all proceed to binding, or none are bound and the entire group returns to the queue.

That’s a genuine change to the model this post has described, and it’s the reason section 1’s “one pod at a time” framing needs an asterisk. Be careful with it anyway: the feature is alpha, off by default, and gated behind GenericWorkload with an alpha API group that has already changed once between releases (v1alpha1 in v1.35 became v1alpha2 in v1.36). The per-pod cycle remains what runs in every cluster that hasn’t opted in, which is nearly all of them. Treat this section as orientation for where the scheduler is going, not as something to build on today.

8. Which extension point your constraints run at

Every placement constraint Kubernetes offers resolves to one of two things: a Filter, which makes a node ineligible, or a Score, which makes a node more or less attractive. That single distinction predicts a constraint’s behaviour better than any amount of documentation about its syntax.

Read down the Filter rows and you have the answer to “why is my pod Pending”. The Score rows answer “why did it go there”. Those are different questions with different diagnostic paths, and the table is the map between them.

Two patterns fall out of the table and carry into the rest of this series.

The first is that “required” and “preferred” are different phases of the cycle, not two intensities of one setting. A preferred constraint doesn’t try harder and give up; it never had the power to make a node ineligible in the first place, so it contributes points and loses to anything expressed as a filter. When someone says a preferred anti-affinity rule “isn’t working”, this is almost always what’s happening.

The second is that filters are cheap and scores are not. A filter can short-circuit on the first failure and skip the rest; a score has to be computed for every feasible node and then normalized against every other node’s score for the same plugin. Combined with the round-robin sampling from section 3.3, this is why constraint-heavy pods schedule measurably slower in large clusters, and why upstream warns specifically about pod affinity at scale.

The rest of this series works through the table one band at a time. Node-targeting constraints, nodeSelector and node affinity and taints, come next. Pod-relative placement follows, covering pod affinity and topology spread. Then resource-based scheduling, where requests, limits, and QoS classes decide what “fits” even means. After that, priority, preemption, and eviction, which is where the three separate mechanisms that pull a pod off a node get told apart. Last is node administration and the day-2 operations that drive scheduling deliberately, cordon and drain and the descheduler.

All of them assume what this post established: an empty spec.nodeName, a serial cycle that decides, a concurrent cycle that commits, and a queue that decides when to try again.