1. The three storage objects

Kubernetes splits storage across three objects because they have three different lifetimes. A volume entry in the pod spec lives and dies with the pod. A PersistentVolume (PV) is a cluster-scoped object representing an actual piece of storage, and it outlives every pod that uses it. A PersistentVolumeClaim (PVC) is a namespaced request for storage that binds to exactly one PV.

The split exists so an application manifest never has to name a disk. The claim states what the workload needs: 20Gi, ReadWriteOnce, this class. Where that actually lives is the PV’s business, and it can be a logical volume on worker-03, an RBD image in a Ceph pool, or a directory on a NAS. The same Deployment runs unchanged on a single-node k3s box and on a Ceph cluster; only the StorageClass differs. Which backend that class should point at is a decision of its own, and section 11 covers it.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgdata
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 20Gi

The pod refers to the claim by name, never to the PV:

spec:
  containers:
    - name: postgres
      image: postgres:17
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: pgdata

Binding is exclusive and permanent. One PVC maps to one PV for the life of the claim, and capacity is not shared: a 100Gi PV bound to a 5Gi request is fully consumed, with the other 95Gi available to nobody. Because claims are namespaced and PVs aren’t, two namespaces can’t share a claim even when they’d happily share the underlying disk.

A PV comes into existence one of two ways. Under static provisioning someone creates PV objects ahead of time and claims bind against whatever matches. Under dynamic provisioning a StorageClass creates the PV on demand when the claim appears. Almost everything is dynamic now; static provisioning survives mostly for local disks, which section 7 covers.

2. StorageClasses and dynamic provisioning

A StorageClass names a provisioner and the parameters it should hand that provisioner. A PVC references the class by name, the provisioner creates real storage, and a matching PV object appears and binds. No administrator is in the loop.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

The storageclass.kubernetes.io/is-default-class annotation makes a class the fallback for claims that don’t name one. If more than one class carries the annotation, PVC creation doesn’t fail; Kubernetes quietly picks the most recently created default. That is almost never a deliberate choice. And storageClassName: "" is not the same as omitting the field. The empty string opts the claim out of dynamic provisioning entirely, so it only ever binds to a pre-created PV that also has an empty class.

2.1. volumeBindingMode: Immediate vs WaitForFirstConsumer

Immediate provisions the volume as soon as the claim is created, before any pod exists. WaitForFirstConsumer holds the claim in Pending until a pod that uses it gets scheduled, then provisions in whatever topology the scheduler chose.

The difference only bites on storage that has a location. Any node can reach any volume on a network-replicated backend like Longhorn or Ceph, so Immediate is fine there. Node-local and zonal storage invert the decision order: the provisioner picks a node first, and the scheduler is then forced to place the pod there. If that node is cordoned, out of CPU, tainted, or doesn’t satisfy the pod’s affinity rules, the pod stays Pending forever against a volume it can never reach.

Any backend whose storage is tied to a node or a zone should use WaitForFirstConsumer. Most local provisioners ship it as the default, but it costs nothing to check the class before you find out the hard way.

3. Access modes: RWO, ROX, RWX, and RWOP

ReadWriteOnce means one node, not one pod. The Kubernetes documentation is explicit that RWO “still can allow multiple pods to access the volume when the pods are running on the same node.” A lot of designs that assume RWO gives them a single-writer guarantee are relying on pod anti-affinity they never wrote down.

ReadWriteOncePod exists because RWO never meant what people assumed. It became stable in Kubernetes 1.29 and is CSI-only, so an in-tree or NFS-backed volume can’t offer it. If a workload genuinely cannot tolerate a second writer, RWOP is the only mode that says so to the scheduler.

A volume is mounted using one access mode at a time even when the PV advertises several, and the mode a claim asks for is a matching filter rather than a guarantee the storage enforces on its own. RWX also needs a backend that can actually do it. Shared-filesystem backends (NFS, CephFS) can; block devices (iSCSI, RBD, LVM, cloud disks) cannot, because two nodes mounting the same non-clustered filesystem corrupts it. A driver that refuses your RWX claim is usually protecting you.

volumeMode is the sibling field on the same claim, and it has two values. Filesystem is the default: the volume gets formatted if it has no filesystem and arrives in the container as a directory. Block skips all of that and hands the raw device through at volumeDevices[].devicePath, which is what a database managing its own on-disk layout or a consumer of a Ceph RBD image wants. Everything else takes the default, and the rest of this post assumes it.

4. The lifecycle of a persistent volume

A volume moves through provisioning, binding, attaching, mounting, and eventually reclaiming. Nearly every storage incident is one of those five steps stuck, and each step has a different owner.

4.1. Binding and PV phases

A PV usually reports one of four phases: Available (unbound), Bound, Released (the claim is gone but the volume hasn’t been reclaimed), and Failed.

Released is the one that traps people. Under the Retain policy the PV keeps a stale claimRef pointing at the deleted claim, and it will not bind to a new PVC while that reference is there, even one with an identical name in the same namespace. The volume looks idle and refuses every claim. Clearing the reference puts it back on the market:

kubectl patch pv local-nvme-0 -p '{"spec":{"claimRef":null}}'

4.2. Attach, stage, publish

Getting a volume into a container takes three steps split across two components, and the error messages only make sense if you know which is which.

Attach is the attach-detach controller in kube-controller-manager making the volume visible to a node, by calling a cloud API or logging into an iSCSI target. It’s tracked by a VolumeAttachment object. Stage (NodeStageVolume) is the kubelet mounting the volume once per node at a global staging path, formatting it first if it has no filesystem. Publish (NodePublishVolume) is the kubelet bind-mounting that staging path into the individual pod directory, which is why several pods on one node can share a single RWO attachment.

This is where Multi-Attach error for volume "pvc-…" comes from. The volume is still attached to another node, and for an RWO volume that means the replacement pod cannot start. The common trigger is a node going NotReady while holding a volume: the controller can’t cleanly detach from a node it can’t talk to, so it waits roughly six minutes before forcing the detach. If you know the node is genuinely gone rather than briefly unreachable, applying the node.kubernetes.io/out-of-service taint tells Kubernetes to stop waiting and detach immediately.

Attach limits are the quieter version of the same class of problem. Every node advertises how many volumes each driver may attach to it through the CSINode object’s allocatable count, and once that budget is spent the next pod requesting a PVC on that node stays Pending. Nothing in the pod’s events points at storage, because as far as the scheduler is concerned the node simply doesn’t fit.

4.3. Reclaim policy

Retain keeps the volume and its data after the claim is deleted, leaving cleanup to you. Delete destroys the PV object and the underlying storage. Recycle is deprecated and can be ignored.

Dynamically provisioned volumes default to Delete, which makes kubectl delete pvc a data-destruction command. Deleting a namespace is the same command with a wider blast radius, since the claims inside it go with it. The kubernetes.io/pvc-protection finalizer holds a claim in Terminating while a pod is still using it, so nothing gets yanked out from under a running workload. Once that pod is gone, the finalizer does nothing. For anything you’d miss, set reclaimPolicy: Retain on the class and accept the manual cleanup.

4.4. Expansion

Growing a volume means editing spec.resources.requests.storage upward on the claim, and it requires allowVolumeExpansion: true on the StorageClass. The controller resizes the backing device and the kubelet resizes the filesystem on top of it. Most CSI drivers do this online; some need the pod to restart before the filesystem picks up the new size.

Shrinking isn’t supported at all. Plan the number you’d regret least.

5. CSI and the driver model

Every storage backend worth using today plugs in through CSI (Container Storage Interface), a gRPC contract between Kubernetes and a driver that runs outside the Kubernetes codebase. The in-tree volume plugins that predate it are gone: awsElasticBlockStore was removed in 1.27, gcePersistentDisk in 1.28, and the Ceph rbd and cephfs plugins in 1.31.

That arc is the same one CRI ran for container runtimes, for the same reason. Vendor code compiled into the kubelet meant vendor bugs shipped on the Kubernetes release cadence and every new backend added surface area to core. An interface moved that cost back to the people who own the storage.

A driver has two halves with different jobs. The controller plugin runs as a Deployment, talks to the storage system’s API, and handles operations that aren’t node-specific: create, delete, attach, snapshot, resize. The node plugin runs as a DaemonSet on every node and does the mounting, since staging and publishing require being on the machine.

Neither half watches the Kubernetes API directly. Sidecars from the kubernetes-csi project do that and translate into CSI calls, so a driver author only implements the storage-specific part:

Drivers can also publish CSIStorageCapacity objects reporting how much space is left per topology segment. The scheduler reads them and stops placing pods on nodes whose storage pool is already full. That matters a great deal for local and LVM-backed drivers, and not at all for a shared network pool.

6. Ephemeral volumes

An ephemeral volume is created when the pod is created and destroyed when the pod is destroyed. It uses the same volumes and volumeMounts grammar as a claim, and comes in three shapes: scratch space, configuration, and a real provisioned volume that happens to have the pod’s lifetime.

emptyDir is created empty when the pod is assigned to a node and deleted when the pod leaves that node. It survives container crashes and restarts, not rescheduling. Setting medium: Memory makes it a tmpfs, which is fast and counts against the pod’s memory limit so a process that writes a few gigabytes into what looks like a directory gets OOM-killed for it.

configMap, secret, downwardAPI, and projected volumes are the same mechanism used for configuration rather than data. Generic ephemeral volumes are the interesting one: an inline PVC template that gets you a real dynamically provisioned volume with the pod’s lifetime, deleted automatically when the pod goes away. It’s the right tool for scratch space that needs to be larger or faster than the node’s disk.

volumes:
  - name: scratch
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          storageClassName: local-nvme
          resources:
            requests:
              storage: 50Gi

Everything a container writes to its own filesystem, plus every emptyDir, draws on the node’s ephemeral storage. You can put requests and limits on ephemeral-storage the way you would on memory, and if you don’t, the kubelet’s eviction manager makes the decision for you under disk pressure, ranking pods by usage relative to their request. A pod with no PVC anywhere near it can be evicted for writing logs too enthusiastically. This is the practical edge of the twelve-factor rule that a process’s filesystem is a brief-lived cache: Kubernetes will enforce that assumption whether or not your application shares it.

7. Node-local storage

Node-local storage is the fastest option available and the one that pins a pod to a machine. There’s no way to have the first property without the second, so the question is always whether the workload can absorb being pinned.

7.1. hostPath

hostPath mounts a path from the node’s filesystem straight into the container. It has no scheduling awareness at all: if the pod moves, it gets whatever happens to be at that path on the new node, which is usually nothing. It’s also a straightforward privilege escalation, since mounting the right host directory is enough to take over the node, and the baseline and restricted Pod Security Standards forbid it for exactly that reason.

It remains the correct tool for workloads whose entire job is touching the node: log collectors, node exporters, CNI installers, anything shipped as a DaemonSet. For application data it’s a trap.

7.2. local PVs

The local volume type is the supported way to use a node’s disk as a real PV. It carries a required nodeAffinity that the scheduler honors, so the pod follows the volume instead of guessing.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-nvme-0
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-nvme
  local:
    path: /mnt/disks/nvme0
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - worker-03

There’s no built-in dynamic provisioner for it, so these PVs are created by hand or by a helper that discovers disks and writes the objects. The matching StorageClass needs volumeBindingMode: WaitForFirstConsumer, otherwise binding happens before scheduling and the affinity above becomes a constraint the scheduler has to satisfy rather than one it got to choose.

7.3. local-path-provisioner

Rancher’s local-path-provisioner is the default StorageClass in k3s and the fastest way to get dynamic provisioning on a cluster with no storage system at all. It creates a directory on whichever node the pod was scheduled to, wraps it in a PV with the right node affinity, and uses WaitForFirstConsumer so the ordering works out.

On a single-node cluster it’s the correct answer and there’s no reason to reach further. Multi-node is where it’s worth being honest about what it is: a directory on one machine, with no replication, no snapshots, and no backup. Lose the node and the workload is down until it comes back; a dead disk takes the data with it. That’s fine for caches, build artifacts and dev environments, and not fine for anything whose loss would ruin an afternoon.

7.4. LVM-backed local volumes

TopoLVM sits between hand-created local PVs and local-path-provisioner, dynamically provisioning LVM (Logical Volume Manager) logical volumes from a volume group on each node. You get real block devices, thin provisioning, snapshots, and expansion, and because it publishes CSIStorageCapacity the scheduler avoids nodes whose volume group is full. The volumes are still node-pinned, so it improves the local-storage experience without changing the underlying tradeoff.

8. Replicated and shared storage

Local storage is the right call when the workload replicates its own data. An etcd member, a Cassandra node, a Kafka broker, or a Postgres replica each hold a copy of a dataset that lives in several places by design, so losing a node loses a replica rather than the data. Replicating underneath a system that already replicates just pays the cost twice.

When the workload can’t replicate its own data, the storage layer has to, and you buy back the ability to reschedule a pod anywhere at the cost of writing every block more than once.

8.1. Longhorn

Longhorn gives each volume its own controller and a configurable number of replicas placed on different nodes, with writes going to all of them synchronously. A node failure costs a replica, not the volume, and the pod can start anywhere. It ships snapshots and scheduled backups to S3-compatible object storage or NFS, plus a web UI that makes volume state legible, and it installs on any cluster with a filesystem and open ports.

It’s a CNCF incubating project and the reasonable default for a self-hosted multi-node cluster. What you pay: three replicas mean three times the raw capacity and synchronous writes across the network, so volume latency becomes network latency. On 1 GbE it’s noticeable under write-heavy load.

8.2. Rook and Ceph

Rook is an operator that runs Ceph inside the cluster and gives you three storage types from one system: RBD (RADOS Block Device) for RWO block volumes, CephFS for RWX shared filesystems, and RGW (RADOS Gateway) for S3-compatible object storage. Nothing else in this list covers that much ground.

It’s also the heaviest thing here. Ceph wants raw disks, several nodes, real memory, and an operator who has read enough of its documentation to act sensibly when the cluster reports itself unhealthy at 2am. Worth it when you need CephFS or object storage in-cluster, hard to justify when Longhorn would have done.

8.3. NFS and NAS-backed drivers

NFS is the pragmatic RWX answer. csi-driver-nfs provisions dynamically by creating a subdirectory per volume on an existing export, which means the storage decision reduces to “point it at the NAS you already run.” There’s no replication in the picture, so the server is a single point of failure, but that failure domain is usually one you already accepted.

democratic-csi is the other common route for anyone with TrueNAS or plain ZFS, provisioning over NFS or iSCSI with a real ZFS dataset or zvol behind each volume. That gets you ZFS snapshots, compression, and send/receive replication under Kubernetes volumes. Better durability than most in-cluster options, at the cost of keeping storage on a box Kubernetes doesn’t manage. The tradeoff is the same one that separates stacked from external etcd, and it’s discussed at more length in the post on etcd topologies.

8.4. OpenEBS

OpenEBS covers both ends: its LocalPV flavors manage node-local volumes on hostpath, LVM, or ZFS, while Mayastor is a replicated NVMe-oF (NVMe over Fabrics) engine aimed at low-latency block storage.

8.5. Managed cloud storage

Managed cloud storage is the same model with someone else running the driver. EBS, Persistent Disk, and Azure Disk are zonal RWO block volumes. EFS and Filestore are NFS under a different name, and Azure Files fills the same role over SMB or NFS, so that’s where cloud RWX comes from. The objects, access modes, and CSI plumbing are identical to everything above. The one behavior worth carrying over is that zonal disks make WaitForFirstConsumer mandatory, because a volume provisioned in eu-central-1a has permanently decided which nodes may run the pod.

Everything from sections 7 and 8 lines up like this:

9. Storage in workloads

How a claim attaches to a workload depends on one question: do the replicas share a volume, or does each one need its own?

9.1. StatefulSets

volumeClaimTemplates gives every replica its own PVC, named {template}-{statefulset}-{ordinal}. Pod postgres-0 always gets data-postgres-0, across rescheduling, node failure, and cluster restarts. That stable pairing of identity and storage is most of the reason StatefulSets exist.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Delete
  template:
    # ... pod spec, mounting the "data" volume at /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: local-nvme
        resources:
          requests:
            storage: 100Gi

Deleting a StatefulSet or scaling it down leaves the PVCs behind, deliberately. persistentVolumeClaimRetentionPolicy became GA in Kubernetes 1.32 and lets you say otherwise, with whenDeleted and whenScaled both defaulting to Retain so nothing changes unless you ask. The corollary catches people during capacity testing: scale from 5 to 3 and back to 5, and the new pods adopt the old claims with the old data still in them.

9.2. Deployments

A Deployment with one PVC and more than one replica means every pod mounts the same volume, which needs RWX and an application that tolerates concurrent writers. Most don’t.

With RWO the failure is more interesting. The default RollingUpdate strategy creates the replacement pod before terminating the old one, and the replacement can’t attach a volume the old pod still holds. The rollout sits there until it times out. Switching to Recreate fixes it by terminating first:

spec:
  strategy:
    type: Recreate

Setting maxSurge: 0 with maxUnavailable: 1 achieves the same ordering if you want to keep a rolling update. This bug survives review because RWO is per node, so it works fine whenever the scheduler happens to place both pods on the same machine. It reproduces on a multi-node cluster and not on the laptop.

9.3. DaemonSets

DaemonSets are the one place hostPath is normal, since a per-node agent genuinely wants that node’s /var/log or /proc. What they don’t have is a volumeClaimTemplates equivalent. The pod template names one claim, so every pod in the set would mount the same PVC, and per-node persistent storage means hand-creating one local PV and one PVC per node with matching node affinity. Generic ephemeral volumes skip all of that whenever the data is scratch.

10. Operating storage: snapshots, permissions, and capacity

Three things go wrong after the volume is working and the pod is running: backups that turn out not to be backups, permission errors that look like application bugs, and full volumes that keep reporting Ready.

10.1. Snapshots and backups

The snapshot API mirrors the volume API exactly: VolumeSnapshot is the namespaced request, VolumeSnapshotContent is the cluster-scoped object, and VolumeSnapshotClass picks the driver. The controller and CRDs that implement it aren’t part of core Kubernetes, so on a self-managed cluster they’re something you install.

Restoring goes through a new claim rather than the old one. Point spec.dataSource at the snapshot and the provisioner populates the volume before it binds:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgdata-restored
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  dataSource:
    name: pgdata-nightly
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  resources:
    requests:
      storage: 20Gi

The request can’t be smaller than the source volume, so a restore can grow the claim but never shrink it.

Snapshots are crash-consistent, not application-consistent. A snapshot of a running database is the state you’d get by pulling the power cord: recoverable through the WAL (write-ahead log) in most cases, corrupt in some, and never something to find out about during a restore. They also usually live on the same storage system as the volume they came from, so they’re protection against DROP TABLE, not against losing the array.

Nothing in an etcd backup helps here either. etcd holds your Kubernetes objects, including the PV and PVC definitions, and not one byte from inside the volumes, which is worth remembering next to the usual advice about backing etcd up. Restoring an etcd snapshot into a fresh cluster recreates claims pointing at storage that may no longer exist.

Real coverage means getting bytes off the cluster: Velero, or Longhorn’s own backup target, writing to object storage, plus application-level dumps for anything transactional.

10.2. Permissions

A non-root container mounting a volume owned by root produces a permission error that looks like a bug in the application. securityContext.fsGroup fixes it by having the kubelet chown the volume’s contents to that group at mount time.

securityContext:
  fsGroup: 2000
  fsGroupChangePolicy: OnRootMismatch

On a volume holding millions of files that recursive chown runs on every single pod start and can add minutes to it. OnRootMismatch walks the tree only when the top-level directory’s ownership doesn’t already match, which turns a per-start cost into a one-time one. NFS ignores fsGroup entirely, since ownership there is the server’s business and gets mapped on the export.

10.3. Capacity

A full volume doesn’t surface as a Kubernetes event. The pod stays Running and Ready while the application throws write errors, because from the cluster’s point of view nothing has failed. The kubelet exports per-volume usage, so kubelet_volume_stats_available_bytes is the metric to alert on. kubelet_volume_stats_inodes_free is the one people forget, until a volume full of small files starts refusing writes with 40% of its bytes free.

11. Choosing a backend

The choice follows from two questions: does the application replicate its own data, and does anything need RWX?

Whichever row you land in, override two defaults: reclaimPolicy: Retain on any class backing data you’d miss, and WaitForFirstConsumer on any backend that isn’t genuinely location-independent. Delete fails toward destroyed data, and Immediate fails toward pods that stay Pending forever.

Before adding replicated storage at all, check whether the state needs to be in the cluster: object storage and managed databases are attached resources, reachable over the network from any pod on any node, and they sidestep this entire chapter of decisions. The storage that gives you the least trouble is the storage you didn’t have to schedule.