Storage capacity is limited and may vary depending on the node on which a pod runs: network-attached storage might not be accessible by all nodes, or storage is local to a node to begin with.
Kubernetes v1.19 [alpha]
This page describes how Kubernetes keeps track of storage capacity and how the scheduler uses that information to schedule Pods onto nodes that have access to enough storage capacity for the remaining missing volumes. Without storage capacity tracking, the scheduler may choose a node that doesn't have enough capacity to provision a volume and multiple scheduling retries will be needed.
Tracking storage capacity is supported for Container Storage InterfaceThe Container Storage Interface (CSI) defines a standard interface to expose storage systems to containers. (CSI) drivers and needs to be enabled when installing a CSI driver.
There are two API extensions for this feature:
- CSIStorageCapacity objects: these get produced by a CSI driver in the namespace where the driver is installed. Each object contains capacity information for one storage class and defines which nodes have access to that storage.
CSIDriverSpec.StorageCapacityfield: when set to
true, the Kubernetes scheduler will consider storage capacity for volumes that use the CSI driver.
Storage capacity information is used by the Kubernetes scheduler if:
CSIStorageCapacityfeature gate is true,
- a Pod uses a volume that has not been created yet,
- that volume uses a StorageClassA StorageClass provides a way for administrators to describe different available storage types. which references a CSI driver and
WaitForFirstConsumervolume binding mode, and
CSIDriverobject for the driver has
StorageCapacityset to true.
In that case, the scheduler only considers nodes for the Pod which
have enough storage available to them. This check is very
simplistic and only compares the size of the volume against the
capacity listed in
CSIStorageCapacity objects with a topology that
includes the node.
For volumes with
Immediate volume binding mode, the storage driver
decides where to create the volume, independently of Pods that will
use the volume. The scheduler then schedules Pods onto nodes where the
volume is available after the volume has been created.
For CSI ephemeral volumes, scheduling always happens without considering storage capacity. This is based on the assumption that this volume type is only used by special CSI drivers which are local to a node and do not need significant resources there.
When a node has been selected for a Pod with
volumes, that decision is still tentative. The next step is that the
CSI storage driver gets asked to create the volume with a hint that the
volume is supposed to be available on the selected node.
Because Kubernetes might have chosen a node based on out-dated capacity information, it is possible that the volume cannot really be created. The node selection is then reset and the Kubernetes scheduler tries again to find a node for the Pod.
Storage capacity tracking increases the chance that scheduling works on the first try, but cannot guarantee this because the scheduler has to decide based on potentially out-dated information. Usually, the same retry mechanism as for scheduling without any storage capacity information handles scheduling failures.
One situation where scheduling can fail permanently is when a Pod uses multiple volumes: one volume might have been created already in a topology segment which then does not have enough capacity left for another volume. Manual intervention is necessary to recover from this, for example by increasing capacity or deleting the volume that was already created. Further work is needed to handle this automatically.
Enabling storage capacity tracking
Storage capacity tracking is an alpha feature and only enabled when
storage.k8s.io/v1alpha1 API groupA set of related paths in the Kubernetes API. are enabled. For details on
that, see the
A quick check whether a Kubernetes cluster supports the feature is to list CSIStorageCapacity objects with:
kubectl get csistoragecapacities --all-namespaces
If your cluster supports CSIStorageCapacity, the response is either a list of CSIStorageCapacity objects or:
No resources found
If not supported, this error is printed instead:
error: the server doesn't have a resource type "csistoragecapacities"
In addition to enabling the feature in the cluster, a CSI driver also has to support it. Please refer to the driver's documentation for details.
- For more information on the design, see the Storage Capacity Constraints for Pod Scheduling KEP.
- For more information on further development of this feature, see the enhancement tracking issue #1472.
- Learn about Kubernetes Scheduler