Reference

• 1: Glossary
• 2: API Overview
• 2.1: Declarative API Validation
• 2.2: Kubernetes API Concepts
• 2.3: Server-Side Apply
• 2.4: Client Libraries
• 2.5: Common Expression Language in Kubernetes
• 2.6: Kubernetes Deprecation Policy
• 2.7: Deprecated API Migration Guide
• 2.8: Kubernetes API health endpoints
• 3: API Access Control
• 3.1: Authenticating
• 3.2: Authenticating with Bootstrap Tokens
• 3.3: Authorization
• 3.4: Using RBAC Authorization
• 3.5: Using Node Authorization
• 3.6: Webhook Mode
• 3.7: Using ABAC Authorization
• 3.8: Admission Control in Kubernetes
• 3.9: Dynamic Admission Control
• 3.10: Managing Service Accounts
• 3.11: Certificates and Certificate Signing Requests
• 3.12: Mapping PodSecurityPolicies to Pod Security Standards
• 3.13: Kubelet authentication/authorization
• 3.14: TLS bootstrapping
• 3.15: Mutating Admission Policy
• 3.16: Validating Admission Policy
• 4: Well-Known Labels, Annotations and Taints
• 4.1: Audit Annotations
• 5: Kubernetes API
• 5.1: Workload Resources
• 5.1.1: Pod
• 5.1.2: Binding
• 5.1.3: PodTemplate
• 5.1.4: ReplicationController
• 5.1.5: ReplicaSet
• 5.1.6: Deployment
• 5.1.7: StatefulSet
• 5.1.8: ControllerRevision
• 5.1.9: DaemonSet
• 5.1.10: Job
• 5.1.11: CronJob
• 5.1.12: HorizontalPodAutoscaler
• 5.1.13: HorizontalPodAutoscaler
• 5.1.14: PriorityClass
• 5.1.15: DeviceTaintRule v1alpha3
• 5.1.16: ResourceClaim v1beta2
• 5.1.17: ResourceClaimTemplate v1beta2
• 5.1.18: ResourceSlice v1beta2
• 5.2: Service Resources
• 5.2.1: Service
• 5.2.2: Endpoints
• 5.2.3: EndpointSlice
• 5.2.4: Ingress
• 5.2.5: IngressClass
• 5.3: Config and Storage Resources
• 5.3.1: ConfigMap
• 5.3.2: Secret
• 5.3.3: CSIDriver
• 5.3.4: CSINode
• 5.3.5: CSIStorageCapacity
• 5.3.6: PersistentVolumeClaim
• 5.3.7: PersistentVolume
• 5.3.8: StorageClass
• 5.3.9: StorageVersionMigration v1alpha1
• 5.3.10: Volume
• 5.3.11: VolumeAttachment
• 5.3.12: VolumeAttributesClass v1beta1
• 5.4: Authentication Resources
• 5.4.1: ServiceAccount
• 5.4.2: TokenRequest
• 5.4.3: TokenReview
• 5.4.4: CertificateSigningRequest
• 5.4.5: ClusterTrustBundle v1beta1
• 5.4.6: SelfSubjectReview
• 5.5: Authorization Resources
• 5.5.1: LocalSubjectAccessReview
• 5.5.2: SelfSubjectAccessReview
• 5.5.3: SelfSubjectRulesReview
• 5.5.4: SubjectAccessReview
• 5.5.5: ClusterRole
• 5.5.6: ClusterRoleBinding
• 5.5.7: Role
• 5.5.8: RoleBinding
• 5.6: Policy Resources
• 5.6.1: FlowSchema
• 5.6.2: LimitRange
• 5.6.3: ResourceQuota
• 5.6.4: NetworkPolicy
• 5.6.5: PodDisruptionBudget
• 5.6.6: PriorityLevelConfiguration
• 5.6.7: ValidatingAdmissionPolicy
• 5.6.8: ValidatingAdmissionPolicyBinding
• 5.6.9: MutatingAdmissionPolicy v1alpha1
• 5.6.10: MutatingAdmissionPolicyBinding v1alpha1
• 5.7: Extend Resources
• 5.7.1: CustomResourceDefinition
• 5.7.2: DeviceClass v1beta2
• 5.7.3: MutatingWebhookConfiguration
• 5.7.4: ValidatingWebhookConfiguration
• 5.8: Cluster Resources
• 5.8.1: APIService
• 5.8.2: ComponentStatus
• 5.8.3: Event
• 5.8.4: IPAddress
• 5.8.5: Lease
• 5.8.6: LeaseCandidate v1beta1
• 5.8.7: Namespace
• 5.8.8: Node
• 5.8.9: RuntimeClass
• 5.8.10: ServiceCIDR
• 5.9: Common Definitions
• 5.9.1: DeleteOptions
• 5.9.2: LabelSelector
• 5.9.3: ListMeta
• 5.9.4: LocalObjectReference
• 5.9.5: NodeSelectorRequirement
• 5.9.6: ObjectFieldSelector
• 5.9.7: ObjectMeta
• 5.9.8: ObjectReference
• 5.9.9: Patch
• 5.9.10: Quantity
• 5.9.11: ResourceFieldSelector
• 5.9.12: Status
• 5.9.13: TypedLocalObjectReference
• 5.10: Common Parameters
• 6: Instrumentation
• 6.1: Kubernetes Component SLI Metrics
• 6.2: CRI Pod & Container Metrics
• 6.3: Node metrics data
• 6.4: Kubernetes z-pages
• 6.5: Kubernetes Metrics Reference
• 7: Kubernetes Issues and Security
• 7.1: Kubernetes Issue Tracker
• 7.2: Kubernetes Security and Disclosure Information
• 7.3: Official CVE Feed
• 8: Node Reference Information
• 8.1: Kubelet Checkpoint API
• 8.2: Linux Kernel Version Requirements
• 8.3: Articles on dockershim Removal and on Using CRI-compatible Runtimes
• 8.4: Node Labels Populated By The Kubelet
• 8.5: Local Files And Paths Used By The Kubelet
• 8.6: Kubelet Configuration Directory Merging
• 8.7: Kubelet Device Manager API Versions
• 8.8: Kubelet Systemd Watchdog
• 8.9: Node Status
• 8.10: Seccomp and Kubernetes
• 9: Networking Reference
• 9.1: Protocols for Services
• 9.2: Ports and Protocols
• 9.3: Virtual IPs and Service Proxies
• 10: Setup tools
• 10.1: Kubeadm
• 10.1.1: Kubeadm Generated
• 10.1.1.1:
• 10.1.1.2:
• 10.1.1.2.1:
• 10.1.1.2.2:
• 10.1.1.2.3:
• 10.1.1.2.4:
• 10.1.1.2.5:
• 10.1.1.2.6:
• 10.1.1.2.7:
• 10.1.1.2.8:
• 10.1.1.2.9:
• 10.1.1.2.10:
• 10.1.1.2.11:
• 10.1.1.2.12:
• 10.1.1.2.13:
• 10.1.1.2.14:
• 10.1.1.2.15:
• 10.1.1.2.16:
• 10.1.1.3:
• 10.1.1.4:
• 10.1.1.4.1:
• 10.1.1.4.2:
• 10.1.1.4.3:
• 10.1.1.4.4:
• 10.1.1.4.5:
• 10.1.1.4.6:
• 10.1.1.4.7:
• 10.1.1.4.8:
• 10.1.1.4.9:
• 10.1.1.4.10:
• 10.1.1.5:
• 10.1.1.5.1:
• 10.1.1.5.2:
• 10.1.1.5.3:
• 10.1.1.5.4:
• 10.1.1.5.5:
• 10.1.1.5.6:
• 10.1.1.5.7:
• 10.1.1.5.8:
• 10.1.1.5.9:
• 10.1.1.5.10:
• 10.1.1.5.11:
• 10.1.1.5.12:
• 10.1.1.5.13:
• 10.1.1.5.14:
• 10.1.1.5.15:
• 10.1.1.5.16:
• 10.1.1.5.17:
• 10.1.1.5.18:
• 10.1.1.5.19:
• 10.1.1.5.20:
• 10.1.1.5.21:
• 10.1.1.5.22:
• 10.1.1.5.23:
• 10.1.1.5.24:
• 10.1.1.5.25:
• 10.1.1.5.26:
• 10.1.1.5.27:
• 10.1.1.5.28:
• 10.1.1.5.29:
• 10.1.1.5.30:
• 10.1.1.5.31:
• 10.1.1.5.32:
• 10.1.1.5.33:
• 10.1.1.5.34:
• 10.1.1.5.35:
• 10.1.1.5.36:
• 10.1.1.5.37:
• 10.1.1.5.38:
• 10.1.1.5.39:
• 10.1.1.5.40:
• 10.1.1.5.41:
• 10.1.1.5.42:
• 10.1.1.5.43:
• 10.1.1.5.44:
• 10.1.1.5.45:
• 10.1.1.6:
• 10.1.1.6.1:
• 10.1.1.6.2:
• 10.1.1.6.3:
• 10.1.1.6.4:
• 10.1.1.6.5:
• 10.1.1.6.6:
• 10.1.1.6.7:
• 10.1.1.6.8:
• 10.1.1.6.9:
• 10.1.1.6.10:
• 10.1.1.6.11:
• 10.1.1.6.12:
• 10.1.1.6.13:
• 10.1.1.6.14:
• 10.1.1.7:
• 10.1.1.7.1:
• 10.1.1.8:
• 10.1.1.8.1:
• 10.1.1.8.2:
• 10.1.1.8.3:
• 10.1.1.8.4:
• 10.1.1.9:
• 10.1.1.9.1:
• 10.1.1.9.2:
• 10.1.1.9.3:
• 10.1.1.9.4:
• 10.1.1.10:
• 10.1.1.10.1:
• 10.1.1.10.2:
• 10.1.1.10.3:
• 10.1.1.10.4:
• 10.1.1.10.5:
• 10.1.1.10.6:
• 10.1.1.10.7:
• 10.1.1.10.8:
• 10.1.1.10.9:
• 10.1.1.10.10:
• 10.1.1.10.11:
• 10.1.1.10.12:
• 10.1.1.10.13:
• 10.1.1.10.14:
• 10.1.1.10.15:
• 10.1.1.10.16:
• 10.1.1.10.17:
• 10.1.1.10.18:
• 10.1.1.10.19:
• 10.1.1.10.20:
• 10.1.1.10.21:
• 10.1.1.10.22:
• 10.1.1.10.23:
• 10.1.1.10.24:
• 10.1.1.10.25:
• 10.1.1.10.26:
• 10.1.1.10.27:
• 10.1.1.11:
• 10.1.1.12:
• 10.1.2: kubeadm init
• 10.1.3: kubeadm join
• 10.1.4: kubeadm upgrade
• 10.1.5: kubeadm upgrade phases
• 10.1.6: kubeadm config
• 10.1.7: kubeadm reset
• 10.1.8: kubeadm token
• 10.1.9: kubeadm version
• 10.1.10: kubeadm alpha
• 10.1.11: kubeadm certs
• 10.1.12: kubeadm init phase
• 10.1.13: kubeadm join phase
• 10.1.14: kubeadm kubeconfig
• 10.1.15: kubeadm reset phase
• 10.1.16: Implementation details
• 11: Command line tool (kubectl)
• 11.1: Introduction to kubectl
• 11.2: kubectl Quick Reference
• 11.3: kubectl reference
• 11.3.1: kubectl
• 11.3.2: kubectl annotate
• 11.3.3: kubectl api-resources
• 11.3.4: kubectl api-versions
• 11.3.5: kubectl apply
• 11.3.5.1: kubectl apply edit-last-applied
• 11.3.5.2: kubectl apply set-last-applied
• 11.3.5.3: kubectl apply view-last-applied
• 11.3.6: kubectl attach
• 11.3.7: kubectl auth
• 11.3.7.1: kubectl auth can-i
• 11.3.7.2: kubectl auth reconcile
• 11.3.7.3: kubectl auth whoami
• 11.3.8: kubectl autoscale
• 11.3.9: kubectl certificate
• 11.3.9.1: kubectl certificate approve
• 11.3.9.2: kubectl certificate deny
• 11.3.10: kubectl cluster-info
• 11.3.10.1: kubectl cluster-info dump
• 11.3.11: kubectl completion
• 11.3.12: kubectl config
• 11.3.12.1: kubectl config current-context
• 11.3.12.2: kubectl config delete-cluster
• 11.3.12.3: kubectl config delete-context
• 11.3.12.4: kubectl config delete-user
• 11.3.12.5: kubectl config get-clusters
• 11.3.12.6: kubectl config get-contexts
• 11.3.12.7: kubectl config get-users
• 11.3.12.8: kubectl config rename-context
• 11.3.12.9: kubectl config set
• 11.3.12.10: kubectl config set-cluster
• 11.3.12.11: kubectl config set-context
• 11.3.12.12: kubectl config set-credentials
• 11.3.12.13: kubectl config unset
• 11.3.12.14: kubectl config use-context
• 11.3.12.15: kubectl config view
• 11.3.13: kubectl cordon
• 11.3.14: kubectl cp
• 11.3.15: kubectl create
• 11.3.15.1: kubectl create clusterrole
• 11.3.15.2: kubectl create clusterrolebinding
• 11.3.15.3: kubectl create configmap
• 11.3.15.4: kubectl create cronjob
• 11.3.15.5: kubectl create deployment
• 11.3.15.6: kubectl create ingress
• 11.3.15.7: kubectl create job
• 11.3.15.8: kubectl create namespace
• 11.3.15.9: kubectl create poddisruptionbudget
• 11.3.15.10: kubectl create priorityclass
• 11.3.15.11: kubectl create quota
• 11.3.15.12: kubectl create role
• 11.3.15.13: kubectl create rolebinding
• 11.3.15.14: kubectl create secret
• 11.3.15.15: kubectl create secret docker-registry
• 11.3.15.16: kubectl create secret generic
• 11.3.15.17: kubectl create secret tls
• 11.3.15.18: kubectl create service
• 11.3.15.19: kubectl create service clusterip
• 11.3.15.20: kubectl create service externalname
• 11.3.15.21: kubectl create service loadbalancer
• 11.3.15.22: kubectl create service nodeport
• 11.3.15.23: kubectl create serviceaccount
• 11.3.15.24: kubectl create token
• 11.3.16: kubectl debug
• 11.3.17: kubectl delete
• 11.3.18: kubectl describe
• 11.3.19: kubectl diff
• 11.3.20: kubectl drain
• 11.3.21: kubectl edit
• 11.3.22: kubectl events
• 11.3.23: kubectl exec
• 11.3.24: kubectl explain
• 11.3.25: kubectl expose
• 11.3.26: kubectl get
• 11.3.27: kubectl kustomize
• 11.3.28: kubectl label
• 11.3.29: kubectl logs
• 11.3.30: kubectl options
• 11.3.31: kubectl patch
• 11.3.32: kubectl plugin
• 11.3.32.1: kubectl plugin list
• 11.3.33: kubectl port-forward
• 11.3.34: kubectl proxy
• 11.3.35: kubectl replace
• 11.3.36: kubectl rollout
• 11.3.36.1: kubectl rollout history
• 11.3.36.2: kubectl rollout pause
• 11.3.36.3: kubectl rollout restart
• 11.3.36.4: kubectl rollout resume
• 11.3.36.5: kubectl rollout status
• 11.3.36.6: kubectl rollout undo
• 11.3.37: kubectl run
• 11.3.38: kubectl scale
• 11.3.39: kubectl set
• 11.3.39.1: kubectl set env
• 11.3.39.2: kubectl set image
• 11.3.39.3: kubectl set resources
• 11.3.39.4: kubectl set selector
• 11.3.39.5: kubectl set serviceaccount
• 11.3.39.6: kubectl set subject
• 11.3.40: kubectl taint
• 11.3.41: kubectl top
• 11.3.41.1: kubectl top node
• 11.3.41.2: kubectl top pod
• 11.3.42: kubectl uncordon
• 11.3.43: kubectl version
• 11.3.44: kubectl wait
• 11.4: kubectl Commands
• 11.5: kubectl
• 11.6: JSONPath Support
• 11.7: kubectl for Docker Users
• 11.8: kubectl Usage Conventions
• 12: Component tools
• 12.1: Feature Gates
• 12.2: Feature Gates (removed)
• 12.3: kubelet
• 12.4: kube-apiserver
• 12.5: kube-controller-manager
• 12.6: kube-proxy
• 12.7: kube-scheduler
• 13: Debug cluster
• 13.1: Flow control
• 14: Configuration APIs
• 14.1: Client Authentication (v1)
• 14.2: Client Authentication (v1beta1)
• 14.3: Event Rate Limit Configuration (v1alpha1)
• 14.4: Image Policy API (v1alpha1)
• 14.5: kube-apiserver Admission (v1)
• 14.6: kube-apiserver Audit Configuration (v1)
• 14.7: kube-apiserver Configuration (v1)
• 14.8: kube-apiserver Configuration (v1alpha1)
• 14.9: kube-apiserver Configuration (v1beta1)
• 14.10: kube-controller-manager Configuration (v1alpha1)
• 14.11: kube-proxy Configuration (v1alpha1)
• 14.12: kube-scheduler Configuration (v1)
• 14.13: kubeadm Configuration (v1beta3)
• 14.14: kubeadm Configuration (v1beta4)
• 14.15: kubeconfig (v1)
• 14.16: Kubelet Configuration (v1)
• 14.17: Kubelet Configuration (v1alpha1)
• 14.18: Kubelet Configuration (v1beta1)
• 14.19: Kubelet CredentialProvider (v1)
• 14.20: WebhookAdmission Configuration (v1)
• 15: External APIs
• 15.1: Kubernetes Custom Metrics (v1beta2)
• 15.2: Kubernetes External Metrics (v1beta1)
• 15.3: Kubernetes Metrics (v1beta1)
• 16: Scheduling
• 16.1: Scheduler Configuration
• 16.2: Scheduling Policies
• 17: Other Tools

1 - Glossary

2 - API Overview

This section provides reference information for the Kubernetes API.

The REST API is the fundamental fabric of Kubernetes. All operations and communications between components, and external user commands are REST API calls that the API Server handles. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API.

The Kubernetes API reference lists the API for Kubernetes version v1.33.

For general background information, read The Kubernetes API. Controlling Access to the Kubernetes API describes how clients can authenticate to the Kubernetes API server, and how their requests are authorized.

API versioning

The JSON and Protobuf serialization schemas follow the same guidelines for schema changes. The following descriptions cover both formats.

The API versioning and software versioning are indirectly related. The API and release versioning proposal describes the relationship between API versioning and software versioning.

Different API versions indicate different levels of stability and support. You can find more information about the criteria for each level in the API Changes documentation.

Here's a summary of each level:

• Alpha:

  • The version names contain alpha (for example, v1alpha1).
  • Built-in alpha API versions are disabled by default and must be explicitly enabled in the kube-apiserver configuration to be used.
  • The software may contain bugs. Enabling a feature may expose bugs.
  • Support for an alpha API may be dropped at any time without notice.
  • The API may change in incompatible ways in a later software release without notice.
  • The software is recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

• Beta:

  • The version names contain beta (for example, v2beta3).

  • Built-in beta API versions are disabled by default and must be explicitly enabled in the kube-apiserver configuration to be used (except for beta versions of APIs introduced prior to Kubernetes 1.22, which were enabled by default).

  • Built-in beta API versions have a maximum lifetime of 9 months or 3 minor releases (whichever is longer) from introduction to deprecation, and 9 months or 3 minor releases (whichever is longer) from deprecation to removal.

  • The software is well tested. Enabling a feature is considered safe.

  • The support for a feature will not be dropped, though the details may change.

  • The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable API version. When this happens, migration instructions are provided. Adapting to a subsequent beta or stable API version may require editing or re-creating API objects, and may not be straightforward. The migration may require downtime for applications that rely on the feature.

  • The software is not recommended for production uses. Subsequent releases may introduce incompatible changes. Use of beta API versions is required to transition to subsequent beta or stable API versions once the beta API version is deprecated and no longer served.

  Note:

  Please try beta features and provide feedback. After the features exit beta, it may not be practical to make more changes.

• Stable:

  • The version name is vX where X is an integer.
  • Stable API versions remain available for all future releases within a Kubernetes major version, and there are no current plans for a major version revision of Kubernetes that removes stable APIs.

API groups

API groups make it easier to extend the Kubernetes API. The API group is specified in a REST path and in the apiVersion field of a serialized object.

There are several API groups in Kubernetes:

• The core (also called legacy) group is found at REST path /api/v1. The core group is not specified as part of the apiVersion field, for example, apiVersion: v1.
• The named groups are at REST path /apis/$GROUP_NAME/$VERSION and use apiVersion: $GROUP_NAME/$VERSION (for example, apiVersion: batch/v1). You can find the full list of supported API groups in Kubernetes API reference.

Enabling or disabling API groups

Certain resources and API groups are enabled by default. You can enable or disable them by setting --runtime-config on the API server. The --runtime-config flag accepts comma separated <key>[=<value>] pairs describing the runtime configuration of the API server. If the =<value> part is omitted, it is treated as if =true is specified. For example:

• to disable batch/v1, set --runtime-config=batch/v1=false
• to enable batch/v2alpha1, set --runtime-config=batch/v2alpha1
• to enable a specific version of an API, such as storage.k8s.io/v1beta1/csistoragecapacities, set --runtime-config=storage.k8s.io/v1beta1/csistoragecapacities

Persistence

Kubernetes stores its serialized state in terms of the API resources by writing them into etcd.

What's next

• Learn more about API conventions
• Read the design documentation for aggregator

2.1 - Declarative API Validation

Kubernetes 1.33 includes optional declarative validation for APIs. When enabled, the Kubernetes API server can use this mechanism rather than the legacy approach that relies on hand-written Go code (validation.go files) to ensure that requests against the API are valid. Kubernetes developers, and people extending the Kubernetes API, can define validation rules directly alongside the API type definitions (types.go files). Code authors define pecial comment tags (e.g., +k8s:minimum=0). A code generator (validation-gen) then uses these tags to produce optimized Go code for API validation.

While primarily a feature impacting Kubernetes contributors and potentially developers of extension API servers, cluster administrators should understand its behavior, especially during its rollout phases.

Declarative validation is being rolled out gradually. In Kubernetes 1.33, the APIs that use declarative validation include:

• ReplicationController

• DeclarativeValidation: (Beta, Default: true) When enabled, the API server runs both the new declarative validation and the old hand-written validation for migrated types/fields. The results are compared internally.
• DeclarativeValidationTakeover: (Beta, Default: false) This gate determines which validation result is authoritative (i.e., returned to the user and used for admission decisions).

Default Behavior (Kubernetes 1.33):

• With DeclarativeValidation=true and DeclarativeValidationTakeover=false (the default values for the gates), both validation systems run.
• The results of the hand-written validation are used. The declarative validation runs in a mismatch mode for comparison.
• Mismatches between the two validation systems are logged by the API server and increment the declarative_validation_mismatch_total metric. This helps developers identify and fix discrepancies during the Beta phase.
• Cluster upgrades should be safe regarding this feature, as the authoritative validation logic doesn't change by default.

Administrators can choose to explicitly enable DeclarativeValidationTakeover=true to make the declarative validation authoritative for migrated fields, typically after verifying stability in their environment (e.g., by monitoring the mismatch metric).

Disabling declarative validation

As a cluster administrator, you might consider disabling declarative validation whilst it is still beta, under specific circumstances:

• Unexpected Validation Behavior: If enabling DeclarativeValidationTakeover leads to unexpected validation errors or allows objects that were previously invalid.
• Performance Regressions: If monitoring indicates significant latency increases (e.g., in apiserver_request_duration_seconds) correlated with the feature's enablement.
• High Mismatch Rate: If the declarative_validation_mismatch_total metric shows frequent mismatches, suggesting potential bugs in the declarative rules affecting the cluster's workloads, even if DeclarativeValidationTakeover is false.

To revert to only using hand-written validation (as used before Kubernetes v1.33), disable the DeclarativeValidation feature gate, for example via command-line arguments: (--feature-gates=DeclarativeValidation=false). This also implicitly disables the effect of DeclarativeValidationTakeover.

Considerations for downgrade and rollback

Disabling the feature acts as a safety mechanism. However, be aware of a potential edge case (considered unlikely due to extensive testing): If a bug in declarative validation (when DeclarativeValidationTakeover=true) incorrectly allowed an invalid object to be persisted, disabling the feature gates might then cause subsequent updates to that specific object to be blocked by the now-authoritative (and correct) hand-written validation. Resolving this might require manual correction of the stored object, potentially via direct etcd modification in rare cases.

For details on managing feature gates, see Feature Gates.

2.2 - Kubernetes API Concepts

The Kubernetes API is a resource-based (RESTful) programmatic interface provided via HTTP. It supports retrieving, creating, updating, and deleting primary resources via the standard HTTP verbs (POST, PUT, PATCH, DELETE, GET).

For some resources, the API includes additional subresources that allow fine-grained authorization (such as separate views for Pod details and log retrievals), and can accept and serve those resources in different representations for convenience or efficiency.

Kubernetes supports efficient change notifications on resources via watches:

in the Kubernetes API, watch is a verb that is used to track changes to an object in Kubernetes as a stream. It is used for the efficient detection of changes.

You can view the API reference online, or read on to learn about the API in general.

Kubernetes API terminology

Kubernetes generally leverages common RESTful terminology to describe the API concepts:

• A resource type is the name used in the URL (pods, namespaces, services)
• All resource types have a concrete representation (their object schema) which is called a kind
• A list of instances of a resource type is known as a collection
• A single instance of a resource type is called a resource, and also usually represents an object
• For some resource types, the API includes one or more sub-resources, which are represented as URI paths below the resource

Most Kubernetes API resource types are objects – they represent a concrete instance of a concept on the cluster, like a pod or namespace. A smaller number of API resource types are virtual in that they often represent operations on objects, rather than objects, such as a permission check (use a POST with a JSON-encoded body of SubjectAccessReview to the subjectaccessreviews resource), or the eviction sub-resource of a Pod (used to trigger API-initiated eviction).

Object names

All objects you can create via the API have a unique object name to allow idempotent creation and retrieval, except that virtual resource types may not have unique names if they are not retrievable, or do not rely on idempotency. Within a namespace, only one object of a given kind can have a given name at a time. However, if you delete the object, you can make a new object with the same name. Some objects are not namespaced (for example: Nodes), and so their names must be unique across the whole cluster.

API verbs

Almost all object resource types support the standard HTTP verbs - GET, POST, PUT, PATCH, and DELETE. Kubernetes also uses its own verbs, which are often written in lowercase to distinguish them from HTTP verbs.

Kubernetes uses the term list to describe the action of returning a collection of resources, to distinguish it from retrieving a single resource which is usually called a get. If you sent an HTTP GET request with the ?watch query parameter, Kubernetes calls this a watch and not a get (see Efficient detection of changes for more details).

For PUT requests, Kubernetes internally classifies these as either create or update based on the state of the existing object. An update is different from a patch; the HTTP verb for a patch is PATCH.

Resource URIs

All resource types are either scoped by the cluster (/apis/GROUP/VERSION/*) or to a namespace (/apis/GROUP/VERSION/namespaces/NAMESPACE/*). A namespace-scoped resource type will be deleted when its namespace is deleted and access to that resource type is controlled by authorization checks on the namespace scope.

Note: core resources use /api instead of /apis and omit the GROUP path segment.

Examples:

• /api/v1/namespaces
• /api/v1/pods
• /api/v1/namespaces/my-namespace/pods
• /apis/apps/v1/deployments
• /apis/apps/v1/namespaces/my-namespace/deployments
• /apis/apps/v1/namespaces/my-namespace/deployments/my-deployment

You can also access collections of resources (for example: listing all Nodes). The following paths are used to retrieve collections and resources:

• Cluster-scoped resources:

  • GET /apis/GROUP/VERSION/RESOURCETYPE - return the collection of resources of the resource type
  • GET /apis/GROUP/VERSION/RESOURCETYPE/NAME - return the resource with NAME under the resource type

• Namespace-scoped resources:

  • GET /apis/GROUP/VERSION/RESOURCETYPE - return the collection of all instances of the resource type across all namespaces
  • GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE - return collection of all instances of the resource type in NAMESPACE
  • GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME - return the instance of the resource type with NAME in NAMESPACE

Since a namespace is a cluster-scoped resource type, you can retrieve the list (“collection”) of all namespaces with GET /api/v1/namespaces and details about a particular namespace with GET /api/v1/namespaces/NAME.

• Cluster-scoped subresource: GET /apis/GROUP/VERSION/RESOURCETYPE/NAME/SUBRESOURCE
• Namespace-scoped subresource: GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME/SUBRESOURCE

The verbs supported for each subresource will differ depending on the object - see the API reference for more information. It is not possible to access sub-resources across multiple resources - generally a new virtual resource type would be used if that becomes necessary.

HTTP media types

Over HTTP, Kubernetes supports JSON and Protobuf wire encodings.

By default, Kubernetes returns objects in JSON serialization, using the application/json media type. Although JSON is the default, clients may request a response in YAML, or use the more efficient binary Protobuf representation for better performance at scale.

The Kubernetes API implements standard HTTP content type negotiation: passing an Accept header with a GET call will request that the server tries to return a response in your preferred media type. If you want to send an object in Protobuf to the server for a PUT or POST request, you must set the Content-Type request header appropriately.

If you request an available media type, the API server returns a response with a suitable Content-Type; if none of the media types you request are supported, the API server returns a 406 Not acceptable error message. All built-in resource types support the application/json media type.

JSON resource encoding

The Kubernetes API defaults to using JSON for encoding HTTP message bodies.

For example:

1. List all of the pods on a cluster, without specifying a preferred format

   GET /api/v1/pods
   

   200 OK
   Content-Type: application/json
   
   … JSON encoded collection of Pods (PodList object)
   

2. Create a pod by sending JSON to the server, requesting a JSON response.

   POST /api/v1/namespaces/test/pods
   Content-Type: application/json
   Accept: application/json
   … JSON encoded Pod object
   

   200 OK
   Content-Type: application/json
   
   {
     "kind": "Pod",
     "apiVersion": "v1",
     …
   }
   

YAML resource encoding

Kubernetes also supports the application/yaml media type for both requests and responses. YAML can be used for defining Kubernetes manifests and API interactions.

For example:

1. List all of the pods on a cluster in YAML format

   GET /api/v1/pods
   Accept: application/yaml
   

   200 OK
   Content-Type: application/yaml
   
   … YAML encoded collection of Pods (PodList object)
   

2. Create a pod by sending YAML-encoded data to the server, requesting a YAML response:

   POST /api/v1/namespaces/test/pods
   Content-Type: application/yaml
   Accept: application/yaml
   … YAML encoded Pod object
   

   200 OK
   Content-Type: application/yaml
   
   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     …
   

Kubernetes Protobuf encoding

Kubernetes uses an envelope wrapper to encode Protobuf responses. That wrapper starts with a 4 byte magic number to help identify content in disk or in etcd as Protobuf (as opposed to JSON). The 4 byte magic number data is followed by a Protobuf encoded wrapper message, which describes the encoding and type of the underlying object. Within the Protobuf wrapper message, the inner object data is recorded using the raw field of Unknown (see the IDL for more detail).

For example:

1. List all of the pods on a cluster in Protobuf format.

   GET /api/v1/pods
   Accept: application/vnd.kubernetes.protobuf
   

   200 OK
   Content-Type: application/vnd.kubernetes.protobuf
   
   … JSON encoded collection of Pods (PodList object)
   

2. Create a pod by sending Protobuf encoded data to the server, but request a response in JSON.

   POST /api/v1/namespaces/test/pods
   Content-Type: application/vnd.kubernetes.protobuf
   Accept: application/json
   … binary encoded Pod object
   

   200 OK
   Content-Type: application/json
   
   {
     "kind": "Pod",
     "apiVersion": "v1",
     ...
   }
   

You can use both techniques together and use Kubernetes' Protobuf encoding to interact with any API that supports it, for both reads and writes. Only some API resource types are compatible with Protobuf.

A four byte magic number prefix:
  Bytes 0-3: "k8s\x00" [0x6b, 0x38, 0x73, 0x00]

An encoded Protobuf message with the following IDL:
  message Unknown {
    // typeMeta should have the string values for "kind" and "apiVersion" as set on the JSON object
    optional TypeMeta typeMeta = 1;

    // raw will hold the complete serialized object in protobuf. See the protobuf definitions in the client libraries for a given kind.
    optional bytes raw = 2;

    // contentEncoding is encoding used for the raw data. Unspecified means no encoding.
    optional string contentEncoding = 3;

    // contentType is the serialization method used to serialize 'raw'. Unspecified means application/vnd.kubernetes.protobuf and is usually
    // omitted.
    optional string contentType = 4;
  }

  message TypeMeta {
    // apiVersion is the group/version for this type
    optional string apiVersion = 1;
    // kind is the name of the object schema. A protobuf definition should exist for this object.
    optional string kind = 2;
  }

Not all API resource types support Kubernetes' Protobuf encoding; specifically, Protobuf isn't available for resources that are defined as CustomResourceDefinitions or are served via the aggregation layer.

As a client, if you might need to work with extension types you should specify multiple content types in the request Accept header to support fallback to JSON. For example:

Accept: application/vnd.kubernetes.protobuf, application/json

CBOR resource encoding

With the CBORServingAndStorage feature gate enabled, request and response bodies for all built-in resource types and all resources defined by a CustomResourceDefinition may be encoded to the CBOR binary data format. CBOR is also supported at the aggregation layer if it is enabled in individual aggregated API servers.

Clients should indicate the IANA media type application/cbor in the Content-Type HTTP request header when the request body contains a single CBOR encoded data item, and in the Accept HTTP request header when prepared to accept a CBOR encoded data item in the response. API servers will use application/cbor in the Content-Type HTTP response header when the response body contains a CBOR-encoded object.

If an API server encodes its response to a watch request using CBOR, the response body will be a CBOR Sequence and the Content-Type HTTP response header will use the IANA media type application/cbor-seq. Each entry of the sequence (if any) is a single CBOR-encoded watch event.

In addition to the existing application/apply-patch+yaml media type for YAML-encoded server-side apply configurations, API servers that enable CBOR will accept the application/apply-patch+cbor media type for CBOR-encoded server-side apply configurations. There is no supported CBOR equivalent for application/json-patch+json or application/merge-patch+json, or application/strategic-merge-patch+json.

Efficient detection of changes

The Kubernetes API allows clients to make an initial request for an object or a collection, and then to track changes since that initial request: a watch. Clients can send a list or a get and then make a follow-up watch request.

To make this change tracking possible, every Kubernetes object has a resourceVersion field representing the version of that resource as stored in the underlying persistence layer. When retrieving a collection of resources (either namespace or cluster scoped), the response from the API server contains a resourceVersion value. The client can use that resourceVersion to initiate a watch against the API server.

When you send a watch request, the API server responds with a stream of changes. These changes itemize the outcome of operations (such as create, delete, and update) that occurred after the resourceVersion you specified as a parameter to the watch request. The overall watch mechanism allows a client to fetch the current state and then subscribe to subsequent changes, without missing any events.

If a client watch is disconnected then that client can start a new watch from the last returned resourceVersion; the client could also perform a fresh get / list request and begin again. See Resource Version Semantics for more detail.

For example:

1. List all of the pods in a given namespace.

   GET /api/v1/namespaces/test/pods
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {"resourceVersion":"10245"},
     "items": [...]
   }
   

2. Starting from resource version 10245, receive notifications of any API operations (such as create, delete, patch or update) that affect Pods in the test namespace. Each change notification is a JSON document. The HTTP response body (served as application/json) consists a series of JSON documents.

   GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
   ---
   200 OK
   Transfer-Encoding: chunked
   Content-Type: application/json
   
   {
     "type": "ADDED",
     "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
   }
   {
     "type": "MODIFIED",
     "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...}
   }
   ...
   

A given Kubernetes server will only preserve a historical record of changes for a limited time. Clusters using etcd 3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code 410 Gone, clearing their local cache, performing a new get or list operation, and starting the watch from the resourceVersion that was returned.

For subscribing to collections, Kubernetes client libraries typically offer some form of standard tool for this list-then-watch logic. (In the Go client library, this is called a Reflector and is located in the k8s.io/client-go/tools/cache package.)

Watch bookmarks

To mitigate the impact of short history window, the Kubernetes API provides a watch event named BOOKMARK. It is a special kind of event to mark that all changes up to a given resourceVersion the client is requesting have already been sent. The document representing the BOOKMARK event is of the type requested by the request, but only includes a .metadata.resourceVersion field. For example:

GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245&allowWatchBookmarks=true
---
200 OK
Transfer-Encoding: chunked
Content-Type: application/json

{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
}
...
{
  "type": "BOOKMARK",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "12746"} }
}

As a client, you can request BOOKMARK events by setting the allowWatchBookmarks=true query parameter to a watch request, but you shouldn't assume bookmarks are returned at any specific interval, nor can clients assume that the API server will send any BOOKMARK event even when requested.

Streaming lists

On large clusters, retrieving the collection of some resource types may result in a significant increase of resource usage (primarily RAM) on the control plane. To alleviate the impact and simplify the user experience of the list + watch pattern, Kubernetes v1.32 promotes to beta the feature that allows requesting the initial state (previously requested via the list request) as part of the watch request.

On the client-side the initial state can be requested by specifying sendInitialEvents=true as query string parameter in a watch request. If set, the API server starts the watch stream with synthetic init events (of type ADDED) to build the whole state of all existing objects followed by a BOOKMARK event (if requested via allowWatchBookmarks=true option). The bookmark event includes the resource version to which is synced. After sending the bookmark event, the API server continues as for any other watch request.

When you set sendInitialEvents=true in the query string, Kubernetes also requires that you set resourceVersionMatch to NotOlderThan value. If you provided resourceVersion in the query string without providing a value or don't provide it at all, this is interpreted as a request for consistent read; the bookmark event is sent when the state is synced at least to the moment of a consistent read from when the request started to be processed. If you specify resourceVersion (in the query string), the bookmark event is sent when the state is synced at least to the provided resource version.

Example

An example: you want to watch a collection of Pods. For that collection, the current resource version is 10245 and there are two pods: foo and bar. Then sending the following request (explicitly requesting consistent read by setting empty resource version using resourceVersion=) could result in the following sequence of events:

GET /api/v1/namespaces/test/pods?watch=1&sendInitialEvents=true&allowWatchBookmarks=true&resourceVersion=&resourceVersionMatch=NotOlderThan
---
200 OK
Transfer-Encoding: chunked
Content-Type: application/json

{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "8467", "name": "foo"}, ...}
}
{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "5726", "name": "bar"}, ...}
}
{
  "type": "BOOKMARK",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10245"} }
}
...
<followed by regular watch stream starting from resourceVersion="10245">

Response compression

APIResponseCompression is an option that allows the API server to compress the responses for get and list requests, reducing the network bandwidth and improving the performance of large-scale clusters. It is enabled by default since Kubernetes 1.16 and it can be disabled by including APIResponseCompression=false in the --feature-gates flag on the API server.

API response compression can significantly reduce the size of the response, especially for large resources or collections. For example, a list request for pods can return hundreds of kilobytes or even megabytes of data, depending on the number of pods and their attributes. By compressing the response, the network bandwidth can be saved and the latency can be reduced.

To verify if APIResponseCompression is working, you can send a get or list request to the API server with an Accept-Encoding header, and check the response size and headers. For example:

GET /api/v1/pods
Accept-Encoding: gzip
---
200 OK
Content-Type: application/json
content-encoding: gzip
...

The content-encoding header indicates that the response is compressed with gzip.

Retrieving large results sets in chunks

On large clusters, retrieving the collection of some resource types may result in very large responses that can impact the server and client. For instance, a cluster may have tens of thousands of Pods, each of which is equivalent to roughly 2 KiB of encoded JSON. Retrieving all pods across all namespaces may result in a very large response (10-20MB) and consume a large amount of server resources.

The Kubernetes API server supports the ability to break a single large collection request into many smaller chunks while preserving the consistency of the total request. Each chunk can be returned sequentially which reduces both the total size of the request and allows user-oriented clients to display results incrementally to improve responsiveness.

You can request that the API server handles a list by serving single collection using pages (which Kubernetes calls chunks). To retrieve a single collection in chunks, two query parameters limit and continue are supported on requests against collections, and a response field continue is returned from all list operations in the collection's metadata field. A client should specify the maximum results they wish to receive in each chunk with limit and the server will return up to limit resources in the result and include a continue value if there are more resources in the collection.

As an API client, you can then pass this continue value to the API server on the next request, to instruct the server to return the next page (chunk) of results. By continuing until the server returns an empty continue value, you can retrieve the entire collection.

Like a watch operation, a continue token will expire after a short amount of time (by default 5 minutes) and return a 410 Gone if more results cannot be returned. In this case, the client will need to start from the beginning or omit the limit parameter.

For example, if there are 1,253 pods on the cluster and you want to receive chunks of 500 pods at a time, request those chunks as follows:

1. List all of the pods on a cluster, retrieving up to 500 pods each time.

   GET /api/v1/pods?limit=500
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {
       "resourceVersion":"10245",
       "continue": "ENCODED_CONTINUE_TOKEN",
       "remainingItemCount": 753,
       ...
     },
     "items": [...] // returns pods 1-500
   }
   

2. Continue the previous call, retrieving the next set of 500 pods.

   GET /api/v1/pods?limit=500&continue=ENCODED_CONTINUE_TOKEN
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {
       "resourceVersion":"10245",
       "continue": "ENCODED_CONTINUE_TOKEN_2",
       "remainingItemCount": 253,
       ...
     },
     "items": [...] // returns pods 501-1000
   }
   

3. Continue the previous call, retrieving the last 253 pods.

   GET /api/v1/pods?limit=500&continue=ENCODED_CONTINUE_TOKEN_2
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {
       "resourceVersion":"10245",
       "continue": "", // continue token is empty because we have reached the end of the list
       ...
     },
     "items": [...] // returns pods 1001-1253
   }
   

Notice that the resourceVersion of the collection remains constant across each request, indicating the server is showing you a consistent snapshot of the pods. Pods that are created, updated, or deleted after version 10245 would not be shown unless you make a separate list request without the continue token. This allows you to break large requests into smaller chunks and then perform a watch operation on the full set without missing any updates.

remainingItemCount is the number of subsequent items in the collection that are not included in this response. If the list request contained label or field selectors then the number of remaining items is unknown and the API server does not include a remainingItemCount field in its response. If the list is complete (either because it is not chunking, or because this is the last chunk), then there are no more remaining items and the API server does not include a remainingItemCount field in its response. The intended use of the remainingItemCount is estimating the size of a collection.

Collections

In Kubernetes terminology, the response you get from a list is a collection. However, Kubernetes defines concrete kinds for collections of different types of resource. Collections have a kind named for the resource kind, with List appended.

When you query the API for a particular type, all items returned by that query are of that type. For example, when you list Services, the collection response has kind set to ServiceList; each item in that collection represents a single Service. For example:

GET /api/v1/services

{
  "kind": "ServiceList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "2947301"
  },
  "items": [
    {
      "metadata": {
        "name": "kubernetes",
        "namespace": "default",
...
      "metadata": {
        "name": "kube-dns",
        "namespace": "kube-system",
...

There are dozens of collection types (such as PodList, ServiceList, and NodeList) defined in the Kubernetes API. You can get more information about each collection type from the Kubernetes API documentation.

Some tools, such as kubectl, represent the Kubernetes collection mechanism slightly differently from the Kubernetes API itself. Because the output of kubectl might include the response from multiple list operations at the API level, kubectl represents a list of items using kind: List. For example:

kubectl get services -A -o yaml

apiVersion: v1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
items:
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: "2021-06-03T14:54:12Z"
    labels:
      component: apiserver
      provider: kubernetes
    name: kubernetes
    namespace: default
...
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      prometheus.io/port: "9153"
      prometheus.io/scrape: "true"
    creationTimestamp: "2021-06-03T14:54:14Z"
    labels:
      k8s-app: kube-dns
      kubernetes.io/cluster-service: "true"
      kubernetes.io/name: CoreDNS
    name: kube-dns
    namespace: kube-system

Receiving resources as Tables

When you run kubectl get, the default output format is a simple tabular representation of one or more instances of a particular resource type. In the past, clients were required to reproduce the tabular and describe output implemented in kubectl to perform simple lists of objects. A few limitations of that approach include non-trivial logic when dealing with certain objects. Additionally, types provided by API aggregation or third party resources are not known at compile time. This means that generic implementations had to be in place for types unrecognized by a client.

In order to avoid potential limitations as described above, clients may request the Table representation of objects, delegating specific details of printing to the server. The Kubernetes API implements standard HTTP content type negotiation: passing an Accept header containing a value of application/json;as=Table;g=meta.k8s.io;v=v1 with a GET call will request that the server return objects in the Table content type.

For example, list all of the pods on a cluster in the Table format.

GET /api/v1/pods
Accept: application/json;as=Table;g=meta.k8s.io;v=v1
---
200 OK
Content-Type: application/json

{
    "kind": "Table",
    "apiVersion": "meta.k8s.io/v1",
    ...
    "columnDefinitions": [
        ...
    ]
}

For API resource types that do not have a custom Table definition known to the control plane, the API server returns a default Table response that consists of the resource's name and creationTimestamp fields.

GET /apis/crd.example.com/v1alpha1/namespaces/default/resources
---
200 OK
Content-Type: application/json
...

{
    "kind": "Table",
    "apiVersion": "meta.k8s.io/v1",
    ...
    "columnDefinitions": [
        {
            "name": "Name",
            "type": "string",
            ...
        },
        {
            "name": "Created At",
            "type": "date",
            ...
        }
    ]
}

Not all API resource types support a Table response; for example, a CustomResourceDefinitions might not define field-to-table mappings, and an APIService that extends the core Kubernetes API might not serve Table responses at all. If you are implementing a client that uses the Table information and must work against all resource types, including extensions, you should make requests that specify multiple content types in the Accept header. For example:

Accept: application/json;as=Table;g=meta.k8s.io;v=v1, application/json

Resource deletion

When you delete a resource this takes place in two phases.

1. finalization
2. removal

{
  "kind": "ConfigMap",
  "apiVersion": "v1",
  "metadata": {
    "finalizers": ["url.io/neat-finalization", "other-url.io/my-finalizer"],
    "deletionTimestamp": nil,
  }
}

When a client first sends a delete to request the removal of a resource, the .metadata.deletionTimestamp is set to the current time. Once the .metadata.deletionTimestamp is set, external controllers that act on finalizers may start performing their cleanup work at any time, in any order.

Order is not enforced between finalizers because it would introduce significant risk of stuck .metadata.finalizers.

The .metadata.finalizers field is shared: any actor with permission can reorder it. If the finalizer list were processed in order, then this might lead to a situation in which the component responsible for the first finalizer in the list is waiting for some signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock.

Without enforced ordering, finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list.

Once the last finalizer is removed, the resource is actually removed from etcd.

Force deletion

By enabling the delete option ignoreStoreReadErrorWithClusterBreakingPotential, the user can perform an unsafe force delete operation of an undecryptable/corrupt resource. This option is behind an ALPHA feature gate, and it is disabled by default. In order to use this option, the cluster operator must enable the feature by setting the command line option --feature-gates=AllowUnsafeMalformedObjectDeletion=true.

A resource is considered corrupt if it can not be successfully retrieved from the storage due to:

• transformation error (for example: decryption failure), or
• the object failed to decode.

The API server first attempts a normal deletion, and if it fails with a corrupt resource error then it triggers the force delete. A force delete operation is unsafe because it ignores finalizer constraints, and skips precondition checks.

The default value for this option is false, this maintains backward compatibility. For a delete request with ignoreStoreReadErrorWithClusterBreakingPotential set to true, the fields dryRun, gracePeriodSeconds, orphanDependents, preconditions, and propagationPolicy must be left unset.

Single resource API

The Kubernetes API verbs get, create, update, patch, delete and proxy support single resources only. These verbs with single resource support have no support for submitting multiple resources together in an ordered or unordered list or transaction.

When clients (including kubectl) act on a set of resources, the client makes a series of single-resource API requests, then aggregates the responses if needed.

By contrast, the Kubernetes API verbs list and watch allow getting multiple resources, and deletecollection allows deleting multiple resources.

Field validation

Kubernetes always validates the type of fields. For example, if a field in the API is defined as a number, you cannot set the field to a text value. If a field is defined as an array of strings, you can only provide an array. Some fields allow you to omit them, other fields are required. Omitting a required field from an API request is an error.

If you make a request with an extra field, one that the cluster's control plane does not recognize, then the behavior of the API server is more complicated.

By default, the API server drops fields that it does not recognize from an input that it receives (for example, the JSON body of a PUT request).

There are two situations where the API server drops fields that you supplied in an HTTP request.

These situations are:

1. The field is unrecognized because it is not in the resource's OpenAPI schema. (One exception to this is for CRDs that explicitly choose not to prune unknown fields via x-kubernetes-preserve-unknown-fields).
2. The field is duplicated in the object.

Validation for unrecognized or duplicate fields

From 1.25 onward, unrecognized or duplicate fields in an object are detected via validation on the server when you use HTTP verbs that can submit data (POST, PUT, and PATCH). Possible levels of validation are Ignore, Warn (default), and Strict.

Ignore

The API server succeeds in handling the request as it would without the erroneous fields being set, dropping all unknown and duplicate fields and giving no indication it has done so.

Warn

(Default) The API server succeeds in handling the request, and reports a warning to the client. The warning is sent using the Warning: response header, adding one warning item for each unknown or duplicate field. For more information about warnings and the Kubernetes API, see the blog article Warning: Helpful Warnings Ahead.

Strict

The API server rejects the request with a 400 Bad Request error when it detects any unknown or duplicate fields. The response message from the API server specifies all the unknown or duplicate fields that the API server has detected.

The field validation level is set by the fieldValidation query parameter.

Tools that submit requests to the server (such as kubectl), might set their own defaults that are different from the Warn validation level that the API server uses by default.

The kubectl tool uses the --validate flag to set the level of field validation. It accepts the values ignore, warn, and strict while also accepting the values true (equivalent to strict) and false (equivalent to ignore). The default validation setting for kubectl is --validate=true, which means strict server-side field validation.

When kubectl cannot connect to an API server with field validation (API servers prior to Kubernetes 1.27), it will fall back to using client-side validation. Client-side validation will be removed entirely in a future version of kubectl.

Dry-run

When you use HTTP verbs that can modify resources (POST, PUT, PATCH, and DELETE), you can submit your request in a dry run mode. Dry run mode helps to evaluate a request through the typical request stages (admission chain, validation, merge conflicts) up until persisting objects to storage. The response body for the request is as close as possible to a non-dry-run response. Kubernetes guarantees that dry-run requests will not be persisted in storage or have any other side effects.

Make a dry-run request

Dry-run is triggered by setting the dryRun query parameter. This parameter is a string, working as an enum, and the only accepted values are:

[no value set]

Allow side effects. You request this with a query string such as ?dryRun or ?dryRun&pretty=true. The response is the final object that would have been persisted, or an error if the request could not be fulfilled.

All

Every stage runs as normal, except for the final storage stage where side effects are prevented.

When you set ?dryRun=All, any relevant admission controllers are run, validating admission controllers check the request post-mutation, merge is performed on PATCH, fields are defaulted, and schema validation occurs. The changes are not persisted to the underlying storage, but the final object which would have been persisted is still returned to the user, along with the normal status code.

If the non-dry-run version of a request would trigger an admission controller that has side effects, the request will be failed rather than risk an unwanted side effect. All built in admission control plugins support dry-run. Additionally, admission webhooks can declare in their configuration object that they do not have side effects, by setting their sideEffects field to None.

Here is an example dry-run request that uses ?dryRun=All:

POST /api/v1/namespaces/test/pods?dryRun=All
Content-Type: application/json
Accept: application/json

The response would look the same as for non-dry-run request, but the values of some generated fields may differ.

Generated values

Some values of an object are typically generated before the object is persisted. It is important not to rely upon the values of these fields set by a dry-run request, since these values will likely be different in dry-run mode from when the real request is made. Some of these fields are:

• name: if generateName is set, name will have a unique random name
• creationTimestamp / deletionTimestamp: records the time of creation/deletion
• UID: uniquely identifies the object and is randomly generated (non-deterministic)
• resourceVersion: tracks the persisted version of the object
• Any field set by a mutating admission controller
• For the Service resource: Ports or IP addresses that the kube-apiserver assigns to Service objects

Dry-run authorization

Authorization for dry-run and non-dry-run requests is identical. Thus, to make a dry-run request, you must be authorized to make the non-dry-run request.

For example, to run a dry-run patch for a Deployment, you must be authorized to perform that patch. Here is an example of a rule for Kubernetes RBAC that allows patching Deployments:

rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["patch"]

See Authorization Overview.

Updates to existing resources

Kubernetes provides several ways to update existing objects. You can read choosing an update mechanism to learn about which approach might be best for your use case.

You can overwrite (update) an existing resource - for example, a ConfigMap - using an HTTP PUT. For a PUT request, it is the client's responsibility to specify the resourceVersion (taking this from the object being updated). Kubernetes uses that resourceVersion information so that the API server can detect lost updates and reject requests made by a client that is out of date with the cluster. In the event that the resource has changed (the resourceVersion the client provided is stale), the API server returns a 409 Conflict error response.

Instead of sending a PUT request, the client can send an instruction to the API server to patch an existing resource. A patch is typically appropriate if the change that the client wants to make isn't conditional on the existing data. Clients that need effective detection of lost updates should consider making their request conditional on the existing resourceVersion (either HTTP PUT or HTTP PATCH), and then handle any retries that are needed in case there is a conflict.

The Kubernetes API supports four different PATCH operations, determined by their corresponding HTTP Content-Type header:

application/apply-patch+yaml

Server Side Apply YAML (a Kubernetes-specific extension, based on YAML). All JSON documents are valid YAML, so you can also submit JSON using this media type. See Server Side Apply serialization for more details. To Kubernetes, this is a create operation if the object does not exist, or a patch operation if the object already exists.

application/json-patch+json

JSON Patch, as defined in RFC6902. A JSON patch is a sequence of operations that are executed on the resource; for example {"op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ]}. To Kubernetes, this is a patch operation.

A patch using application/json-patch+json can include conditions to validate consistency, allowing the operation to fail if those conditions are not met (for example, to avoid a lost update).

application/merge-patch+json

JSON Merge Patch, as defined in RFC7386. A JSON Merge Patch is essentially a partial representation of the resource. The submitted JSON is combined with the current resource to create a new one, then the new one is saved. To Kubernetes, this is a patch operation.

application/strategic-merge-patch+json

Strategic Merge Patch (a Kubernetes-specific extension based on JSON). Strategic Merge Patch is a custom implementation of JSON Merge Patch. You can only use Strategic Merge Patch with built-in APIs, or with aggregated API servers that have special support for it. You cannot use application/strategic-merge-patch+json with any API defined using a CustomResourceDefinition.

Note:

The Kubernetes server side apply mechanism has superseded Strategic Merge Patch.

Kubernetes' Server Side Apply feature allows the control plane to track managed fields for newly created objects. Server Side Apply provides a clear pattern for managing field conflicts, offers server-side apply and update operations, and replaces the client-side functionality of kubectl apply.

For Server-Side Apply, Kubernetes treats the request as a create if the object does not yet exist, and a patch otherwise. For other requests that use PATCH at the HTTP level, the logical Kubernetes operation is always patch.

See Server Side Apply for more details.

Choosing an update mechanism

HTTP PUT to replace existing resource

The update (HTTP PUT) operation is simple to implement and flexible, but has drawbacks:

• You need to handle conflicts where the resourceVersion of the object changes between your client reading it and trying to write it back. Kubernetes always detects the conflict, but you as the client author need to implement retries.
• You might accidentally drop fields if you decode an object locally (for example, using client-go, you could receive fields that your client does not know how to handle - and then drop them as part of your update.
• If there's a lot of contention on the object (even on a field, or set of fields, that you're not trying to edit), you might have trouble sending the update. The problem is worse for larger objects and for objects with many fields.

HTTP PATCH using JSON Patch

A patch update is helpful, because:

• As you're only sending differences, you have less data to send in the PATCH request.
• You can make changes that rely on existing values, such as copying the value of a particular field into an annotation.
• Unlike with an update (HTTP PUT), making your change can happen right away even if there are frequent changes to unrelated fields): you usually would not need to retry.
  • You might still need to specify the resourceVersion (to match an existing object) if you want to be extra careful to avoid lost updates
  • It's still good practice to write in some retry logic in case of errors.
• You can use test conditions to careful craft specific update conditions. For example, you can increment a counter without reading it if the existing value matches what you expect. You can do this with no lost update risk, even if the object has changed in other ways since you last wrote to it. (If the test condition fails, you can fall back to reading the current value and then write back the changed number).

However:

• You need more local (client) logic to build the patch; it helps a lot if you have a library implementation of JSON Patch, or even for making a JSON Patch specifically against Kubernetes.
• As the author of client software, you need to be careful when building the patch (the HTTP request body) not to drop fields (the order of operations matters).

HTTP PATCH using Server-Side Apply

Server-Side Apply has some clear benefits:

• A single round trip: it rarely requires making a GET request first.
  • and you can still detect conflicts for unexpected changes
  • you have the option to force override a conflict, if appropriate
• Client implementations are easy to make.
• You get an atomic create-or-update operation without extra effort (similar to UPSERT in some SQL dialects).

However:

• Server-Side Apply does not work at all for field changes that depend on a current value of the object.
• You can only apply updates to objects. Some resources in the Kubernetes HTTP API are not objects (they do not have a .metadata field), and Server-Side Apply is only relevant for Kubernetes objects.

Resource versions

Resource versions are strings that identify the server's internal version of an object. Resource versions can be used by clients to determine when objects have changed, or to express data consistency requirements when getting, listing and watching resources. Resource versions must be treated as opaque by clients and passed unmodified back to the server.

You must not assume resource versions are numeric or collatable. API clients may only compare two resource versions for equality (this means that you must not compare resource versions for greater-than or less-than relationships).

resourceVersion fields in metadata

Clients find resource versions in resources, including the resources from the response stream for a watch, or when using list to enumerate resources.

v1.meta/ObjectMeta - The metadata.resourceVersion of a resource instance identifies the resource version the instance was last modified at.

v1.meta/ListMeta - The metadata.resourceVersion of a resource collection (the response to a list) identifies the resource version at which the collection was constructed.

resourceVersion parameters in query strings

The get, list, and watch operations support the resourceVersion parameter. From version v1.19, Kubernetes API servers also support the resourceVersionMatch parameter on list requests.

The API server interprets the resourceVersion parameter differently depending on the operation you request, and on the value of resourceVersion. If you set resourceVersionMatch then this also affects the way matching happens.

Semantics for get and list

For get and list, the semantics of resourceVersion are:

get:

list:

From version v1.19, Kubernetes API servers support the resourceVersionMatch parameter on list requests. If you set both resourceVersion and resourceVersionMatch, the resourceVersionMatch parameter determines how the API server interprets resourceVersion.

You should always set the resourceVersionMatch parameter when setting resourceVersion on a list request. However, be prepared to handle the case where the API server that responds is unaware of resourceVersionMatch and ignores it.

Unless you have strong consistency requirements, using resourceVersionMatch=NotOlderThan and a known resourceVersion is preferable since it can achieve better performance and scalability of your cluster than leaving resourceVersion and resourceVersionMatch unset, which requires quorum read to be served.

Setting the resourceVersionMatch parameter without setting resourceVersion is not valid.

This table explains the behavior of list requests with various combinations of resourceVersion and resourceVersionMatch:

The meaning of the get and list semantics are:

Any

Return data at any resource version. The newest available resource version is preferred, but strong consistency is not required; data at any resource version may be served. It is possible for the request to return data at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches. Clients that cannot tolerate this should not use this semantic. Always served from watch cache, improving performance and reducing etcd load.

Most recent

Return data at the most recent resource version. The returned data must be consistent (in detail: served from etcd via a quorum read). For etcd v3.4.31+ and v3.5.13+, Kubernetes 1.33 serves “most recent” reads from the watch cache: an internal, in-memory store within the API server that caches and mirrors the state of data persisted into etcd. Kubernetes requests progress notification to maintain cache consistency against the etcd persistence layer. Kubernetes v1.28 through to v1.30 also supported this feature, although as Alpha it was not recommended for production nor enabled by default until the v1.31 release.

Not older than

Return data at least as new as the provided resourceVersion. The newest available data is preferred, but any data not older than the provided resourceVersion may be served. For list requests to servers that honor the resourceVersionMatch parameter, this guarantees that the collection's .metadata.resourceVersion is not older than the requested resourceVersion, but does not make any guarantee about the .metadata.resourceVersion of any of the items in that collection. Always served from watch cache, improving performance and reducing etcd load.

Exact

Return data at the exact resource version provided. If the provided resourceVersion is unavailable, the server responds with HTTP 410 Gone. For list requests to servers that honor the resourceVersionMatch parameter, this guarantees that the collection's .metadata.resourceVersion is the same as the resourceVersion you requested in the query string. That guarantee does not apply to the .metadata.resourceVersion of any items within that collection. By default served from etcd, but with the ListFromCacheSnapshot feature gate enabled, API server will attempt to serve the response from snapshot if available. This improves performance and reduces etcd load. Cache snapshots are kept by default for 75 seconds, so if the provided resourceVersion is unavailable, the server will fallback to etcd.

Continuation

Return the next page of data for a paginated list request, ensuring consistency with the exact resourceVersion established by the initial request in the sequence. Response to list requests with limit include continue token, that encodes the resourceVersion and last observed position from which to resume the list. If the resourceVersion in the provided continue token is unavailable, the server responds with HTTP 410 Gone. By default served from etcd, but with the ListFromCacheSnapshot feature gate enabled, API server will attempt to serve the response from snapshot if available. This improves performance and reduces etcd load. Cache snapshots are kept by default for 75 seconds, so if the resourceVersion in provided continue token is unavailable, the server will fallback to etcd.

When using resourceVersionMatch=NotOlderThan and limit is set, clients must handle HTTP 410 Gone responses. For example, the client might retry with a newer resourceVersion or fall back to resourceVersion="".

When using resourceVersionMatch=Exact and limit is unset, clients must verify that the collection's .metadata.resourceVersion matches the requested resourceVersion, and handle the case where it does not. For example, the client might fall back to a request with limit set.

Semantics for watch

For watch, the semantics of resource version are:

watch:

The meaning of those watch semantics are:

Get State and Start at Any

Start a watch at any resource version; the most recent resource version available is preferred, but not required. Any starting resource version is allowed. It is possible for the watch to start at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches. Clients that cannot tolerate this apparent rewinding should not start a watch with this semantic. To establish initial state, the watch begins with synthetic "Added" events for all resource instances that exist at the starting resource version. All following watch events are for all changes that occurred after the resource version the watch started at.

Caution:

watches initialized this way may return arbitrarily stale data. Please review this semantic before using it, and favor the other semantics where possible.

Get State and Start at Most Recent

Start a watch at the most recent resource version, which must be consistent (in detail: served from etcd via a quorum read). To establish initial state, the watch begins with synthetic "Added" events of all resources instances that exist at the starting resource version. All following watch events are for all changes that occurred after the resource version the watch started at.

Start at Exact

Start a watch at an exact resource version. The watch events are for all changes after the provided resource version. Unlike "Get State and Start at Most Recent" and "Get State and Start at Any", the watch is not started with synthetic "Added" events for the provided resource version. The client is assumed to already have the initial state at the starting resource version since the client provided the resource version.

"410 Gone" responses

Servers are not required to serve all older resource versions and may return a HTTP 410 (Gone) status code if a client requests a resourceVersion older than the server has retained. Clients must be able to tolerate 410 (Gone) responses. See Efficient detection of changes for details on how to handle 410 (Gone) responses when watching resources.

If you request a resourceVersion outside the applicable limit then, depending on whether a request is served from cache or not, the API server may reply with a 410 Gone HTTP response.

Unavailable resource versions

Servers are not required to serve unrecognized resource versions. If you request list or get for a resource version that the API server does not recognize, then the API server may either:

• wait briefly for the resource version to become available, then timeout with a 504 (Gateway Timeout) if the provided resource versions does not become available in a reasonable amount of time;
• respond with a Retry-After response header indicating how many seconds a client should wait before retrying the request.

If you request a resource version that an API server does not recognize, the kube-apiserver additionally identifies its error responses with a message Too large resource version.

If you make a watch request for an unrecognized resource version, the API server may wait indefinitely (until the request timeout) for the resource version to become available.

2.3 - Server-Side Apply

Kubernetes supports multiple appliers collaborating to manage the fields of a single object.

Server-Side Apply provides an optional mechanism for your cluster's control plane to track changes to an object's fields. At the level of a specific resource, Server-Side Apply records and tracks information about control over the fields of that object.

Server-Side Apply helps users and controllers manage their resources through declarative configuration. Clients can create and modify objects declaratively by submitting their fully specified intent.

A fully specified intent is a partial object that only includes the fields and values for which the user has an opinion. That intent either creates a new object (using default values for unspecified fields), or is combined, by the API server, with the existing object.

Comparison with Client-Side Apply explains how Server-Side Apply differs from the original, client-side kubectl apply implementation.

Field management

The Kubernetes API server tracks managed fields for all newly created objects.

When trying to apply an object, fields that have a different value and are owned by another manager will result in a conflict. This is done in order to signal that the operation might undo another collaborator's changes. Writes to objects with managed fields can be forced, in which case the value of any conflicted field will be overridden, and the ownership will be transferred.

Whenever a field's value does change, ownership moves from its current manager to the manager making the change.

Apply checks if there are any other field managers that also own the field. If the field is not owned by any other field managers, that field is set to its default value (if there is one), or otherwise is deleted from the object. The same rule applies to fields that are lists, associative lists, or maps.

For a user to manage a field, in the Server-Side Apply sense, means that the user relies on and expects the value of the field not to change. The user who last made an assertion about the value of a field will be recorded as the current field manager. This can be done by changing the field manager details explicitly using HTTP POST (create), PUT (update), or non-apply PATCH (patch). You can also declare and record a field manager by including a value for that field in a Server-Side Apply operation.

A Server-Side Apply patch request requires the client to provide its identity as a field manager. When using Server-Side Apply, trying to change a field that is controlled by a different manager results in a rejected request unless the client forces an override. For details of overrides, see Conflicts.

When two or more appliers set a field to the same value, they share ownership of that field. Any subsequent attempt to change the value of the shared field, by any of the appliers, results in a conflict. Shared field owners may give up ownership of a field by making a Server-Side Apply patch request that doesn't include that field.

Field management details are stored in a managedFields field that is part of an object's metadata.

If you remove a field from a manifest and apply that manifest, Server-Side Apply checks if there are any other field managers that also own the field. If the field is not owned by any other field managers, it is either deleted from the live object or reset to its default value, if it has one. The same rule applies to associative list or map items.

Compared to the (legacy) kubectl.kubernetes.io/last-applied-configuration annotation managed by kubectl, Server-Side Apply uses a more declarative approach, that tracks a user's (or client's) field management, rather than a user's last applied state. As a side effect of using Server-Side Apply, information about which field manager manages each field in an object also becomes available.

Example

A simple example of an object created using Server-Side Apply could look like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test-cm
  namespace: default
  labels:
    test-label: test
  managedFields:
  - manager: kubectl
    operation: Apply # note capitalization: "Apply" (or "Update")
    apiVersion: v1
    time: "2010-10-10T0:00:00Z"
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:test-label: {}
      f:data:
        f:key: {}
data:
  key: some value

That example ConfigMap object contains a single field management record in .metadata.managedFields. The field management record consists of basic information about the managing entity itself, plus details about the fields being managed and the relevant operation (Apply or Update). If the request that last changed that field was a Server-Side Apply patch then the value of operation is Apply; otherwise, it is Update.

There is another possible outcome. A client could submit an invalid request body. If the fully specified intent does not produce a valid object, the request fails.

It is however possible to change .metadata.managedFields through an update, or through a patch operation that does not use Server-Side Apply. Doing so is highly discouraged, but might be a reasonable option to try if, for example, the .metadata.managedFields get into an inconsistent state (which should not happen in normal operations).

The format of managedFields is described in the Kubernetes API reference.

Conflicts

A conflict is a special status error that occurs when an Apply operation tries to change a field that another manager also claims to manage. This prevents an applier from unintentionally overwriting the value set by another user. When this occurs, the applier has 3 options to resolve the conflicts:

• Overwrite value, become sole manager: If overwriting the value was intentional (or if the applier is an automated process like a controller) the applier should set the force query parameter to true (for kubectl apply, you use the --force-conflicts command line parameter), and make the request again. This forces the operation to succeed, changes the value of the field, and removes the field from all other managers' entries in managedFields.

• Don't overwrite value, give up management claim: If the applier doesn't care about the value of the field any more, the applier can remove it from their local model of the resource, and make a new request with that particular field omitted. This leaves the value unchanged, and causes the field to be removed from the applier's entry in managedFields.

• Don't overwrite value, become shared manager: If the applier still cares about the value of a field, but doesn't want to overwrite it, they can change the value of that field in their local model of the resource so as to match the value of the object on the server, and then make a new request that takes into account that local update. Doing so leaves the value unchanged, and causes that field's management to be shared by the applier along with all other field managers that already claimed to manage it.

Field managers

Managers identify distinct workflows that are modifying the object (especially useful on conflicts!), and can be specified through the fieldManager query parameter as part of a modifying request. When you Apply to a resource, the fieldManager parameter is required. For other updates, the API server infers a field manager identity from the "User-Agent:" HTTP header (if present).

When you use the kubectl tool to perform a Server-Side Apply operation, kubectl sets the manager identity to "kubectl" by default.

Serialization

At the protocol level, Kubernetes represents Server-Side Apply message bodies as YAML, with the media type application/apply-patch+yaml.

The serialization is the same as for Kubernetes objects, with the exception that clients are not required to send a complete object.

Here's an example of a Server-Side Apply message body (fully specified intent):

{
  "apiVersion": "v1",
  "kind": "ConfigMap"
}

(this would make a no-change update, provided that it was sent as the body of a patch request to a valid v1/configmaps resource, and with the appropriate request Content-Type).

Operations in scope for field management

The Kubernetes API operations where field management is considered are:

1. Server-Side Apply (HTTP PATCH, with content type application/apply-patch+yaml)
2. Replacing an existing object (update to Kubernetes; PUT at the HTTP level)

Both operations update .metadata.managedFields, but behave a little differently.

Unless you specify a forced override, an apply operation that encounters field-level conflicts always fails; by contrast, if you make a change using update that would affect a managed field, a conflict never provokes failure of the operation.

All Server-Side Apply patch requests are required to identify themselves by providing a fieldManager query parameter, while the query parameter is optional for update operations. Finally, when using the Apply operation you cannot define managedFields in the body of the request that you submit.

An example object with multiple managers could look like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test-cm
  namespace: default
  labels:
    test-label: test
  managedFields:
  - manager: kubectl
    operation: Apply
    time: '2019-03-30T15:00:00.000Z'
    apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:test-label: {}
  - manager: kube-controller-manager
    operation: Update
    apiVersion: v1
    time: '2019-03-30T16:00:00.000Z'
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key: {}
data:
  key: new value

In this example, a second operation was run as an update by the manager called kube-controller-manager. The update request succeeded and changed a value in the data field, which caused that field's management to change to the kube-controller-manager.

If this update has instead been attempted using Server-Side Apply, the request would have failed due to conflicting ownership.

Merge strategy

The merging strategy, implemented with Server-Side Apply, provides a generally more stable object lifecycle. Server-Side Apply tries to merge fields based on the actor who manages them instead of overruling based on values. This way multiple actors can update the same object without causing unexpected interference.

When a user sends a fully-specified intent object to the Server-Side Apply endpoint, the server merges it with the live object favoring the value from the request body if it is specified in both places. If the set of items present in the applied config is not a superset of the items applied by the same user last time, each missing item not managed by any other appliers is removed. For more information about how an object's schema is used to make decisions when merging, see sigs.k8s.io/structured-merge-diff.

The Kubernetes API (and the Go code that implements that API for Kubernetes) allows defining merge strategy markers. These markers describe the merge strategy supported for fields within Kubernetes objects. For a CustomResourceDefinition, you can set these markers when you define the custom resource.

If listType is missing, the API server interprets a patchStrategy=merge marker as a listType=map and the corresponding patchMergeKey marker as a listMapKey.

The atomic list type is recursive.

(In the Go code for Kubernetes, these markers are specified as comments and code authors need not repeat them as field tags).

Custom resources and Server-Side Apply

By default, Server-Side Apply treats custom resources as unstructured data. All keys are treated the same as struct fields, and all lists are considered atomic.

If the CustomResourceDefinition defines a schema that contains annotations as defined in the previous Merge Strategy section, these annotations will be used when merging objects of this type.

Compatibility across topology changes

On rare occurrences, the author for a CustomResourceDefinition (CRD) or built-in may want to change the specific topology of a field in their resource, without incrementing its API version. Changing the topology of types, by upgrading the cluster or updating the CRD, has different consequences when updating existing objects. There are two categories of changes: when a field goes from map/set/granular to atomic, and the other way around.

When the listType, mapType, or structType changes from map/set/granular to atomic, the whole list, map, or struct of existing objects will end-up being owned by actors who owned an element of these types. This means that any further change to these objects would cause a conflict.

When a listType, mapType, or structType changes from atomic to map/set/granular, the API server is unable to infer the new ownership of these fields. Because of that, no conflict will be produced when objects have these fields updated. For that reason, it is not recommended to change a type from atomic to map/set/granular.

Take for example, the custom resource:

---
apiVersion: example.com/v1
kind: Foo
metadata:
  name: foo-sample
  managedFields:
  - manager: "manager-one"
    operation: Apply
    apiVersion: example.com/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:data: {}
spec:
  data:
    key1: val1
    key2: val2

Before spec.data gets changed from atomic to granular, manager-one owns the field spec.data, and all the fields within it (key1 and key2). When the CRD gets changed to make spec.data granular, manager-one continues to own the top-level field spec.data (meaning no other managers can delete the map called data without a conflict), but it no longer owns key1 and key2, so another manager can then modify or delete those fields without conflict.

Using Server-Side Apply in a controller

As a developer of a controller, you can use Server-Side Apply as a way to simplify the update logic of your controller. The main differences with a read-modify-write and/or patch are the following:

• the applied object must contain all the fields that the controller cares about.
• there is no way to remove fields that haven't been applied by the controller before (controller can still send a patch or update for these use-cases).
• the object doesn't have to be read beforehand; resourceVersion doesn't have to be specified.

It is strongly recommended for controllers to always force conflicts on objects that they own and manage, since they might not be able to resolve or act on these conflicts.

Transferring ownership

In addition to the concurrency controls provided by conflict resolution, Server-Side Apply provides ways to perform coordinated field ownership transfers from users to controllers.

This is best explained by example. Let's look at how to safely transfer ownership of the replicas field from a user to a controller while enabling automatic horizontal scaling for a Deployment, using the HorizontalPodAutoscaler resource and its accompanying controller.

Say a user has defined Deployment with replicas set to the desired value:

application/ssa/nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

And the user has created the Deployment using Server-Side Apply, like so:

kubectl apply -f https://k8s.io/examples/application/ssa/nginx-deployment.yaml --server-side

Then later, automatic scaling is enabled for the Deployment; for example:

kubectl autoscale deployment nginx-deployment --cpu-percent=50 --min=1 --max=10

Now, the user would like to remove replicas from their configuration, so they don't accidentally fight with the HorizontalPodAutoscaler (HPA) and its controller. However, there is a race: it might take some time before the HPA feels the need to adjust .spec.replicas; if the user removes .spec.replicas before the HPA writes to the field and becomes its owner, then the API server would set .spec.replicas to 1 (the default replica count for Deployment). This is not what the user wants to happen, even temporarily - it might well degrade a running workload.

There are two solutions:

• (basic) Leave replicas in the configuration; when the HPA eventually writes to that field, the system gives the user a conflict over it. At that point, it is safe to remove from the configuration.

• (more advanced) If, however, the user doesn't want to wait, for example because they want to keep the cluster legible to their colleagues, then they can take the following steps to make it safe to remove replicas from their configuration:

First, the user defines a new manifest containing only the replicas field:

# Save this file as 'nginx-deployment-replicas-only.yaml'.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3

The user applies that manifest using a private field manager name. In this example, the user picked handover-to-hpa:

kubectl apply -f nginx-deployment-replicas-only.yaml \
  --server-side --field-manager=handover-to-hpa \
  --validate=false

If the apply results in a conflict with the HPA controller, then do nothing. The conflict indicates the controller has claimed the field earlier in the process than it sometimes does.

At this point the user may remove the replicas field from their manifest:

application/ssa/nginx-deployment-no-replicas.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

Note that whenever the HPA controller sets the replicas field to a new value, the temporary field manager will no longer own any fields and will be automatically deleted. No further clean up is required.

Transferring ownership between managers

Field managers can transfer ownership of a field between each other by setting the field to the same value in both of their applied configurations, causing them to share ownership of the field. Once the managers share ownership of the field, one of them can remove the field from their applied configuration to give up ownership and complete the transfer to the other field manager.

Comparison with Client-Side Apply

Server-Side Apply is meant both as a replacement for the original client-side implementation of the kubectl apply subcommand, and as simple and effective mechanism for controllers to enact their changes.

Compared to the last-applied annotation managed by kubectl, Server-Side Apply uses a more declarative approach, which tracks an object's field management, rather than a user's last applied state. This means that as a side effect of using Server-Side Apply, information about which field manager manages each field in an object also becomes available.

A consequence of the conflict detection and resolution implemented by Server-Side Apply is that an applier always has up to date field values in their local state. If they don't, they get a conflict the next time they apply. Any of the three options to resolve conflicts results in the applied configuration being an up to date subset of the object on the server's fields.

This is different from Client-Side Apply, where outdated values which have been overwritten by other users are left in an applier's local config. These values only become accurate when the user updates that specific field, if ever, and an applier has no way of knowing whether their next apply will overwrite other users' changes.

Another difference is that an applier using Client-Side Apply is unable to change the API version they are using, but Server-Side Apply supports this use case.

Migration between client-side and server-side apply

Upgrading from client-side apply to server-side apply

Client-side apply users who manage a resource with kubectl apply can start using server-side apply with the following flag.

kubectl apply --server-side [--dry-run=server]

By default, field management of the object transfers from client-side apply to kubectl server-side apply, without encountering conflicts.

This behavior applies to server-side apply with the kubectl field manager. As an exception, you can opt-out of this behavior by specifying a different, non-default field manager, as seen in the following example. The default field manager for kubectl server-side apply is kubectl.

kubectl apply --server-side --field-manager=my-manager [--dry-run=server]

Downgrading from server-side apply to client-side apply

If you manage a resource with kubectl apply --server-side, you can downgrade to client-side apply directly with kubectl apply.

Downgrading works because kubectl Server-Side Apply keeps the last-applied-configuration annotation up-to-date if you use kubectl apply.

This behavior applies to Server-Side Apply with the kubectl field manager. As an exception, you can opt-out of this behavior by specifying a different, non-default field manager, as seen in the following example. The default field manager for kubectl server-side apply is kubectl.

kubectl apply --server-side --field-manager=my-manager [--dry-run=server]

API implementation

The PATCH verb for a resource that supports Server-Side Apply can accepts the unofficial application/apply-patch+yaml content type. Users of Server-Side Apply can send partially specified objects as YAML as the body of a PATCH request to the URI of a resource. When applying a configuration, you should always include all the fields that are important to the outcome (such as a desired state) that you want to define.

All JSON messages are valid YAML. Therefore, in addition to using YAML request bodies for Server-Side Apply requests, you can also use JSON request bodies, as they are also valid YAML. In either case, use the media type application/apply-patch+yaml for the HTTP request.

Access control and permissions

Since Server-Side Apply is a type of PATCH, a principal (such as a Role for Kubernetes RBAC) requires the patch permission to edit existing resources, and also needs the create verb permission in order to create new resources with Server-Side Apply.

Clearing managedFields

It is possible to strip all managedFields from an object by overwriting them using a patch (JSON Merge Patch, Strategic Merge Patch, JSON Patch), or through an update (HTTP PUT); in other words, through every write operation other than apply. This can be done by overwriting the managedFields field with an empty entry. Two examples are:

PATCH /api/v1/namespaces/default/configmaps/example-cm
Accept: application/json
Content-Type: application/merge-patch+json

{
  "metadata": {
    "managedFields": [
      {}
    ]
  }
}

PATCH /api/v1/namespaces/default/configmaps/example-cm
Accept: application/json
Content-Type: application/json-patch+json
If-Match: 1234567890123456789

[{"op": "replace", "path": "/metadata/managedFields", "value": [{}]}]

This will overwrite the managedFields with a list containing a single empty entry that then results in the managedFields being stripped entirely from the object. Note that setting the managedFields to an empty list will not reset the field. This is on purpose, so managedFields never get stripped by clients not aware of the field.

In cases where the reset operation is combined with changes to other fields than the managedFields, this will result in the managedFields being reset first and the other changes being processed afterwards. As a result the applier takes ownership of any fields updated in the same request.

What's next

You can read about managedFields within the Kubernetes API reference for the metadata top level field.

2.4 - Client Libraries

This page contains an overview of the client libraries for using the Kubernetes API from various programming languages.

To write applications using the Kubernetes REST API, you do not need to implement the API calls and request/response types yourself. You can use a client library for the programming language you are using.

Client libraries often handle common tasks such as authentication for you. Most client libraries can discover and use the Kubernetes Service Account to authenticate if the API client is running inside the Kubernetes cluster, or can understand the kubeconfig file format to read the credentials and the API Server address.

Officially-supported Kubernetes client libraries

The following client libraries are officially maintained by Kubernetes SIG API Machinery.

Community-maintained client libraries

The following Kubernetes API client libraries are provided and maintained by their authors, not the Kubernetes team.

2.5 - Common Expression Language in Kubernetes

The Common Expression Language (CEL) is used in the Kubernetes API to declare validation rules, policy rules, and other constraints or conditions.

CEL expressions are evaluated directly in the API server, making CEL a convenient alternative to out-of-process mechanisms, such as webhooks, for many extensibility use cases. Your CEL expressions continue to execute so long as the control plane's API server component remains available.

Language overview

The CEL language has a straightforward syntax that is similar to the expressions in C, C++, Java, JavaScript and Go.

CEL was designed to be embedded into applications. Each CEL "program" is a single expression that evaluates to a single value. CEL expressions are typically short "one-liners" that inline well into the string fields of Kubernetes API resources.

Inputs to a CEL program are "variables". Each Kubernetes API field that contains CEL declares in the API documentation which variables are available to use for that field. For example, in the x-kubernetes-validations[i].rules field of CustomResourceDefinitions, the self and oldSelf variables are available and refer to the previous and current state of the custom resource data to be validated by the CEL expression. Other Kubernetes API fields may declare different variables. See the API documentation of the API fields to learn which variables are available for that field.

Example CEL expressions:

CEL options, language features, and libraries

CEL is configured with the following options, libraries and language features, introduced at the specified Kubernetes versions:

CEL functions, features and language settings support Kubernetes control plane rollbacks. For example, CEL Optional Values was introduced at Kubernetes 1.29 and so only API servers at that version or newer will accept write requests to CEL expressions that use CEL Optional Values. However, when a cluster is rolled back to Kubernetes 1.28 CEL expressions using "CEL Optional Values" that are already stored in API resources will continue to evaluate correctly.

Kubernetes CEL libraries

In additional to the CEL community libraries, Kubernetes includes CEL libraries that are available everywhere CEL is used in Kubernetes.

Kubernetes list library

The list library includes indexOf and lastIndexOf, which work similar to the strings functions of the same names. These functions either the first or last positional index of the provided element in the list.

The list library also includes min, max and sum. Sum is supported on all number types as well as the duration type. Min and max are supported on all comparable types.

isSorted is also provided as a convenience function and is supported on all comparable types.

Examples:

See the Kubernetes List Library godoc for more information.

Kubernetes regex library

In addition to the matches function provided by the CEL standard library, the regex library provides find and findAll, enabling a much wider range of regex operations.

Examples:

See the Kubernetes regex library godoc for more information.

Kubernetes URL library

To make it easier and safer to process URLs, the following functions have been added:

• isURL(string) checks if a string is a valid URL according to the Go's net/url package. The string must be an absolute URL.
• url(string) URL converts a string to a URL or results in an error if the string is not a valid URL.

Once parsed via the url function, the resulting URL object has getScheme, getHost, getHostname, getPort, getEscapedPath and getQuery accessors.

Examples:

See the Kubernetes URL library godoc for more information.

Kubernetes authorizer library

For CEL expressions in the API where a variable of type Authorizer is available, the authorizer may be used to perform authorization checks for the principal (authenticated user) of the request.

API resource checks are performed as follows:

1. Specify the group and resource to check: Authorizer.group(string).resource(string) ResourceCheck
2. Optionally call any combination of the following builder functions to further narrow the authorization check. Note that these functions return the receiver type and can be chained:
   • ResourceCheck.subresource(string) ResourceCheck
   • ResourceCheck.namespace(string) ResourceCheck
   • ResourceCheck.name(string) ResourceCheck
3. Call ResourceCheck.check(verb string) Decision to perform the authorization check.
4. Call allowed() bool or reason() string to inspect the result of the authorization check.

Non-resource authorization performed are used as follows:

1. Specify only a path: Authorizer.path(string) PathCheck
2. Call PathCheck.check(httpVerb string) Decision to perform the authorization check.
3. Call allowed() bool or reason() string to inspect the result of the authorization check.

To perform an authorization check for a service account:

• Authorizer.serviceAccount(namespace string, name string) Authorizer

With the alpha AuthorizeWithSelectors feature enabled, field and label selectors can be added to authorization checks.

See the Kubernetes Authz library and Kubernetes AuthzSelectors library godoc for more information.

Kubernetes quantity library

Kubernetes 1.28 adds support for manipulating quantity strings (ex 1.5G, 512k, 20Mi)

• isQuantity(string) checks if a string is a valid Quantity according to Kubernetes' resource.Quantity.
• quantity(string) Quantity converts a string to a Quantity or results in an error if the string is not a valid quantity.

Once parsed via the quantity function, the resulting Quantity object has the following library of member functions:

Examples:

Type checking

CEL is a gradually typed language.

Some Kubernetes API fields contain fully type checked CEL expressions. For example, CustomResourceDefinitions Validation Rules are fully type checked.

Some Kubernetes API fields contain partially type checked CEL expressions. A partially type checked expression is an expressions where some of the variables are statically typed but others are dynamically typed. For example, in the CEL expressions of ValidatingAdmissionPolicies the request variable is typed, but the object variable is dynamically typed. As a result, an expression containing request.namex would fail type checking because the namex field is not defined. However, object.namex would pass type checking even when the namex field is not defined for the resource kinds that object refers to, because object is dynamically typed.

The has() macro in CEL may be used in CEL expressions to check if a field of a dynamically typed variable is accessible before attempting to access the field's value. For example:

has(object.namex) ? object.namex == 'special' : request.name == 'special'

Type system integration

Also see: CEL types, OpenAPI types, Kubernetes Structural Schemas.

Equality comparison for arrays with x-kubernetes-list-type of set or map ignores element order. For example [1, 2] == [2, 1] if the arrays represent Kubernetes set values.

Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:

set

X + Y performs a union where the array positions of all elements in X are preserved and non-intersecting elements in Y are appended, retaining their partial order.

map

X + Y performs a merge where the array positions of all keys in X are preserved but the values are overwritten by values in Y when the key sets of X and Y intersect. Elements in Y with non-intersecting keys are appended, retaining their partial order.

Escaping

Only Kubernetes resource property names of the form [a-zA-Z_.-/][a-zA-Z0-9_.-/]* are accessible from CEL. Accessible property names are escaped according to the following rules when accessed in the expression:

When you escape any of CEL's RESERVED keywords you need to match the exact property name use the underscore escaping (for example, int in the word sprint would not be escaped and nor would it need to be).

Examples on escaping:

Resource constraints

CEL is non-Turing complete and offers a variety of production safety controls to limit execution time. CEL's resource constraint features provide feedback to developers about expression complexity and help protect the API server from excessive resource consumption during evaluation. CEL's resource constraint features are used to prevent CEL evaluation from consuming excessive API server resources.

A key element of the resource constraint features is a cost unit that CEL defines as a way of tracking CPU utilization. Cost units are independent of system load and hardware. Cost units are also deterministic; for any given CEL expression and input data, evaluation of the expression by the CEL interpreter will always result in the same cost.

Many of CEL's core operations have fixed costs. The simplest operations, such as comparisons (e.g. <) have a cost of 1. Some have a higher fixed cost, for example list literal declarations have a fixed base cost of 40 cost units.

Calls to functions implemented in native code approximate cost based on the time complexity of the operation. For example: operations that use regular expressions, such as match and find, are estimated using an approximated cost of length(regexString)*length(inputString). The approximated cost reflects the worst case time complexity of Go's RE2 implementation.

Runtime cost budget

All CEL expressions evaluated by Kubernetes are constrained by a runtime cost budget. The runtime cost budget is an estimate of actual CPU utilization computed by incrementing a cost unit counter while interpreting a CEL expression. If the CEL interpreter executes too many instructions, the runtime cost budget will be exceeded, execution of the expressions will be halted, and an error will result.

Some Kubernetes resources define an additional runtime cost budget that bounds the execution of multiple expressions. If the sum total of the cost of expressions exceed the budget, execution of the expressions will be halted, and an error will result. For example the validation of a custom resource has a per-validation runtime cost budget for all Validation Rules evaluated to validate the custom resource.

Estimated cost limits

For some Kubernetes resources, the API server may also check if worst case estimated running time of CEL expressions would be prohibitively expensive to execute. If so, the API server prevent the CEL expression from being written to API resources by rejecting create or update operations containing the CEL expression to the API resources. This feature offers a stronger assurance that CEL expressions written to the API resource will be evaluated at runtime without exceeding the runtime cost budget.

2.6 - Kubernetes Deprecation Policy

This document details the deprecation policy for various facets of the system.

Kubernetes is a large system with many components and many contributors. As with any such software, the feature set naturally evolves over time, and sometimes a feature may need to be removed. This could include an API, a flag, or even an entire feature. To avoid breaking existing users, Kubernetes follows a deprecation policy for aspects of the system that are slated to be removed.

Deprecating parts of the API

Since Kubernetes is an API-driven system, the API has evolved over time to reflect the evolving understanding of the problem space. The Kubernetes API is actually a set of APIs, called "API groups", and each API group is independently versioned. API versions fall into 3 main tracks, each of which has different policies for deprecation:

A given release of Kubernetes can support any number of API groups and any number of versions of each.

The following rules govern the deprecation of elements of the API. This includes:

• REST resources (aka API objects)
• Fields of REST resources
• Annotations on REST resources, including "beta" annotations but not including "alpha" annotations.
• Enumerated or constant values
• Component config structures

These rules are enforced between official releases, not between arbitrary commits to master or release branches.

Rule #1: API elements may only be removed by incrementing the version of the API group.

Once an API element has been added to an API group at a particular version, it can not be removed from that version or have its behavior significantly changed, regardless of track.

Rule #2: API objects must be able to round-trip between API versions in a given release without information loss, with the exception of whole REST resources that do not exist in some versions.

For example, an object can be written as v1 and then read back as v2 and converted to v1, and the resulting v1 resource will be identical to the original. The representation in v2 might be different from v1, but the system knows how to convert between them in both directions. Additionally, any new field added in v2 must be able to round-trip to v1 and back, which means v1 might have to add an equivalent field or represent it as an annotation.

Rule #3: An API version in a given track may not be deprecated in favor of a less stable API version.

• GA API versions can replace beta and alpha API versions.
• Beta API versions can replace earlier beta and alpha API versions, but may not replace GA API versions.
• Alpha API versions can replace earlier alpha API versions, but may not replace GA or beta API versions.

Rule #4a: API lifetime is determined by the API stability level

• GA API versions may be marked as deprecated, but must not be removed within a major version of Kubernetes
• Beta API versions are deprecated no more than 9 months or 3 minor releases after introduction (whichever is longer), and are no longer served 9 months or 3 minor releases after deprecation (whichever is longer)
• Alpha API versions may be removed in any release without prior deprecation notice

This ensures beta API support covers the maximum supported version skew of 2 releases, and that APIs don't stagnate on unstable beta versions, accumulating production usage that will be disrupted when support for the beta API ends.

Rule #4b: The "preferred" API version and the "storage version" for a given group may not advance until after a release has been made that supports both the new version and the previous version

Users must be able to upgrade to a new release of Kubernetes and then roll back to a previous release, without converting anything to the new API version or suffering breakages (unless they explicitly used features only available in the newer version). This is particularly evident in the stored representation of objects.

All of this is best illustrated by examples. Imagine a Kubernetes release, version X, which introduces a new API group. A new Kubernetes release is made every approximately 4 months (3 per year). The following table describes which API versions are supported in a series of subsequent releases.

REST resources (aka API objects)

Consider a hypothetical REST resource named Widget, which was present in API v1 in the above timeline, and which needs to be deprecated. We document and announce the deprecation in sync with release X+1. The Widget resource still exists in API version v1 (deprecated) but not in v2alpha1. The Widget resource continues to exist and function in releases up to and including X+8. Only in release X+9, when API v1 has aged out, does the Widget resource cease to exist, and the behavior get removed.

Starting in Kubernetes v1.19, making an API request to a deprecated REST API endpoint:

1. Returns a Warning header (as defined in RFC7234, Section 5.5) in the API response.

2. Adds a "k8s.io/deprecated":"true" annotation to the audit event recorded for the request.

3. Sets an apiserver_requested_deprecated_apis gauge metric to 1 in the kube-apiserver process. The metric has labels for group, version, resource, subresource that can be joined to the apiserver_request_total metric, and a removed_release label that indicates the Kubernetes release in which the API will no longer be served. The following Prometheus query returns information about requests made to deprecated APIs which will be removed in v1.22:

   apiserver_requested_deprecated_apis{removed_release="1.22"} * on(group,version,resource,subresource) group_right() apiserver_request_total
   

Fields of REST resources

As with whole REST resources, an individual field which was present in API v1 must exist and function until API v1 is removed. Unlike whole resources, the v2 APIs may choose a different representation for the field, as long as it can be round-tripped. For example a v1 field named "magnitude" which was deprecated might be named "deprecatedMagnitude" in API v2. When v1 is eventually removed, the deprecated field can be removed from v2.

Enumerated or constant values

As with whole REST resources and fields thereof, a constant value which was supported in API v1 must exist and function until API v1 is removed.

Component config structures

Component configs are versioned and managed similar to REST resources.

Future work

Over time, Kubernetes will introduce more fine-grained API versions, at which point these rules will be adjusted as needed.

Deprecating a flag or CLI

The Kubernetes system is comprised of several different programs cooperating. Sometimes, a Kubernetes release might remove flags or CLI commands (collectively "CLI elements") in these programs. The individual programs naturally sort into two main groups - user-facing and admin-facing programs, which vary slightly in their deprecation policies. Unless a flag is explicitly prefixed or documented as "alpha" or "beta", it is considered GA.

CLI elements are effectively part of the API to the system, but since they are not versioned in the same way as the REST API, the rules for deprecation are as follows:

Rule #5a: CLI elements of user-facing components (e.g. kubectl) must function after their announced deprecation for no less than:

• GA: 12 months or 2 releases (whichever is longer)
• Beta: 3 months or 1 release (whichever is longer)
• Alpha: 0 releases

Rule #5b: CLI elements of admin-facing components (e.g. kubelet) must function after their announced deprecation for no less than:

• GA: 6 months or 1 release (whichever is longer)
• Beta: 3 months or 1 release (whichever is longer)
• Alpha: 0 releases

Rule #5c: Command line interface (CLI) elements cannot be deprecated in favor of less stable CLI elements

Similar to the Rule #3 for APIs, if an element of a command line interface is being replaced with an alternative implementation, such as by renaming an existing element, or by switching to use configuration sourced from a file instead of a command line argument, that recommended alternative must be of the same or higher stability level.

Rule #6: Deprecated CLI elements must emit warnings (optionally disable) when used.

Deprecating a feature or behavior

Occasionally a Kubernetes release needs to deprecate some feature or behavior of the system that is not controlled by the API or CLI. In this case, the rules for deprecation are as follows:

Rule #7: Deprecated behaviors must function for no less than 1 year after their announced deprecation.

If the feature or behavior is being replaced with an alternative implementation that requires work to adopt the change, there should be an effort to simplify the transition whenever possible. If an alternative implementation is under Kubernetes organization control, the following rules apply:

Rule #8: The feature of behavior must not be deprecated in favor of an alternative implementation that is less stable

For example, a generally available feature cannot be deprecated in favor of a Beta replacement. The Kubernetes project does, however, encourage users to adopt and transitions to alternative implementations even before they reach the same maturity level. This is particularly important for exploring new use cases of a feature or getting an early feedback on the replacement.

Alternative implementations may sometimes be external tools or products, for example a feature may move from the kubelet to container runtime that is not under Kubernetes project control. In such cases, the rule cannot be applied, but there must be an effort to ensure that there is a transition path that does not compromise on components' maturity levels. In the example with container runtimes, the effort may involve trying to ensure that popular container runtimes have versions that offer the same level of stability while implementing that replacement behavior.

Deprecation rules for features and behaviors do not imply that all changes to the system are governed by this policy. These rules apply only to significant, user-visible behaviors which impact the correctness of applications running on Kubernetes or that impact the administration of Kubernetes clusters, and which are being removed entirely.

An exception to the above rule is feature gates. Feature gates are key=value pairs that allow for users to enable/disable experimental features.

Feature gates are intended to cover the development life cycle of a feature - they are not intended to be long-term APIs. As such, they are expected to be deprecated and removed after a feature becomes GA or is dropped.

As a feature moves through the stages, the associated feature gate evolves. The feature life cycle matched to its corresponding feature gate is:

• Alpha: the feature gate is disabled by default and can be enabled by the user.
• Beta: the feature gate is enabled by default and can be disabled by the user.
• GA: the feature gate is deprecated (see "Deprecation") and becomes non-operational.
• GA, deprecation window complete: the feature gate is removed and calls to it are no longer accepted.

Deprecation

Features can be removed at any point in the life cycle prior to GA. When features are removed prior to GA, their associated feature gates are also deprecated.

When an invocation tries to disable a non-operational feature gate, the call fails in order to avoid unsupported scenarios that might otherwise run silently.

In some cases, removing pre-GA features requires considerable time. Feature gates can remain operational until their associated feature is fully removed, at which point the feature gate itself can be deprecated.

When removing a feature gate for a GA feature also requires considerable time, calls to feature gates may remain operational if the feature gate has no effect on the feature, and if the feature gate causes no errors.

Features intended to be disabled by users should include a mechanism for disabling the feature in the associated feature gate.

Versioning for feature gates is different from the previously discussed components, therefore the rules for deprecation are as follows:

Rule #9: Feature gates must be deprecated when the corresponding feature they control transitions a lifecycle stage as follows. Feature gates must function for no less than:

• Beta feature to GA: 6 months or 2 releases (whichever is longer)
• Beta feature to EOL: 3 months or 1 release (whichever is longer)
• Alpha feature to EOL: 0 releases

Rule #10: Deprecated feature gates must respond with a warning when used. When a feature gate is deprecated it must be documented in both in the release notes and the corresponding CLI help. Both warnings and documentation must indicate whether a feature gate is non-operational.

Deprecating a metric

Each component of the Kubernetes control-plane exposes metrics (usually the /metrics endpoint), which are typically ingested by cluster administrators. Not all metrics are the same: some metrics are commonly used as SLIs or used to determine SLOs, these tend to have greater import. Other metrics are more experimental in nature or are used primarily in the Kubernetes development process.

Accordingly, metrics fall under three stability classes (ALPHA, BETA STABLE); this impacts removal of a metric during a Kubernetes release. These classes are determined by the perceived importance of the metric. The rules for deprecating and removing a metric are as follows:

Rule #11a: Metrics, for the corresponding stability class, must function for no less than:

• STABLE: 4 releases or 12 months (whichever is longer)
• BETA: 2 releases or 8 months (whichever is longer)
• ALPHA: 0 releases

Rule #11b: Metrics, after their announced deprecation, must function for no less than:

• STABLE: 3 releases or 9 months (whichever is longer)
• BETA: 1 releases or 4 months (whichever is longer)
• ALPHA: 0 releases

Deprecated metrics will have their description text prefixed with a deprecation notice string '(Deprecated from x.y)' and a warning log will be emitted during metric registration. Like their stable undeprecated counterparts, deprecated metrics will be automatically registered to the metrics endpoint and therefore visible.

On a subsequent release (when the metric's deprecatedVersion is equal to current_kubernetes_version - 3), a deprecated metric will become a hidden metric. Unlike their deprecated counterparts, hidden metrics will no longer be automatically registered to the metrics endpoint (hence hidden). However, they can be explicitly enabled through a command line flag on the binary (--show-hidden-metrics-for-version=). This provides cluster admins an escape hatch to properly migrate off of a deprecated metric, if they were not able to react to the earlier deprecation warnings. Hidden metrics should be deleted after one release.

Exceptions

No policy can cover every possible situation. This policy is a living document, and will evolve over time. In practice, there will be situations that do not fit neatly into this policy, or for which this policy becomes a serious impediment. Such situations should be discussed with SIGs and project leaders to find the best solutions for those specific cases, always bearing in mind that Kubernetes is committed to being a stable system that, as much as possible, never breaks users. Exceptions will always be announced in all relevant release notes.

2.7 - Deprecated API Migration Guide

As the Kubernetes API evolves, APIs are periodically reorganized or upgraded. When APIs evolve, the old API is deprecated and eventually removed. This page contains information you need to know when migrating from deprecated API versions to newer and more stable API versions.

Removed APIs by release

v1.32

The v1.32 release stopped serving the following deprecated API versions:

Flow control resources

The flowcontrol.apiserver.k8s.io/v1beta3 API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.32.

• Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1 API version, available since v1.29.
• All existing persisted objects are accessible via the new API
• Notable changes in flowcontrol.apiserver.k8s.io/v1:
  • The PriorityLevelConfiguration spec.limited.nominalConcurrencyShares field only defaults to 30 when unspecified, and an explicit value of 0 is not changed to 30.

v1.29

The v1.29 release stopped serving the following deprecated API versions:

Flow control resources

The flowcontrol.apiserver.k8s.io/v1beta2 API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.29.

• Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1 API version, available since v1.29, or the flowcontrol.apiserver.k8s.io/v1beta3 API version, available since v1.26.
• All existing persisted objects are accessible via the new API
• Notable changes in flowcontrol.apiserver.k8s.io/v1:
  • The PriorityLevelConfiguration spec.limited.assuredConcurrencyShares field is renamed to spec.limited.nominalConcurrencyShares and only defaults to 30 when unspecified, and an explicit value of 0 is not changed to 30.
• Notable changes in flowcontrol.apiserver.k8s.io/v1beta3:
  • The PriorityLevelConfiguration spec.limited.assuredConcurrencyShares field is renamed to spec.limited.nominalConcurrencyShares

v1.27

The v1.27 release stopped serving the following deprecated API versions:

CSIStorageCapacity

The storage.k8s.io/v1beta1 API version of CSIStorageCapacity is no longer served as of v1.27.

• Migrate manifests and API clients to use the storage.k8s.io/v1 API version, available since v1.24.
• All existing persisted objects are accessible via the new API
• No notable changes

v1.26

The v1.26 release stopped serving the following deprecated API versions:

Flow control resources

The flowcontrol.apiserver.k8s.io/v1beta1 API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.26.

• Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1beta2 API version.
• All existing persisted objects are accessible via the new API
• No notable changes

HorizontalPodAutoscaler

The autoscaling/v2beta2 API version of HorizontalPodAutoscaler is no longer served as of v1.26.

• Migrate manifests and API clients to use the autoscaling/v2 API version, available since v1.23.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • targetAverageUtilization is replaced with target.averageUtilization and target.type: Utilization. See Autoscaling on multiple metrics and custom metrics.

v1.25

The v1.25 release stopped serving the following deprecated API versions:

CronJob

The batch/v1beta1 API version of CronJob is no longer served as of v1.25.

• Migrate manifests and API clients to use the batch/v1 API version, available since v1.21.
• All existing persisted objects are accessible via the new API
• No notable changes

EndpointSlice

The discovery.k8s.io/v1beta1 API version of EndpointSlice is no longer served as of v1.25.

• Migrate manifests and API clients to use the discovery.k8s.io/v1 API version, available since v1.21.
• All existing persisted objects are accessible via the new API
• Notable changes in discovery.k8s.io/v1:
  • use per Endpoint nodeName field instead of deprecated topology["kubernetes.io/hostname"] field
  • use per Endpoint zone field instead of deprecated topology["topology.kubernetes.io/zone"] field
  • topology is replaced with the deprecatedTopology field which is not writable in v1

Event

The events.k8s.io/v1beta1 API version of Event is no longer served as of v1.25.

• Migrate manifests and API clients to use the events.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• Notable changes in events.k8s.io/v1:
  • type is limited to Normal and Warning
  • involvedObject is renamed to regarding
  • action, reason, reportingController, and reportingInstance are required when creating new events.k8s.io/v1 Events
  • use eventTime instead of the deprecated firstTimestamp field (which is renamed to deprecatedFirstTimestamp and not permitted in new events.k8s.io/v1 Events)
  • use series.lastObservedTime instead of the deprecated lastTimestamp field (which is renamed to deprecatedLastTimestamp and not permitted in new events.k8s.io/v1 Events)
  • use series.count instead of the deprecated count field (which is renamed to deprecatedCount and not permitted in new events.k8s.io/v1 Events)
  • use reportingController instead of the deprecated source.component field (which is renamed to deprecatedSource.component and not permitted in new events.k8s.io/v1 Events)
  • use reportingInstance instead of the deprecated source.host field (which is renamed to deprecatedSource.host and not permitted in new events.k8s.io/v1 Events)

HorizontalPodAutoscaler

The autoscaling/v2beta1 API version of HorizontalPodAutoscaler is no longer served as of v1.25.

• Migrate manifests and API clients to use the autoscaling/v2 API version, available since v1.23.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • targetAverageUtilization is replaced with target.averageUtilization and target.type: Utilization. See Autoscaling on multiple metrics and custom metrics.

PodDisruptionBudget

The policy/v1beta1 API version of PodDisruptionBudget is no longer served as of v1.25.

• Migrate manifests and API clients to use the policy/v1 API version, available since v1.21.
• All existing persisted objects are accessible via the new API
• Notable changes in policy/v1:
  • an empty spec.selector ({}) written to a policy/v1 PodDisruptionBudget selects all pods in the namespace (in policy/v1beta1 an empty spec.selector selected no pods). An unset spec.selector selects no pods in either API version.

PodSecurityPolicy

PodSecurityPolicy in the policy/v1beta1 API version is no longer served as of v1.25, and the PodSecurityPolicy admission controller will be removed.

Migrate to Pod Security Admission or a 3rd party admission webhook. For a migration guide, see Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller. For more information on the deprecation, see PodSecurityPolicy Deprecation: Past, Present, and Future.

RuntimeClass

RuntimeClass in the node.k8s.io/v1beta1 API version is no longer served as of v1.25.

• Migrate manifests and API clients to use the node.k8s.io/v1 API version, available since v1.20.
• All existing persisted objects are accessible via the new API
• No notable changes

v1.22

The v1.22 release stopped serving the following deprecated API versions:

Webhook resources

The admissionregistration.k8s.io/v1beta1 API version of MutatingWebhookConfiguration and ValidatingWebhookConfiguration is no longer served as of v1.22.

• Migrate manifests and API clients to use the admissionregistration.k8s.io/v1 API version, available since v1.16.
• All existing persisted objects are accessible via the new APIs
• Notable changes:
  • webhooks[*].failurePolicy default changed from Ignore to Fail for v1
  • webhooks[*].matchPolicy default changed from Exact to Equivalent for v1
  • webhooks[*].timeoutSeconds default changed from 30s to 10s for v1
  • webhooks[*].sideEffects default value is removed, and the field made required, and only None and NoneOnDryRun are permitted for v1
  • webhooks[*].admissionReviewVersions default value is removed and the field made required for v1 (supported versions for AdmissionReview are v1 and v1beta1)
  • webhooks[*].name must be unique in the list for objects created via admissionregistration.k8s.io/v1

CustomResourceDefinition

The apiextensions.k8s.io/v1beta1 API version of CustomResourceDefinition is no longer served as of v1.22.

• Migrate manifests and API clients to use the apiextensions.k8s.io/v1 API version, available since v1.16.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.scope is no longer defaulted to Namespaced and must be explicitly specified
  • spec.version is removed in v1; use spec.versions instead
  • spec.validation is removed in v1; use spec.versions[*].schema instead
  • spec.subresources is removed in v1; use spec.versions[*].subresources instead
  • spec.additionalPrinterColumns is removed in v1; use spec.versions[*].additionalPrinterColumns instead
  • spec.conversion.webhookClientConfig is moved to spec.conversion.webhook.clientConfig in v1
  • spec.conversion.conversionReviewVersions is moved to spec.conversion.webhook.conversionReviewVersions in v1
  • spec.versions[*].schema.openAPIV3Schema is now required when creating v1 CustomResourceDefinition objects, and must be a structural schema
  • spec.preserveUnknownFields: true is disallowed when creating v1 CustomResourceDefinition objects; it must be specified within schema definitions as x-kubernetes-preserve-unknown-fields: true
  • In additionalPrinterColumns items, the JSONPath field was renamed to jsonPath in v1 (fixes #66531)

APIService

The apiregistration.k8s.io/v1beta1 API version of APIService is no longer served as of v1.22.

• Migrate manifests and API clients to use the apiregistration.k8s.io/v1 API version, available since v1.10.
• All existing persisted objects are accessible via the new API
• No notable changes

TokenReview

The authentication.k8s.io/v1beta1 API version of TokenReview is no longer served as of v1.22.

• Migrate manifests and API clients to use the authentication.k8s.io/v1 API version, available since v1.6.
• No notable changes

SubjectAccessReview resources

The authorization.k8s.io/v1beta1 API version of LocalSubjectAccessReview, SelfSubjectAccessReview, SubjectAccessReview, and SelfSubjectRulesReview is no longer served as of v1.22.

• Migrate manifests and API clients to use the authorization.k8s.io/v1 API version, available since v1.6.
• Notable changes:
  • spec.group was renamed to spec.groups in v1 (fixes #32709)

CertificateSigningRequest

The certificates.k8s.io/v1beta1 API version of CertificateSigningRequest is no longer served as of v1.22.

• Migrate manifests and API clients to use the certificates.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• Notable changes in certificates.k8s.io/v1:
  • For API clients requesting certificates:
    • spec.signerName is now required (see known Kubernetes signers), and requests for kubernetes.io/legacy-unknown are not allowed to be created via the certificates.k8s.io/v1 API
    • spec.usages is now required, may not contain duplicate values, and must only contain known usages
  • For API clients approving or signing certificates:
    • status.conditions may not contain duplicate types
    • status.conditions[*].status is now required
    • status.certificate must be PEM-encoded, and contain only CERTIFICATE blocks

Lease

The coordination.k8s.io/v1beta1 API version of Lease is no longer served as of v1.22.

• Migrate manifests and API clients to use the coordination.k8s.io/v1 API version, available since v1.14.
• All existing persisted objects are accessible via the new API
• No notable changes

Ingress

The extensions/v1beta1 and networking.k8s.io/v1beta1 API versions of Ingress is no longer served as of v1.22.

• Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.backend is renamed to spec.defaultBackend
  • The backend serviceName field is renamed to service.name
  • Numeric backend servicePort fields are renamed to service.port.number
  • String backend servicePort fields are renamed to service.port.name
  • pathType is now required for each specified path. Options are Prefix, Exact, and ImplementationSpecific. To match the undefined v1beta1 behavior, use ImplementationSpecific.

IngressClass

The networking.k8s.io/v1beta1 API version of IngressClass is no longer served as of v1.22.

• Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• No notable changes

RBAC resources

The rbac.authorization.k8s.io/v1beta1 API version of ClusterRole, ClusterRoleBinding, Role, and RoleBinding is no longer served as of v1.22.

• Migrate manifests and API clients to use the rbac.authorization.k8s.io/v1 API version, available since v1.8.
• All existing persisted objects are accessible via the new APIs
• No notable changes

PriorityClass

The scheduling.k8s.io/v1beta1 API version of PriorityClass is no longer served as of v1.22.

• Migrate manifests and API clients to use the scheduling.k8s.io/v1 API version, available since v1.14.
• All existing persisted objects are accessible via the new API
• No notable changes

Storage resources

The storage.k8s.io/v1beta1 API version of CSIDriver, CSINode, StorageClass, and VolumeAttachment is no longer served as of v1.22.

• Migrate manifests and API clients to use the storage.k8s.io/v1 API version
  • CSIDriver is available in storage.k8s.io/v1 since v1.19.
  • CSINode is available in storage.k8s.io/v1 since v1.17
  • StorageClass is available in storage.k8s.io/v1 since v1.6
  • VolumeAttachment is available in storage.k8s.io/v1 v1.13
• All existing persisted objects are accessible via the new APIs
• No notable changes

v1.16

The v1.16 release stopped serving the following deprecated API versions:

NetworkPolicy

The extensions/v1beta1 API version of NetworkPolicy is no longer served as of v1.16.

• Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.8.
• All existing persisted objects are accessible via the new API

DaemonSet

The extensions/v1beta1 and apps/v1beta2 API versions of DaemonSet are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.templateGeneration is removed
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  • spec.updateStrategy.type now defaults to RollingUpdate (the default in extensions/v1beta1 was OnDelete)

Deployment

The extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions of Deployment are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.rollbackTo is removed
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  • spec.progressDeadlineSeconds now defaults to 600 seconds (the default in extensions/v1beta1 was no deadline)
  • spec.revisionHistoryLimit now defaults to 10 (the default in apps/v1beta1 was 2, the default in extensions/v1beta1 was to retain all)
  • maxSurge and maxUnavailable now default to 25% (the default in extensions/v1beta1 was 1)

StatefulSet

The apps/v1beta1 and apps/v1beta2 API versions of StatefulSet are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  • spec.updateStrategy.type now defaults to RollingUpdate (the default in apps/v1beta1 was OnDelete)

ReplicaSet

The extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions of ReplicaSet are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades

PodSecurityPolicy

The extensions/v1beta1 API version of PodSecurityPolicy is no longer served as of v1.16.

• Migrate manifests and API client to use the policy/v1beta1 API version, available since v1.10.
• Note that the policy/v1beta1 API version of PodSecurityPolicy will be removed in v1.25.

What to do

Test with deprecated APIs disabled

You can test your clusters by starting an API server with specific API versions disabled to simulate upcoming removals. Add the following flag to the API server startup arguments:

--runtime-config=<group>/<version>=false

For example:

--runtime-config=admissionregistration.k8s.io/v1beta1=false,apiextensions.k8s.io/v1beta1,...

Locate use of deprecated APIs

Use client warnings, metrics, and audit information available in 1.19+ to locate use of deprecated APIs.

Migrate to non-deprecated APIs

• Update custom integrations and controllers to call the non-deprecated APIs

• Change YAML files to reference the non-deprecated APIs

  You can use the kubectl convert command to automatically convert an existing object:

  kubectl convert -f <file> --output-version <group>/<version>.

  For example, to convert an older Deployment to apps/v1, you can run:

  kubectl convert -f ./my-deployment.yaml --output-version apps/v1

  This conversion may use non-ideal default values. To learn more about a specific resource, check the Kubernetes API reference.

  Note:

  The kubectl convert tool is not installed by default, although in fact it once was part of kubectl itself. For more details, you can read the deprecation and removal issue for the built-in subcommand.

  To learn how to set up kubectl convert on your computer, visit the page that is right for your operating system: Linux, macOS, or Windows.

2.8 - Kubernetes API health endpoints

The Kubernetes API server provides API endpoints to indicate the current status of the API server. This page describes these API endpoints and explains how you can use them.

API endpoints for health

The Kubernetes API server provides 3 API endpoints (healthz, livez and readyz) to indicate the current status of the API server. The healthz endpoint is deprecated (since Kubernetes v1.16), and you should use the more specific livez and readyz endpoints instead. The livez endpoint can be used with the --livez-grace-period flag to specify the startup duration. For a graceful shutdown you can specify the --shutdown-delay-duration flag with the /readyz endpoint. Machines that check the healthz/livez/readyz of the API server should rely on the HTTP status code. A status code 200 indicates the API server is healthy/live/ready, depending on the called endpoint. The more verbose options shown below are intended to be used by human operators to debug their cluster or understand the state of the API server.

The following examples will show how you can interact with the health API endpoints.

For all endpoints, you can use the verbose parameter to print out the checks and their status. This can be useful for a human operator to debug the current status of the API server, it is not intended to be consumed by a machine:

curl -k https://localhost:6443/livez?verbose

or from a remote host with authentication:

kubectl get --raw='/readyz?verbose'

The output will look like this:

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
healthz check passed

The Kubernetes API server also supports to exclude specific checks. The query parameters can also be combined like in this example:

curl -k 'https://localhost:6443/readyz?verbose&exclude=etcd'

The output show that the etcd check is excluded:

[+]ping ok
[+]log ok
[+]etcd excluded: ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]shutdown ok
healthz check passed

Individual health checks

Each individual health check exposes an HTTP endpoint and can be checked individually. The schema for the individual health checks is /livez/<healthcheck-name> or /readyz/<healthcheck-name>, where livez and readyz can be used to indicate if you want to check the liveness or the readiness of the API server, respectively. The <healthcheck-name> path can be discovered using the verbose flag from above and take the path between [+] and ok. These individual health checks should not be consumed by machines but can be helpful for a human operator to debug a system:

curl -k https://localhost:6443/livez/etcd

3 - API Access Control

For an introduction to how Kubernetes implements and controls API access, read Controlling Access to the Kubernetes API.

Reference documentation:

• Authenticating
  • Authenticating with Bootstrap Tokens
• Admission Controllers
  • Dynamic Admission Control
• Authorization
  • Role Based Access Control
  • Attribute Based Access Control
  • Node Authorization
  • Webhook Authorization
• Certificate Signing Requests
  • including CSR approval and certificate signing
• Service accounts
  • Developer guide
  • Administration
• Kubelet Authentication & Authorization
  • including kubelet TLS bootstrapping

3.1 - Authenticating

This page provides an overview of authentication.

Users in Kubernetes

All Kubernetes clusters have two categories of users: service accounts managed by Kubernetes, and normal users.

It is assumed that a cluster-independent service manages normal users in the following ways:

• an administrator distributing private keys
• a user store like Keystone or Google Accounts
• a file with a list of usernames and passwords

In this regard, Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call.

Even though a normal user cannot be added via an API call, any user that presents a valid certificate signed by the cluster's certificate authority (CA) is considered authenticated. In this configuration, Kubernetes determines the username from the common name field in the 'subject' of the cert (e.g., "/CN=bob"). From there, the role based access control (RBAC) sub-system would determine whether the user is authorized to perform a specific operation on a resource.

In contrast, service accounts are users managed by the Kubernetes API. They are bound to specific namespaces, and created automatically by the API server or manually through API calls. Service accounts are tied to a set of credentials stored as Secrets, which are mounted into pods allowing in-cluster processes to talk to the Kubernetes API.

API requests are tied to either a normal user or a service account, or are treated as anonymous requests. This means every process inside or outside the cluster, from a human user typing kubectl on a workstation, to kubelets on nodes, to members of the control plane, must authenticate when making requests to the API server, or be treated as an anonymous user.

Authentication strategies

Kubernetes uses client certificates, bearer tokens, or an authenticating proxy to authenticate API requests through authentication plugins. As HTTP requests are made to the API server, plugins attempt to associate the following attributes with the request:

• Username: a string which identifies the end user. Common values might be kube-admin or jane@example.com.
• UID: a string which identifies the end user and attempts to be more consistent and unique than username.
• Groups: a set of strings, each of which indicates the user's membership in a named logical collection of users. Common values might be system:masters or devops-team.
• Extra fields: a map of strings to list of strings which holds additional information authorizers may find useful.

All values are opaque to the authentication system and only hold significance when interpreted by an authorizer.

You can enable multiple authentication methods at once. You should usually use at least two methods:

• service account tokens for service accounts
• at least one other method for user authentication.

When multiple authenticator modules are enabled, the first module to successfully authenticate the request short-circuits evaluation. The API server does not guarantee the order authenticators run in.

The system:authenticated group is included in the list of groups for all authenticated users.

Integrations with other authentication protocols (LDAP, SAML, Kerberos, alternate x509 schemes, etc) can be accomplished using an authenticating proxy or the authentication webhook.

X509 client certificates

Client certificate authentication is enabled by passing the --client-ca-file=SOMEFILE option to API server. The referenced file must contain one or more certificate authorities to use to validate client certificates presented to the API server. If a client certificate is presented and verified, the common name of the subject is used as the user name for the request. As of Kubernetes 1.4, client certificates can also indicate a user's group memberships using the certificate's organization fields. To include multiple group memberships for a user, include multiple organization fields in the certificate.

For example, using the openssl command line tool to generate a certificate signing request:

openssl req -new -key jbeda.pem -out jbeda-csr.pem -subj "/CN=jbeda/O=app1/O=app2"

This would create a CSR for the username "jbeda", belonging to two groups, "app1" and "app2".

See Managing Certificates for how to generate a client cert.

Static token file

The API server reads bearer tokens from a file when given the --token-auth-file=SOMEFILE option on the command line. Currently, tokens last indefinitely, and the token list cannot be changed without restarting the API server.

The token file is a csv file with a minimum of 3 columns: token, user name, user uid, followed by optional group names.

token,user,uid,"group1,group2,group3"

Putting a bearer token in a request

When using bearer token authentication from an http client, the API server expects an Authorization header with a value of Bearer <token>. The bearer token must be a character sequence that can be put in an HTTP header value using no more than the encoding and quoting facilities of HTTP. For example: if the bearer token is 31ada4fd-adec-460c-809a-9e56ceb75269 then it would appear in an HTTP header as shown below.

Authorization: Bearer 31ada4fd-adec-460c-809a-9e56ceb75269

Bootstrap tokens

To allow for streamlined bootstrapping for new clusters, Kubernetes includes a dynamically-managed Bearer token type called a Bootstrap Token. These tokens are stored as Secrets in the kube-system namespace, where they can be dynamically managed and created. Controller Manager contains a TokenCleaner controller that deletes bootstrap tokens as they expire.

The tokens are of the form [a-z0-9]{6}.[a-z0-9]{16}. The first component is a Token ID and the second component is the Token Secret. You specify the token in an HTTP header as follows:

Authorization: Bearer 781292.db7bc3a58fc5f07e

You must enable the Bootstrap Token Authenticator with the --enable-bootstrap-token-auth flag on the API Server. You must enable the TokenCleaner controller via the --controllers flag on the Controller Manager. This is done with something like --controllers=*,tokencleaner. kubeadm will do this for you if you are using it to bootstrap a cluster.

The authenticator authenticates as system:bootstrap:<Token ID>. It is included in the system:bootstrappers group. The naming and groups are intentionally limited to discourage users from using these tokens past bootstrapping. The user names and group can be used (and are used by kubeadm) to craft the appropriate authorization policies to support bootstrapping a cluster.

Please see Bootstrap Tokens for in depth documentation on the Bootstrap Token authenticator and controllers along with how to manage these tokens with kubeadm.

Service account tokens

A service account is an automatically enabled authenticator that uses signed bearer tokens to verify requests. The plugin takes two optional flags:

• --service-account-key-file File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used.
• --service-account-lookup If enabled, tokens which are deleted from the API will be revoked.

Service accounts are usually created automatically by the API server and associated with pods running in the cluster through the ServiceAccount Admission Controller. Bearer tokens are mounted into pods at well-known locations, and allow in-cluster processes to talk to the API server. Accounts may be explicitly associated with pods using the serviceAccountName field of a PodSpec.

apiVersion: apps/v1 # this apiVersion is relevant as of Kubernetes 1.9
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 3
  template:
    metadata:
    # ...
    spec:
      serviceAccountName: bob-the-bot
      containers:
      - name: nginx
        image: nginx:1.14.2

Service account bearer tokens are perfectly valid to use outside the cluster and can be used to create identities for long standing jobs that wish to talk to the Kubernetes API. To manually create a service account, use the kubectl create serviceaccount (NAME) command. This creates a service account in the current namespace.

kubectl create serviceaccount jenkins

serviceaccount/jenkins created

Create an associated token:

kubectl create token jenkins

eyJhbGciOiJSUzI1NiIsImtp...

The created token is a signed JSON Web Token (JWT).

The signed JWT can be used as a bearer token to authenticate as the given service account. See above for how the token is included in a request. Normally these tokens are mounted into pods for in-cluster access to the API server, but can be used from outside the cluster as well.

Service accounts authenticate with the username system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT), and are assigned to the groups system:serviceaccounts and system:serviceaccounts:(NAMESPACE).

OpenID Connect Tokens

OpenID Connect is a flavor of OAuth2 supported by some OAuth2 providers, notably Microsoft Entra ID, Salesforce, and Google. The protocol's main extension of OAuth2 is an additional field returned with the access token called an ID Token. This token is a JSON Web Token (JWT) with well known fields, such as a user's email, signed by the server.

To identify the user, the authenticator uses the id_token (not the access_token) from the OAuth2 token response as a bearer token. See above for how the token is included in a request.

1. Log in to your identity provider

2. Your identity provider will provide you with an access_token, id_token and a refresh_token

3. When using kubectl, use your id_token with the --token flag or add it directly to your kubeconfig

4. kubectl sends your id_token in a header called Authorization to the API server

5. The API server will make sure the JWT signature is valid

6. Check to make sure the id_token hasn't expired

   Perform claim and/or user validation if CEL expressions are configured with AuthenticationConfiguration.

7. Make sure the user is authorized

8. Once authorized the API server returns a response to kubectl

9. kubectl provides feedback to the user

Since all of the data needed to validate who you are is in the id_token, Kubernetes doesn't need to "phone home" to the identity provider. In a model where every request is stateless this provides a very scalable solution for authentication. It does offer a few challenges:

1. Kubernetes has no "web interface" to trigger the authentication process. There is no browser or interface to collect credentials which is why you need to authenticate to your identity provider first.
2. The id_token can't be revoked, it's like a certificate so it should be short-lived (only a few minutes) so it can be very annoying to have to get a new token every few minutes.
3. To authenticate to the Kubernetes dashboard, you must use the kubectl proxy command or a reverse proxy that injects the id_token.

Configuring the API Server

Using flags

To enable the plugin, configure the following flags on the API server:

Authentication configuration from a file

JWT Authenticator is an authenticator to authenticate Kubernetes users using JWT compliant tokens. The authenticator will attempt to parse a raw ID token, verify it's been signed by the configured issuer. The public key to verify the signature is discovered from the issuer's public endpoint using OIDC discovery.

The minimum valid JWT payload must contain the following claims:

{
  "iss": "https://example.com",   // must match the issuer.url
  "aud": ["my-app"],              // at least one of the entries in issuer.audiences must match the "aud" claim in presented JWTs.
  "exp": 1234567890,              // token expiration as Unix time (the number of seconds elapsed since January 1, 1970 UTC)
  "<username-claim>": "user"      // this is the username claim configured in the claimMappings.username.claim or claimMappings.username.expression
}

The configuration file approach allows you to configure multiple JWT authenticators, each with a unique issuer.url and issuer.discoveryURL. The configuration file even allows you to specify CEL expressions to map claims to user attributes, and to validate claims and user information. The API server also automatically reloads the authenticators when the configuration file is modified. You can use apiserver_authentication_config_controller_automatic_reload_last_timestamp_seconds metric to monitor the last time the configuration was reloaded by the API server.

You must specify the path to the authentication configuration using the --authentication-config flag on the API server. If you want to use command line flags instead of the configuration file, those will continue to work as-is. To access the new capabilities like configuring multiple authenticators, setting multiple audiences for an issuer, switch to using the configuration file.

For Kubernetes v1.33, the structured authentication configuration file format is beta-level, and the mechanism for using that configuration is also beta. Provided you didn't specifically disable the StructuredAuthenticationConfiguration feature gate for your cluster, you can turn on structured authentication by specifying the --authentication-config command line argument to the kube-apiserver. An example of the structured authentication configuration file is shown below.

---
#
# CAUTION: this is an example configuration.
#          Do not use this for your own cluster!
#
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
# list of authenticators to authenticate Kubernetes users using JWT compliant tokens.
# the maximum number of allowed authenticators is 64.
jwt:
- issuer:
    # url must be unique across all authenticators.
    # url must not conflict with issuer configured in --service-account-issuer.
    url: https://example.com # Same as --oidc-issuer-url.
    # discoveryURL, if specified, overrides the URL used to fetch discovery
    # information instead of using "{url}/.well-known/openid-configuration".
    # The exact value specified is used, so "/.well-known/openid-configuration"
    # must be included in discoveryURL if needed.
    #
    # The "issuer" field in the fetched discovery information must match the "issuer.url" field
    # in the AuthenticationConfiguration and will be used to validate the "iss" claim in the presented JWT.
    # This is for scenarios where the well-known and jwks endpoints are hosted at a different
    # location than the issuer (such as locally in the cluster).
    # discoveryURL must be different from url if specified and must be unique across all authenticators.
    discoveryURL: https://discovery.example.com/.well-known/openid-configuration
    # PEM encoded CA certificates used to validate the connection when fetching
    # discovery information. If not set, the system verifier will be used.
    # Same value as the content of the file referenced by the --oidc-ca-file flag.
    certificateAuthority: <PEM encoded CA certificates>    
    # audiences is the set of acceptable audiences the JWT must be issued to.
    # At least one of the entries must match the "aud" claim in presented JWTs.
    audiences:
    - my-app # Same as --oidc-client-id.
    - my-other-app
    # this is required to be set to "MatchAny" when multiple audiences are specified.
    audienceMatchPolicy: MatchAny
  # rules applied to validate token claims to authenticate users.
  claimValidationRules:
    # Same as --oidc-required-claim key=value.
  - claim: hd
    requiredValue: example.com
    # Instead of claim and requiredValue, you can use expression to validate the claim.
    # expression is a CEL expression that evaluates to a boolean.
    # all the expressions must evaluate to true for validation to succeed.
  - expression: 'claims.hd == "example.com"'
    # Message customizes the error message seen in the API server logs when the validation fails.
    message: the hd claim must be set to example.com
  - expression: 'claims.exp - claims.nbf <= 86400'
    message: total token lifetime must not exceed 24 hours
  claimMappings:
    # username represents an option for the username attribute.
    # This is the only required attribute.
    username:
      # Same as --oidc-username-claim. Mutually exclusive with username.expression.
      claim: "sub"
      # Same as --oidc-username-prefix. Mutually exclusive with username.expression.
      # if username.claim is set, username.prefix is required.
      # Explicitly set it to "" if no prefix is desired.
      prefix: ""
      # Mutually exclusive with username.claim and username.prefix.
      # expression is a CEL expression that evaluates to a string.
      #
      # 1.  If username.expression uses 'claims.email', then 'claims.email_verified' must be used in
      #     username.expression or extra[*].valueExpression or claimValidationRules[*].expression.
      #     An example claim validation rule expression that matches the validation automatically
      #     applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true) == true'.
      #     By explicitly comparing the value to true, we let type-checking see the result will be a boolean, and
      #     to make sure a non-boolean email_verified claim will be caught at runtime.
      # 2.  If the username asserted based on username.expression is the empty string, the authentication
      #     request will fail.
      expression: 'claims.username + ":external-user"'
    # groups represents an option for the groups attribute.
    groups:
      # Same as --oidc-groups-claim. Mutually exclusive with groups.expression.
      claim: "sub"
      # Same as --oidc-groups-prefix. Mutually exclusive with groups.expression.
      # if groups.claim is set, groups.prefix is required.
      # Explicitly set it to "" if no prefix is desired.
      prefix: ""
      # Mutually exclusive with groups.claim and groups.prefix.
      # expression is a CEL expression that evaluates to a string or a list of strings.
      expression: 'claims.roles.split(",")'
    # uid represents an option for the uid attribute.
    uid:
      # Mutually exclusive with uid.expression.
      claim: 'sub'
      # Mutually exclusive with uid.claim
      # expression is a CEL expression that evaluates to a string.
      expression: 'claims.sub'
    # extra attributes to be added to the UserInfo object. Keys must be domain-prefix path and must be unique.
    extra:
      # key is a string to use as the extra attribute key.
      # key must be a domain-prefix path (e.g. example.org/foo). All characters before the first "/" must be a valid
      # subdomain as defined by RFC 1123. All characters trailing the first "/" must
      # be valid HTTP Path characters as defined by RFC 3986.
      # k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use and cannot be used.
      # key must be lowercase and unique across all extra attributes.
    - key: 'example.com/tenant'
      # valueExpression is a CEL expression that evaluates to a string or a list of strings.
      valueExpression: 'claims.tenant'
  # validation rules applied to the final user object.
  userValidationRules:
    # expression is a CEL expression that evaluates to a boolean.
    # all the expressions must evaluate to true for the user to be valid.
  - expression: "!user.username.startsWith('system:')"
    # Message customizes the error message seen in the API server logs when the validation fails.
    message: 'username cannot used reserved system: prefix'
  - expression: "user.groups.all(group, !group.startsWith('system:'))"
    message: 'groups cannot used reserved system: prefix'

• Claim validation rule expression

  jwt.claimValidationRules[i].expression represents the expression which will be evaluated by CEL. CEL expressions have access to the contents of the token payload, organized into claims CEL variable. claims is a map of claim names (as strings) to claim values (of any type).

• User validation rule expression

  jwt.userValidationRules[i].expression represents the expression which will be evaluated by CEL. CEL expressions have access to the contents of userInfo, organized into user CEL variable. Refer to the UserInfo API documentation for the schema of user.

• Claim mapping expression

  jwt.claimMappings.username.expression, jwt.claimMappings.groups.expression, jwt.claimMappings.uid.expression jwt.claimMappings.extra[i].valueExpression represents the expression which will be evaluated by CEL. CEL expressions have access to the contents of the token payload, organized into claims CEL variable. claims is a map of claim names (as strings) to claim values (of any type).

  To learn more, see the Documentation on CEL

  Here are examples of the AuthenticationConfiguration with different token payloads.

  • Valid token
  • Fails claim validation
  • Fails user validation

  apiVersion: apiserver.config.k8s.io/v1beta1
  kind: AuthenticationConfiguration
  jwt:
  - issuer:
      url: https://example.com
      audiences:
      - my-app
    claimMappings:
      username:
        expression: 'claims.username + ":external-user"'
      groups:
        expression: 'claims.roles.split(",")'
      uid:
        expression: 'claims.sub'
      extra:
      - key: 'example.com/tenant'
        valueExpression: 'claims.tenant'
    userValidationRules:
    - expression: "!user.username.startsWith('system:')" # the expression will evaluate to true, so validation will succeed.
      message: 'username cannot used reserved system: prefix'
  

  TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6ImY3dF9tOEROWmFTQk1oWGw5QXZTWGhBUC04Y0JmZ0JVbFVpTG5oQkgxdXMiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJrdWJlcm5ldGVzIiwiZXhwIjoxNzAzMjMyOTQ5LCJpYXQiOjE3MDExMDcyMzMsImlzcyI6Imh0dHBzOi8vZXhhbXBsZS5jb20iLCJqdGkiOiI3YzMzNzk0MjgwN2U3M2NhYTJjMzBjODY4YWMwY2U5MTBiY2UwMmRkY2JmZWJlOGMyM2I4YjVmMjdhZDYyODczIiwibmJmIjoxNzAxMTA3MjMzLCJyb2xlcyI6InVzZXIsYWRtaW4iLCJzdWIiOiJhdXRoIiwidGVuYW50IjoiNzJmOTg4YmYtODZmMS00MWFmLTkxYWItMmQ3Y2QwMTFkYjRhIiwidXNlcm5hbWUiOiJmb28ifQ.TBWF2RkQHm4QQz85AYPcwLxSk-VLvQW-mNDHx7SEOSv9LVwcPYPuPajJpuQn9C_gKq1R94QKSQ5F6UgHMILz8OfmPKmX_00wpwwNVGeevJ79ieX2V-__W56iNR5gJ-i9nn6FYk5pwfVREB0l4HSlpTOmu80gbPWAXY5hLW0ZtcE1JTEEmefORHV2ge8e3jp1xGafNy6LdJWabYuKiw8d7Qga__HxtKB-t0kRMNzLRS7rka_SfQg0dSYektuxhLbiDkqhmRffGlQKXGVzUsuvFw7IGM5ZWnZgEMDzCI357obHeM3tRqpn5WRjtB8oM7JgnCymaJi-P3iCd88iu1xnzA
  

  where the token payload is:

    {
      "aud": "kubernetes",
      "exp": 1703232949,
      "iat": 1701107233,
      "iss": "https://example.com",
      "jti": "7c337942807e73caa2c30c868ac0ce910bce02ddcbfebe8c23b8b5f27ad62873",
      "nbf": 1701107233,
      "roles": "user,admin",
      "sub": "auth",
      "tenant": "72f988bf-86f1-41af-91ab-2d7cd011db4a",
      "username": "foo"
    }
  

  The token with the above AuthenticationConfiguration will produce the following UserInfo object and successfully authenticate the user.

  {
      "username": "foo:external-user",
      "uid": "auth",
      "groups": [
          "user",
          "admin"
      ],
      "extra": {
          "example.com/tenant": "72f988bf-86f1-41af-91ab-2d7cd011db4a"
      }
  }
  

  apiVersion: apiserver.config.k8s.io/v1beta1
  kind: AuthenticationConfiguration
  jwt:
  - issuer:
      url: https://example.com
      audiences:
      - my-app
    claimValidationRules:
    - expression: 'claims.hd == "example.com"' # the token below does not have this claim, so validation will fail.
      message: the hd claim must be set to example.com
    claimMappings:
      username:
        expression: 'claims.username + ":external-user"'
      groups:
        expression: 'claims.roles.split(",")'
      uid:
        expression: 'claims.sub'
      extra:
      - key: 'example.com/tenant'
        valueExpression: 'claims.tenant'
    userValidationRules:
    - expression: "!user.username.startsWith('system:')" # the expression will evaluate to true, so validation will succeed.
      message: 'username cannot used reserved system: prefix'
  

  TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6ImY3dF9tOEROWmFTQk1oWGw5QXZTWGhBUC04Y0JmZ0JVbFVpTG5oQkgxdXMiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJrdWJlcm5ldGVzIiwiZXhwIjoxNzAzMjMyOTQ5LCJpYXQiOjE3MDExMDcyMzMsImlzcyI6Imh0dHBzOi8vZXhhbXBsZS5jb20iLCJqdGkiOiI3YzMzNzk0MjgwN2U3M2NhYTJjMzBjODY4YWMwY2U5MTBiY2UwMmRkY2JmZWJlOGMyM2I4YjVmMjdhZDYyODczIiwibmJmIjoxNzAxMTA3MjMzLCJyb2xlcyI6InVzZXIsYWRtaW4iLCJzdWIiOiJhdXRoIiwidGVuYW50IjoiNzJmOTg4YmYtODZmMS00MWFmLTkxYWItMmQ3Y2QwMTFkYjRhIiwidXNlcm5hbWUiOiJmb28ifQ.TBWF2RkQHm4QQz85AYPcwLxSk-VLvQW-mNDHx7SEOSv9LVwcPYPuPajJpuQn9C_gKq1R94QKSQ5F6UgHMILz8OfmPKmX_00wpwwNVGeevJ79ieX2V-__W56iNR5gJ-i9nn6FYk5pwfVREB0l4HSlpTOmu80gbPWAXY5hLW0ZtcE1JTEEmefORHV2ge8e3jp1xGafNy6LdJWabYuKiw8d7Qga__HxtKB-t0kRMNzLRS7rka_SfQg0dSYektuxhLbiDkqhmRffGlQKXGVzUsuvFw7IGM5ZWnZgEMDzCI357obHeM3tRqpn5WRjtB8oM7JgnCymaJi-P3iCd88iu1xnzA
  

  where the token payload is:

    {
      "aud": "kubernetes",
      "exp": 1703232949,
      "iat": 1701107233,
      "iss": "https://example.com",
      "jti": "7c337942807e73caa2c30c868ac0ce910bce02ddcbfebe8c23b8b5f27ad62873",
      "nbf": 1701107233,
      "roles": "user,admin",
      "sub": "auth",
      "tenant": "72f988bf-86f1-41af-91ab-2d7cd011db4a",
      "username": "foo"
    }
  

  The token with the above AuthenticationConfiguration will fail to authenticate because the hd claim is not set to example.com. The API server will return 401 Unauthorized error.

  apiVersion: apiserver.config.k8s.io/v1beta1
  kind: AuthenticationConfiguration
  jwt:
  - issuer:
      url: https://example.com
      audiences:
      - my-app
    claimValidationRules:
    - expression: 'claims.hd == "example.com"'
      message: the hd claim must be set to example.com
    claimMappings:
      username:
        expression: '"system:" + claims.username' # this will prefix the username with "system:" and will fail user validation.
      groups:
        expression: 'claims.roles.split(",")'
      uid:
        expression: 'claims.sub'
      extra:
      - key: 'example.com/tenant'
        valueExpression: 'claims.tenant'
    userValidationRules:
    - expression: "!user.username.startsWith('system:')" # the username will be system:foo and expression will evaluate to false, so validation will fail.
      message: 'username cannot used reserved system: prefix'
  

  TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6ImY3dF9tOEROWmFTQk1oWGw5QXZTWGhBUC04Y0JmZ0JVbFVpTG5oQkgxdXMiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJrdWJlcm5ldGVzIiwiZXhwIjoxNzAzMjMyOTQ5LCJoZCI6ImV4YW1wbGUuY29tIiwiaWF0IjoxNzAxMTEzMTAxLCJpc3MiOiJodHRwczovL2V4YW1wbGUuY29tIiwianRpIjoiYjViMDY1MjM3MmNkMjBlMzQ1YjZmZGZmY2RjMjE4MWY0YWZkNmYyNTlhYWI0YjdlMzU4ODEyMzdkMjkyMjBiYyIsIm5iZiI6MTcwMTExMzEwMSwicm9sZXMiOiJ1c2VyLGFkbWluIiwic3ViIjoiYXV0aCIsInRlbmFudCI6IjcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI0YSIsInVzZXJuYW1lIjoiZm9vIn0.FgPJBYLobo9jnbHreooBlvpgEcSPWnKfX6dc0IvdlRB-F0dCcgy91oCJeK_aBk-8zH5AKUXoFTlInfLCkPivMOJqMECA1YTrMUwt_IVqwb116AqihfByUYIIqzMjvUbthtbpIeHQm2fF0HbrUqa_Q0uaYwgy8mD807h7sBcUMjNd215ff_nFIHss-9zegH8GI1d9fiBf-g6zjkR1j987EP748khpQh9IxPjMJbSgG_uH5x80YFuqgEWwq-aYJPQxXX6FatP96a2EAn7wfPpGlPRt0HcBOvq5pCnudgCgfVgiOJiLr_7robQu4T1bis0W75VPEvwWtgFcLnvcQx0JWg
  

  where the token payload is:

    {
      "aud": "kubernetes",
      "exp": 1703232949,
      "hd": "example.com",
      "iat": 1701113101,
      "iss": "https://example.com",
      "jti": "b5b0652372cd20e345b6fdffcdc2181f4afd6f259aab4b7e35881237d29220bc",
      "nbf": 1701113101,
      "roles": "user,admin",
      "sub": "auth",
      "tenant": "72f988bf-86f1-41af-91ab-2d7cd011db4a",
      "username": "foo"
    }
  

  The token with the above AuthenticationConfiguration will produce the following UserInfo object:

  {
      "username": "system:foo",
      "uid": "auth",
      "groups": [
          "user",
          "admin"
      ],
      "extra": {
          "example.com/tenant": "72f988bf-86f1-41af-91ab-2d7cd011db4a"
      }
  }
  

  which will fail user validation because the username starts with system:. The API server will return 401 Unauthorized error.

Limitations

1. Distributed claims do not work via CEL expressions.
2. Egress selector configuration is not supported for calls to issuer.url and issuer.discoveryURL.

Kubernetes does not provide an OpenID Connect Identity Provider. You can use an existing public OpenID Connect Identity Provider (such as Google, or others). Or, you can run your own Identity Provider, such as dex, Keycloak, CloudFoundry UAA, or Tremolo Security's OpenUnison.

For an identity provider to work with Kubernetes it must:

1. Support OpenID connect discovery

   The public key to verify the signature is discovered from the issuer's public endpoint using OIDC discovery. If you're using the authentication configuration file, the identity provider doesn't need to publicly expose the discovery endpoint. You can host the discovery endpoint at a different location than the issuer (such as locally in the cluster) and specify the issuer.discoveryURL in the configuration file.

2. Run in TLS with non-obsolete ciphers

3. Have a CA signed certificate (even if the CA is not a commercial CA or is self signed)

A note about requirement #3 above, requiring a CA signed certificate. If you deploy your own identity provider (as opposed to one of the cloud providers like Google or Microsoft) you MUST have your identity provider's web server certificate signed by a certificate with the CA flag set to TRUE, even if it is self signed. This is due to GoLang's TLS client implementation being very strict to the standards around certificate validation. If you don't have a CA handy, you can use the gencert script from the Dex team to create a simple CA and a signed certificate and key pair. Or you can use this similar script that generates SHA256 certs with a longer life and larger key size.

Refer to setup instructions for specific systems:

• UAA
• Dex
• OpenUnison

Using kubectl

Option 1 - OIDC Authenticator

The first option is to use the kubectl oidc authenticator, which sets the id_token as a bearer token for all requests and refreshes the token once it expires. After you've logged into your provider, use kubectl to add your id_token, refresh_token, client_id, and client_secret to configure the plugin.

Providers that don't return an id_token as part of their refresh token response aren't supported by this plugin and should use "Option 2" below.

kubectl config set-credentials USER_NAME \
   --auth-provider=oidc \
   --auth-provider-arg=idp-issuer-url=( issuer url ) \
   --auth-provider-arg=client-id=( your client id ) \
   --auth-provider-arg=client-secret=( your client secret ) \
   --auth-provider-arg=refresh-token=( your refresh token ) \
   --auth-provider-arg=idp-certificate-authority=( path to your ca certificate ) \
   --auth-provider-arg=id-token=( your id_token )