1 - Feature Gates

This page contains an overview of the various feature gates an administrator can specify on different Kubernetes components.

See feature stages for an explanation of the stages for a feature.

Overview

Feature gates are a set of key=value pairs that describe Kubernetes features. You can turn these features on or off using the --feature-gates command line flag on each Kubernetes component.

Each Kubernetes component lets you enable or disable a set of feature gates that are relevant to that component. Use -h flag to see a full set of feature gates for all components. To set feature gates for a component, such as kubelet, use the --feature-gates flag assigned to a list of feature pairs:

--feature-gates="...,DynamicKubeletConfig=true"

The following tables are a summary of the feature gates that you can set on different Kubernetes components.

  • The "Since" column contains the Kubernetes release when a feature is introduced or its release stage is changed.
  • The "Until" column, if not empty, contains the last Kubernetes release in which you can still use a feature gate.
  • If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
  • If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
  • The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.

Feature gates for Alpha or Beta features

Feature gates for features in Alpha or Beta states
FeatureDefaultStageSinceUntil
APIListChunkingfalseAlpha1.81.8
APIListChunkingtrueBeta1.9
APIPriorityAndFairnessfalseAlpha1.171.19
APIPriorityAndFairnesstrueBeta1.20
APIResponseCompressionfalseAlpha1.71.15
APIResponseCompressionfalseBeta1.16
APIServerIdentityfalseAlpha1.20
AllowInsecureBackendProxytrueBeta1.17
AnyVolumeDataSourcefalseAlpha1.18
AppArmortrueBeta1.4
BalanceAttachedNodeVolumesfalseAlpha1.11
BoundServiceAccountTokenVolumefalseAlpha1.131.20
BoundServiceAccountTokenVolumetrueBeta1.21
CPUManagerfalseAlpha1.81.9
CPUManagertrueBeta1.10
CSIInlineVolumefalseAlpha1.151.15
CSIInlineVolumetrueBeta1.16-
CSIMigrationfalseAlpha1.141.16
CSIMigrationtrueBeta1.17
CSIMigrationAWSfalseAlpha1.14
CSIMigrationAWSfalseBeta1.17
CSIMigrationAWSCompletefalseAlpha1.17
CSIMigrationAzureDiskfalseAlpha1.151.18
CSIMigrationAzureDiskfalseBeta1.19
CSIMigrationAzureDiskCompletefalseAlpha1.17
CSIMigrationAzureFilefalseAlpha1.151.19
CSIMigrationAzureFilefalseBeta1.21
CSIMigrationAzureFileCompletefalseAlpha1.17
CSIMigrationGCEfalseAlpha1.141.16
CSIMigrationGCEfalseBeta1.17
CSIMigrationGCECompletefalseAlpha1.17
CSIMigrationOpenStackfalseAlpha1.141.17
CSIMigrationOpenStacktrueBeta1.18
CSIMigrationOpenStackCompletefalseAlpha1.17
CSIMigrationvSpherefalseBeta1.19
CSIMigrationvSphereCompletefalseBeta1.19
CSIServiceAccountTokenfalseAlpha1.201.20
CSIServiceAccountTokentrueBeta1.21
CSIStorageCapacityfalseAlpha1.191.20
CSIStorageCapacitytrueBeta1.21
CSIVolumeFSGroupPolicyfalseAlpha1.191.19
CSIVolumeFSGroupPolicytrueBeta1.20
ConfigurableFSGroupPolicyfalseAlpha1.181.19
ConfigurableFSGroupPolicytrueBeta1.20
CronJobControllerV2falseAlpha1.201.20
CronJobControllerV2trueBeta1.21
CustomCPUCFSQuotaPeriodfalseAlpha1.12
DefaultPodTopologySpreadfalseAlpha1.191.19
DefaultPodTopologySpreadtrueBeta1.20
DevicePluginsfalseAlpha1.81.9
DevicePluginstrueBeta1.10
DisableAcceleratorUsageMetricsfalseAlpha1.191.19
DisableAcceleratorUsageMetricstrueBeta1.20
DownwardAPIHugePagesfalseAlpha1.201.20
DownwardAPIHugePagesfalseBeta1.21
DynamicKubeletConfigfalseAlpha1.41.10
DynamicKubeletConfigtrueBeta1.11
EfficientWatchResumptionfalseAlpha1.20
EndpointSliceProxyingfalseAlpha1.181.18
EndpointSliceProxyingtrueBeta1.19
EndpointSliceTerminatingConditionfalseAlpha1.20
EphemeralContainersfalseAlpha1.16
ExpandCSIVolumesfalseAlpha1.141.15
ExpandCSIVolumestrueBeta1.16
ExpandInUsePersistentVolumesfalseAlpha1.111.14
ExpandInUsePersistentVolumestrueBeta1.15
ExpandPersistentVolumesfalseAlpha1.81.10
ExpandPersistentVolumestrueBeta1.11
ExperimentalHostUserNamespaceDefaultingfalseBeta1.5
GenericEphemeralVolumefalseAlpha1.191.20
GenericEphemeralVolumetrueBeta1.21
GracefulNodeShutdownfalseAlpha1.201.20
GracefulNodeShutdowntrueBeta1.21
HPAContainerMetricsfalseAlpha1.20
HPAScaleToZerofalseAlpha1.16
HugePageStorageMediumSizefalseAlpha1.181.18
HugePageStorageMediumSizetrueBeta1.19
IngressClassNamespacedParamsfalseAlpha1.21
IPv6DualStackfalseAlpha1.151.20
IPv6DualStacktrueBeta1.21
KubeletCredentialProvidersfalseAlpha1.20
KubeletPodResourcestrueAlpha1.131.14
KubeletPodResourcestrueBeta1.15
LegacyNodeRoleBehaviorfalseAlpha1.161.18
LegacyNodeRoleBehaviortrueTrue1.19
LocalStorageCapacityIsolationfalseAlpha1.71.9
LocalStorageCapacityIsolationtrueBeta1.10
LocalStorageCapacityIsolationFSQuotaMonitoringfalseAlpha1.15
LogarithmicScaleDownfalseAlpha1.21
KubeletPodResourcesGetAllocatablefalseAlpha1.21
MixedProtocolLBServicefalseAlpha1.20
NamespaceDefaultLabelNametrueBeta1.21
NetworkPolicyEndPortfalseAlpha1.21
NodeDisruptionExclusionfalseAlpha1.161.18
NodeDisruptionExclusiontrueBeta1.19
NonPreemptingPriorityfalseAlpha1.151.18
NonPreemptingPrioritytrueBeta1.19
PodDeletionCostfalseAlpha1.21
PodAffinityNamespaceSelectorfalseAlpha1.21
PodOverheadfalseAlpha1.161.17
PodOverheadtrueBeta1.18
ProbeTerminationGracePeriodfalseAlpha1.21
ProcMountTypefalseAlpha1.12
QOSReservedfalseAlpha1.11
RemainingItemCountfalseAlpha1.15
RemoveSelfLinkfalseAlpha1.161.19
RemoveSelfLinktrueBeta1.20
RotateKubeletServerCertificatefalseAlpha1.71.11
RotateKubeletServerCertificatetrueBeta1.12
RunAsGrouptrueBeta1.14
SCTPSupportfalseAlpha1.121.18
SCTPSupporttrueBeta1.19
ServerSideApplyfalseAlpha1.141.15
ServerSideApplytrueBeta1.16
ServiceInternalTrafficPolicyfalseAlpha1.21
ServiceLBNodePortControlfalseAlpha1.20
ServiceLoadBalancerClassfalseAlpha1.21
ServiceNodeExclusionfalseAlpha1.81.18
ServiceNodeExclusiontrueBeta1.19
ServiceTopologyfalseAlpha1.17
SetHostnameAsFQDNfalseAlpha1.191.19
SetHostnameAsFQDNtrueBeta1.20
SizeMemoryBackedVolumesfalseAlpha1.20
StorageVersionAPIfalseAlpha1.20
StorageVersionHashfalseAlpha1.141.14
StorageVersionHashtrueBeta1.15
SuspendJobfalseAlpha1.21
TTLAfterFinishedfalseAlpha1.12
TopologyAwareHintsfalseAlpha1.21
TopologyManagerfalseAlpha1.161.17
TopologyManagertrueBeta1.18
ValidateProxyRedirectsfalseAlpha1.121.13
ValidateProxyRedirectstrueBeta1.14
WarningHeaderstrueBeta1.19
WinDSRfalseAlpha1.14
WinOverlayfalseAlpha1.141.19
WinOverlaytrueBeta1.20
WindowsEndpointSliceProxyingfalseAlpha1.191.20
WindowsEndpointSliceProxyingtruebeta1.21

Feature gates for graduated or deprecated features

Feature Gates for Graduated or Deprecated Features
FeatureDefaultStageSinceUntil
AcceleratorsfalseAlpha1.61.10
Accelerators-Deprecated1.11-
AdvancedAuditingfalseAlpha1.71.7
AdvancedAuditingtrueBeta1.81.11
AdvancedAuditingtrueGA1.12-
AffinityInAnnotationsfalseAlpha1.61.7
AffinityInAnnotations-Deprecated1.8-
AllowExtTrafficLocalEndpointsfalseBeta1.41.6
AllowExtTrafficLocalEndpointstrueGA1.7-
BlockVolumefalseAlpha1.91.12
BlockVolumetrueBeta1.131.17
BlockVolumetrueGA1.18-
CRIContainerLogRotationfalseAlpha1.101.10
CRIContainerLogRotationtrueBeta1.111.20
CRIContainerLogRotationtrueGA1.21-
CSIBlockVolumefalseAlpha1.111.13
CSIBlockVolumetrueBeta1.141.17
CSIBlockVolumetrueGA1.18-
CSIDriverRegistryfalseAlpha1.121.13
CSIDriverRegistrytrueBeta1.141.17
CSIDriverRegistrytrueGA1.18
CSINodeInfofalseAlpha1.121.13
CSINodeInfotrueBeta1.141.16
CSINodeInfotrueGA1.17
AttachVolumeLimitfalseAlpha1.111.11
AttachVolumeLimittrueBeta1.121.16
AttachVolumeLimittrueGA1.17-
CSIPersistentVolumefalseAlpha1.91.9
CSIPersistentVolumetrueBeta1.101.12
CSIPersistentVolumetrueGA1.13-
CSIVolumeHealthfalseAlpha1.21-
CustomPodDNSfalseAlpha1.91.9
CustomPodDNStrueBeta1.101.13
CustomPodDNStrueGA1.14-
CustomResourceDefaultingfalseAlpha1.151.15
CustomResourceDefaultingtrueBeta1.161.16
CustomResourceDefaultingtrueGA1.17-
CustomResourcePublishOpenAPIfalseAlpha1.141.14
CustomResourcePublishOpenAPItrueBeta1.151.15
CustomResourcePublishOpenAPItrueGA1.16-
CustomResourceSubresourcesfalseAlpha1.101.10
CustomResourceSubresourcestrueBeta1.111.15
CustomResourceSubresourcestrueGA1.16-
CustomResourceValidationfalseAlpha1.81.8
CustomResourceValidationtrueBeta1.91.15
CustomResourceValidationtrueGA1.16-
CustomResourceWebhookConversionfalseAlpha1.131.14
CustomResourceWebhookConversiontrueBeta1.151.15
CustomResourceWebhookConversiontrueGA1.16-
DryRunfalseAlpha1.121.12
DryRuntrueBeta1.131.18
DryRuntrueGA1.19-
DynamicAuditingfalseAlpha1.131.18
DynamicAuditing-Deprecated1.19-
DynamicProvisioningSchedulingfalseAlpha1.111.11
DynamicProvisioningScheduling-Deprecated1.12-
DynamicVolumeProvisioningtrueAlpha1.31.7
DynamicVolumeProvisioningtrueGA1.8-
EnableAggregatedDiscoveryTimeouttrueDeprecated1.16-
EnableEquivalenceClassCachefalseAlpha1.81.14
EnableEquivalenceClassCache-Deprecated1.15-
EndpointSlicefalseAlpha1.161.16
EndpointSlicefalseBeta1.171.17
EndpointSlicetrueBeta1.181.21
EndpointSlicetrueGA1.21-
EndpointSliceNodeNamefalseAlpha1.201.21
EndpointSliceNodeNametrueGA1.21-
ExperimentalCriticalPodAnnotationfalseAlpha1.51.12
ExperimentalCriticalPodAnnotationfalseDeprecated1.13-
EvenPodsSpreadfalseAlpha1.161.17
EvenPodsSpreadtrueBeta1.181.18
EvenPodsSpreadtrueGA1.19-
ExecProbeTimeouttrueGA1.20-
GCERegionalPersistentDisktrueBeta1.101.12
GCERegionalPersistentDisktrueGA1.13-
HugePagesfalseAlpha1.81.9
HugePagestrueBeta1.101.13
HugePagestrueGA1.14-
HyperVContainerfalseAlpha1.101.19
HyperVContainerfalseDeprecated1.20-
ImmutableEphemeralVolumesfalseAlpha1.181.18
ImmutableEphemeralVolumestrueBeta1.191.20
ImmutableEphemeralVolumestrueGA1.21
IndexedJobfalseAlpha1.21
InitializersfalseAlpha1.71.13
Initializers-Deprecated1.14-
KubeletConfigFilefalseAlpha1.81.9
KubeletConfigFile-Deprecated1.10-
KubeletPluginsWatcherfalseAlpha1.111.11
KubeletPluginsWatchertrueBeta1.121.12
KubeletPluginsWatchertrueGA1.13-
KubeletPodResourcesfalseAlpha1.131.14
KubeletPodResourcestrueBeta1.15
KubeletPodResourcestrueGA1.20
MountContainersfalseAlpha1.91.16
MountContainersfalseDeprecated1.17-
MountPropagationfalseAlpha1.81.9
MountPropagationtrueBeta1.101.11
MountPropagationtrueGA1.12-
NodeLeasefalseAlpha1.121.13
NodeLeasetrueBeta1.141.16
NodeLeasetrueGA1.17-
PVCProtectionfalseAlpha1.91.9
PVCProtection-Deprecated1.10-
PersistentLocalVolumesfalseAlpha1.71.9
PersistentLocalVolumestrueBeta1.101.13
PersistentLocalVolumestrueGA1.14-
PodDisruptionBudgetfalseAlpha1.31.4
PodDisruptionBudgettrueBeta1.51.20
PodDisruptionBudgettrueGA1.21-
PodPriorityfalseAlpha1.81.10
PodPrioritytrueBeta1.111.13
PodPrioritytrueGA1.14-
PodReadinessGatesfalseAlpha1.111.11
PodReadinessGatestrueBeta1.121.13
PodReadinessGatestrueGA1.14-
PodShareProcessNamespacefalseAlpha1.101.11
PodShareProcessNamespacetrueBeta1.121.16
PodShareProcessNamespacetrueGA1.17-
RequestManagementfalseAlpha1.151.16
ResourceLimitsPriorityFunctionfalseAlpha1.91.18
ResourceLimitsPriorityFunction-Deprecated1.19-
ResourceQuotaScopeSelectorsfalseAlpha1.111.11
ResourceQuotaScopeSelectorstrueBeta1.121.16
ResourceQuotaScopeSelectorstrueGA1.17-
RootCAConfigMapfalseAlpha1.131.19
RootCAConfigMaptrueBeta1.201.20
RootCAConfigMaptrueGA1.21-
RotateKubeletClientCertificatetrueBeta1.81.18
RotateKubeletClientCertificatetrueGA1.19-
RuntimeClassfalseAlpha1.121.13
RuntimeClasstrueBeta1.141.19
RuntimeClasstrueGA1.20-
ScheduleDaemonSetPodsfalseAlpha1.111.11
ScheduleDaemonSetPodstrueBeta1.121.16
ScheduleDaemonSetPodstrueGA1.17-
SCTPSupportfalseAlpha1.121.18
SCTPSupporttrueBeta1.191.19
SCTPSupporttrueGA1.20-
ServiceAccountIssuerDiscoveryfalseAlpha1.181.19
ServiceAccountIssuerDiscoverytrueBeta1.201.20
ServiceAccountIssuerDiscoverytrueGA1.21-
ServiceAppProtocolfalseAlpha1.181.18
ServiceAppProtocoltrueBeta1.19
ServiceAppProtocoltrueGA1.20-
ServiceLoadBalancerFinalizerfalseAlpha1.151.15
ServiceLoadBalancerFinalizertrueBeta1.161.16
ServiceLoadBalancerFinalizertrueGA1.17-
StartupProbefalseAlpha1.161.17
StartupProbetrueBeta1.181.19
StartupProbetrueGA1.20-
StorageObjectInUseProtectiontrueBeta1.101.10
StorageObjectInUseProtectiontrueGA1.11-
StreamingProxyRedirectsfalseBeta1.51.5
StreamingProxyRedirectstrueBeta1.61.18
StreamingProxyRedirects-Deprecated1.19-
SupportIPVSProxyModefalseAlpha1.81.8
SupportIPVSProxyModefalseBeta1.91.9
SupportIPVSProxyModetrueBeta1.101.10
SupportIPVSProxyModetrueGA1.11-
SupportNodePidsLimitfalseAlpha1.141.14
SupportNodePidsLimittrueBeta1.151.19
SupportNodePidsLimittrueGA1.20-
SupportPodPidsLimitfalseAlpha1.101.13
SupportPodPidsLimittrueBeta1.141.19
SupportPodPidsLimittrueGA1.20-
SysctlstrueBeta1.111.20
SysctlstrueGA1.21
TaintBasedEvictionsfalseAlpha1.61.12
TaintBasedEvictionstrueBeta1.131.17
TaintBasedEvictionstrueGA1.18-
TaintNodesByConditionfalseAlpha1.81.11
TaintNodesByConditiontrueBeta1.121.16
TaintNodesByConditiontrueGA1.17-
TokenRequestfalseAlpha1.101.11
TokenRequesttrueBeta1.121.19
TokenRequesttrueGA1.20-
TokenRequestProjectionfalseAlpha1.111.11
TokenRequestProjectiontrueBeta1.121.19
TokenRequestProjectiontrueGA1.20-
VolumeCapacityPriorityfalseAlpha1.21-
VolumeSnapshotDataSourcefalseAlpha1.121.16
VolumeSnapshotDataSourcetrueBeta1.171.19
VolumeSnapshotDataSourcetrueGA1.20-
VolumePVCDataSourcefalseAlpha1.151.15
VolumePVCDataSourcetrueBeta1.161.17
VolumePVCDataSourcetrueGA1.18-
VolumeSchedulingfalseAlpha1.91.9
VolumeSchedulingtrueBeta1.101.12
VolumeSchedulingtrueGA1.13-
VolumeSubpathtrueGA1.10-
VolumeSubpathEnvExpansionfalseAlpha1.141.14
VolumeSubpathEnvExpansiontrueBeta1.151.16
VolumeSubpathEnvExpansiontrueGA1.17-
WatchBookmarkfalseAlpha1.151.15
WatchBookmarktrueBeta1.161.16
WatchBookmarktrueGA1.17-
WindowsGMSAfalseAlpha1.141.15
WindowsGMSAtrueBeta1.161.17
WindowsGMSAtrueGA1.18-
WindowsRunAsUserNamefalseAlpha1.161.16
WindowsRunAsUserNametrueBeta1.171.17
WindowsRunAsUserNametrueGA1.18-

Using a feature

Feature stages

A feature can be in Alpha, Beta or GA stage. An Alpha feature means:

  • Disabled by default.
  • Might be buggy. Enabling the feature may expose bugs.
  • Support for feature may be dropped at any time without notice.
  • The API may change in incompatible ways in a later software release without notice.
  • Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

A Beta feature means:

  • Enabled by default.
  • The feature is well tested. Enabling the feature is considered safe.
  • Support for the overall feature will not be dropped, though details may change.
  • The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
  • Recommended for only non-business-critical uses because of potential for incompatible changes in subsequent releases. If you have multiple clusters that can be upgraded independently, you may be able to relax this restriction.
Note: Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.

A General Availability (GA) feature is also referred to as a stable feature. It means:

  • The feature is always enabled; you cannot disable it.
  • The corresponding feature gate is no longer needed.
  • Stable versions of features will appear in released software for many subsequent versions.

List of feature gates

Each feature gate is designed for enabling/disabling a specific feature:

  • APIListChunking: Enable the API clients to retrieve (LIST or GET) resources from API server in chunks.
  • APIPriorityAndFairness: Enable managing request concurrency with prioritization and fairness at each server. (Renamed from RequestManagement)
  • APIResponseCompression: Compress the API responses for LIST or GET requests.
  • APIServerIdentity: Assign each API server an ID in a cluster.
  • Accelerators: Enable Nvidia GPU support when using Docker
  • AdvancedAuditing: Enable advanced auditing
  • AffinityInAnnotations(deprecated): Enable setting Pod affinity or anti-affinity.
  • AllowExtTrafficLocalEndpoints: Enable a service to route external requests to node local endpoints.
  • AllowInsecureBackendProxy: Enable the users to skip TLS verification of kubelets on Pod log requests.
  • AnyVolumeDataSource: Enable use of any custom resource as the DataSource of a PVC.
  • AppArmor: Enable AppArmor based mandatory access control on Linux nodes when using Docker. See AppArmor Tutorial for more details.
  • AttachVolumeLimit: Enable volume plugins to report limits on number of volumes that can be attached to a node. See dynamic volume limits for more details.
  • BalanceAttachedNodeVolumes: Include volume count on node to be considered for balanced resource allocation while scheduling. A node which has closer CPU, memory utilization, and volume count is favored by the scheduler while making decisions.
  • BlockVolume: Enable the definition and consumption of raw block devices in Pods. See Raw Block Volume Support for more details.
  • BoundServiceAccountTokenVolume: Migrate ServiceAccount volumes to use a projected volume consisting of a ServiceAccountTokenVolumeProjection. Cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false. Check Bound Service Account Tokens for more details.
  • CPUManager: Enable container level CPU affinity support, see CPU Management Policies.
  • CRIContainerLogRotation: Enable container log rotation for CRI container runtime. The default max size of a log file is 10MB and the default max number of log files allowed for a container is 5. These values can be configured in the kubelet config. See the logging at node level documentation for more details.
  • CSIBlockVolume: Enable external CSI volume drivers to support block storage. See the csi raw block volume support documentation for more details.
  • CSIDriverRegistry: Enable all logic related to the CSIDriver API object in csi.storage.k8s.io.
  • CSIInlineVolume: Enable CSI Inline volumes support for pods.
  • CSIMigration: Enables shims and translation logic to route volume operations from in-tree plugins to corresponding pre-installed CSI plugins
  • CSIMigrationAWS: Enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Supports falling back to in-tree EBS plugin if a node does not have EBS CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
  • CSIMigrationAWSComplete: Stops registering the EBS in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Requires CSIMigration and CSIMigrationAWS feature flags enabled and EBS CSI plugin installed and configured on all nodes in the cluster.
  • CSIMigrationAzureDisk: Enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. Supports falling back to in-tree AzureDisk plugin if a node does not have AzureDisk CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
  • CSIMigrationAzureDiskComplete: Stops registering the Azure-Disk in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. Requires CSIMigration and CSIMigrationAzureDisk feature flags enabled and AzureDisk CSI plugin installed and configured on all nodes in the cluster.
  • CSIMigrationAzureFile: Enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. Supports falling back to in-tree AzureFile plugin if a node does not have AzureFile CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
  • CSIMigrationAzureFileComplete: Stops registering the Azure-File in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. Requires CSIMigration and CSIMigrationAzureFile feature flags enabled and AzureFile CSI plugin installed and configured on all nodes in the cluster.
  • CSIMigrationGCE: Enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Supports falling back to in-tree GCE plugin if a node does not have PD CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
  • CSIMigrationGCEComplete: Stops registering the GCE-PD in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Requires CSIMigration and CSIMigrationGCE feature flags enabled and PD CSI plugin installed and configured on all nodes in the cluster.
  • CSIMigrationOpenStack: Enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Supports falling back to in-tree Cinder plugin if a node does not have Cinder CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
  • CSIMigrationOpenStackComplete: Stops registering the Cinder in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Requires CSIMigration and CSIMigrationOpenStack feature flags enabled and Cinder CSI plugin installed and configured on all nodes in the cluster.
  • CSIMigrationvSphere: Enables shims and translation logic to route volume operations from the vSphere in-tree plugin to vSphere CSI plugin. Supports falling back to in-tree vSphere plugin if a node does not have vSphere CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
  • CSIMigrationvSphereComplete: Stops registering the vSphere in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the vSphere in-tree plugin to vSphere CSI plugin. Requires CSIMigration and CSIMigrationvSphere feature flags enabled and vSphere CSI plugin installed and configured on all nodes in the cluster.
  • CSINodeInfo: Enable all logic related to the CSINodeInfo API object in csi.storage.k8s.io.
  • CSIPersistentVolume: Enable discovering and mounting volumes provisioned through a CSI (Container Storage Interface) compatible volume plugin.
  • CSIServiceAccountToken: Enable CSI drivers to receive the pods' service account token that they mount volumes for. See Token Requests.
  • CSIStorageCapacity: Enables CSI drivers to publish storage capacity information and the Kubernetes scheduler to use that information when scheduling pods. See Storage Capacity. Check the csi volume type documentation for more details.
  • CSIVolumeFSGroupPolicy: Allows CSIDrivers to use the fsGroupPolicy field. This field controls whether volumes created by a CSIDriver support volume ownership and permission modifications when these volumes are mounted.
  • CSIVolumeHealth: Enable support for CSI volume health monitoring on node.
  • ConfigurableFSGroupPolicy: Allows user to configure volume permission change policy for fsGroups when mounting a volume in a Pod. See Configure volume permission and ownership change policy for Pods for more details.
  • CronJobControllerV2: Use an alternative implementation of the CronJob controller. Otherwise, version 1 of the same controller is selected. The version 2 controller provides experimental performance improvements.
  • CustomCPUCFSQuotaPeriod: Enable nodes to change cpuCFSQuotaPeriod in kubelet config.
  • CustomPodDNS: Enable customizing the DNS settings for a Pod using its dnsConfig property. Check Pod's DNS Config for more details.
  • CustomResourceDefaulting: Enable CRD support for default values in OpenAPI v3 validation schemas.
  • CustomResourcePublishOpenAPI: Enables publishing of CRD OpenAPI specs.
  • CustomResourceSubresources: Enable /status and /scale subresources on resources created from CustomResourceDefinition.
  • CustomResourceValidation: Enable schema based validation on resources created from CustomResourceDefinition.
  • CustomResourceWebhookConversion: Enable webhook-based conversion on resources created from CustomResourceDefinition.
  • DefaultPodTopologySpread: Enables the use of PodTopologySpread scheduling plugin to do default spreading.
  • DevicePlugins: Enable the device-plugins based resource provisioning on nodes.
  • DisableAcceleratorUsageMetrics: Disable accelerator metrics collected by the kubelet.
  • DownwardAPIHugePages: Enables usage of hugepages in downward API.
  • DryRun: Enable server-side dry run requests so that validation, merging, and mutation can be tested without committing.
  • DynamicAuditing(deprecated): Used to enable dynamic auditing before v1.19.
  • DynamicKubeletConfig: Enable the dynamic configuration of kubelet. See Reconfigure kubelet.
  • DynamicProvisioningScheduling: Extend the default scheduler to be aware of volume topology and handle PV provisioning. This feature is superseded by the VolumeScheduling feature completely in v1.12.
  • DynamicVolumeProvisioning(deprecated): Enable the dynamic provisioning of persistent volumes to Pods.
  • EfficientWatchResumption: Allows for storage-originated bookmark (progress notify) events to be delivered to the users. This is only applied to watch operations.
  • EnableAggregatedDiscoveryTimeout (deprecated): Enable the five second timeout on aggregated discovery calls.
  • EnableEquivalenceClassCache: Enable the scheduler to cache equivalence of nodes when scheduling Pods.
  • EndpointSlice: Enables EndpointSlices for more scalable and extensible network endpoints. See Enabling EndpointSlices.
  • EndpointSliceNodeName: Enables EndpointSlice nodeName field.
  • EndpointSliceProxying: When enabled, kube-proxy running on Linux will use EndpointSlices as the primary data source instead of Endpoints, enabling scalability and performance improvements. See Enabling Endpoint Slices.
  • EndpointSliceTerminatingCondition: Enables EndpointSlice terminating and serving condition fields.
  • EphemeralContainers: Enable the ability to add ephemeral containers to running pods.
  • EvenPodsSpread: Enable pods to be scheduled evenly across topology domains. See Pod Topology Spread Constraints.
  • ExecProbeTimeout: Ensure kubelet respects exec probe timeouts. This feature gate exists in case any of your existing workloads depend on a now-corrected fault where Kubernetes ignored exec probe timeouts. See readiness probes.
  • ExpandCSIVolumes: Enable the expanding of CSI volumes.
  • ExpandInUsePersistentVolumes: Enable expanding in-use PVCs. See Resizing an in-use PersistentVolumeClaim.
  • ExpandPersistentVolumes: Enable the expanding of persistent volumes. See Expanding Persistent Volumes Claims.
  • ExperimentalCriticalPodAnnotation: Enable annotating specific pods as critical so that their scheduling is guaranteed. This feature is deprecated by Pod Priority and Preemption as of v1.13.
  • ExperimentalHostUserNamespaceDefaulting: Enabling the defaulting user namespace to host. This is for containers that are using other host namespaces, host mounts, or containers that are privileged or using specific non-namespaced capabilities (e.g. MKNODE, SYS_MODULE etc.). This should only be enabled if user namespace remapping is enabled in the Docker daemon.
  • GCERegionalPersistentDisk: Enable the regional PD feature on GCE.
  • GenericEphemeralVolume: Enables ephemeral, inline volumes that support all features of normal volumes (can be provided by third-party storage vendors, storage capacity tracking, restore from snapshot, etc.). See Ephemeral Volumes.
  • GracefulNodeShutdown: Enables support for graceful shutdown in kubelet. During a system shutdown, kubelet will attempt to detect the shutdown event and gracefully terminate pods running on the node. See Graceful Node Shutdown for more details.
  • HPAContainerMetrics: Enable the HorizontalPodAutoscaler to scale based on metrics from individual containers in target pods.
  • HPAScaleToZero: Enables setting minReplicas to 0 for HorizontalPodAutoscaler resources when using custom or external metrics.
  • HugePages: Enable the allocation and consumption of pre-allocated huge pages.
  • HugePageStorageMediumSize: Enable support for multiple sizes pre-allocated huge pages.
  • HyperVContainer: Enable Hyper-V isolation for Windows containers.
  • ImmutableEphemeralVolumes: Allows for marking individual Secrets and ConfigMaps as immutable for better safety and performance.
  • IndexedJob: Allows the Job controller to manage Pod completions per completion index.
  • IngressClassNamespacedParams: Allow namespace-scoped parameters reference in IngressClass resource. This feature adds two fields - Scope and Namespace to IngressClass.spec.parameters.
  • IPv6DualStack: Enable dual stack support for IPv6.
  • KubeletConfigFile (deprecated): Enable loading kubelet configuration from a file specified using a config file. See setting kubelet parameters via a config file for more details.
  • KubeletCredentialProviders: Enable kubelet exec credential providers for image pull credentials.
  • KubeletPluginsWatcher: Enable probe-based plugin watcher utility to enable kubelet to discover plugins such as CSI volume drivers.
  • KubeletPodResources: Enable the kubelet's pod resources gRPC endpoint. See Support Device Monitoring for more details.
  • KubeletPodResourcesGetAllocatable: Enable the kubelet's pod resources GetAllocatableResources functionality. This API augments the resource allocation reporting with informations about the allocatable resources, enabling clients to properly track the free compute resources on a node.
  • LegacyNodeRoleBehavior: When disabled, legacy behavior in service load balancers and node disruption will ignore the node-role.kubernetes.io/master label in favor of the feature-specific labels provided by NodeDisruptionExclusion and ServiceNodeExclusion.
  • LocalStorageCapacityIsolation: Enable the consumption of local ephemeral storage and also the sizeLimit property of an emptyDir volume.
  • LocalStorageCapacityIsolationFSQuotaMonitoring: When LocalStorageCapacityIsolation is enabled for local ephemeral storage and the backing filesystem for emptyDir volumes supports project quotas and they are enabled, use project quotas to monitor emptyDir volume storage consumption rather than filesystem walk for better performance and accuracy.
  • LogarithmicScaleDown: Enable semi-random selection of pods to evict on controller scaledown based on logarithmic bucketing of pod timestamps.
  • MixedProtocolLBService: Enable using different protocols in the same LoadBalancer type Service instance.
  • MountContainers (deprecated): Enable using utility containers on host as the volume mounter.
  • MountPropagation: Enable sharing volume mounted by one container to other containers or pods. For more details, please see mount propagation.
  • NamespaceDefaultLabelName: Configure the API Server to set an immutable label kubernetes.io/metadata.name on all namespaces, containing the namespace name.
  • NetworkPolicyEndPort: Enable use of the field endPort in NetworkPolicy objects, allowing the selection of a port range instead of a single port.
  • NodeDisruptionExclusion: Enable use of the Node label node.kubernetes.io/exclude-disruption which prevents nodes from being evacuated during zone failures.
  • NodeLease: Enable the new Lease API to report node heartbeats, which could be used as a node health signal.
  • NonPreemptingPriority: Enable preemptionPolicy field for PriorityClass and Pod.
  • PVCProtection: Enable the prevention of a PersistentVolumeClaim (PVC) from being deleted when it is still used by any Pod.
  • PodDeletionCost: Enable the Pod Deletion Cost feature which allows users to influence ReplicaSet downscaling order.
  • PersistentLocalVolumes: Enable the usage of local volume type in Pods. Pod affinity has to be specified if requesting a local volume.
  • PodDisruptionBudget: Enable the PodDisruptionBudget feature.
  • PodAffinityNamespaceSelector: Enable the Pod Affinity Namespace Selector and CrossNamespacePodAffinity quota scope features.
  • PodOverhead: Enable the PodOverhead feature to account for pod overheads.
  • PodPriority: Enable the descheduling and preemption of Pods based on their priorities.
  • PodReadinessGates: Enable the setting of PodReadinessGate field for extending Pod readiness evaluation. See Pod readiness gate for more details.
  • PodShareProcessNamespace: Enable the setting of shareProcessNamespace in a Pod for sharing a single process namespace between containers running in a pod. More details can be found in Share Process Namespace between Containers in a Pod.
  • ProbeTerminationGracePeriod: Enable setting probe-level terminationGracePeriodSeconds on pods. See the enhancement proposal for more details.
  • ProcMountType: Enables control over the type proc mounts for containers by setting the procMount field of a SecurityContext.
  • QOSReserved: Allows resource reservations at the QoS level preventing pods at lower QoS levels from bursting into resources requested at higher QoS levels (memory only for now).
  • RemainingItemCount: Allow the API servers to show a count of remaining items in the response to a chunking list request.
  • RemoveSelfLink: Deprecates and removes selfLink from ObjectMeta and ListMeta.
  • ResourceLimitsPriorityFunction (deprecated): Enable a scheduler priority function that assigns a lowest possible score of 1 to a node that satisfies at least one of the input Pod's cpu and memory limits. The intent is to break ties between nodes with same scores.
  • ResourceQuotaScopeSelectors: Enable resource quota scope selectors.
  • RootCAConfigMap: Configure the kube-controller-manager to publish a ConfigMap named kube-root-ca.crt to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. See Bound Service Account Tokens for more details.
  • RotateKubeletClientCertificate: Enable the rotation of the client TLS certificate on the kubelet. See kubelet configuration for more details.
  • RotateKubeletServerCertificate: Enable the rotation of the server TLS certificate on the kubelet. See kubelet configuration for more details.
  • RunAsGroup: Enable control over the primary group ID set on the init processes of containers.
  • RuntimeClass: Enable the RuntimeClass feature for selecting container runtime configurations.
  • ScheduleDaemonSetPods: Enable DaemonSet Pods to be scheduled by the default scheduler instead of the DaemonSet controller.
  • SCTPSupport: Enables the SCTP protocol value in Pod, Service, Endpoints, EndpointSlice, and NetworkPolicy definitions.
  • ServerSideApply: Enables the Sever Side Apply (SSA) feature on the API Server.
  • ServiceAccountIssuerDiscovery: Enable OIDC discovery endpoints (issuer and JWKS URLs) for the service account issuer in the API server. See Configure Service Accounts for Pods for more details.
  • ServiceAppProtocol: Enables the AppProtocol field on Services and Endpoints.
  • ServiceInternalTrafficPolicy: Enables the InternalTrafficPolicy field on Services
  • ServiceLBNodePortControl: Enables the spec.allocateLoadBalancerNodePorts field on Services.
  • ServiceLoadBalancerClass: Enables the LoadBalancerClass field on Services. See Specifying class of load balancer implementation for more details.
  • ServiceLoadBalancerFinalizer: Enable finalizer protection for Service load balancers.
  • ServiceNodeExclusion: Enable the exclusion of nodes from load balancers created by a cloud provider. A node is eligible for exclusion if labelled with "node.kubernetes.io/exclude-from-external-load-balancers".
  • ServiceTopology: Enable service to route traffic based upon the Node topology of the cluster. See ServiceTopology for more details.
  • SetHostnameAsFQDN: Enable the ability of setting Fully Qualified Domain Name(FQDN) as the hostname of a pod. See Pod's setHostnameAsFQDN field.
  • SizeMemoryBackedVolumes: Enable kubelets to determine the size limit for memory-backed volumes (mainly emptyDir volumes).
  • StartupProbe: Enable the startup probe in the kubelet.
  • StorageObjectInUseProtection: Postpone the deletion of PersistentVolume or PersistentVolumeClaim objects if they are still being used.
  • StorageVersionAPI: Enable the storage version API.
  • StorageVersionHash: Allow API servers to expose the storage version hash in the discovery.
  • StreamingProxyRedirects: Instructs the API server to intercept (and follow) redirects from the backend (kubelet) for streaming requests. Examples of streaming requests include the exec, attach and port-forward requests.
  • SupportIPVSProxyMode: Enable providing in-cluster service load balancing using IPVS. See service proxies for more details.
  • SupportPodPidsLimit: Enable the support to limiting PIDs in Pods.
  • SupportNodePidsLimit: Enable the support to limiting PIDs on the Node. The parameter pid=<number> in the --system-reserved and --kube-reserved options can be specified to ensure that the specified number of process IDs will be reserved for the system as a whole and for Kubernetes system daemons respectively.
  • SuspendJob: Enable support to suspend and resume Jobs. See the Jobs docs for more details.
  • Sysctls: Enable support for namespaced kernel parameters (sysctls) that can be set for each pod. See sysctls for more details.
  • TTLAfterFinished: Allow a TTL controller to clean up resources after they finish execution.
  • TaintBasedEvictions: Enable evicting pods from nodes based on taints on Nodes and tolerations on Pods. See taints and tolerations for more details.
  • TaintNodesByCondition: Enable automatic tainting nodes based on node conditions.
  • TokenRequest: Enable the TokenRequest endpoint on service account resources.
  • TokenRequestProjection: Enable the injection of service account tokens into a Pod through a projected volume.
  • TopologyAwareHints: Enables topology aware routing based on topology hints in EndpointSlices. See Topology Aware Hints for more details.
  • TopologyManager: Enable a mechanism to coordinate fine-grained hardware resource assignments for different components in Kubernetes. See Control Topology Management Policies on a node.
  • VolumeCapacityPriority: Enable support for prioritizing nodes in different topologies based on available PV capacity.
  • VolumePVCDataSource: Enable support for specifying an existing PVC as a DataSource.
  • VolumeScheduling: Enable volume topology aware scheduling and make the PersistentVolumeClaim (PVC) binding aware of scheduling decisions. It also enables the usage of local volume type when used together with the PersistentLocalVolumes feature gate.
  • VolumeSnapshotDataSource: Enable volume snapshot data source support.
  • VolumeSubpathEnvExpansion: Enable subPathExpr field for expanding environment variables into a subPath.
  • WarningHeaders: Allow sending warning headers in API responses.
  • WatchBookmark: Enable support for watch bookmark events.
  • WinDSR: Allows kube-proxy to create DSR loadbalancers for Windows.
  • WinOverlay: Allows kube-proxy to run in overlay mode for Windows.
  • WindowsGMSA: Enables passing of GMSA credential specs from pods to container runtimes.
  • WindowsRunAsUserName : Enable support for running applications in Windows containers with as a non-default user. See Configuring RunAsUserName for more details.
  • WindowsEndpointSliceProxying: When enabled, kube-proxy running on Windows will use EndpointSlices as the primary data source instead of Endpoints, enabling scalability and performance improvements. See Enabling Endpoint Slices.

What's next

  • The deprecation policy for Kubernetes explains the project's approach to removing features and components.

2 - kubelet

Synopsis

The kubelet is the primary "node agent" that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.

The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn't manage containers which were not created by Kubernetes.

Other than from a PodSpec from the apiserver, there are three ways that a container manifest can be provided to the Kubelet.

File: Path passed as a flag on the command line. Files under this path will be monitored periodically for updates. The monitoring period is 20s by default and is configurable via a flag.

HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint is checked every 20 seconds (also configurable with a flag).

HTTP server: The kubelet can also listen for HTTP and respond to a simple API (underspec'd currently) to submit a new manifest.

kubelet [flags]

Options

--add-dir-header
If true, adds the file directory to the header of the log messages
--address ip     Default: 0.0.0.0
The IP address for the Kubelet to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--allowed-unsafe-sysctls strings
Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in `*`). Use these at your own risk. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--alsologtostderr
log to standard error as well as files
--anonymous-auth     Default: true
Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of `system:anonymous`, and a group name of `system:unauthenticated`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--authentication-token-webhook
Use the `TokenReview` API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--authentication-token-webhook-cache-ttl duration     Default: `2m0s`
The duration to cache responses from the webhook token authenticator. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--authorization-mode string
Authorization mode for Kubelet server. Valid options are `AlwaysAllow` or `Webhook`. `Webhook` mode uses the `SubjectAccessReview` API to determine authorization. (default "AlwaysAllow" when `--config` flag is not provided; "Webhook" when `--config` flag presents.) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--authorization-webhook-cache-authorized-ttl duration     Default: `5m0s`
The duration to cache 'authorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--authorization-webhook-cache-unauthorized-ttl duration     Default: `30s`
The duration to cache 'unauthorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--azure-container-registry-config string
Path to the file containing Azure container registry configuration information.
--bootstrap-kubeconfig string
Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by `--kubeconfig` does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by `--kubeconfig`. The client certificate and key file will be stored in the directory pointed by `--cert-dir`.
--cert-dir string     Default: `/var/lib/kubelet/pki`
The directory where the TLS certs are located. If `--tls-cert-file` and `--tls-private-key-file` are provided, this flag will be ignored.
--cgroup-driver string     Default: `cgroupfs`
Driver that the kubelet uses to manipulate cgroups on the host. Possible values: `cgroupfs`, `systemd`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)/td>
--cgroup-root string     Default: `''`
Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cgroups-per-qos     Default: `true`
Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--chaos-chance float
If > 0.0, introduce random client errors and latency. Intended for testing. (DEPRECATED: will be removed in a future version.)
--client-ca-file string
If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cloud-config string
The path to the cloud provider configuration file. Empty string for no configuration file. (DEPRECATED: will be removed in 1.23, in favor of removing cloud providers code from Kubelet.)
--cloud-provider string
The provider for cloud services. Set to empty string for running with no cloud provider. If set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used). (DEPRECATED: will be removed in 1.23, in favor of removing cloud provider code from Kubelet.)
--cluster-dns strings
Comma-separated list of DNS server IP address. This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst". Note: all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cluster-domain string
Domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cni-bin-dir string     Default: `/opt/cni/bin`
<Warning: Alpha feature> A comma-separated list of full paths of directories in which to search for CNI plugin binaries. This docker-specific flag only works when container-runtime is set to `docker`.
--cni-cache-dir string     Default: `/var/lib/cni/cache`
<Warning: Alpha feature> The full path of the directory in which CNI should store cache files. This docker-specific flag only works when container-runtime is set to `docker`.
--cni-conf-dir string     Default: `/etc/cni/net.d`
<Warning: Alpha feature> The full path of the directory in which to search for CNI config files. This docker-specific flag only works when container-runtime is set to `docker`.
--config string
The Kubelet will load its initial configuration from this file. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Omit this flag to use the built-in default configuration values. Command-line flags override configuration from this file.
--container-log-max-files int32     Default: 5
Set the maximum number of container log files that can be present for a container. The number must be ≥ 2. This flag can only be used with `--container-runtime=remote`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--container-log-max-size string     Default: `10Mi`
Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with `--container-runtime=remote`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--container-runtime string     Default: `docker`
The container runtime to use. Possible values: `docker`, `remote`.
--container-runtime-endpoint string     Default: `unix:///var/run/dockershim.sock`
[Experimental] The endpoint of remote runtime service. Currently unix socket endpoint is supported on Linux, while npipe and tcp endpoints are supported on windows. Examples: `unix:///var/run/dockershim.sock`, `npipe:////./pipe/dockershim`.
--contention-profiling
Enable lock contention profiling, if profiling is enabled (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cpu-cfs-quota     Default: `true`
Enable CPU CFS quota enforcement for containers that specify CPU limits (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cpu-cfs-quota-period duration     Default: `100ms`
Sets CPU CFS quota period value, `cpu.cfs_period_us`, defaults to Linux Kernel default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cpu-manager-policy string     Default: `none`
CPU Manager policy to use. Possible values: `none`, `static`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--cpu-manager-reconcile-period duration     Default: `10s`
<Warning: Alpha feature> CPU Manager reconciliation period. Examples: `10s`, or `1m`. If not supplied, defaults to node status update frequency. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--docker-endpoint string     Default: `unix:///var/run/docker.sock`
Use this for the `docker` endpoint to communicate with. This docker-specific flag only works when container-runtime is set to `docker`.
--dynamic-config-dir string
The Kubelet will use this directory for checkpointing downloaded configurations and tracking configuration health. The Kubelet will create this directory if it does not already exist. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Providing this flag enables dynamic Kubelet configuration. The `DynamicKubeletConfig` feature gate must be enabled to pass this flag; this gate currently defaults to `true` because the feature is beta.
--enable-controller-attach-detach     Default: `true`
Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations.
--enable-debugging-handlers     Default: `true`
Enables server endpoints for log collection and local running of containers and commands. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--enable-server     Default: `true`
Enable the Kubelet's server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--enforce-node-allocatable strings     Default: `pods`
A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptable options are `none`, `pods`, `system-reserved`, and `kube-reserved`. If the latter two options are specified, `--system-reserved-cgroup` and `--kube-reserved-cgroup` must also be set, respectively. If `none` is specified, no additional options should be set. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--event-burst int32     Default: 10
Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding `--event-qps`. Only used if `--event-qps` > 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--event-qps int32     Default: 5
If > `0`, limit event creations per second to this value. If `0`, unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-hard mapStringString     Default: `imagefs.available<15%,memory.available<100Mi,nodefs.available<10%`
A set of eviction thresholds (e.g. `memory.available<1Gi`) that if met would trigger a pod eviction. On a Linux node, the default value also includes `nodefs.inodesFree<5%`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-max-pod-grace-period int32
Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-minimum-reclaim mapStringString
A set of minimum reclaims (e.g. `imagefs.available=2Gi`) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-pressure-transition-period duration     Default: `5m0s`
Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-soft mapStringString
A set of eviction thresholds (e.g. `memory.available>1.5Gi`) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-soft-grace-period mapStringString
A set of eviction grace periods (e.g. `memory.available=1m30s`) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--exit-on-lock-contention
Whether kubelet should exit upon lock-file contention.
--experimental-allocatable-ignore-eviction     Default: `false`
When set to `true`, Hard eviction thresholds will be ignored while calculating node allocatable. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (DEPRECATED: will be removed in 1.23)
--experimental-bootstrap-kubeconfig string
DEPRECATED: Use `--bootstrap-kubeconfig`
--experimental-check-node-capabilities-before-mount
[Experimental] if set to `true`, the kubelet will check the underlying node for required components (binaries, etc.) before performing the mount (DEPRECATED: will be removed in 1.23, in favor of using CSI.)
--experimental-kernel-memcg-notification
If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. This flag will be removed in 1.23. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--experimental-log-sanitization bool
[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--experimental-mounter-path string     Default: `mount`
[Experimental] Path of mounter binary. Leave empty to use the default `mount`. (DEPRECATED: will be removed in 1.23, in favor of using CSI.)
--fail-swap-on     Default: `true`
Makes the Kubelet fail to start if swap is enabled on the node. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--feature-gates mapStringBool
A set of `key=value` pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AllowInsecureBackendProxy=true|false (BETA - default=true)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAWSComplete=true|false (ALPHA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureDiskComplete=true|false (ALPHA - default=false)
CSIMigrationAzureFile=true|false (ALPHA - default=false)
CSIMigrationAzureFileComplete=true|false (ALPHA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationGCEComplete=true|false (ALPHA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=false)
CSIMigrationOpenStackComplete=true|false (ALPHA - default=false)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIMigrationvSphereComplete=true|false (BETA - default=false)
CSIServiceAccountToken=true|false (ALPHA - default=false)
CSIStorageCapacity=true|false (ALPHA - default=false)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
CronJobControllerV2=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DefaultPodTopologySpread=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DownwardAPIHugePages=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EfficientWatchResumption=true|false (ALPHA - default=false)
EndpointSlice=true|false (BETA - default=true)
EndpointSliceNodeName=true|false (ALPHA - default=false)
EndpointSliceProxying=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (ALPHA - default=false)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (ALPHA - default=false)
GracefulNodeShutdown=true|false (ALPHA - default=false)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HugePageStorageMediumSize=true|false (BETA - default=true)
IPv6DualStack=true|false (ALPHA - default=false)
ImmutableEphemeralVolumes=true|false (BETA - default=true)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
LegacyNodeRoleBehavior=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NodeDisruptionExclusion=true|false (BETA - default=true)
NonPreemptingPriority=true|false (BETA - default=true)
PodDisruptionBudget=true|false (BETA - default=true)
PodOverhead=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RootCAConfigMap=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (BETA - default=true)
ServerSideApply=true|false (BETA - default=true)
ServiceAccountIssuerDiscovery=true|false (BETA - default=true)
ServiceLBNodePortControl=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (BETA - default=true)
ServiceTopology=true|false (ALPHA - default=false)
SetHostnameAsFQDN=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (BETA - default=true)
WarningHeaders=true|false (BETA - default=true)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsEndpointSliceProxying=true|false (ALPHA - default=false)
(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--file-check-frequency duration     Default: `20s`
Duration between checking config files for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--hairpin-mode string     Default: `promiscuous-bridge`
How should the kubelet setup hairpin NAT. This allows endpoints of a Service to load balance back to themselves if they should try to access their own Service. Valid values are `promiscuous-bridge`, `hairpin-veth` and `none`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--healthz-bind-address ip     Default: `127.0.0.1`
The IP address for the healthz server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--healthz-port int32     Default: 10248
The port of the localhost healthz endpoint (set to `0` to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
-h, --help
help for kubelet
--hostname-override string
If non-empty, will use this string as identification instead of the actual hostname. If `--cloud-provider` is set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used).
--housekeeping-interval duration     Default: `10s`
Interval between container housekeepings.
--http-check-frequency duration     Default: `20s`
Duration between checking HTTP for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--image-credential-provider-bin-dir string
The path to the directory where credential provider plugin binaries are located.
--image-credential-provider-config string
The path to the credential provider plugin config file.
--image-gc-high-threshold int32     Default: 85
The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--image-gc-low-threshold int32     Default: 80
The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of `--image-gc-high-threshold`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--image-pull-progress-deadline duration     Default: `1m0s`
If no pulling progress is made before this deadline, the image pulling will be cancelled. This docker-specific flag only works when container-runtime is set to `docker`.
--image-service-endpoint string
[Experimental] The endpoint of remote image service. If not specified, it will be the same with `--container-runtime-endpoint` by default. Currently UNIX socket endpoint is supported on Linux, while npipe and TCP endpoints are supported on Windows. Examples: `unix:///var/run/dockershim.sock`, `npipe:////./pipe/dockershim`
--iptables-drop-bit int32     Default: 15
The bit of the `fwmark` space to mark packets for dropping. Must be within the range [0, 31]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--iptables-masquerade-bit int32     Default: 14
The bit of the `fwmark` space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in `kube-proxy`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--keep-terminated-pod-volumes
Keep terminated pod volumes mounted to the node after the pod terminates. Can be useful for debugging volume related issues. (DEPRECATED: will be removed in a future version)
--kernel-memcg-notification
If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--kube-api-burst int32     Default: 10
Burst to use while talking with kubernetes API server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--kube-api-content-type string     Default: `application/vnd.kubernetes.protobuf`
Content type of requests sent to apiserver. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--kube-api-qps int32     Default: 5
QPS to use while talking with kubernetes API server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--kube-reserved mapStringString     Default: <None>
A set of `=` (e.g. `cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100'`) pairs that describe resources reserved for kubernetes system components. Currently `cpu`, `memory` and local `ephemeral-storage` for root file system are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--kube-reserved-cgroup string     Default: `''`
Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via `--kube-reserved` flag. Ex. `/kube-reserved`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--kubeconfig string
Path to a kubeconfig file, specifying how to connect to the API server. Providing `--kubeconfig` enables API server mode, omitting `--kubeconfig` enables standalone mode.
--kubelet-cgroups string
Optional absolute name of cgroups to create and run the Kubelet in. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--lock-file string
<Warning: Alpha feature> The path to file for kubelet to use as a lock file.
--log-backtrace-at traceLocation     Default: `:0`
When logging hits line `:`, emit a stack trace.
--log-dir string
If non-empty, write log files in this directory
--log-file string
If non-empty, use this log file
--log-file-max-size uint     Default: 1800
Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.
--log-flush-frequency duration     Default: `5s`
Maximum number of seconds between log flushes.
--logging-format string     Default: `text`
Sets the log format. Permitted formats: `text`, `json`.\nNon-default formats don't honor these flags: `--add-dir-header`, `--alsologtostderr`, `--log-backtrace-at`, `--log-dir`, `--log-file`, `--log-file-max-size`, `--logtostderr`, `--skip_headers`, `--skip_log_headers`, `--stderrthreshold`, `--log-flush-frequency`.\nNon-default choices are currently alpha and subject to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--logtostderr     Default: `true`
log to standard error instead of files.
--make-iptables-util-chains     Default: `true`
If true, kubelet will ensure `iptables` utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--manifest-url string
URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--manifest-url-header string
Comma-separated list of HTTP headers to use when accessing the URL provided to `--manifest-url`. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: `--manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful'` (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--master-service-namespace string     Default: `default`
The namespace from which the kubernetes master services should be injected into pods. (DEPRECATED: This flag will be removed in a future version.)
--max-open-files int     Default: 1000000
Number of files that can be opened by Kubelet process. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--max-pods int32     Default: 110
Number of Pods that can run on this Kubelet. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--maximum-dead-containers int32     Default: -1
Maximum number of old instances of containers to retain globally. Each container takes up some disk space. To disable, set to a negative number. (DEPRECATED: Use `--eviction-hard` or `--eviction-soft` instead. Will be removed in a future version.)
--maximum-dead-containers-per-container int32     Default: 1
Maximum number of old instances to retain per container. Each container takes up some disk space. (DEPRECATED: Use `--eviction-hard` or `--eviction-soft` instead. Will be removed in a future version.)
--minimum-container-ttl-duration duration
Minimum age for a finished container before it is garbage collected. Examples: `300ms`, `10s` or `2h45m` (DEPRECATED: Use `--eviction-hard` or `--eviction-soft` instead. Will be removed in a future version.)
--minimum-image-ttl-duration duration     Default: `2m0s`
Minimum age for an unused image before it is garbage collected. Examples: `300ms`, `10s` or `2h45m`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--network-plugin string
<Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle. This docker-specific flag only works when container-runtime is set to `docker`.
--network-plugin-mtu int32
<Warning: Alpha feature> The MTU to be passed to the network plugin, to override the default. Set to `0` to use the default 1460 MTU. This docker-specific flag only works when container-runtime is set to `docker`.
--node-ip string
IP address of the node. If set, kubelet will use this IP address for the node
--node-labels mapStringString
<Warning: Alpha feature>Labels to add when registering the node in the cluster. Labels must be `key=value pairs` separated by `,`. Labels in the `kubernetes.io` namespace must begin with an allowed prefix (`kubelet.kubernetes.io`, `node.kubernetes.io`) or be in the specifically allowed set (`beta.kubernetes.io/arch`, `beta.kubernetes.io/instance-type`, `beta.kubernetes.io/os`, `failure-domain.beta.kubernetes.io/region`, `failure-domain.beta.kubernetes.io/zone`, `kubernetes.io/arch`, `kubernetes.io/hostname`, `kubernetes.io/os`, `node.kubernetes.io/instance-type`, `topology.kubernetes.io/region`, `topology.kubernetes.io/zone`)
--node-status-max-images int32     Default: 50
The maximum number of images to report in `node.status.images`. If `-1` is specified, no cap will be applied. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--node-status-update-frequency duration     Default: `10s`
Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in Node controller. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--non-masquerade-cidr string     Default: `10.0.0.0/8`
Traffic to IPs outside this range will use IP masquerade. Set to `0.0.0.0/0` to never masquerade. (DEPRECATED: will be removed in a future version)
--one-output
If true, only write logs to their native severity level (vs also writing to each lower severity level.
--oom-score-adj int32     Default: -999
The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--pod-cidr string
The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--pod-infra-container-image string     Default: `k8s.gcr.io/pause:3.2`
Specified image will not be pruned by the image garbage collector. When container-runtime is set to `docker`, all containers in each pod will use the network/ipc namespaces from this image. Other CRI implementations have their own configuration to set this image.
--pod-manifest-path string
Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--pod-max-pids int     Default: -1
Set the maximum number of processes per pod. If `-1`, the kubelet defaults to the node allocatable PID capacity. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--pods-per-core int32
Number of Pods per core that can run on this Kubelet. The total number of Pods on this Kubelet cannot exceed `--max-pods`, so `--max-pods` will be used if this calculation results in a larger number of Pods allowed on the Kubelet. A value of `0` disables this limit. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--port int32     Default: 10250
The port for the Kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--protect-kernel-defaults
Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--provider-id string
Unique identifier for identifying the node in a machine database, i.e cloud provider. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--qos-reserved mapStringString
<Warning: Alpha feature> A set of `=` (e.g. `memory=50%`) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. Requires the `QOSReserved` feature gate to be enabled. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--read-only-port int32     Default: 10255
The read-only port for the Kubelet to serve on with no authentication/authorization (set to `0` to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--really-crash-for-testing
If true, when panics occur crash. Intended for testing. (DEPRECATED: will be removed in a future version.)
--redirect-container-streaming
Enables container streaming redirect. If false, kubelet will proxy container streaming data between the API server and container runtime; if `true`, kubelet will return an HTTP redirect to the API server, and the API server will access container runtime directly. The proxy approach is more secure, but introduces some overhead. The redirect approach is more performant, but less secure because the connection between apiserver and container runtime may not be authenticated. (DEPRECATED: Container streaming redirection will be removed from the kubelet in v1.20, and this flag will be removed in v1.22. For more details, see http://git.k8s.io/enhancements/keps/sig-node/20191205-container-streaming-requests.md)
--register-node     Default: `true`
Register the node with the API server. If `--kubeconfig` is not provided, this flag is irrelevant, as the Kubelet won't have an API server to register with. Default to `true`.
--register-schedulable     Default: `true`
Register the node as schedulable. Won't have any effect if `--register-node` is false. (DEPRECATED: will be removed in a future version)
--register-with-taints mapStringString
Register the node with the given list of taints (comma separated `=:`). No-op if `--register-node` is `false`.
--registry-burst int32     Default: 10
Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding `--registry-qps`. Only used if `--registry-qps > 0`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--registry-qps int32     Default: 5
If > 0, limit registry pull QPS to this value. If `0`, unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--reserved-cpus string
A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in `--system-reserved` and `--kube-reserved`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--resolv-conf string     Default: `/etc/resolv.conf`
Resolver configuration file used as the basis for the container DNS resolution configuration. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--root-dir string     Default: `/var/lib/kubelet`
Directory path for managing kubelet files (volume mounts, etc).
--rotate-certificates
<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the `kube-apiserver` when the certificate expiration approaches. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--rotate-server-certificates
Auto-request and rotate the kubelet serving certificates by requesting new certificates from the `kube-apiserver` when the certificate expiration approaches. Requires the `RotateKubeletServerCertificate` feature gate to be enabled, and approval of the submitted `CertificateSigningRequest` objects. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--runonce
If `true`, exit after spawning pods from local manifests or remote urls. Exclusive with `--enable-server` (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--runtime-cgroups string
Optional absolute name of cgroups to create and run the runtime in.
--runtime-request-timeout duration     Default: `2m0s`
Timeout of all runtime requests except long running request - `pull`, `logs`, `exec` and `attach`. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--seccomp-profile-root string     Default: `/var/lib/kubelet/seccomp`
<Warning: Alpha feature> Directory path for seccomp profiles. (DEPRECATED: will be removed in 1.23, in favor of using the `/seccomp` directory)
--serialize-image-pulls     Default: `true`
Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an `aufs` storage backend. Issue #10959 has more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--skip-headers
If `true`, avoid header prefixes in the log messages
--skip-log-headers
If `true`, avoid headers when opening log files
--stderrthreshold int     Default: 2
logs at or above this threshold go to stderr.
--streaming-connection-idle-timeout duration     Default: `4h0m0s`
Maximum time a streaming connection can be idle before the connection is automatically closed. `0` indicates no timeout. Example: `5m`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--sync-frequency duration     Default: `1m0s`
Max period between synchronizing running containers and config. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--system-cgroups string
Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under `/`. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--system-reserved mapStringString     Default: \
A set of `=` (e.g. `cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100'`) pairs that describe resources reserved for non-kubernetes components. Currently only `cpu` and `memory` are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--system-reserved-cgroup string     Default: `''`
Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via `--system-reserved` flag. Ex. `/system-reserved`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--tls-cert-file string
File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If `--tls-cert-file` and `--tls-private-key-file` are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to `--cert-dir`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--tls-cipher-suites stringSlice
Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--tls-min-version string
Minimum TLS version supported. Possible values: `VersionTLS10`, `VersionTLS11`, `VersionTLS12`, `VersionTLS13` (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--tls-private-key-file string
File containing x509 private key matching `--tls-cert-file`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--topology-manager-policy string     Default: `none`
Topology Manager policy to use. Possible values: `none`, `best-effort`, `restricted`, `single-numa-node`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--topology-manager-scope string     Default: `container`
Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container' (default), 'pod'. (default "container") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
-v, --v Level
Number for the log level verbosity
--version version[=true]
Print version information and quit
--vmodule moduleSpec
Comma-separated list of `pattern=N` settings for file-filtered logging
--volume-plugin-dir string     Default: `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/`
The full path of the directory in which to search for additional third party volume plugins. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--volume-stats-agg-period duration     Default: `1m0s`
Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to `0`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

3 - kube-apiserver

Synopsis

The Kubernetes API server validates and configures data for the api objects which include pods, services, replicationcontrollers, and others. The API Server services REST operations and provides the frontend to the cluster's shared state through which all other components interact.

kube-apiserver [flags]

Options

--add_dir_header

If true, adds the file directory to the header of the log messages

--admission-control-config-file string

File with admission control configuration.

--advertise-address string

The IP address on which to advertise the apiserver to members of the cluster. This address must be reachable by the rest of the cluster. If blank, the --bind-address will be used. If --bind-address is unspecified, the host's default interface will be used.

--allow-metric-labels stringToString     Default: []

The map from metric-label to value allow-list of this label. The key's format is ,. The value's format is <allowed_value>,<allowed_value>...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.

--allow-privileged

If true, allow privileged containers. [default=false]

--alsologtostderr

log to standard error as well as files

--anonymous-auth     Default: true

Enables anonymous requests to the secure port of the API server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated.

--api-audiences strings

Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.

--apiserver-count int     Default: 1

The number of apiservers running in the cluster, must be a positive number. (In use when --endpoint-reconciler-type=master-count is enabled.)

--audit-log-batch-buffer-size int     Default: 10000

The size of the buffer to store events before batching and writing. Only used in batch mode.

--audit-log-batch-max-size int     Default: 1

The maximum size of a batch. Only used in batch mode.

--audit-log-batch-max-wait duration

The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.

--audit-log-batch-throttle-burst int

Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.

--audit-log-batch-throttle-enable

Whether batching throttling is enabled. Only used in batch mode.

--audit-log-batch-throttle-qps float

Maximum average number of batches per second. Only used in batch mode.

--audit-log-compress

If set, the rotated log files will be compressed using gzip.

--audit-log-format string     Default: "json"

Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Known formats are legacy,json.

--audit-log-maxage int

The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.

--audit-log-maxbackup int

The maximum number of old audit log files to retain.

--audit-log-maxsize int

The maximum size in megabytes of the audit log file before it gets rotated.

--audit-log-mode string     Default: "blocking"

Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict.

--audit-log-path string

If set, all requests coming to the apiserver will be logged to this file. '-' means standard out.

--audit-log-truncate-enabled

Whether event and batch truncating is enabled.

--audit-log-truncate-max-batch-size int     Default: 10485760

Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size.

--audit-log-truncate-max-event-size int     Default: 102400

Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded.

--audit-log-version string     Default: "audit.k8s.io/v1"

API group and version used for serializing audit events written to log.

--audit-policy-file string

Path to the file that defines the audit policy configuration.

--audit-webhook-batch-buffer-size int     Default: 10000

The size of the buffer to store events before batching and writing. Only used in batch mode.

--audit-webhook-batch-max-size int     Default: 400

The maximum size of a batch. Only used in batch mode.

--audit-webhook-batch-max-wait duration     Default: 30s

The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.

--audit-webhook-batch-throttle-burst int     Default: 15

Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.

--audit-webhook-batch-throttle-enable     Default: true

Whether batching throttling is enabled. Only used in batch mode.

--audit-webhook-batch-throttle-qps float     Default: 10

Maximum average number of batches per second. Only used in batch mode.

--audit-webhook-config-file string

Path to a kubeconfig formatted file that defines the audit webhook configuration.

--audit-webhook-initial-backoff duration     Default: 10s

The amount of time to wait before retrying the first failed request.

--audit-webhook-mode string     Default: "batch"

Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict.

--audit-webhook-truncate-enabled

Whether event and batch truncating is enabled.

--audit-webhook-truncate-max-batch-size int     Default: 10485760

Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size.

--audit-webhook-truncate-max-event-size int     Default: 102400

Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded.

--audit-webhook-version string     Default: "audit.k8s.io/v1"

API group and version used for serializing audit events written to webhook.

--authentication-token-webhook-cache-ttl duration     Default: 2m0s

The duration to cache responses from the webhook token authenticator.

--authentication-token-webhook-config-file string

File with webhook configuration for token authentication in kubeconfig format. The API server will query the remote service to determine authentication for bearer tokens.

--authentication-token-webhook-version string     Default: "v1beta1"

The API version of the authentication.k8s.io TokenReview to send to and expect from the webhook.

--authorization-mode strings     Default: "AlwaysAllow"

Ordered list of plug-ins to do authorization on secure port. Comma-delimited list of: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC,Node.

--authorization-policy-file string

File with authorization policy in json line by line format, used with --authorization-mode=ABAC, on the secure port.

--authorization-webhook-cache-authorized-ttl duration     Default: 5m0s

The duration to cache 'authorized' responses from the webhook authorizer.

--authorization-webhook-cache-unauthorized-ttl duration     Default: 30s

The duration to cache 'unauthorized' responses from the webhook authorizer.

--authorization-webhook-config-file string

File with webhook configuration in kubeconfig format, used with --authorization-mode=Webhook. The API server will query the remote service to determine access on the API server's secure port.

--authorization-webhook-version string     Default: "v1beta1"

The API version of the authorization.k8s.io SubjectAccessReview to send to and expect from the webhook.

--azure-container-registry-config string

Path to the file containing Azure container registry configuration information.

--bind-address string     Default: 0.0.0.0

The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used.

--cert-dir string     Default: "/var/run/kubernetes"

The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.

--client-ca-file string

If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.

--cloud-config string

The path to the cloud provider configuration file. Empty string for no configuration file.

--cloud-provider string

The provider for cloud services. Empty string for no provider.

--cloud-provider-gce-l7lb-src-cidrs cidrs     Default: 130.211.0.0/22,35.191.0.0/16

CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks

--contention-profiling

Enable lock contention profiling, if profiling is enabled

--cors-allowed-origins strings

List of allowed origins for CORS, comma separated. An allowed origin can be a regular expression to support subdomain matching. If this list is empty CORS will not be enabled.

--default-not-ready-toleration-seconds int     Default: 300

Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration.

--default-unreachable-toleration-seconds int     Default: 300

Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.

--default-watch-cache-size int     Default: 100

Default watch cache size. If zero, watch cache will be disabled for resources that do not have a default watch size set.

--delete-collection-workers int     Default: 1

Number of workers spawned for DeleteCollection call. These are used to speed up namespace cleanup.

--disable-admission-plugins strings

admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.

--disabled-metrics strings

This flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.

--egress-selector-config-file string

File with apiserver egress selector configuration.

--enable-admission-plugins strings

admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.

--enable-aggregator-routing

Turns on aggregator routing requests to endpoints IP rather than cluster IP.

--enable-bootstrap-token-auth

Enable to allow secrets of type 'bootstrap.kubernetes.io/token' in the 'kube-system' namespace to be used for TLS bootstrapping authentication.

--enable-garbage-collector     Default: true

Enables the generic garbage collector. MUST be synced with the corresponding flag of the kube-controller-manager.

--enable-priority-and-fairness     Default: true

If true and the APIPriorityAndFairness feature gate is enabled, replace the max-in-flight handler with an enhanced one that queues and dispatches with priority and fairness

--encryption-provider-config string

The file containing configuration for encryption providers to be used for storing secrets in etcd

--endpoint-reconciler-type string     Default: "lease"

Use an endpoint reconciler (master-count, lease, none)

--etcd-cafile string

SSL Certificate Authority file used to secure etcd communication.

--etcd-certfile string

SSL certification file used to secure etcd communication.

--etcd-compaction-interval duration     Default: 5m0s

The interval of compaction requests. If 0, the compaction request from apiserver is disabled.

--etcd-count-metric-poll-period duration     Default: 1m0s

Frequency of polling etcd for number of resources per type. 0 disables the metric collection.

--etcd-db-metric-poll-interval duration     Default: 30s

The interval of requests to poll etcd and update metric. 0 disables the metric collection

--etcd-healthcheck-timeout duration     Default: 2s

The timeout to use when checking etcd health.

--etcd-keyfile string

SSL key file used to secure etcd communication.

--etcd-prefix string     Default: "/registry"

The prefix to prepend to all resource paths in etcd.

--etcd-servers strings

List of etcd servers to connect with (scheme://ip:port), comma separated.

--etcd-servers-overrides strings

Per-resource etcd servers overrides, comma separated. The individual override format: group/resource#servers, where servers are URLs, semicolon separated. Note that this applies only to resources compiled into this server binary.

--event-ttl duration     Default: 1h0m0s

Amount of time to retain events.

--experimental-logging-sanitization

[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

--external-hostname string

The hostname to use when generating externalized URLs for this master (e.g. Swagger API Docs or OpenID Discovery).

--feature-gates <comma-separated 'key=True|False' pairs>

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BoundServiceAccountTokenVolume=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIMigrationvSphereComplete=true|false (BETA - default=false)
CSIServiceAccountToken=true|false (BETA - default=true)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (ALPHA - default=false)
CronJobControllerV2=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (ALPHA - default=false)
DefaultPodTopologySpread=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DownwardAPIHugePages=true|false (BETA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceProxying=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (ALPHA - default=false)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HugePageStorageMediumSize=true|false (BETA - default=true)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (ALPHA - default=false)
IngressClassNamespacedParams=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (ALPHA - default=false)
MemoryManager=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NamespaceDefaultLabelName=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (ALPHA - default=false)
PodDeletionCost=true|false (ALPHA - default=false)
PodOverhead=true|false (BETA - default=true)
PreferNominatedNode=true|false (ALPHA - default=false)
ProbeTerminationGracePeriod=true|false (ALPHA - default=false)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
ServerSideApply=true|false (BETA - default=true)
ServiceInternalTrafficPolicy=true|false (ALPHA - default=false)
ServiceLBNodePortControl=true|false (ALPHA - default=false)
ServiceLoadBalancerClass=true|false (ALPHA - default=false)
ServiceTopology=true|false (ALPHA - default=false)
SetHostnameAsFQDN=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (ALPHA - default=false)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WarningHeaders=true|false (BETA - default=true)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsEndpointSliceProxying=true|false (BETA - default=true)

--goaway-chance float

To prevent HTTP/2 clients from getting stuck on a single apiserver, randomly close a connection (GOAWAY). The client's other in-flight requests won't be affected, and the client will reconnect, likely landing on a different apiserver after going through the load balancer again. This argument sets the fraction of requests that will be sent a GOAWAY. Clusters with single apiservers, or which don't use a load balancer, should NOT enable this. Min is 0 (off), Max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.

-h, --help

help for kube-apiserver

--http2-max-streams-per-connection int

The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.

--identity-lease-duration-seconds int     Default: 3600

The duration of kube-apiserver lease in seconds, must be a positive number. (In use when the APIServerIdentity feature gate is enabled.)

--identity-lease-renew-interval-seconds int     Default: 10

The interval of kube-apiserver renewing its lease in seconds, must be a positive number. (In use when the APIServerIdentity feature gate is enabled.)

--kubelet-certificate-authority string

Path to a cert file for the certificate authority.

--kubelet-client-certificate string

Path to a client cert file for TLS.

--kubelet-client-key string

Path to a client key file for TLS.

--kubelet-preferred-address-types strings     Default: "Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP"

List of the preferred NodeAddressTypes to use for kubelet connections.

--kubelet-timeout duration     Default: 5s

Timeout for kubelet operations.

--kubernetes-service-node-port int

If non-zero, the Kubernetes master service (which apiserver creates/maintains) will be of type NodePort, using this as the value of the port. If zero, the Kubernetes master service will be of type ClusterIP.

--lease-reuse-duration-seconds int     Default: 60

The time in seconds that each lease is reused. A lower value could avoid large number of objects reusing the same lease. Notice that a too small value may cause performance problems at storage layer.

--livez-grace-period duration

This option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has elapsed, /livez will assume that unfinished post-start hooks will complete successfully and therefore return true.

--log-flush-frequency duration     Default: 5s

Maximum number of seconds between log flushes

--log_backtrace_at <a string in the form 'file:N'>     Default: :0

when logging hits line file:N, emit a stack trace

--log_dir string

If non-empty, write log files in this directory

--log_file string

If non-empty, use this log file

--log_file_max_size uint     Default: 1800

Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

--logging-format string     Default: "text"

Sets the log format. Permitted formats: "json", "text".
Non-default formats don't honor these flags: --add_dir_header, --alsologtostderr, --log_backtrace_at, --log_dir, --log_file, --log_file_max_size, --logtostderr, --one_output, --skip_headers, --skip_log_headers, --stderrthreshold, --vmodule, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning.

--logtostderr     Default: true

log to standard error instead of files

--master-service-namespace string     Default: "default"

DEPRECATED: the namespace from which the Kubernetes master services should be injected into pods.

--max-connection-bytes-per-sec int

If non-zero, throttle each user connection to this number of bytes/sec. Currently only applies to long-running requests.

--max-mutating-requests-inflight int     Default: 200

The maximum number of mutating requests in flight at a given time. When the server exceeds this, it rejects requests. Zero for no limit.

--max-requests-inflight int     Default: 400

The maximum number of non-mutating requests in flight at a given time. When the server exceeds this, it rejects requests. Zero for no limit.

--min-request-timeout int     Default: 1800

An optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. Currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.

--oidc-ca-file string

If set, the OpenID server's certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host's root CA set will be used.

--oidc-client-id string

The client ID for the OpenID Connect client, must be set if oidc-issuer-url is set.

--oidc-groups-claim string

If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This flag is experimental, please see the authentication documentation for further details.

--oidc-groups-prefix string

If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.

--oidc-issuer-url string

The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT).

--oidc-required-claim <comma-separated 'key=value' pairs>

A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. Repeat this flag to specify multiple claims.

--oidc-signing-algs strings     Default: "RS256"

Comma-separated list of allowed JOSE asymmetric signing algorithms. JWTs with a 'alg' header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1.

--oidc-username-claim string     Default: "sub"

The OpenID claim to use as the user name. Note that claims other than the default ('sub') is not guaranteed to be unique and immutable. This flag is experimental, please see the authentication documentation for further details.

--oidc-username-prefix string

If provided, all usernames will be prefixed with this value. If not provided, username claims other than 'email' are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value '-'.

--one_output

If true, only write logs to their native severity level (vs also writing to each lower severity level)

--permit-address-sharing

If true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]

--permit-port-sharing

If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]

--profiling     Default: true

Enable profiling via web interface host:port/debug/pprof/

--proxy-client-cert-file string

Client certificate used to prove the identity of the aggregator or kube-apiserver when it must call out during a request. This includes proxying requests to a user api-server and calling out to webhook admission plugins. It is expected that this cert includes a signature from the CA in the --requestheader-client-ca-file flag. That CA is published in the 'extension-apiserver-authentication' configmap in the kube-system namespace. Components receiving calls from kube-aggregator should use that CA to perform their half of the mutual TLS verification.

--proxy-client-key-file string

Private key for the client certificate used to prove the identity of the aggregator or kube-apiserver when it must call out during a request. This includes proxying requests to a user api-server and calling out to webhook admission plugins.

--request-timeout duration     Default: 1m0s

An optional field indicating the duration a handler must keep a request open before timing it out. This is the default request timeout for requests but may be overridden by flags such as --min-request-timeout for specific types of requests.

--requestheader-allowed-names strings

List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.

--requestheader-client-ca-file string

Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.

--requestheader-extra-headers-prefix strings

List of request header prefixes to inspect. X-Remote-Extra- is suggested.

--requestheader-group-headers strings

List of request headers to inspect for groups. X-Remote-Group is suggested.

--requestheader-username-headers strings

List of request headers to inspect for usernames. X-Remote-User is common.

--runtime-config <comma-separated 'key=value' pairs>

A set of key=value pairs that enable or disable built-in APIs. Supported options are:
v1=true|false for the core API group
/=true|false for a specific API group and version (e.g. apps/v1=true)
api/all=true|false controls all API versions
api/ga=true|false controls all API versions of the form v[0-9]+
api/beta=true|false controls all API versions of the form v[0-9]+beta[0-9]+
api/alpha=true|false controls all API versions of the form v[0-9]+alpha[0-9]+
api/legacy is deprecated, and will be removed in a future version

--secure-port int     Default: 6443

The port on which to serve HTTPS with authentication and authorization. It cannot be switched off with 0.

--service-account-extend-token-expiration     Default: true

Turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.

--service-account-issuer string

Identifier of the service account token issuer. The issuer will assert this identifier in "iss" claim of issued tokens. This value is a string or URI. If this option is not a valid URI per the OpenID Discovery 1.0 spec, the ServiceAccountIssuerDiscovery feature will remain disabled, even if the feature gate is set to true. It is highly recommended that this value comply with the OpenID spec: https://openid.net/specs/openid-connect-discovery-1_0.html. In practice, this means that service-account-issuer must be an https URL. It is also highly recommended that this URL be capable of serving OpenID discovery documents at {service-account-issuer}/.well-known/openid-configuration.

--service-account-jwks-uri string

Overrides the URI for the JSON Web Key Set in the discovery doc served at /.well-known/openid-configuration. This flag is useful if the discovery docand key set are served to relying parties from a URL other than the API server's external (as auto-detected or overridden with external-hostname). Only valid if the ServiceAccountIssuerDiscovery feature gate is enabled.

--service-account-key-file strings

File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key is provided

--service-account-lookup     Default: true

If true, validate ServiceAccount tokens exist in etcd as part of authentication.

--service-account-max-token-expiration duration

The maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value.

--service-account-signing-key-file string

Path to the file that contains the current private key of the service account token issuer. The issuer will sign issued ID tokens with this private key.

--service-cluster-ip-range string

A CIDR notation IP range from which to assign service cluster IPs. This must not overlap with any IP ranges assigned to nodes or pods. Max of two dual-stack CIDRs is allowed.

--service-node-port-range <a string in the form 'N1-N2'>     Default: 30000-32767

A port range to reserve for services with NodePort visibility. Example: '30000-32767'. Inclusive at both ends of the range.

--show-hidden-metrics-for-version string

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is ., e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

--shutdown-delay-duration duration

Time to delay the termination. During that time the server keeps serving requests normally. The endpoints /healthz and /livez will return success, but /readyz immediately returns failure. Graceful termination starts after this delay has elapsed. This can be used to allow load balancer to stop sending traffic to this server.

--skip_headers

If true, avoid header prefixes in the log messages

--skip_log_headers

If true, avoid headers when opening log files

--stderrthreshold int     Default: 2

logs at or above this threshold go to stderr

--storage-backend string

The storage backend for persistence. Options: 'etcd3' (default).

--storage-media-type string     Default: "application/vnd.kubernetes.protobuf"

The media type to use to store objects in storage. Some resources or storage backends may only support a specific media type and will ignore this setting.

--strict-transport-security-directives strings

List of directives for HSTS, comma separated. If this list is empty, then HSTS directives will not be added. Example: 'max-age=31536000,includeSubDomains,preload'

--tls-cert-file string

File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.

--tls-cipher-suites strings

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

--tls-min-version string

Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13

--tls-private-key-file string

File containing the default x509 private key matching --tls-cert-file.

--tls-sni-cert-key string

A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".

--token-auth-file string

If set, the file that will be used to secure the secure port of the API server via token authentication.

-v, --v int

number for the log level verbosity

--version version[=true]

Print version information and quit

--vmodule <comma-separated 'pattern=N' settings>

comma-separated list of pattern=N settings for file-filtered logging

--watch-cache     Default: true

Enable watch caching in the apiserver

--watch-cache-sizes strings

Watch cache size settings for some resources (pods, nodes, etc.), comma separated. The individual setting format: resource[.group]#size, where resource is lowercase plural (no version), group is omitted for resources of apiVersion v1 (the legacy core API) and included for others, and size is a number. It takes effect when watch-cache is enabled. Some resources (replicationcontrollers, endpoints, nodes, pods, services, apiservices.apiregistration.k8s.io) have system defaults set by heuristics, others default to default-watch-cache-size

4 - kube-controller-manager

Synopsis

The Kubernetes controller manager is a daemon that embeds the core control loops shipped with Kubernetes. In applications of robotics and automation, a control loop is a non-terminating loop that regulates the state of the system. In Kubernetes, a controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state. Examples of controllers that ship with Kubernetes today are the replication controller, endpoints controller, namespace controller, and serviceaccounts controller.

kube-controller-manager [flags]

Options

--add_dir_header

If true, adds the file directory to the header of the log messages

--allocate-node-cidrs

Should CIDRs for Pods be allocated and set on the cloud provider.

--allow-metric-labels stringToString     Default: []

The map from metric-label to value allow-list of this label. The key's format is ,. The value's format is <allowed_value>,<allowed_value>...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.

--alsologtostderr

log to standard error as well as files

--attach-detach-reconcile-sync-period duration     Default: 1m0s

The reconciler sync wait time between volume attach detach. This duration must be larger than one second, and increasing this value from the default may allow for volumes to be mismatched with pods.

--authentication-kubeconfig string

kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no client CA is looked up in the cluster.

--authentication-skip-lookup

If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.

--authentication-token-webhook-cache-ttl duration     Default: 10s

The duration to cache responses from the webhook token authenticator.

--authentication-tolerate-lookup-failure

If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.

--authorization-always-allow-paths strings     Default: "/healthz,/readyz,/livez"

A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.

--authorization-kubeconfig string

kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped by authorization are forbidden.

--authorization-webhook-cache-authorized-ttl duration     Default: 10s

The duration to cache 'authorized' responses from the webhook authorizer.

--authorization-webhook-cache-unauthorized-ttl duration     Default: 10s

The duration to cache 'unauthorized' responses from the webhook authorizer.

--azure-container-registry-config string

Path to the file containing Azure container registry configuration information.

--bind-address string     Default: 0.0.0.0

The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used.

--cert-dir string

The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.

--cidr-allocator-type string     Default: "RangeAllocator"

Type of CIDR allocator to use

--client-ca-file string

If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.

--cloud-config string

The path to the cloud provider configuration file. Empty string for no configuration file.

--cloud-provider string

The provider for cloud services. Empty string for no provider.

--cluster-cidr string

CIDR Range for Pods in cluster. Requires --allocate-node-cidrs to be true

--cluster-name string     Default: "kubernetes"

The instance prefix for the cluster.

--cluster-signing-cert-file string

Filename containing a PEM-encoded X509 CA certificate used to issue cluster-scoped certificates. If specified, no more specific --cluster-signing-* flag may be specified.

--cluster-signing-duration duration     Default: 8760h0m0s

The length of duration signed certificates will be given.

--cluster-signing-key-file string

Filename containing a PEM-encoded RSA or ECDSA private key used to sign cluster-scoped certificates. If specified, no more specific --cluster-signing-* flag may be specified.

--cluster-signing-kube-apiserver-client-cert-file string

Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/kube-apiserver-client signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--cluster-signing-kube-apiserver-client-key-file string

Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/kube-apiserver-client signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--cluster-signing-kubelet-client-cert-file string

Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/kube-apiserver-client-kubelet signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--cluster-signing-kubelet-client-key-file string

Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/kube-apiserver-client-kubelet signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--cluster-signing-kubelet-serving-cert-file string

Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/kubelet-serving signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--cluster-signing-kubelet-serving-key-file string

Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/kubelet-serving signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--cluster-signing-legacy-unknown-cert-file string

Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/legacy-unknown signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--cluster-signing-legacy-unknown-key-file string

Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/legacy-unknown signer. If specified, --cluster-signing-{cert,key}-file must not be set.

--concurrent-deployment-syncs int32     Default: 5

The number of deployment objects that are allowed to sync concurrently. Larger number = more responsive deployments, but more CPU (and network) load

--concurrent-endpoint-syncs int32     Default: 5

The number of endpoint syncing operations that will be done concurrently. Larger number = faster endpoint updating, but more CPU (and network) load

--concurrent-gc-syncs int32     Default: 20

The number of garbage collector workers that are allowed to sync concurrently.

--concurrent-namespace-syncs int32     Default: 10

The number of namespace objects that are allowed to sync concurrently. Larger number = more responsive namespace termination, but more CPU (and network) load

--concurrent-replicaset-syncs int32     Default: 5

The number of replica sets that are allowed to sync concurrently. Larger number = more responsive replica management, but more CPU (and network) load

--concurrent-resource-quota-syncs int32     Default: 5

The number of resource quotas that are allowed to sync concurrently. Larger number = more responsive quota management, but more CPU (and network) load

--concurrent-service-endpoint-syncs int32     Default: 5

The number of service endpoint syncing operations that will be done concurrently. Larger number = faster endpoint slice updating, but more CPU (and network) load. Defaults to 5.

--concurrent-service-syncs int32     Default: 1

The number of services that are allowed to sync concurrently. Larger number = more responsive service management, but more CPU (and network) load

--concurrent-serviceaccount-token-syncs int32     Default: 5

The number of service account token objects that are allowed to sync concurrently. Larger number = more responsive token generation, but more CPU (and network) load

--concurrent-statefulset-syncs int32     Default: 5

The number of statefulset objects that are allowed to sync concurrently. Larger number = more responsive statefulsets, but more CPU (and network) load

--concurrent-ttl-after-finished-syncs int32     Default: 5

The number of TTL-after-finished controller workers that are allowed to sync concurrently.

--concurrent_rc_syncs int32     Default: 5

The number of replication controllers that are allowed to sync concurrently. Larger number = more responsive replica management, but more CPU (and network) load

--configure-cloud-routes     Default: true

Should CIDRs allocated by allocate-node-cidrs be configured on the cloud provider.

--contention-profiling

Enable lock contention profiling, if profiling is enabled

--controller-start-interval duration

Interval between starting controller managers.

--controllers strings     Default: "*"

A list of controllers to enable. '*' enables all on-by-default controllers, 'foo' enables the controller named 'foo', '-foo' disables the controller named 'foo'.
All controllers: attachdetach, bootstrapsigner, cloud-node-lifecycle, clusterrole-aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, endpointslice, endpointslicemirroring, ephemeral-volume, garbagecollector, horizontalpodautoscaling, job, namespace, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv-protection, pvc-protection, replicaset, replicationcontroller, resourcequota, root-ca-cert-publisher, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after-finished
Disabled-by-default controllers: bootstrapsigner, tokencleaner

--deployment-controller-sync-period duration     Default: 30s

Period for syncing the deployments.

--disable-attach-detach-reconcile-sync

Disable volume attach detach reconciler sync. Disabling this may cause volumes to be mismatched with pods. Use wisely.

--disabled-metrics strings

This flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.

--enable-dynamic-provisioning     Default: true

Enable dynamic provisioning for environments that support it.

--enable-garbage-collector     Default: true

Enables the generic garbage collector. MUST be synced with the corresponding flag of the kube-apiserver.

--enable-hostpath-provisioner

Enable HostPath PV provisioning when running without a cloud provider. This allows testing and development of provisioning features. HostPath provisioning is not supported in any way, won't work in a multi-node cluster, and should not be used for anything other than testing or development.

--enable-leader-migration

Whether to enable controller leader migration.

--enable-taint-manager     Default: true

WARNING: Beta feature. If set to true enables NoExecute Taints and will evict all not-tolerating Pod running on Nodes tainted with this kind of Taints.

--endpoint-updates-batch-period duration

The length of endpoint updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated

--endpointslice-updates-batch-period duration

The length of endpoint slice updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated

--experimental-logging-sanitization

[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

--external-cloud-volume-plugin string

The plugin to use when cloud provider is set to external. Can be empty, should only be set when cloud-provider is external. Currently used to allow node and volume controllers to work for in tree cloud providers.

--feature-gates <comma-separated 'key=True|False' pairs>

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BoundServiceAccountTokenVolume=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIMigrationvSphereComplete=true|false (BETA - default=false)
CSIServiceAccountToken=true|false (BETA - default=true)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (ALPHA - default=false)
CronJobControllerV2=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (ALPHA - default=false)
DefaultPodTopologySpread=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DownwardAPIHugePages=true|false (BETA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceProxying=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (ALPHA - default=false)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HugePageStorageMediumSize=true|false (BETA - default=true)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (ALPHA - default=false)
IngressClassNamespacedParams=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (ALPHA - default=false)
MemoryManager=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NamespaceDefaultLabelName=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (ALPHA - default=false)
PodDeletionCost=true|false (ALPHA - default=false)
PodOverhead=true|false (BETA - default=true)
PreferNominatedNode=true|false (ALPHA - default=false)
ProbeTerminationGracePeriod=true|false (ALPHA - default=false)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
ServerSideApply=true|false (BETA - default=true)
ServiceInternalTrafficPolicy=true|false (ALPHA - default=false)
ServiceLBNodePortControl=true|false (ALPHA - default=false)
ServiceLoadBalancerClass=true|false (ALPHA - default=false)
ServiceTopology=true|false (ALPHA - default=false)
SetHostnameAsFQDN=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (ALPHA - default=false)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WarningHeaders=true|false (BETA - default=true)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsEndpointSliceProxying=true|false (BETA - default=true)

--flex-volume-plugin-dir string     Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"

Full path of the directory in which the flex volume plugin should search for additional third party volume plugins.

-h, --help

help for kube-controller-manager

--horizontal-pod-autoscaler-cpu-initialization-period duration     Default: 5m0s

The period after pod start when CPU samples might be skipped.

--horizontal-pod-autoscaler-downscale-stabilization duration     Default: 5m0s

The period for which autoscaler will look backwards and not scale down below any recommendation it made during that period.

--horizontal-pod-autoscaler-initial-readiness-delay duration     Default: 30s

The period after pod start during which readiness changes will be treated as initial readiness.

--horizontal-pod-autoscaler-sync-period duration     Default: 15s

The period for syncing the number of pods in horizontal pod autoscaler.

--horizontal-pod-autoscaler-tolerance float     Default: 0.1

The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling.

--http2-max-streams-per-connection int

The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.

--kube-api-burst int32     Default: 30

Burst to use while talking with kubernetes apiserver.

--kube-api-content-type string     Default: "application/vnd.kubernetes.protobuf"

Content type of requests sent to apiserver.

--kube-api-qps float     Default: 20

QPS to use while talking with kubernetes apiserver.

--kubeconfig string

Path to kubeconfig file with authorization and master location information.

--large-cluster-size-threshold int32     Default: 50

Number of nodes from which NodeController treats the cluster as large for the eviction logic purposes. --secondary-node-eviction-rate is implicitly overridden to 0 for clusters this size or smaller.

--leader-elect     Default: true

Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability.

--leader-elect-lease-duration duration     Default: 15s

The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.

--leader-elect-renew-deadline duration     Default: 10s

The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled.

--leader-elect-resource-lock string     Default: "leases"

The type of resource object that is used for locking during leader election. Supported options are 'endpoints', 'configmaps', 'leases', 'endpointsleases' and 'configmapsleases'.

--leader-elect-resource-name string     Default: "kube-controller-manager"

The name of resource object that is used for locking during leader election.

--leader-elect-resource-namespace string     Default: "kube-system"

The namespace of resource object that is used for locking during leader election.

--leader-elect-retry-period duration     Default: 2s

The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.

--leader-migration-config string

Path to the config file for controller leader migration, or empty to use the value that reflects default configuration of the controller manager. The config file should be of type LeaderMigrationConfiguration, group controllermanager.config.k8s.io, version v1alpha1.

--log-flush-frequency duration     Default: 5s

Maximum number of seconds between log flushes

--log_backtrace_at <a string in the form 'file:N'>     Default: :0

when logging hits line file:N, emit a stack trace

--log_dir string

If non-empty, write log files in this directory

--log_file string

If non-empty, use this log file

--log_file_max_size uint     Default: 1800

Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

--logging-format string     Default: "text"

Sets the log format. Permitted formats: "json", "text".
Non-default formats don't honor these flags: --add_dir_header, --alsologtostderr, --log_backtrace_at, --log_dir, --log_file, --log_file_max_size, --logtostderr, --one_output, --skip_headers, --skip_log_headers, --stderrthreshold, --vmodule, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning.

--logtostderr     Default: true

log to standard error instead of files

--master string

The address of the Kubernetes API server (overrides any value in kubeconfig).

--max-endpoints-per-slice int32     Default: 100

The maximum number of endpoints that will be added to an EndpointSlice. More endpoints per slice will result in less endpoint slices, but larger resources. Defaults to 100.

--min-resync-period duration     Default: 12h0m0s

The resync period in reflectors will be random between MinResyncPeriod and 2*MinResyncPeriod.

--mirroring-concurrent-service-endpoint-syncs int32     Default: 5

The number of service endpoint syncing operations that will be done concurrently by the EndpointSliceMirroring controller. Larger number = faster endpoint slice updating, but more CPU (and network) load. Defaults to 5.

--mirroring-endpointslice-updates-batch-period duration

The length of EndpointSlice updates batching period for EndpointSliceMirroring controller. Processing of EndpointSlice changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of EndpointSlice updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated

--mirroring-max-endpoints-per-subset int32     Default: 1000

The maximum number of endpoints that will be added to an EndpointSlice by the EndpointSliceMirroring controller. More endpoints per slice will result in less endpoint slices, but larger resources. Defaults to 100.

--namespace-sync-period duration     Default: 5m0s

The period for syncing namespace life-cycle updates

--node-cidr-mask-size int32

Mask size for node cidr in cluster. Default is 24 for IPv4 and 64 for IPv6.

--node-cidr-mask-size-ipv4 int32

Mask size for IPv4 node cidr in dual-stack cluster. Default is 24.

--node-cidr-mask-size-ipv6 int32

Mask size for IPv6 node cidr in dual-stack cluster. Default is 64.

--node-eviction-rate float     Default: 0.1

Number of nodes per second on which pods are deleted in case of node failure when a zone is healthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters.

--node-monitor-grace-period duration     Default: 40s

Amount of time which we allow running Node to be unresponsive before marking it unhealthy. Must be N times more than kubelet's nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet to post node status.

--node-monitor-period duration     Default: 5s

The period for syncing NodeStatus in NodeController.

--node-startup-grace-period duration     Default: 1m0s

Amount of time which we allow starting Node to be unresponsive before marking it unhealthy.

--one_output

If true, only write logs to their native severity level (vs also writing to each lower severity level)

--permit-address-sharing

If true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]

--permit-port-sharing

If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]

--pod-eviction-timeout duration     Default: 5m0s

The grace period for deleting pods on failed nodes.

--profiling     Default: true

Enable profiling via web interface host:port/debug/pprof/

--pv-recycler-increment-timeout-nfs int32     Default: 30

the increment of time added per Gi to ActiveDeadlineSeconds for an NFS scrubber pod

--pv-recycler-minimum-timeout-hostpath int32     Default: 60

The minimum ActiveDeadlineSeconds to use for a HostPath Recycler pod. This is for development and testing only and will not work in a multi-node cluster.

--pv-recycler-minimum-timeout-nfs int32     Default: 300

The minimum ActiveDeadlineSeconds to use for an NFS Recycler pod

--pv-recycler-pod-template-filepath-hostpath string

The file path to a pod definition used as a template for HostPath persistent volume recycling. This is for development and testing only and will not work in a multi-node cluster.

--pv-recycler-pod-template-filepath-nfs string

The file path to a pod definition used as a template for NFS persistent volume recycling

--pv-recycler-timeout-increment-hostpath int32     Default: 30

the increment of time added per Gi to ActiveDeadlineSeconds for a HostPath scrubber pod. This is for development and testing only and will not work in a multi-node cluster.

--pvclaimbinder-sync-period duration     Default: 15s

The period for syncing persistent volumes and persistent volume claims

--requestheader-allowed-names strings

List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.

--requestheader-client-ca-file string

Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.

--requestheader-extra-headers-prefix strings     Default: "x-remote-extra-"

List of request header prefixes to inspect. X-Remote-Extra- is suggested.

--requestheader-group-headers strings     Default: "x-remote-group"

List of request headers to inspect for groups. X-Remote-Group is suggested.

--requestheader-username-headers strings     Default: "x-remote-user"

List of request headers to inspect for usernames. X-Remote-User is common.

--resource-quota-sync-period duration     Default: 5m0s

The period for syncing quota usage status in the system

--root-ca-file string

If set, this root certificate authority will be included in service account's token secret. This must be a valid PEM-encoded CA bundle.

--route-reconciliation-period duration     Default: 10s

The period for reconciling routes created for Nodes by cloud provider.

--secondary-node-eviction-rate float     Default: 0.01

Number of nodes per second on which pods are deleted in case of node failure when a zone is unhealthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters. This value is implicitly overridden to 0 if the cluster size is smaller than --large-cluster-size-threshold.

--secure-port int     Default: 10257

The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all.

--service-account-private-key-file string

Filename containing a PEM-encoded private RSA or ECDSA key used to sign service account tokens.

--service-cluster-ip-range string

CIDR Range for Services in cluster. Requires --allocate-node-cidrs to be true

--show-hidden-metrics-for-version string

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is ., e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

--skip_headers

If true, avoid header prefixes in the log messages

--skip_log_headers

If true, avoid headers when opening log files

--stderrthreshold int     Default: 2

logs at or above this threshold go to stderr

--terminated-pod-gc-threshold int32     Default: 12500

Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If <= 0, the terminated pod garbage collector is disabled.

--tls-cert-file string

File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.

--tls-cipher-suites strings

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

--tls-min-version string

Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13

--tls-private-key-file string

File containing the default x509 private key matching --tls-cert-file.

--tls-sni-cert-key string

A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".

--unhealthy-zone-threshold float     Default: 0.55

Fraction of Nodes in a zone which needs to be not Ready (minimum 3) for zone to be treated as unhealthy.

--use-service-account-credentials

If true, use individual service account credentials for each controller.

-v, --v int

number for the log level verbosity

--version version[=true]

Print version information and quit

--vmodule <comma-separated 'pattern=N' settings>

comma-separated list of pattern=N settings for file-filtered logging

--volume-host-allow-local-loopback     Default: true

If false, deny local loopback IPs in addition to any CIDR ranges in --volume-host-cidr-denylist

--volume-host-cidr-denylist strings

A comma-separated list of CIDR ranges to avoid from volume plugins.

5 - kube-proxy

Synopsis

The Kubernetes network proxy runs on each node. This reflects services as defined in the Kubernetes API on each node and can do simple TCP, UDP, and SCTP stream forwarding or round robin TCP, UDP, and SCTP forwarding across a set of backends. Service cluster IPs and ports are currently found through Docker-links-compatible environment variables specifying ports opened by the service proxy. There is an optional addon that provides cluster DNS for these cluster IPs. The user must create a service with the apiserver API to configure the proxy.

kube-proxy [flags]

Options

--azure-container-registry-config string

Path to the file containing Azure container registry configuration information.

--bind-address string     Default: 0.0.0.0

The IP address for the proxy server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces)

--bind-address-hard-fail

If true kube-proxy will treat failure to bind to a port as fatal and exit

--cleanup

If true cleanup iptables and ipvs rules and exit.

--cluster-cidr string

The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead

--config string

The path to the configuration file.

--config-sync-period duration     Default: 15m0s

How often configuration from the apiserver is refreshed. Must be greater than 0.

--conntrack-max-per-core int32     Default: 32768

Maximum number of NAT connections to track per CPU core (0 to leave the limit as-is and ignore conntrack-min).

--conntrack-min int32     Default: 131072

Minimum number of conntrack entries to allocate, regardless of conntrack-max-per-core (set conntrack-max-per-core=0 to leave the limit as-is).

--conntrack-tcp-timeout-close-wait duration     Default: 1h0m0s

NAT timeout for TCP connections in the CLOSE_WAIT state

--conntrack-tcp-timeout-established duration     Default: 24h0m0s

Idle timeout for established TCP connections (0 to leave as-is)

--detect-local-mode LocalMode

Mode to use to detect local traffic

--feature-gates <comma-separated 'key=True|False' pairs>

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BoundServiceAccountTokenVolume=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIMigrationvSphereComplete=true|false (BETA - default=false)
CSIServiceAccountToken=true|false (BETA - default=true)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (ALPHA - default=false)
CronJobControllerV2=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (ALPHA - default=false)
DefaultPodTopologySpread=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DownwardAPIHugePages=true|false (BETA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceProxying=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (ALPHA - default=false)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HugePageStorageMediumSize=true|false (BETA - default=true)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (ALPHA - default=false)
IngressClassNamespacedParams=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (ALPHA - default=false)
MemoryManager=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NamespaceDefaultLabelName=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (ALPHA - default=false)
PodDeletionCost=true|false (ALPHA - default=false)
PodOverhead=true|false (BETA - default=true)
PreferNominatedNode=true|false (ALPHA - default=false)
ProbeTerminationGracePeriod=true|false (ALPHA - default=false)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
ServerSideApply=true|false (BETA - default=true)
ServiceInternalTrafficPolicy=true|false (ALPHA - default=false)
ServiceLBNodePortControl=true|false (ALPHA - default=false)
ServiceLoadBalancerClass=true|false (ALPHA - default=false)
ServiceTopology=true|false (ALPHA - default=false)
SetHostnameAsFQDN=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (ALPHA - default=false)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WarningHeaders=true|false (BETA - default=true)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsEndpointSliceProxying=true|false (BETA - default=true)

--healthz-bind-address ipport     Default: 0.0.0.0:10256

The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '[::]:10256' for all IPv6 interfaces). Set empty to disable.

-h, --help

help for kube-proxy

--hostname-override string

If non-empty, will use this string as identification instead of the actual hostname.

--iptables-masquerade-bit int32     Default: 14

If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with. Must be within the range [0, 31].

--iptables-min-sync-period duration     Default: 1s

The minimum interval of how often the iptables rules can be refreshed as endpoints and services change (e.g. '5s', '1m', '2h22m').

--iptables-sync-period duration     Default: 30s

The maximum interval of how often iptables rules are refreshed (e.g. '5s', '1m', '2h22m'). Must be greater than 0.

--ipvs-exclude-cidrs strings

A comma-separated list of CIDR's which the ipvs proxier should not touch when cleaning up IPVS rules.

--ipvs-min-sync-period duration

The minimum interval of how often the ipvs rules can be refreshed as endpoints and services change (e.g. '5s', '1m', '2h22m').

--ipvs-scheduler string

The ipvs scheduler type when proxy mode is ipvs

--ipvs-strict-arp

Enable strict ARP by setting arp_ignore to 1 and arp_announce to 2

--ipvs-sync-period duration     Default: 30s

The maximum interval of how often ipvs rules are refreshed (e.g. '5s', '1m', '2h22m'). Must be greater than 0.

--ipvs-tcp-timeout duration

The timeout for idle IPVS TCP connections, 0 to leave as-is. (e.g. '5s', '1m', '2h22m').

--ipvs-tcpfin-timeout duration

The timeout for IPVS TCP connections after receiving a FIN packet, 0 to leave as-is. (e.g. '5s', '1m', '2h22m').

--ipvs-udp-timeout duration

The timeout for IPVS UDP packets, 0 to leave as-is. (e.g. '5s', '1m', '2h22m').

--kube-api-burst int32     Default: 10

Burst to use while talking with kubernetes apiserver

--kube-api-content-type string     Default: "application/vnd.kubernetes.protobuf"

Content type of requests sent to apiserver.

--kube-api-qps float     Default: 5

QPS to use while talking with kubernetes apiserver

--kubeconfig string

Path to kubeconfig file with authorization information (the master location can be overridden by the master flag).

--log-flush-frequency duration     Default: 5s

Maximum number of seconds between log flushes

--masquerade-all

If using the pure iptables proxy, SNAT all traffic sent via Service cluster IPs (this not commonly needed)

--master string

The address of the Kubernetes API server (overrides any value in kubeconfig)

--metrics-bind-address ipport     Default: 127.0.0.1:10249

The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable.

--nodeport-addresses strings

A string slice of values which specify the addresses to use for NodePorts. Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32). The default empty string slice ([]) means to use all local addresses.

--oom-score-adj int32     Default: -999

The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]

--profiling

If true enables profiling via web interface on /debug/pprof handler.

--proxy-mode ProxyMode

Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' or 'kernelspace' (windows). If blank, use the best-available proxy (currently iptables). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.

--proxy-port-range port-range

Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.

--show-hidden-metrics-for-version string

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is ., e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

--udp-timeout duration     Default: 250ms

How long an idle UDP connection will be kept open (e.g. '250ms', '2s'). Must be greater than 0. Only applicable for proxy-mode=userspace

--version version[=true]

Print version information and quit

--write-config-to string

If set, write the default configuration values to this file and exit.

6 - kube-scheduler

Synopsis

The Kubernetes scheduler is a control plane process which assigns Pods to Nodes. The scheduler determines which Nodes are valid placements for each Pod in the scheduling queue according to constraints and available resources. The scheduler then ranks each valid Node and binds the Pod to a suitable Node. Multiple different schedulers may be used within a cluster; kube-scheduler is the reference implementation. See scheduling for more information about scheduling and the kube-scheduler component.

kube-scheduler [flags]

Options

--add_dir_header

If true, adds the file directory to the header of the log messages

--address string     Default: "0.0.0.0"

DEPRECATED: the IP address on which to listen for the --port port (set to 0.0.0.0 or :: for listening in all interfaces and IP families). See --bind-address instead. This parameter is ignored if a config file is specified in --config.

--algorithm-provider string

DEPRECATED: the scheduling algorithm provider to use, this sets the default plugins for component config profiles. Choose one of: ClusterAutoscalerProvider | DefaultProvider

--allow-metric-labels stringToString     Default: []

The map from metric-label to value allow-list of this label. The key's format is ,. The value's format is <allowed_value>,<allowed_value>...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.

--alsologtostderr

log to standard error as well as files

--authentication-kubeconfig string

kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no client CA is looked up in the cluster.

--authentication-skip-lookup

If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.

--authentication-token-webhook-cache-ttl duration     Default: 10s

The duration to cache responses from the webhook token authenticator.

--authentication-tolerate-lookup-failure     Default: true

If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.

--authorization-always-allow-paths strings     Default: "/healthz,/readyz,/livez"

A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.

--authorization-kubeconfig string

kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped by authorization are forbidden.

--authorization-webhook-cache-authorized-ttl duration     Default: 10s

The duration to cache 'authorized' responses from the webhook authorizer.

--authorization-webhook-cache-unauthorized-ttl duration     Default: 10s

The duration to cache 'unauthorized' responses from the webhook authorizer.

--azure-container-registry-config string

Path to the file containing Azure container registry configuration information.

--bind-address string     Default: 0.0.0.0

The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used.

--cert-dir string

The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.

--client-ca-file string

If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.

--config string

The path to the configuration file. The following flags can overwrite fields in this file:
--algorithm-provider
--policy-config-file
--policy-configmap
--policy-configmap-namespace

--contention-profiling     Default: true

DEPRECATED: enable lock contention profiling, if profiling is enabled. This parameter is ignored if a config file is specified in --config.

--disabled-metrics strings

This flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.

--experimental-logging-sanitization

[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

--feature-gates <comma-separated 'key=True|False' pairs>

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BoundServiceAccountTokenVolume=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIMigrationvSphereComplete=true|false (BETA - default=false)
CSIServiceAccountToken=true|false (BETA - default=true)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (ALPHA - default=false)
CronJobControllerV2=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (ALPHA - default=false)
DefaultPodTopologySpread=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DownwardAPIHugePages=true|false (BETA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceProxying=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (ALPHA - default=false)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HugePageStorageMediumSize=true|false (BETA - default=true)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (ALPHA - default=false)
IngressClassNamespacedParams=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (ALPHA - default=false)
MemoryManager=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NamespaceDefaultLabelName=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (ALPHA - default=false)
PodDeletionCost=true|false (ALPHA - default=false)
PodOverhead=true|false (BETA - default=true)
PreferNominatedNode=true|false (ALPHA - default=false)
ProbeTerminationGracePeriod=true|false (ALPHA - default=false)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
ServerSideApply=true|false (BETA - default=true)
ServiceInternalTrafficPolicy=true|false (ALPHA - default=false)
ServiceLBNodePortControl=true|false (ALPHA - default=false)
ServiceLoadBalancerClass=true|false (ALPHA - default=false)
ServiceTopology=true|false (ALPHA - default=false)
SetHostnameAsFQDN=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (ALPHA - default=false)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WarningHeaders=true|false (BETA - default=true)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsEndpointSliceProxying=true|false (BETA - default=true)

--hard-pod-affinity-symmetric-weight int32     Default: 1

DEPRECATED: RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule corresponding to every RequiredDuringScheduling affinity rule. --hard-pod-affinity-symmetric-weight represents the weight of implicit PreferredDuringScheduling affinity rule. Must be in the range 0-100.This parameter is ignored if a config file is specified in --config.

-h, --help

help for kube-scheduler

--http2-max-streams-per-connection int

The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.

--kube-api-burst int32     Default: 100

DEPRECATED: burst to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.

--kube-api-content-type string     Default: "application/vnd.kubernetes.protobuf"

DEPRECATED: content type of requests sent to apiserver. This parameter is ignored if a config file is specified in --config.

--kube-api-qps float     Default: 50

DEPRECATED: QPS to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.

--kubeconfig string

DEPRECATED: path to kubeconfig file with authorization and master location information. This parameter is ignored if a config file is specified in --config.

--leader-elect     Default: true

Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability.

--leader-elect-lease-duration duration     Default: 15s

The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.

--leader-elect-renew-deadline duration     Default: 10s

The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled.

--leader-elect-resource-lock string     Default: "leases"

The type of resource object that is used for locking during leader election. Supported options are 'endpoints', 'configmaps', 'leases', 'endpointsleases' and 'configmapsleases'.

--leader-elect-resource-name string     Default: "kube-scheduler"

The name of resource object that is used for locking during leader election.

--leader-elect-resource-namespace string     Default: "kube-system"

The namespace of resource object that is used for locking during leader election.

--leader-elect-retry-period duration     Default: 2s

The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.

--lock-object-name string     Default: "kube-scheduler"

DEPRECATED: define the name of the lock object. Will be removed in favor of leader-elect-resource-name. This parameter is ignored if a config file is specified in --config.

--lock-object-namespace string     Default: "kube-system"

DEPRECATED: define the namespace of the lock object. Will be removed in favor of leader-elect-resource-namespace. This parameter is ignored if a config file is specified in --config.

--log-flush-frequency duration     Default: 5s

Maximum number of seconds between log flushes

--log_backtrace_at <a string in the form 'file:N'>     Default: :0

when logging hits line file:N, emit a stack trace

--log_dir string

If non-empty, write log files in this directory

--log_file string

If non-empty, use this log file

--log_file_max_size uint     Default: 1800

Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

--logging-format string     Default: "text"

Sets the log format. Permitted formats: "json", "text".
Non-default formats don't honor these flags: --add_dir_header, --alsologtostderr, --log_backtrace_at, --log_dir, --log_file, --log_file_max_size, --logtostderr, --one_output, --skip_headers, --skip_log_headers, --stderrthreshold, --vmodule, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning.

--logtostderr     Default: true

log to standard error instead of files

--master string

The address of the Kubernetes API server (overrides any value in kubeconfig)

--one_output

If true, only write logs to their native severity level (vs also writing to each lower severity level)

--permit-address-sharing

If true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]

--permit-port-sharing

If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]

--policy-config-file string

DEPRECATED: file with scheduler policy configuration. This file is used if policy ConfigMap is not provided or --use-legacy-policy-config=true. Note: The scheduler will fail if this is combined with Plugin configs

--policy-configmap string

DEPRECATED: name of the ConfigMap object that contains scheduler's policy configuration. It must exist in the system namespace before scheduler initialization if --use-legacy-policy-config=false. The config must be provided as the value of an element in 'Data' map with the key='policy.cfg'. Note: The scheduler will fail if this is combined with Plugin configs

--policy-configmap-namespace string     Default: "kube-system"

DEPRECATED: the namespace where policy ConfigMap is located. The kube-system namespace will be used if this is not provided or is empty. Note: The scheduler will fail if this is combined with Plugin configs

--port int     Default: 10251

DEPRECATED: the port on which to serve HTTP insecurely without authentication and authorization. If 0, don't serve plain HTTP at all. See --secure-port instead. This parameter is ignored if a config file is specified in --config.

--profiling     Default: true

DEPRECATED: enable profiling via web interface host:port/debug/pprof/. This parameter is ignored if a config file is specified in --config.

--requestheader-allowed-names strings

List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.

--requestheader-client-ca-file string

Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.

--requestheader-extra-headers-prefix strings     Default: "x-remote-extra-"

List of request header prefixes to inspect. X-Remote-Extra- is suggested.

--requestheader-group-headers strings     Default: "x-remote-group"

List of request headers to inspect for groups. X-Remote-Group is suggested.

--requestheader-username-headers strings     Default: "x-remote-user"

List of request headers to inspect for usernames. X-Remote-User is common.

--scheduler-name string     Default: "default-scheduler"

DEPRECATED: name of the scheduler, used to select which pods will be processed by this scheduler, based on pod's "spec.schedulerName". This parameter is ignored if a config file is specified in --config.

--secure-port int     Default: 10259

The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all.

--show-hidden-metrics-for-version string

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is ., e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

--skip_headers

If true, avoid header prefixes in the log messages

--skip_log_headers

If true, avoid headers when opening log files

--stderrthreshold int     Default: 2

logs at or above this threshold go to stderr

--tls-cert-file string

File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.

--tls-cipher-suites strings

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

--tls-min-version string

Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13

--tls-private-key-file string

File containing the default x509 private key matching --tls-cert-file.

--tls-sni-cert-key string

A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".

--use-legacy-policy-config

DEPRECATED: when set to true, scheduler will ignore policy ConfigMap and uses policy config file. Note: The scheduler will fail if this is combined with Plugin configs

-v, --v int

number for the log level verbosity

--version version[=true]

Print version information and quit

--vmodule <comma-separated 'pattern=N' settings>

comma-separated list of pattern=N settings for file-filtered logging

--write-config-to string

If set, write the configuration values to this file and exit.

7 - Kubelet authentication/authorization

Overview

A kubelet's HTTPS endpoint exposes APIs which give access to data of varying sensitivity, and allow you to perform operations with varying levels of power on the node and within containers.

This document describes how to authenticate and authorize access to the kubelet's HTTPS endpoint.

Kubelet authentication

By default, requests to the kubelet's HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated.

To disable anonymous access and send 401 Unauthorized responses to unauthenticated requests:

  • start the kubelet with the --anonymous-auth=false flag

To enable X509 client certificate authentication to the kubelet's HTTPS endpoint:

  • start the kubelet with the --client-ca-file flag, providing a CA bundle to verify client certificates with
  • start the apiserver with --kubelet-client-certificate and --kubelet-client-key flags
  • see the apiserver authentication documentation for more details

To enable API bearer tokens (including service account tokens) to be used to authenticate to the kubelet's HTTPS endpoint:

  • ensure the authentication.k8s.io/v1beta1 API group is enabled in the API server
  • start the kubelet with the --authentication-token-webhook and --kubeconfig flags
  • the kubelet calls the TokenReview API on the configured API server to determine user information from bearer tokens

Kubelet authorization

Any request that is successfully authenticated (including an anonymous request) is then authorized. The default authorization mode is AlwaysAllow, which allows all requests.

There are many possible reasons to subdivide access to the kubelet API:

  • anonymous auth is enabled, but anonymous users' ability to call the kubelet API should be limited
  • bearer token auth is enabled, but arbitrary API users' (like service accounts) ability to call the kubelet API should be limited
  • client certificate auth is enabled, but only some of the client certificates signed by the configured CA should be allowed to use the kubelet API

To subdivide access to the kubelet API, delegate authorization to the API server:

  • ensure the authorization.k8s.io/v1beta1 API group is enabled in the API server
  • start the kubelet with the --authorization-mode=Webhook and the --kubeconfig flags
  • the kubelet calls the SubjectAccessReview API on the configured API server to determine whether each request is authorized

The kubelet authorizes API requests using the same request attributes approach as the apiserver.

The verb is determined from the incoming request's HTTP verb:

HTTP verbrequest verb
POSTcreate
GET, HEADget
PUTupdate
PATCHpatch
DELETEdelete

The resource and subresource is determined from the incoming request's path:

Kubelet APIresourcesubresource
/stats/*nodesstats
/metrics/*nodesmetrics
/logs/*nodeslog
/spec/*nodesspec
all othersnodesproxy

The namespace and API group attributes are always an empty string, and the resource name is always the name of the kubelet's Node API object.

When running in this mode, ensure the user identified by the --kubelet-client-certificate and --kubelet-client-key flags passed to the apiserver is authorized for the following attributes:

  • verb=*, resource=nodes, subresource=proxy
  • verb=*, resource=nodes, subresource=stats
  • verb=*, resource=nodes, subresource=log
  • verb=*, resource=nodes, subresource=spec
  • verb=*, resource=nodes, subresource=metrics

8 - kubelet.config.k8s.io/v1beta1

Resource Types

KubeletConfiguration

KubeletConfiguration contains the configuration for the Kubelet

FieldDescription
apiVersion
string
kubelet.config.k8s.io/v1beta1
kind
string
KubeletConfiguration
enableServer*
bool
enableServer enables Kubelet's secured server. Note: Kubelet's insecure port is controlled by the readOnlyPort option. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: true
staticPodPath
string
staticPodPath is the path to the directory containing local (static) pods to run, or the path to a single static pod file. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that the set of static pods specified at the new path may be different than the ones the Kubelet initially started with, and this may disrupt your node. Default: ""
syncFrequency
meta/v1.Duration
syncFrequency is the max period between synchronizing running containers and config. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that shortening this duration may have a negative performance impact, especially as the number of Pods on the node increases. Alternatively, increasing this duration will result in longer refresh times for ConfigMaps and Secrets. Default: "1m"
fileCheckFrequency
meta/v1.Duration
fileCheckFrequency is the duration between checking config files for new data Dynamic Kubelet Config (beta): If dynamically updating this field, consider that shortening the duration will cause the Kubelet to reload local Static Pod configurations more frequently, which may have a negative performance impact. Default: "20s"
httpCheckFrequency
meta/v1.Duration
httpCheckFrequency is the duration between checking http for new data Dynamic Kubelet Config (beta): If dynamically updating this field, consider that shortening the duration will cause the Kubelet to poll staticPodURL more frequently, which may have a negative performance impact. Default: "20s"
staticPodURL
string
staticPodURL is the URL for accessing static pods to run Dynamic Kubelet Config (beta): If dynamically updating this field, consider that the set of static pods specified at the new URL may be different than the ones the Kubelet initially started with, and this may disrupt your node. Default: ""
staticPodURLHeader
map[string][]string
staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt the ability to read the latest set of static pods from StaticPodURL. Default: nil
address
string
address is the IP address for the Kubelet to serve on (set to 0.0.0.0 for all interfaces). Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: "0.0.0.0"
port
int32
port is the port for the Kubelet to serve on. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: 10250
readOnlyPort
int32
readOnlyPort is the read-only port for the Kubelet to serve on with no authentication/authorization. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: 0 (disabled)
tlsCertFile
string
tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If tlsCertFile and tlsPrivateKeyFile are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to the Kubelet's --cert-dir flag. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: ""
tlsPrivateKeyFile
string
tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: ""
tlsCipherSuites
[]string
TLSCipherSuites is the list of allowed cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: nil
tlsMinVersion
string
TLSMinVersion is the minimum TLS version supported. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: ""
rotateCertificates
bool
rotateCertificates enables client certificate rotation. The Kubelet will request a new certificate from the certificates.k8s.io API. This requires an approver to approve the certificate signing requests. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that disabling it may disrupt the Kubelet's ability to authenticate with the API server after the current certificate expires. Default: false
serverTLSBootstrap
bool
serverTLSBootstrap enables server certificate bootstrap. Instead of self signing a serving certificate, the Kubelet will request a certificate from the certificates.k8s.io API. This requires an approver to approve the certificate signing requests. The RotateKubeletServerCertificate feature must be enabled. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that disabling it will stop the renewal of Kubelet server certificates, which can disrupt components that interact with the Kubelet server in the long term, due to certificate expiration. Default: false
authentication
KubeletAuthentication
authentication specifies how requests to the Kubelet's server are authenticated Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Defaults: anonymous: enabled: false webhook: enabled: true cacheTTL: "2m"
authorization
KubeletAuthorization
authorization specifies how requests to the Kubelet's server are authorized Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Defaults: mode: Webhook webhook: cacheAuthorizedTTL: "5m" cacheUnauthorizedTTL: "30s"
registryPullQPS
int32
registryPullQPS is the limit of registry pulls per second. Set to 0 for no limit. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact scalability by changing the amount of traffic produced by image pulls. Default: 5
registryBurst
int32
registryBurst is the maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registryPullQPS. Only used if registryPullQPS > 0. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact scalability by changing the amount of traffic produced by image pulls. Default: 10
eventRecordQPS
int32
eventRecordQPS is the maximum event creations per second. If 0, there is no limit enforced. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact scalability by changing the amount of traffic produced by event creations. Default: 5
eventBurst
int32
eventBurst is the maximum size of a burst of event creations, temporarily allows event creations to burst to this number, while still not exceeding eventRecordQPS. Only used if eventRecordQPS > 0. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact scalability by changing the amount of traffic produced by event creations. Default: 10
enableDebuggingHandlers
bool
enableDebuggingHandlers enables server endpoints for log access and local running of containers and commands, including the exec, attach, logs, and portforward features. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that disabling it may disrupt components that interact with the Kubelet server. Default: true
enableContentionProfiling
bool
enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that enabling it may carry a performance impact. Default: false
healthzPort
int32
healthzPort is the port of the localhost healthz endpoint (set to 0 to disable) Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that monitor Kubelet health. Default: 10248
healthzBindAddress
string
healthzBindAddress is the IP address for the healthz server to serve on Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that monitor Kubelet health. Default: "127.0.0.1"
oomScoreAdj
int32
oomScoreAdj is The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact the stability of nodes under memory pressure. Default: -999
clusterDomain
string
clusterDomain is the DNS domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains. Dynamic Kubelet Config (beta): Dynamically updating this field is not recommended, as it should be kept in sync with the rest of the cluster. Default: ""
clusterDNS
[]string
clusterDNS is a list of IP addresses for the cluster DNS server. If set, kubelet will configure all containers to use this for DNS resolution instead of the host's DNS servers. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that changes will only take effect on Pods created after the update. Draining the node is recommended before changing this field. Default: nil
streamingConnectionIdleTimeout
meta/v1.Duration
streamingConnectionIdleTimeout is the maximum time a streaming connection can be idle before the connection is automatically closed. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact components that rely on infrequent updates over streaming connections to the Kubelet server. Default: "4h"
nodeStatusUpdateFrequency
meta/v1.Duration
nodeStatusUpdateFrequency is the frequency that kubelet computes node status. If node lease feature is not enabled, it is also the frequency that kubelet posts node status to master. Note: When node lease feature is not enabled, be cautious when changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact node scalability, and also that the node controller's nodeMonitorGracePeriod must be set to N∗NodeStatusUpdateFrequency, where N is the number of retries before the node controller marks the node unhealthy. Default: "10s"
nodeStatusReportFrequency
meta/v1.Duration
nodeStatusReportFrequency is the frequency that kubelet posts node status to master if node status does not change. Kubelet will ignore this frequency and post node status immediately if any change is detected. It is only used when node lease feature is enabled. nodeStatusReportFrequency's default value is 1m. But if nodeStatusUpdateFrequency is set explicitly, nodeStatusReportFrequency's default value will be set to nodeStatusUpdateFrequency for backward compatibility. Default: "1m"
nodeLeaseDurationSeconds
int32
nodeLeaseDurationSeconds is the duration the Kubelet will set on its corresponding Lease, when the NodeLease feature is enabled. This feature provides an indicator of node health by having the Kubelet create and periodically renew a lease, named after the node, in the kube-node-lease namespace. If the lease expires, the node can be considered unhealthy. The lease is currently renewed every 10s, per KEP-0009. In the future, the lease renewal interval may be set based on the lease duration. Requires the NodeLease feature gate to be enabled. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that decreasing the duration may reduce tolerance for issues that temporarily prevent the Kubelet from renewing the lease (e.g. a short-lived network issue). Default: 40
imageMinimumGCAge
meta/v1.Duration
imageMinimumGCAge is the minimum age for an unused image before it is garbage collected. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger or delay garbage collection, and may change the image overhead on the node. Default: "2m"
imageGCHighThresholdPercent
int32
imageGCHighThresholdPercent is the percent of disk usage after which image garbage collection is always run. The percent is calculated as this field value out of 100. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger or delay garbage collection, and may change the image overhead on the node. Default: 85
imageGCLowThresholdPercent
int32
imageGCLowThresholdPercent is the percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. The percent is calculated as this field value out of 100. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger or delay garbage collection, and may change the image overhead on the node. Default: 80
volumeStatsAggPeriod
meta/v1.Duration
How frequently to calculate and cache volume disk usage for all pods Dynamic Kubelet Config (beta): If dynamically updating this field, consider that shortening the period may carry a performance impact. Default: "1m"
kubeletCgroups
string
kubeletCgroups is the absolute name of cgroups to isolate the kubelet in Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: ""
systemCgroups
string
systemCgroups is absolute name of cgroups in which to place all non-kernel processes that are not already in a container. Empty for no container. Rolling back the flag requires a reboot. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: ""
cgroupRoot
string
cgroupRoot is the root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: ""
cgroupsPerQOS
bool
Enable QoS based Cgroup hierarchy: top level cgroups for QoS Classes And all Burstable and BestEffort pods are brought up under their specific top level QoS cgroup. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: true
cgroupDriver
string
driver that the kubelet uses to manipulate cgroups on the host (cgroupfs or systemd) Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: "cgroupfs"
cpuManagerPolicy
string
CPUManagerPolicy is the name of the policy to use. Requires the CPUManager feature gate to be enabled. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: "none"
cpuManagerReconcilePeriod
meta/v1.Duration
CPU Manager reconciliation period. Requires the CPUManager feature gate to be enabled. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that shortening the period may carry a performance impact. Default: "10s"
topologyManagerPolicy
string
TopologyManagerPolicy is the name of the policy to use. Policies other than "none" require the TopologyManager feature gate to be enabled. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: "none"
topologyManagerScope
string
TopologyManagerScope represents the scope of topology hint generation that topology manager requests and hint providers generate. "pod" scope requires the TopologyManager feature gate to be enabled. Default: "container"
qosReserved
map[string]string
qosReserved is a set of resource name to percentage pairs that specify the minimum percentage of a resource reserved for exclusive use by the guaranteed QoS tier. Currently supported resources: "memory" Requires the QOSReserved feature gate to be enabled. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: nil
runtimeRequestTimeout
meta/v1.Duration
runtimeRequestTimeout is the timeout for all runtime requests except long running requests - pull, logs, exec and attach. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may disrupt components that interact with the Kubelet server. Default: "2m"
hairpinMode
string
hairpinMode specifies how the Kubelet should configure the container bridge for hairpin packets. Setting this flag allows endpoints in a Service to loadbalance back to themselves if they should try to access their own Service. Values: "promiscuous-bridge": make the container bridge promiscuous. "hairpin-veth": set the hairpin flag on container veth interfaces. "none": do nothing. Generally, one must set --hairpin-mode=hairpin-veth to achieve hairpin NAT, because promiscuous-bridge assumes the existence of a container bridge named cbr0. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may require a node reboot, depending on the network plugin. Default: "promiscuous-bridge"
maxPods
int32
maxPods is the number of pods that can run on this Kubelet. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that changes may cause Pods to fail admission on Kubelet restart, and may change the value reported in Node.Status.Capacity[v1.ResourcePods], thus affecting future scheduling decisions. Increasing this value may also decrease performance, as more Pods can be packed into a single node. Default: 110
podCIDR
string
The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. Dynamic Kubelet Config (beta): This field should always be set to the empty default. It should only set for standalone Kubelets, which cannot use Dynamic Kubelet Config. Default: ""
podPidsLimit
int64
PodPidsLimit is the maximum number of pids in any pod. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that lowering it may prevent container processes from forking after the change. Default: -1
resolvConf
string
ResolverConfig is the resolver configuration file used as the basis for the container DNS resolution configuration. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that changes will only take effect on Pods created after the update. Draining the node is recommended before changing this field. Default: "/etc/resolv.conf"
runOnce
bool
RunOnce causes the Kubelet to check the API server once for pods, run those in addition to the pods specified by static pod files, and exit. Default: false
cpuCFSQuota
bool
cpuCFSQuota enables CPU CFS quota enforcement for containers that specify CPU limits. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that disabling it may reduce node stability. Default: true
cpuCFSQuotaPeriod
meta/v1.Duration
CPUCFSQuotaPeriod is the CPU CFS quota period value, cpu.cfs_period_us. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that limits set for containers will result in different cpu.cfs_quota settings. This will trigger container restarts on the node being reconfigured. Default: "100ms"
nodeStatusMaxImages
int32
nodeStatusMaxImages caps the number of images reported in Node.Status.Images. Note: If -1 is specified, no cap will be applied. If 0 is specified, no image is returned. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that different values can be reported on node status. Default: 50
maxOpenFiles
int64
maxOpenFiles is Number of files that can be opened by Kubelet process. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact the ability of the Kubelet to interact with the node's filesystem. Default: 1000000
contentType
string
contentType is contentType of requests sent to apiserver. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact the ability for the Kubelet to communicate with the API server. If the Kubelet loses contact with the API server due to a change to this field, the change cannot be reverted via dynamic Kubelet config. Default: "application/vnd.kubernetes.protobuf"
kubeAPIQPS
int32
kubeAPIQPS is the QPS to use while talking with kubernetes apiserver Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact scalability by changing the amount of traffic the Kubelet sends to the API server. Default: 5
kubeAPIBurst
int32
kubeAPIBurst is the burst to allow while talking with kubernetes apiserver Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact scalability by changing the amount of traffic the Kubelet sends to the API server. Default: 10
serializeImagePulls
bool
serializeImagePulls when enabled, tells the Kubelet to pull images one at a time. We recommend ∗not∗ changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact the performance of image pulls. Default: true
evictionHard
map[string]string
Map of signal names to quantities that defines hard eviction thresholds. For example: {"memory.available": "300Mi"}. To explicitly disable, pass a 0% or 100% threshold on an arbitrary resource. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger or delay Pod evictions. Default: memory.available: "100Mi" nodefs.available: "10%" nodefs.inodesFree: "5%" imagefs.available: "15%"
evictionSoft
map[string]string
Map of signal names to quantities that defines soft eviction thresholds. For example: {"memory.available": "300Mi"}. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger or delay Pod evictions, and may change the allocatable reported by the node. Default: nil
evictionSoftGracePeriod
map[string]string
Map of signal names to quantities that defines grace periods for each soft eviction signal. For example: {"memory.available": "30s"}. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger or delay Pod evictions. Default: nil
evictionPressureTransitionPeriod
meta/v1.Duration
Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that lowering it may decrease the stability of the node when the node is overcommitted. Default: "5m"
evictionMaxPodGracePeriod
int32
Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. This value effectively caps the Pod's TerminationGracePeriodSeconds value during soft evictions. Note: Due to issue #64530, the behavior has a bug where this value currently just overrides the grace period during soft eviction, which can increase the grace period from what is set on the Pod. This bug will be fixed in a future release. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that lowering it decreases the amount of time Pods will have to gracefully clean up before being killed during a soft eviction. Default: 0
evictionMinimumReclaim
map[string]string
Map of signal names to quantities that defines minimum reclaims, which describe the minimum amount of a given resource the kubelet will reclaim when performing a pod eviction while that resource is under pressure. For example: {"imagefs.available": "2Gi"} Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may change how well eviction can manage resource pressure. Default: nil
podsPerCore
int32
podsPerCore is the maximum number of pods per core. Cannot exceed MaxPods. If 0, this field is ignored. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that changes may cause Pods to fail admission on Kubelet restart, and may change the value reported in Node.Status.Capacity[v1.ResourcePods], thus affecting future scheduling decisions. Increasing this value may also decrease performance, as more Pods can be packed into a single node. Default: 0
enableControllerAttachDetach
bool
enableControllerAttachDetach enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations Dynamic Kubelet Config (beta): If dynamically updating this field, consider that changing which component is responsible for volume management on a live node may result in volumes refusing to detach if the node is not drained prior to the update, and if Pods are scheduled to the node before the volumes.kubernetes.io/controller-managed-attach-detach annotation is updated by the Kubelet. In general, it is safest to leave this value set the same as local config. Default: true
protectKernelDefaults
bool
protectKernelDefaults, if true, causes the Kubelet to error if kernel flags are not as it expects. Otherwise the Kubelet will attempt to modify kernel flags to match its expectation. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that enabling it may cause the Kubelet to crash-loop if the Kernel is not configured as Kubelet expects. Default: false
makeIPTablesUtilChains
bool
If true, Kubelet ensures a set of iptables rules are present on host. These rules will serve as utility rules for various components, e.g. KubeProxy. The rules will be created based on IPTablesMasqueradeBit and IPTablesDropBit. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that disabling it will prevent the Kubelet from healing locally misconfigured iptables rules. Default: true
iptablesMasqueradeBit
int32
iptablesMasqueradeBit is the bit of the iptables fwmark space to mark for SNAT Values must be within the range [0, 31]. Must be different from other mark bits. Warning: Please match the value of the corresponding parameter in kube-proxy. TODO: clean up IPTablesMasqueradeBit in kube-proxy Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it needs to be coordinated with other components, like kube-proxy, and the update will only be effective if MakeIPTablesUtilChains is enabled. Default: 14
iptablesDropBit
int32
iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets. Values must be within the range [0, 31]. Must be different from other mark bits. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it needs to be coordinated with other components, like kube-proxy, and the update will only be effective if MakeIPTablesUtilChains is enabled. Default: 15
featureGates
map[string]bool
featureGates is a map of feature names to bools that enable or disable alpha/experimental features. This field modifies piecemeal the built-in default values from "k8s.io/kubernetes/pkg/features/kube_features.go". Dynamic Kubelet Config (beta): If dynamically updating this field, consider the documentation for the features you are enabling or disabling. While we encourage feature developers to make it possible to dynamically enable and disable features, some changes may require node reboots, and some features may require careful coordination to retroactively disable. Default: nil
failSwapOn
bool
failSwapOn tells the Kubelet to fail to start if swap is enabled on the node. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that setting it to true will cause the Kubelet to crash-loop if swap is enabled. Default: true
containerLogMaxSize
string
A quantity defines the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki". Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger log rotation. Default: "10Mi"
containerLogMaxFiles
int32
Maximum number of container log files that can be present for a container. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that lowering it may cause log files to be deleted. Default: 5
configMapAndSecretChangeDetectionStrategy
ResourceChangeDetectionStrategy
ConfigMapAndSecretChangeDetectionStrategy is a mode in which config map and secret managers are running. Default: "Watch"
systemReserved
map[string]string
systemReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may not be possible to increase the reserved resources, because this requires resizing cgroups. Always look for a NodeAllocatableEnforced event after updating this field to ensure that the update was successful. Default: nil
kubeReserved
map[string]string
A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local storage for root file system are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may not be possible to increase the reserved resources, because this requires resizing cgroups. Always look for a NodeAllocatableEnforced event after updating this field to ensure that the update was successful. Default: nil
reservedSystemCPUs*
string
This ReservedSystemCPUs option specifies the cpu list reserved for the host level system threads and kubernetes related threads. This provide a "static" CPU list rather than the "dynamic" list by system-reserved and kube-reserved. This option overwrites CPUs provided by system-reserved and kube-reserved.
showHiddenMetricsForVersion
string
The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is ., e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that. Default: ""
systemReservedCgroup
string
This flag helps kubelet identify absolute name of top level cgroup used to enforce `SystemReserved` compute resource reservation for OS system daemons. Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) doc for more information. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: ""
kubeReservedCgroup
string
This flag helps kubelet identify absolute name of top level cgroup used to enforce `KubeReserved` compute resource reservation for Kubernetes node system daemons. Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) doc for more information. Dynamic Kubelet Config (beta): This field should not be updated without a full node reboot. It is safest to keep this value the same as the local config. Default: ""
enforceNodeAllocatable
[]string
This flag specifies the various Node Allocatable enforcements that Kubelet needs to perform. This flag accepts a list of options. Acceptable options are `none`, `pods`, `system-reserved` & `kube-reserved`. If `none` is specified, no other options may be specified. Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) doc for more information. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that removing enforcements may reduce the stability of the node. Alternatively, adding enforcements may reduce the stability of components which were using more than the reserved amount of resources; for example, enforcing kube-reserved may cause Kubelets to OOM if it uses more than the reserved resources, and enforcing system-reserved may cause system daemons to OOM if they use more than the reserved resources. Default: ["pods"]
allowedUnsafeSysctls
[]string
A comma separated whitelist of unsafe sysctls or sysctl patterns (ending in ∗). Unsafe sysctl groups are kernel.shm∗, kernel.msg∗, kernel.sem, fs.mqueue.∗, and net.∗. These sysctls are namespaced but not allowed by default. For example: "kernel.msg∗,net.ipv4.route.min_pmtu" Default: []
volumePluginDir
string
volumePluginDir is the full path of the directory in which to search for additional third party volume plugins. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that changing the volumePluginDir may disrupt workloads relying on third party volume plugins. Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
providerID
string
providerID, if set, sets the unique id of the instance that an external provider (i.e. cloudprovider) can use to identify a specific node. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact the ability of the Kubelet to interact with cloud providers. Default: ""
kernelMemcgNotification
bool
kernelMemcgNotification, if set, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may impact the way Kubelet interacts with the kernel. Default: false
logging*
LoggingConfiguration
Logging specifies the options of logging. Refer [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) for more information. Defaults: Format: text
enableSystemLogHandler
bool
enableSystemLogHandler enables system logs via web interface host:port/logs/ Default: true
shutdownGracePeriod
meta/v1.Duration
ShutdownGracePeriod specifies the total duration that the node should delay the shutdown and total grace period for pod termination during a node shutdown. Default: "30s"
shutdownGracePeriodCriticalPods
meta/v1.Duration
ShutdownGracePeriodCriticalPods specifies the duration used to terminate critical pods during a node shutdown. This should be less than ShutdownGracePeriod. For example, if ShutdownGracePeriod=30s, and ShutdownGracePeriodCriticalPods=10s, during a node shutdown the first 20 seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating critical pods. Default: "10s"

SerializedNodeConfigSource

SerializedNodeConfigSource allows us to serialize v1.NodeConfigSource. This type is used internally by the Kubelet for tracking checkpointed dynamic configs. It exists in the kubeletconfig API group because it is classified as a versioned input to the Kubelet.

FieldDescription
apiVersion
string
kubelet.config.k8s.io/v1beta1
kind
string
SerializedNodeConfigSource
source
core/v1.NodeConfigSource
Source is the source that we are serializing

HairpinMode

(Alias of string)

HairpinMode denotes how the kubelet should configure networking to handle hairpin packets.

KubeletAnonymousAuthentication

Appears in:

FieldDescription
enabled
bool
enabled allows anonymous requests to the kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated.

KubeletAuthentication

Appears in:

FieldDescription
x509
KubeletX509Authentication
x509 contains settings related to x509 client certificate authentication
webhook
KubeletWebhookAuthentication
webhook contains settings related to webhook bearer token authentication
anonymous
KubeletAnonymousAuthentication
anonymous contains settings related to anonymous authentication

KubeletAuthorization

Appears in:

FieldDescription
mode
KubeletAuthorizationMode
mode is the authorization mode to apply to requests to the kubelet server. Valid values are AlwaysAllow and Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization.
webhook
KubeletWebhookAuthorization
webhook contains settings related to Webhook authorization.

KubeletAuthorizationMode

(Alias of string)

Appears in:

KubeletWebhookAuthentication

Appears in:

FieldDescription
enabled
bool
enabled allows bearer token authentication backed by the tokenreviews.authentication.k8s.io API
cacheTTL
meta/v1.Duration
cacheTTL enables caching of authentication results

KubeletWebhookAuthorization

Appears in:

FieldDescription
cacheAuthorizedTTL
meta/v1.Duration
cacheAuthorizedTTL is the duration to cache 'authorized' responses from the webhook authorizer.
cacheUnauthorizedTTL
meta/v1.Duration
cacheUnauthorizedTTL is the duration to cache 'unauthorized' responses from the webhook authorizer.

KubeletX509Authentication

Appears in:

FieldDescription
clientCAFile
string
clientCAFile is the path to a PEM-encoded certificate bundle. If set, any request presenting a client certificate signed by one of the authorities in the bundle is authenticated with a username corresponding to the CommonName, and groups corresponding to the Organization in the client certificate.

ResourceChangeDetectionStrategy

(Alias of string)

Appears in:

ResourceChangeDetectionStrategy denotes a mode in which internal managers (secret, configmap) are discovering object changes.

LoggingConfiguration

Appears in:

LoggingConfiguration contains logging options Refer Logs Options for more information.

FieldDescription
format*
string
Format Flag specifies the structure of log messages. default value of format is `text`
sanitization*
bool
[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)

9 - TLS bootstrapping

In a Kubernetes cluster, the components on the worker nodes - kubelet and kube-proxy - need to communicate with Kubernetes master components, specifically kube-apiserver. In order to ensure that communication is kept private, not interfered with, and ensure that each component of the cluster is talking to another trusted component, we strongly recommend using client TLS certificates on nodes.

The normal process of bootstrapping these components, especially worker nodes that need certificates so they can communicate safely with kube-apiserver, can be a challenging process as it is often outside of the scope of Kubernetes and requires significant additional work. This in turn, can make it challenging to initialize or scale a cluster.

In order to simplify the process, beginning in version 1.4, Kubernetes introduced a certificate request and signing API to simplify the process. The proposal can be found here.

This document describes the process of node initialization, how to set up TLS client certificate bootstrapping for kubelets, and how it works.

Initialization Process

When a worker node starts up, the kubelet does the following:

  1. Look for its kubeconfig file
  2. Retrieve the URL of the API server and credentials, normally a TLS key and signed certificate from the kubeconfig file
  3. Attempt to communicate with the API server using the credentials.

Assuming that the kube-apiserver successfully validates the kubelet's credentials, it will treat the kubelet as a valid node, and begin to assign pods to it.

Note that the above process depends upon:

  • Existence of a key and certificate on the local host in the kubeconfig
  • The certificate having been signed by a Certificate Authority (CA) trusted by the kube-apiserver

All of the following are responsibilities of whoever sets up and manages the cluster:

  1. Creating the CA key and certificate
  2. Distributing the CA certificate to the master nodes, where kube-apiserver is running
  3. Creating a key and certificate for each kubelet; strongly recommended to have a unique one, with a unique CN, for each kubelet
  4. Signing the kubelet certificate using the CA key
  5. Distributing the kubelet key and signed certificate to the specific node on which the kubelet is running

The TLS Bootstrapping described in this document is intended to simplify, and partially or even completely automate, steps 3 onwards, as these are the most common when initializing or scaling a cluster.

Bootstrap Initialization

In the bootstrap initialization process, the following occurs:

  1. kubelet begins
  2. kubelet sees that it does not have a kubeconfig file
  3. kubelet searches for and finds a bootstrap-kubeconfig file
  4. kubelet reads its bootstrap file, retrieving the URL of the API server and a limited usage "token"
  5. kubelet connects to the API server, authenticates using the token
  6. kubelet now has limited credentials to create and retrieve a certificate signing request (CSR)
  7. kubelet creates a CSR for itself with the signerName set to kubernetes.io/kube-apiserver-client-kubelet
  8. CSR is approved in one of two ways:
  • If configured, kube-controller-manager automatically approves the CSR
  • If configured, an outside process, possibly a person, approves the CSR using the Kubernetes API or via kubectl
  1. Certificate is created for the kubelet
  2. Certificate is issued to the kubelet
  3. kubelet retrieves the certificate
  4. kubelet creates a proper kubeconfig with the key and signed certificate
  5. kubelet begins normal operation
  6. Optional: if configured, kubelet automatically requests renewal of the certificate when it is close to expiry
  7. The renewed certificate is approved and issued, either automatically or manually, depending on configuration.

The rest of this document describes the necessary steps to configure TLS Bootstrapping, and its limitations.

Configuration

To configure for TLS bootstrapping and optional automatic approval, you must configure options on the following components:

  • kube-apiserver
  • kube-controller-manager
  • kubelet
  • in-cluster resources: ClusterRoleBinding and potentially ClusterRole

In addition, you need your Kubernetes Certificate Authority (CA).

Certificate Authority

As without bootstrapping, you will need a Certificate Authority (CA) key and certificate. As without bootstrapping, these will be used to sign the kubelet certificate. As before, it is your responsibility to distribute them to master nodes.

For the purposes of this document, we will assume these have been distributed to master nodes at /var/lib/kubernetes/ca.pem (certificate) and /var/lib/kubernetes/ca-key.pem (key). We will refer to these as "Kubernetes CA certificate and key".

All Kubernetes components that use these certificates - kubelet, kube-apiserver, kube-controller-manager - assume the key and certificate to be PEM-encoded.

kube-apiserver configuration

The kube-apiserver has several requirements to enable TLS bootstrapping:

  • Recognizing CA that signs the client certificate
  • Authenticating the bootstrapping kubelet to the system:bootstrappers group
  • Authorize the bootstrapping kubelet to create a certificate signing request (CSR)

Recognizing client certificates

This is normal for all client certificate authentication. If not already set, add the --client-ca-file=FILENAME flag to the kube-apiserver command to enable client certificate authentication, referencing a certificate authority bundle containing the signing certificate, for example --client-ca-file=/var/lib/kubernetes/ca.pem.

Initial bootstrap authentication

In order for the bootstrapping kubelet to connect to kube-apiserver and request a certificate, it must first authenticate to the server. You can use any authenticator that can authenticate the kubelet.

While any authentication strategy can be used for the kubelet's initial bootstrap credentials, the following two authenticators are recommended for ease of provisioning.

  1. Bootstrap Tokens
  2. Token authentication file

Bootstrap tokens are a simpler and more easily managed method to authenticate kubelets, and do not require any additional flags when starting kube-apiserver. Using bootstrap tokens is currently beta as of Kubernetes version 1.12.

Whichever method you choose, the requirement is that the kubelet be able to authenticate as a user with the rights to:

  1. create and retrieve CSRs
  2. be automatically approved to request node client certificates, if automatic approval is enabled.

A kubelet authenticating using bootstrap tokens is authenticated as a user in the group system:bootstrappers, which is the standard method to use.

As this feature matures, you should ensure tokens are bound to a Role Based Access Control (RBAC) policy which limits requests (using the bootstrap token) strictly to client requests related to certificate provisioning. With RBAC in place, scoping the tokens to a group allows for great flexibility. For example, you could disable a particular bootstrap group's access when you are done provisioning the nodes.

Bootstrap tokens

Bootstrap tokens are described in detail here. These are tokens that are stored as secrets in the Kubernetes cluster, and then issued to the individual kubelet. You can use a single token for an entire cluster, or issue one per worker node.

The process is two-fold:

  1. Create a Kubernetes secret with the token ID, secret and scope(s).
  2. Issue the token to the kubelet

From the kubelet's perspective, one token is like another and has no special meaning. From the kube-apiserver's perspective, however, the bootstrap token is special. Due to its type, namespace and name, kube-apiserver recognizes it as a special token, and grants anyone authenticating with that token special bootstrap rights, notably treating them as a member of the system:bootstrappers group. This fulfills a basic requirement for TLS bootstrapping.

The details for creating the secret are available here.

If you want to use bootstrap tokens, you must enable it on kube-apiserver with the flag:

--enable-bootstrap-token-auth=true

Token authentication file

kube-apiserver has an ability to accept tokens as authentication. These tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number generator (such as /dev/urandom on most modern Linux systems). There are multiple ways you can generate a token. For example:

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

will generate tokens that look like 02b50b05283e98dd0fd71db496ef01e8.

The token file should look like the following example, where the first three values can be anything and the quoted group name should be as depicted:

02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:bootstrappers"

Add the --token-auth-file=FILENAME flag to the kube-apiserver command (in your systemd unit file perhaps) to enable the token file. See docs here for further details.

Authorize kubelet to create CSR

Now that the bootstrapping node is authenticated as part of the system:bootstrappers group, it needs to be authorized to create a certificate signing request (CSR) as well as retrieve it when done. Fortunately, Kubernetes ships with a ClusterRole with precisely these (and only these) permissions, system:node-bootstrapper.

To do this, you only need to create a ClusterRoleBinding that binds the system:bootstrappers group to the cluster role system:node-bootstrapper.

# enable bootstrapping nodes to create CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: create-csrs-for-bootstrapping
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:node-bootstrapper
  apiGroup: rbac.authorization.k8s.io

kube-controller-manager configuration

While the apiserver receives the requests for certificates from the kubelet and authenticates those requests, the controller-manager is responsible for issuing actual signed certificates.

The controller-manager performs this function via a certificate-issuing control loop. This takes the form of a cfssl local signer using assets on disk. Currently, all certificates issued have one year validity and a default set of key usages.

In order for the controller-manager to sign certificates, it needs the following:

  • access to the "Kubernetes CA key and certificate" that you created and distributed
  • enabling CSR signing

Access to key and certificate

As described earlier, you need to create a Kubernetes CA key and certificate, and distribute it to the master nodes. These will be used by the controller-manager to sign the kubelet certificates.

Since these signed certificates will, in turn, be used by the kubelet to authenticate as a regular kubelet to kube-apiserver, it is important that the CA provided to the controller-manager at this stage also be trusted by kube-apiserver for authentication. This is provided to kube-apiserver with the flag --client-ca-file=FILENAME (for example, --client-ca-file=/var/lib/kubernetes/ca.pem), as described in the kube-apiserver configuration section.

To provide the Kubernetes CA key and certificate to kube-controller-manager, use the following flags:

--cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key"

for example:

--cluster-signing-cert-file="/var/lib/kubernetes/ca.pem" --cluster-signing-key-file="/var/lib/kubernetes/ca-key.pem"

The validity duration of signed certificates can be configured with flag:

--cluster-signing-duration

Approval

In order to approve CSRs, you need to tell the controller-manager that it is acceptable to approve them. This is done by granting RBAC permissions to the correct group.

There are two distinct sets of permissions:

  • nodeclient: If a node is creating a new certificate for a node, then it does not have a certificate yet. It is authenticating using one of the tokens listed above, and thus is part of the group system:bootstrappers.
  • selfnodeclient: If a node is renewing its certificate, then it already has a certificate (by definition), which it uses continuously to authenticate as part of the group system:nodes.

To enable the kubelet to request and receive a new certificate, create a ClusterRoleBinding that binds the group in which the bootstrapping node is a member system:bootstrappers to the ClusterRole that grants it permission, system:certificates.k8s.io:certificatesigningrequests:nodeclient:

# Approve all CSRs for the group "system:bootstrappers"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-csrs-for-group
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  apiGroup: rbac.authorization.k8s.io

To enable the kubelet to renew its own client certificate, create a ClusterRoleBinding that binds the group in which the fully functioning node is a member system:nodes to the ClusterRole that grants it permission, system:certificates.k8s.io:certificatesigningrequests:selfnodeclient:

# Approve renewal CSRs for the group "system:nodes"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  apiGroup: rbac.authorization.k8s.io

The csrapproving controller that ships as part of kube-controller-manager and is enabled by default. The controller uses the SubjectAccessReview API to determine if a given user is authorized to request a CSR, then approves based on the authorization outcome. To prevent conflicts with other approvers, the builtin approver doesn't explicitly deny CSRs. It only ignores unauthorized requests. The controller also prunes expired certificates as part of garbage collection.

kubelet configuration

Finally, with the master nodes properly set up and all of the necessary authentication and authorization in place, we can configure the kubelet.

The kubelet requires the following configuration to bootstrap:

  • A path to store the key and certificate it generates (optional, can use default)
  • A path to a kubeconfig file that does not yet exist; it will place the bootstrapped config file here
  • A path to a bootstrap kubeconfig file to provide the URL for the server and bootstrap credentials, e.g. a bootstrap token
  • Optional: instructions to rotate certificates

The bootstrap kubeconfig should be in a path available to the kubelet, for example /var/lib/kubelet/bootstrap-kubeconfig.

Its format is identical to a normal kubeconfig file. A sample file might look as follows:

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority: /var/lib/kubernetes/ca.pem
    server: https://my.server.example.com:6443
  name: bootstrap
contexts:
- context:
    cluster: bootstrap
    user: kubelet-bootstrap
  name: bootstrap
current-context: bootstrap
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: 07401b.f395accd246ae52d

The important elements to note are:

  • certificate-authority: path to a CA file, used to validate the server certificate presented by kube-apiserver
  • server: URL to kube-apiserver
  • token: the token to use

The format of the token does not matter, as long as it matches what kube-apiserver expects. In the above example, we used a bootstrap token. As stated earlier, any valid authentication method can be used, not only tokens.

Because the bootstrap kubeconfig is a standard kubeconfig, you can use kubectl to generate it. To create the above example file:

kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-cluster bootstrap --server='https://my.server.example.com:6443' --certificate-authority=/var/lib/kubernetes/ca.pem
kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-credentials kubelet-bootstrap --token=07401b.f395accd246ae52d
kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-context bootstrap --user=kubelet-bootstrap --cluster=bootstrap
kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig use-context bootstrap

To indicate to the kubelet to use the bootstrap kubeconfig, use the following kubelet flag:

--bootstrap-kubeconfig="/var/lib/kubelet/bootstrap-kubeconfig" --kubeconfig="/var/lib/kubelet/kubeconfig"

When starting the kubelet, if the file specified via --kubeconfig does not exist, the bootstrap kubeconfig specified via --bootstrap-kubeconfig is used to request a client certificate from the API server. On approval of the certificate request and receipt back by the kubelet, a kubeconfig file referencing the generated key and obtained certificate is written to the path specified by --kubeconfig. The certificate and key file will be placed in the directory specified by --cert-dir.

Client and Serving Certificates

All of the above relate to kubelet client certificates, specifically, the certificates a kubelet uses to authenticate to kube-apiserver.

A kubelet also can use serving certificates. The kubelet itself exposes an https endpoint for certain features. To secure these, the kubelet can do one of:

  • use provided key and certificate, via the --tls-private-key-file and --tls-cert-file flags
  • create self-signed key and certificate, if a key and certificate are not provided
  • request serving certificates from the cluster server, via the CSR API

The client certificate provided by TLS bootstrapping is signed, by default, for client auth only, and thus cannot be used as serving certificates, or server auth.

However, you can enable its server certificate, at least partially, via certificate rotation.

Certificate Rotation

Kubernetes v1.8 and higher kubelet implements beta features for enabling rotation of its client and/or serving certificates. These can be enabled through the respective RotateKubeletClientCertificate and RotateKubeletServerCertificate feature flags on the kubelet and are enabled by default.

RotateKubeletClientCertificate causes the kubelet to rotate its client certificates by creating new CSRs as its existing credentials expire. To enable this feature pass the following flag to the kubelet:

--rotate-certificates

RotateKubeletServerCertificate causes the kubelet both to request a serving certificate after bootstrapping its client credentials and to rotate that certificate. To enable this feature pass the following flag to the kubelet:

--rotate-server-certificates
Note:

The CSR approving controllers implemented in core Kubernetes do not approve node serving certificates for security reasons. To use RotateKubeletServerCertificate operators need to run a custom approving controller, or manually approve the serving certificate requests.

A deployment-specific approval process for kubelet serving certificates should typically only approve CSRs which:

  1. are requested by nodes (ensure the spec.username field is of the form system:node:<nodeName> and spec.groups contains system:nodes)
  2. request usages for a serving certificate (ensure spec.usages contains server auth, optionally contains digital signature and key encipherment, and contains no other usages)
  3. only have IP and DNS subjectAltNames that belong to the requesting node, and have no URI and Email subjectAltNames (parse the x509 Certificate Signing Request in spec.request to verify subjectAltNames)

Other authenticating components

All of TLS bootstrapping described in this document relates to the kubelet. However, other components may need to communicate directly with kube-apiserver. Notable is kube-proxy, which is part of the Kubernetes control plane and runs on every node, but may also include other components such as monitoring or networking.

Like the kubelet, these other components also require a method of authenticating to kube-apiserver. You have several options for generating these credentials:

  • The old way: Create and distribute certificates the same way you did for kubelet before TLS bootstrapping
  • DaemonSet: Since the kubelet itself is loaded on each node, and is sufficient to start base services, you can run kube-proxy and other node-specific services not as a standalone process, but rather as a daemonset in the kube-system namespace. Since it will be in-cluster, you can give it a proper service account with appropriate permissions to perform its activities. This may be the simplest way to configure such services.

kubectl approval

CSRs can be approved outside of the approval flows builtin to the controller manager.

The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an "Approved" status by an appropriately-privileged user. This flow is intended to allow for automated approval handled by an external approval controller or the approval controller implemented in the core controller-manager. However cluster administrators can also manually approve certificate requests using kubectl. An administrator can list CSRs with kubectl get csr and describe one in detail with kubectl describe csr <name>. An administrator can approve or deny a CSR with kubectl certificate approve <name> and kubectl certificate deny <name>.