Dynamic Admission Control
In addition to compiled-in admission plugins, admission plugins can be developed as extensions and run as webhooks configured at runtime. This page describes how to build, configure, use, and monitor admission webhooks.
What are admission webhooks?
Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook. Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked and can reject requests to enforce custom policies.
Note:
Admission webhooks that need to guarantee they see the final state of the object in order to enforce policy should use a validating admission webhook, since objects can be modified after being seen by mutating webhooks.Experimenting with admission webhooks
Admission webhooks are essentially part of the cluster control-plane. You should write and deploy them with great caution. Please read the user guides for instructions if you intend to write/deploy production-grade admission webhooks. In the following, we describe how to quickly experiment with admission webhooks.
Prerequisites
Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers are enabled. Here is a recommended set of admission controllers to enable in general.
Ensure that the
admissionregistration.k8s.io/v1
API is enabled.
Write an admission webhook server
Please refer to the implementation of the admission webhook server
that is validated in a Kubernetes e2e test. The webhook handles the
AdmissionReview
request sent by the API servers, and sends back its decision
as an AdmissionReview
object in the same version it received.
See the webhook request section for details on the data sent to webhooks.
See the webhook response section for the data expected from webhooks.
The example admission webhook server leaves the ClientAuth
field
empty,
which defaults to NoClientCert
. This means that the webhook server does not
authenticate the identity of the clients, supposedly API servers. If you need
mutual TLS or other ways to authenticate the clients, see
how to authenticate API servers.
Deploy the admission webhook service
The webhook server in the e2e test is deployed in the Kubernetes cluster, via the deployment API. The test also creates a service as the front-end of the webhook server. See code.
You may also deploy your webhooks outside of the cluster. You will need to update your webhook configurations accordingly.
Configure admission webhooks on the fly
You can dynamically configure what resources are subject to what admission webhooks via ValidatingWebhookConfiguration or MutatingWebhookConfiguration.
The following is an example ValidatingWebhookConfiguration
, a mutating webhook configuration is similar.
See the webhook configuration section for details about each config field.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: "pod-policy.example.com"
webhooks:
- name: "pod-policy.example.com"
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
scope: "Namespaced"
clientConfig:
service:
namespace: "example-namespace"
name: "example-service"
caBundle: <CA_BUNDLE>
admissionReviewVersions: ["v1"]
sideEffects: None
timeoutSeconds: 5
Note:
You must replace the<CA_BUNDLE>
in the above example by a valid CA bundle
which is a PEM-encoded (field value is Base64 encoded) CA bundle for validating the webhook's server certificate.The scope
field specifies if only cluster-scoped resources ("Cluster") or namespace-scoped
resources ("Namespaced") will match this rule. "∗" means that there are no scope restrictions.
Note:
When usingclientConfig.service
, the server cert must be valid for
<svc_name>.<svc_namespace>.svc
.Note:
Default timeout for a webhook call is 10 seconds, You can set thetimeout
and it is encouraged to use a short timeout for webhooks.
If the webhook call times out, the request is handled according to the webhook's
failure policy.When an API server receives a request that matches one of the rules
, the
API server sends an admissionReview
request to webhook as specified in the
clientConfig
.
After you create the webhook configuration, the system will take a few seconds to honor the new configuration.
Authenticate API servers
If your admission webhooks require authentication, you can configure the API servers to use basic auth, bearer token, or a cert to authenticate itself to the webhooks. There are three steps to complete the configuration.
When starting the API server, specify the location of the admission control configuration file via the
--admission-control-config-file
flag.In the admission control configuration file, specify where the MutatingAdmissionWebhook controller and ValidatingAdmissionWebhook controller should read the credentials. The credentials are stored in kubeConfig files (yes, the same schema that's used by kubectl), so the field name is
kubeConfigFile
. Here is an example admission control configuration file:
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ValidatingAdmissionWebhook
configuration:
apiVersion: apiserver.config.k8s.io/v1
kind: WebhookAdmissionConfiguration
kubeConfigFile: "<path-to-kubeconfig-file>"
- name: MutatingAdmissionWebhook
configuration:
apiVersion: apiserver.config.k8s.io/v1
kind: WebhookAdmissionConfiguration
kubeConfigFile: "<path-to-kubeconfig-file>"
# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
apiVersion: apiserver.k8s.io/v1alpha1
kind: AdmissionConfiguration
plugins:
- name: ValidatingAdmissionWebhook
configuration:
# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1, kind=WebhookAdmissionConfiguration
apiVersion: apiserver.config.k8s.io/v1alpha1
kind: WebhookAdmission
kubeConfigFile: "<path-to-kubeconfig-file>"
- name: MutatingAdmissionWebhook
configuration:
# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1, kind=WebhookAdmissionConfiguration
apiVersion: apiserver.config.k8s.io/v1alpha1
kind: WebhookAdmission
kubeConfigFile: "<path-to-kubeconfig-file>"
For more information about AdmissionConfiguration
, see the
AdmissionConfiguration (v1) reference.
See the webhook configuration section for details about each config field.
In the kubeConfig file, provide the credentials:
apiVersion: v1
kind: Config
users:
# name should be set to the DNS name of the service or the host (including port) of the URL the webhook is configured to speak to.
# If a non-443 port is used for services, it must be included in the name when configuring 1.16+ API servers.
#
# For a webhook configured to speak to a service on the default port (443), specify the DNS name of the service:
# - name: webhook1.ns1.svc
# user: ...
#
# For a webhook configured to speak to a service on non-default port (e.g. 8443), specify the DNS name and port of the service in 1.16+:
# - name: webhook1.ns1.svc:8443
# user: ...
# and optionally create a second stanza using only the DNS name of the service for compatibility with 1.15 API servers:
# - name: webhook1.ns1.svc
# user: ...
#
# For webhooks configured to speak to a URL, match the host (and port) specified in the webhook's URL. Examples:
# A webhook with `url: https://www.example.com`:
# - name: www.example.com
# user: ...
#
# A webhook with `url: https://www.example.com:443`:
# - name: www.example.com:443
# user: ...
#
# A webhook with `url: https://www.example.com:8443`:
# - name: www.example.com:8443
# user: ...
#
- name: 'webhook1.ns1.svc'
user:
client-certificate-data: "<pem encoded certificate>"
client-key-data: "<pem encoded key>"
# The `name` supports using * to wildcard-match prefixing segments.
- name: '*.webhook-company.org'
user:
password: "<password>"
username: "<name>"
# '*' is the default match.
- name: '*'
user:
token: "<token>"
Of course you need to set up the webhook server to handle these authentication requests.
Webhook request and response
Request
Webhooks are sent as POST requests, with Content-Type: application/json
,
with an AdmissionReview
API object in the admission.k8s.io
API group
serialized to JSON as the body.
Webhooks can specify what versions of AdmissionReview
objects they accept
with the admissionReviewVersions
field in their configuration:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
admissionReviewVersions: ["v1", "v1beta1"]
admissionReviewVersions
is a required field when creating webhook configurations.
Webhooks are required to support at least one AdmissionReview
version understood by the current and previous API server.
API servers send the first AdmissionReview
version in the admissionReviewVersions
list they support.
If none of the versions in the list are supported by the API server, the configuration will not be allowed to be created.
If an API server encounters a webhook configuration that was previously created and does not support any of the AdmissionReview
versions the API server knows how to send, attempts to call to the webhook will fail and be subject to the failure policy.
This example shows the data contained in an AdmissionReview
object
for a request to update the scale
subresource of an apps/v1
Deployment
:
apiVersion: admission.k8s.io/v1
kind: AdmissionReview
request:
# Random uid uniquely identifying this admission call
uid: 705ab4f5-6393-11e8-b7cc-42010a800002
# Fully-qualified group/version/kind of the incoming object
kind:
group: autoscaling
version: v1
kind: Scale
# Fully-qualified group/version/kind of the resource being modified
resource:
group: apps
version: v1
resource: deployments
# subresource, if the request is to a subresource
subResource: scale
# Fully-qualified group/version/kind of the incoming object in the original request to the API server.
# This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the
# original request to the API server was converted to a version the webhook registered for.
requestKind:
group: autoscaling
version: v1
kind: Scale
# Fully-qualified group/version/kind of the resource being modified in the original request to the API server.
# This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the
# original request to the API server was converted to a version the webhook registered for.
requestResource:
group: apps
version: v1
resource: deployments
# subresource, if the request is to a subresource
# This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the
# original request to the API server was converted to a version the webhook registered for.
requestSubResource: scale
# Name of the resource being modified
name: my-deployment
# Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object)
namespace: my-namespace
# operation can be CREATE, UPDATE, DELETE, or CONNECT
operation: UPDATE
userInfo:
# Username of the authenticated user making the request to the API server
username: admin
# UID of the authenticated user making the request to the API server
uid: 014fbff9a07c
# Group memberships of the authenticated user making the request to the API server
groups:
- system:authenticated
- my-admin-group
# Arbitrary extra info associated with the user making the request to the API server.
# This is populated by the API server authentication layer and should be included
# if any SubjectAccessReview checks are performed by the webhook.
extra:
some-key:
- some-value1
- some-value2
# object is the new object being admitted.
# It is null for DELETE operations.
object:
apiVersion: autoscaling/v1
kind: Scale
# oldObject is the existing object.
# It is null for CREATE and CONNECT operations.
oldObject:
apiVersion: autoscaling/v1
kind: Scale
# options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions.
# It is null for CONNECT operations.
options:
apiVersion: meta.k8s.io/v1
kind: UpdateOptions
# dryRun indicates the API request is running in dry run mode and will not be persisted.
# Webhooks with side effects should avoid actuating those side effects when dryRun is true.
# See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details.
dryRun: False
Response
Webhooks respond with a 200 HTTP status code, Content-Type: application/json
,
and a body containing an AdmissionReview
object (in the same version they were sent),
with the response
stanza populated, serialized to JSON.
At a minimum, the response
stanza must contain the following fields:
uid
, copied from therequest.uid
sent to the webhookallowed
, either set totrue
orfalse
Example of a minimal response from a webhook to allow a request:
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "<value from request.uid>",
"allowed": true
}
}
Example of a minimal response from a webhook to forbid a request:
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "<value from request.uid>",
"allowed": false
}
}
When rejecting a request, the webhook can customize the http code and message returned to the user
using the status
field. The specified status object is returned to the user.
See the API documentation
for details about the status
type.
Example of a response to forbid a request, customizing the HTTP status code and message presented to the user:
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "<value from request.uid>",
"allowed": false,
"status": {
"code": 403,
"message": "You cannot do this because it is Tuesday and your name starts with A"
}
}
}
When allowing a request, a mutating admission webhook may optionally modify the incoming object as well.
This is done using the patch
and patchType
fields in the response.
The only currently supported patchType
is JSONPatch
.
See JSON patch documentation for more details.
For patchType: JSONPatch
, the patch
field contains a base64-encoded array of JSON patch operations.
As an example, a single patch operation that would set spec.replicas
would be
[{"op": "add", "path": "/spec/replicas", "value": 3}]
Base64-encoded, this would be W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0=
So a webhook response to add that label would be:
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "<value from request.uid>",
"allowed": true,
"patchType": "JSONPatch",
"patch": "W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0="
}
}
Admission webhooks can optionally return warning messages that are returned to the requesting client
in HTTP Warning
headers with a warning code of 299. Warnings can be sent with allowed or rejected admission responses.
If you're implementing a webhook that returns a warning:
- Don't include a "Warning:" prefix in the message
- Use warning messages to describe problems the client making the API request should correct or be aware of
- Limit warnings to 120 characters if possible
Caution:
Individual warning messages over 256 characters may be truncated by the API server before being returned to clients. If more than 4096 characters of warning messages are added (from all sources), additional warning messages are ignored.{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "<value from request.uid>",
"allowed": true,
"warnings": [
"duplicate envvar entries specified with name MY_ENV",
"memory request less than 4MB specified for container mycontainer, which will not start successfully"
]
}
}
Webhook configuration
To register admission webhooks, create MutatingWebhookConfiguration
or ValidatingWebhookConfiguration
API objects.
The name of a MutatingWebhookConfiguration
or a ValidatingWebhookConfiguration
object must be a valid
DNS subdomain name.
Each configuration can contain one or more webhooks. If multiple webhooks are specified in a single configuration, each must be given a unique name. This is required in order to make resulting audit logs and metrics easier to match up to active configurations.
Each webhook defines the following things.
Matching requests: rules
Each webhook must specify a list of rules used to determine if a request to the API server should be sent to the webhook. Each rule specifies one or more operations, apiGroups, apiVersions, and resources, and a resource scope:
operations
lists one or more operations to match. Can be"CREATE"
,"UPDATE"
,"DELETE"
,"CONNECT"
, or"*"
to match all.apiGroups
lists one or more API groups to match.""
is the core API group."*"
matches all API groups.apiVersions
lists one or more API versions to match."*"
matches all API versions.resources
lists one or more resources to match."*"
matches all resources, but not subresources."*/*"
matches all resources and subresources."pods/*"
matches all subresources of pods."*/status"
matches all status subresources.
scope
specifies a scope to match. Valid values are"Cluster"
,"Namespaced"
, and"*"
. Subresources match the scope of their parent resource. Default is"*"
."Cluster"
means that only cluster-scoped resources will match this rule (Namespace API objects are cluster-scoped)."Namespaced"
means that only namespaced resources will match this rule."*"
means that there are no scope restrictions.
If an incoming request matches one of the specified operations
, groups
, versions
,
resources
, and scope
for any of a webhook's rules
, the request is sent to the webhook.
Here are other examples of rules that could be used to specify which resources should be intercepted.
Match CREATE
or UPDATE
requests to apps/v1
and apps/v1beta1
deployments
and replicasets
:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
...
webhooks:
- name: my-webhook.example.com
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: ["apps"]
apiVersions: ["v1", "v1beta1"]
resources: ["deployments", "replicasets"]
scope: "Namespaced"
...
Match create requests for all resources (but not subresources) in all API groups and versions:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
rules:
- operations: ["CREATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
scope: "*"
Match update requests for all status
subresources in all API groups and versions:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
rules:
- operations: ["UPDATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*/status"]
scope: "*"
Matching requests: objectSelector
Webhooks may optionally limit which requests are intercepted based on the labels of the
objects they would be sent, by specifying an objectSelector
. If specified, the objectSelector
is evaluated against both the object and oldObject that would be sent to the webhook,
and is considered to match if either object matches the selector.
A null object (oldObject
in the case of create, or newObject
in the case of delete),
or an object that cannot have labels (like a DeploymentRollback
or a PodProxyOptions
object)
is not considered to match.
Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels.
This example shows a mutating webhook that would match a CREATE
of any resource (but not subresources) with the label foo: bar
:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
objectSelector:
matchLabels:
foo: bar
rules:
- operations: ["CREATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
scope: "*"
See labels concept for more examples of label selectors.
Matching requests: namespaceSelector
Webhooks may optionally limit which requests for namespaced resources are intercepted,
based on the labels of the containing namespace, by specifying a namespaceSelector
.
The namespaceSelector
decides whether to run the webhook on a request for a namespaced resource
(or a Namespace object), based on whether the namespace's labels match the selector.
If the object itself is a namespace, the matching is performed on object.metadata.labels.
If the object is a cluster scoped resource other than a Namespace, namespaceSelector
has no effect.
This example shows a mutating webhook that matches a CREATE
of any namespaced resource inside a namespace
that does not have a "runlevel" label of "0" or "1":
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
namespaceSelector:
matchExpressions:
- key: runlevel
operator: NotIn
values: ["0","1"]
rules:
- operations: ["CREATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
scope: "Namespaced"
This example shows a validating webhook that matches a CREATE
of any namespaced resource inside
a namespace that is associated with the "environment" of "prod" or "staging":
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
namespaceSelector:
matchExpressions:
- key: environment
operator: In
values: ["prod","staging"]
rules:
- operations: ["CREATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
scope: "Namespaced"
See labels concept for more examples of label selectors.
Matching requests: matchPolicy
API servers can make objects available via multiple API groups or versions.
For example, if a webhook only specified a rule for some API groups/versions
(like apiGroups:["apps"], apiVersions:["v1","v1beta1"]
),
and a request was made to modify the resource via another API group/version (like extensions/v1beta1
),
the request would not be sent to the webhook.
The matchPolicy
lets a webhook define how its rules
are used to match incoming requests.
Allowed values are Exact
or Equivalent
.
Exact
means a request should be intercepted only if it exactly matches a specified rule.Equivalent
means a request should be intercepted if it modifies a resource listed inrules
, even via another API group or version.
In the example given above, the webhook that only registered for apps/v1
could use matchPolicy
:
matchPolicy: Exact
would mean theextensions/v1beta1
request would not be sent to the webhookmatchPolicy: Equivalent
means theextensions/v1beta1
request would be sent to the webhook (with the objects converted to a version the webhook had specified:apps/v1
)
Specifying Equivalent
is recommended, and ensures that webhooks continue to intercept the
resources they expect when upgrades enable new versions of the resource in the API server.
When a resource stops being served by the API server, it is no longer considered equivalent to
other versions of that resource that are still served.
For example, extensions/v1beta1
deployments were first deprecated and then removed (in Kubernetes v1.16).
Since that removal, a webhook with a apiGroups:["extensions"], apiVersions:["v1beta1"], resources:["deployments"]
rule
does not intercept deployments created via apps/v1
APIs. For that reason, webhooks should prefer registering
for stable versions of resources.
This example shows a validating webhook that intercepts modifications to deployments (no matter the API group or version),
and is always sent an apps/v1
Deployment
object:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
matchPolicy: Equivalent
rules:
- operations: ["CREATE","UPDATE","DELETE"]
apiGroups: ["apps"]
apiVersions: ["v1"]
resources: ["deployments"]
scope: "Namespaced"
The matchPolicy
for an admission webhooks defaults to Equivalent
.
Matching requests: matchConditions
Kubernetes v1.30 [stable]
You can define match conditions for webhooks if you need fine-grained request filtering. These
conditions are useful if you find that match rules, objectSelectors
and namespaceSelectors
still
doesn't provide the filtering you want over when to call out over HTTP. Match conditions are
CEL expressions. All match conditions must evaluate to true for the
webhook to be called.
Here is an example illustrating a few different uses for match conditions:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
matchPolicy: Equivalent
rules:
- operations: ['CREATE','UPDATE']
apiGroups: ['*']
apiVersions: ['*']
resources: ['*']
failurePolicy: 'Ignore' # Fail-open (optional)
sideEffects: None
clientConfig:
service:
namespace: my-namespace
name: my-webhook
caBundle: '<omitted>'
# You can have up to 64 matchConditions per webhook
matchConditions:
- name: 'exclude-leases' # Each match condition must have a unique name
expression: '!(request.resource.group == "coordination.k8s.io" && request.resource.resource == "leases")' # Match non-lease resources.
- name: 'exclude-kubelet-requests'
expression: '!("system:nodes" in request.userInfo.groups)' # Match requests made by non-node users.
- name: 'rbac' # Skip RBAC requests, which are handled by the second webhook.
expression: 'request.resource.group != "rbac.authorization.k8s.io"'
# This example illustrates the use of the 'authorizer'. The authorization check is more expensive
# than a simple expression, so in this example it is scoped to only RBAC requests by using a second
# webhook. Both webhooks can be served by the same endpoint.
- name: rbac.my-webhook.example.com
matchPolicy: Equivalent
rules:
- operations: ['CREATE','UPDATE']
apiGroups: ['rbac.authorization.k8s.io']
apiVersions: ['*']
resources: ['*']
failurePolicy: 'Fail' # Fail-closed (the default)
sideEffects: None
clientConfig:
service:
namespace: my-namespace
name: my-webhook
caBundle: '<omitted>'
# You can have up to 64 matchConditions per webhook
matchConditions:
- name: 'breakglass'
# Skip requests made by users authorized to 'breakglass' on this webhook.
# The 'breakglass' API verb does not need to exist outside this check.
expression: '!authorizer.group("admissionregistration.k8s.io").resource("validatingwebhookconfigurations").name("my-webhook.example.com").check("breakglass").allowed()'
Note:
You can define up to 64 elements in thematchConditions
field per webhook.Match conditions have access to the following CEL variables:
object
- The object from the incoming request. The value is null for DELETE requests. The object version may be converted based on the matchPolicy.oldObject
- The existing object. The value is null for CREATE requests.request
- The request portion of the AdmissionReview, excludingobject
andoldObject
.authorizer
- A CEL Authorizer. May be used to perform authorization checks for the principal (authenticated user) of the request. See Authz in the Kubernetes CEL library documentation for more details.authorizer.requestResource
- A shortcut for an authorization check configured with the request resource (group, resource, (subresource), namespace, name).
For more information on CEL expressions, refer to the Common Expression Language in Kubernetes reference.
In the event of an error evaluating a match condition the webhook is never called. Whether to reject the request is determined as follows:
- If any match condition evaluated to
false
(regardless of other errors), the API server skips the webhook. - Otherwise:
- for
failurePolicy: Fail
, reject the request (without calling the webhook). - for
failurePolicy: Ignore
, proceed with the request but skip the webhook.
- for
Contacting the webhook
Once the API server has determined a request should be sent to a webhook,
it needs to know how to contact the webhook. This is specified in the clientConfig
stanza of the webhook configuration.
Webhooks can either be called via a URL or a service reference, and can optionally include a custom CA bundle to use to verify the TLS connection.
URL
url
gives the location of the webhook, in standard URL form
(scheme://host:port/path
).
The host
should not refer to a service running in the cluster; use
a service reference by specifying the service
field instead.
The host might be resolved via external DNS in some API servers
(e.g., kube-apiserver
cannot resolve in-cluster DNS as that would
be a layering violation). host
may also be an IP address.
Please note that using localhost
or 127.0.0.1
as a host
is
risky unless you take great care to run this webhook on all hosts
which run an API server which might need to make calls to this
webhook. Such installations are likely to be non-portable or not readily
run in a new cluster.
The scheme must be "https"; the URL must begin with "https://".
Attempting to use a user or basic auth (for example user:password@
) is not allowed.
Fragments (#...
) and query parameters (?...
) are also not allowed.
Here is an example of a mutating webhook configured to call a URL (and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle):
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
clientConfig:
url: "https://my-webhook.example.com:9443/my-webhook-path"
Service reference
The service
stanza inside clientConfig
is a reference to the service for this webhook.
If the webhook is running within the cluster, then you should use service
instead of url
.
The service namespace and name are required. The port is optional and defaults to 443.
The path is optional and defaults to "/".
Here is an example of a mutating webhook configured to call a service on port "1234"
at the subpath "/my-path", and to verify the TLS connection against the ServerName
my-service-name.my-service-namespace.svc
using a custom CA bundle:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
clientConfig:
caBundle: <CA_BUNDLE>
service:
namespace: my-service-namespace
name: my-service-name
path: /my-path
port: 1234
Note:
You must replace the<CA_BUNDLE>
in the above example by a valid CA bundle
which is a PEM-encoded CA bundle for validating the webhook's server certificate.Side effects
Webhooks typically operate only on the content of the AdmissionReview
sent to them.
Some webhooks, however, make out-of-band changes as part of processing admission requests.
Webhooks that make out-of-band changes ("side effects") must also have a reconciliation mechanism (like a controller) that periodically determines the actual state of the world, and adjusts the out-of-band data modified by the admission webhook to reflect reality. This is because a call to an admission webhook does not guarantee the admitted object will be persisted as is, or at all. Later webhooks can modify the content of the object, a conflict could be encountered while writing to storage, or the server could power off before persisting the object.
Additionally, webhooks with side effects must skip those side-effects when dryRun: true
admission requests are handled.
A webhook must explicitly indicate that it will not have side-effects when run with dryRun
,
or the dry-run request will not be sent to the webhook and the API request will fail instead.
Webhooks indicate whether they have side effects using the sideEffects
field in the webhook configuration:
None
: calling the webhook will have no side effects.NoneOnDryRun
: calling the webhook will possibly have side effects, but if a request withdryRun: true
is sent to the webhook, the webhook will suppress the side effects (the webhook isdryRun
-aware).
Here is an example of a validating webhook indicating it has no side effects on dryRun: true
requests:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
sideEffects: NoneOnDryRun
Timeouts
Because webhooks add to API request latency, they should evaluate as quickly as possible.
timeoutSeconds
allows configuring how long the API server should wait for a webhook to respond
before treating the call as a failure.
If the timeout expires before the webhook responds, the webhook call will be ignored or the API call will be rejected based on the failure policy.
The timeout value must be between 1 and 30 seconds.
Here is an example of a validating webhook with a custom timeout of 2 seconds:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
timeoutSeconds: 2
The timeout for an admission webhook defaults to 10 seconds.
Reinvocation policy
A single ordering of mutating admissions plugins (including webhooks) does not work for all cases
(see https://issue.k8s.io/64333 as an example). A mutating webhook can add a new sub-structure
to the object (like adding a container
to a pod
), and other mutating plugins which have already
run may have opinions on those new structures (like setting an imagePullPolicy
on all containers).
To allow mutating admission plugins to observe changes made by other plugins,
built-in mutating admission plugins are re-run if a mutating webhook modifies an object,
and mutating webhooks can specify a reinvocationPolicy
to control whether they are reinvoked as well.
reinvocationPolicy
may be set to Never
or IfNeeded
. It defaults to Never
.
Never
: the webhook must not be called more than once in a single admission evaluation.IfNeeded
: the webhook may be called again as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call.
The important elements to note are:
- The number of additional invocations is not guaranteed to be exactly one.
- If additional invocations result in further modifications to the object, webhooks are not guaranteed to be invoked again.
- Webhooks that use this option may be reordered to minimize the number of additional invocations.
- To validate an object after all mutations are guaranteed complete, use a validating admission webhook instead (recommended for webhooks with side-effects).
Here is an example of a mutating webhook opting into being re-invoked if later admission plugins modify the object:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
reinvocationPolicy: IfNeeded
Mutating webhooks must be idempotent, able to successfully process an object they have already admitted and potentially modified. This is true for all mutating admission webhooks, since any change they can make in an object could already exist in the user-provided object, but it is essential for webhooks that opt into reinvocation.
Failure policy
failurePolicy
defines how unrecognized errors and timeout errors from the admission webhook
are handled. Allowed values are Ignore
or Fail
.
Ignore
means that an error calling the webhook is ignored and the API request is allowed to continue.Fail
means that an error calling the webhook causes the admission to fail and the API request to be rejected.
Here is a mutating webhook configured to reject an API request if errors are encountered calling the admission webhook:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
failurePolicy: Fail
The default failurePolicy
for an admission webhooks is Fail
.
Monitoring admission webhooks
The API server provides ways to monitor admission webhook behaviors. These monitoring mechanisms help cluster admins to answer questions like:
Which mutating webhook mutated the object in a API request?
What change did the mutating webhook applied to the object?
Which webhooks are frequently rejecting API requests? What's the reason for a rejection?
Mutating webhook auditing annotations
Sometimes it's useful to know which mutating webhook mutated the object in a API request, and what change did the webhook apply.
The Kubernetes API server performs auditing on each mutating webhook invocation. Each invocation generates an auditing annotation capturing if a request object is mutated by the invocation, and optionally generates an annotation capturing the applied patch from the webhook admission response. The annotations are set in the audit event for given request on given stage of its execution, which is then pre-processed according to a certain policy and written to a backend.
The audit level of a event determines which annotations get recorded:
At
Metadata
audit level or higher, an annotation with keymutation.webhook.admission.k8s.io/round_{round idx}_index_{order idx}
gets logged with JSON payload indicating a webhook gets invoked for given request and whether it mutated the object or not.For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is ordered the third in the mutating webhook chain, and didn't mutated the request object during the invocation.
# the audit event recorded { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "annotations": { "mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook.example.com\",\"mutated\": false}" # other annotations ... } # other fields ... }
# the annotation value deserialized { "configuration": "my-mutating-webhook-configuration.example.com", "webhook": "my-webhook.example.com", "mutated": false }
The following annotation gets recorded for a webhook being invoked in the first round. The webhook is ordered the first in the mutating webhook chain, and mutated the request object during the invocation.
# the audit event recorded { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "annotations": { "mutation.webhook.admission.k8s.io/round_0_index_0": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"mutated\": true}" # other annotations ... } # other fields ... }
# the annotation value deserialized { "configuration": "my-mutating-webhook-configuration.example.com", "webhook": "my-webhook-always-mutate.example.com", "mutated": true }
At
Request
audit level or higher, an annotation with keypatch.webhook.admission.k8s.io/round_{round idx}_index_{order idx}
gets logged with JSON payload indicating a webhook gets invoked for given request and what patch gets applied to the request object.For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is ordered the fourth in the mutating webhook chain, and responded with a JSON patch which got applied to the request object.
# the audit event recorded { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "annotations": { "patch.webhook.admission.k8s.io/round_1_index_3": "{\"configuration\":\"my-other-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"patch\":[{\"op\":\"add\",\"path\":\"/data/mutation-stage\",\"value\":\"yes\"}],\"patchType\":\"JSONPatch\"}" # other annotations ... } # other fields ... }
# the annotation value deserialized { "configuration": "my-other-mutating-webhook-configuration.example.com", "webhook": "my-webhook-always-mutate.example.com", "patchType": "JSONPatch", "patch": [ { "op": "add", "path": "/data/mutation-stage", "value": "yes" } ] }
Admission webhook metrics
The API server exposes Prometheus metrics from the /metrics
endpoint, which can be used for monitoring and
diagnosing API server status. The following metrics record status related to admission webhooks.
API server admission webhook rejection count
Sometimes it's useful to know which admission webhooks are frequently rejecting API requests, and the reason for a rejection.
The API server exposes a Prometheus counter metric recording admission webhook rejections. The metrics are labelled to identify the causes of webhook rejection(s):
name
: the name of the webhook that rejected a request.operation
: the operation type of the request, can be one ofCREATE
,UPDATE
,DELETE
andCONNECT
.type
: the admission webhook type, can be one ofadmit
andvalidating
.error_type
: identifies if an error occurred during the webhook invocation that caused the rejection. Its value can be one of:calling_webhook_error
: unrecognized errors or timeout errors from the admission webhook happened and the webhook's Failure policy is set toFail
.no_error
: no error occurred. The webhook rejected the request withallowed: false
in the admission response. The metrics labelrejection_code
records the.status.code
set in the admission response.apiserver_internal_error
: an API server internal error happened.
rejection_code
: the HTTP status code set in the admission response when a webhook rejected a request.
Example of the rejection count metrics:
# HELP apiserver_admission_webhook_rejection_count [ALPHA] Admission webhook rejection count, identified by name and broken out for each admission type (validating or admit) and operation. Additional labels specify an error type (calling_webhook_error or apiserver_internal_error if an error occurred; no_error otherwise) and optionally a non-zero rejection code if the webhook rejects the request with an HTTP status code (honored by the apiserver when the code is greater or equal to 400). Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
# TYPE apiserver_admission_webhook_rejection_count counter
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="always-timeout-webhook.example.com",operation="CREATE",rejection_code="0",type="validating"} 1
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="invalid-admission-response-webhook.example.com",operation="CREATE",rejection_code="0",type="validating"} 1
apiserver_admission_webhook_rejection_count{error_type="no_error",name="deny-unwanted-configmap-data.example.com",operation="CREATE",rejection_code="400",type="validating"} 13
Best practices and warnings
Idempotence
An idempotent mutating admission webhook is able to successfully process an object it has already admitted and potentially modified. The admission can be applied multiple times without changing the result beyond the initial application.
Example of idempotent mutating admission webhooks:
For a
CREATE
pod request, set the field.spec.securityContext.runAsNonRoot
of the pod to true, to enforce security best practices.For a
CREATE
pod request, if the field.spec.containers[].resources.limits
of a container is not set, set default resource limits.For a
CREATE
pod request, inject a sidecar container with namefoo-sidecar
if no container with the namefoo-sidecar
already exists.
In the cases above, the webhook can be safely reinvoked, or admit an object that already has the fields set.
Example of non-idempotent mutating admission webhooks:
For a
CREATE
pod request, inject a sidecar container with namefoo-sidecar
suffixed with the current timestamp (e.g.foo-sidecar-19700101-000000
).For a
CREATE
/UPDATE
pod request, reject if the pod has label"env"
set, otherwise add an"env": "prod"
label to the pod.For a
CREATE
pod request, blindly append a sidecar container namedfoo-sidecar
without looking to see if there is already afoo-sidecar
container in the pod.
In the first case above, reinvoking the webhook can result in the same sidecar being injected multiple times to a pod, each time with a different container name. Similarly the webhook can inject duplicated containers if the sidecar already exists in a user-provided pod.
In the second case above, reinvoking the webhook will result in the webhook failing on its own output.
In the third case above, reinvoking the webhook will result in duplicated containers in the pod spec, which makes the request invalid and rejected by the API server.
Intercepting all versions of an object
It is recommended that admission webhooks should always intercept all versions of an object by setting .webhooks[].matchPolicy
to Equivalent
. It is also recommended that admission webhooks should prefer registering for stable versions of resources.
Failure to intercept all versions of an object can result in admission policies not being enforced for requests in certain
versions. See Matching requests: matchPolicy for examples.
Availability
It is recommended that admission webhooks should evaluate as quickly as possible (typically in milliseconds), since they add to API request latency. It is encouraged to use a small timeout for webhooks. See Timeouts for more detail.
It is recommended that admission webhooks should leverage some format of load-balancing, to provide high availability and performance benefits. If a webhook is running within the cluster, you can run multiple webhook backends behind a service to leverage the load-balancing that service supports.
Guaranteeing the final state of the object is seen
Admission webhooks that need to guarantee they see the final state of the object in order to enforce policy should use a validating admission webhook, since objects can be modified after being seen by mutating webhooks.
For example, a mutating admission webhook is configured to inject a sidecar container with name
"foo-sidecar" on every CREATE
pod request. If the sidecar must be present, a validating
admisson webhook should also be configured to intercept CREATE
pod requests, and validate that a
container with name "foo-sidecar" with the expected configuration exists in the to-be-created
object.
Avoiding deadlocks in self-hosted webhooks
A webhook running inside the cluster might cause deadlocks for its own deployment if it is configured to intercept resources required to start its own pods.
For example, a mutating admission webhook is configured to admit CREATE
pod requests only if a certain label is set in the
pod (e.g. "env": "prod"
). The webhook server runs in a deployment which doesn't set the "env"
label.
When a node that runs the webhook server pods
becomes unhealthy, the webhook deployment will try to reschedule the pods to another node. However the requests will
get rejected by the existing webhook server since the "env"
label is unset, and the migration cannot happen.
It is recommended to exclude the namespace where your webhook is running with a namespaceSelector.
Side effects
It is recommended that admission webhooks should avoid side effects if possible, which means the webhooks operate only on the
content of the AdmissionReview
sent to them, and do not make out-of-band changes. The .webhooks[].sideEffects
field should
be set to None
if a webhook doesn't have any side effect.
If side effects are required during the admission evaluation, they must be suppressed when processing an
AdmissionReview
object with dryRun
set to true
, and the .webhooks[].sideEffects
field should be
set to NoneOnDryRun
. See Side effects for more detail.
Avoiding operating on the kube-system namespace
The kube-system
namespace contains objects created by the Kubernetes system,
e.g. service accounts for the control plane components, pods like kube-dns
.
Accidentally mutating or rejecting requests in the kube-system
namespace may
cause the control plane components to stop functioning or introduce unknown behavior.
If your admission webhooks don't intend to modify the behavior of the Kubernetes control
plane, exclude the kube-system
namespace from being intercepted using a
namespaceSelector
.