This is the multi-page printable view of this section. Click here to print.
Manage Kubernetes Objects
- 1: Declarative Management of Kubernetes Objects Using Configuration Files
- 2: Declarative Management of Kubernetes Objects Using Kustomize
- 3: Managing Kubernetes Objects Using Imperative Commands
- 4: Imperative Management of Kubernetes Objects Using Configuration Files
- 5: Update API Objects in Place Using kubectl patch
- 6: Migrate Kubernetes Objects Using Storage Version Migration
1 - Declarative Management of Kubernetes Objects Using Configuration Files
Kubernetes objects can be created, updated, and deleted by storing multiple
object configuration files in a directory and using kubectl apply
to
recursively create and update those objects as needed. This method
retains writes made to live objects without merging the changes
back into the object configuration files. kubectl diff
also gives you a
preview of what changes apply
will make.
Before you begin
Install kubectl
.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
To check the version, enterkubectl version
.Trade-offs
The kubectl
tool supports three kinds of object management:
- Imperative commands
- Imperative object configuration
- Declarative object configuration
See Kubernetes Object Management for a discussion of the advantages and disadvantage of each kind of object management.
Overview
Declarative object configuration requires a firm understanding of the Kubernetes object definitions and configuration. Read and complete the following documents if you have not already:
- Managing Kubernetes Objects Using Imperative Commands
- Imperative Management of Kubernetes Objects Using Configuration Files
Following are definitions for terms used in this document:
- object configuration file / configuration file: A file that defines the
configuration for a Kubernetes object. This topic shows how to pass configuration
files to
kubectl apply
. Configuration files are typically stored in source control, such as Git. - live object configuration / live configuration: The live configuration values of an object, as observed by the Kubernetes cluster. These are kept in the Kubernetes cluster storage, typically etcd.
- declarative configuration writer / declarative writer: A person or software component
that makes updates to a live object. The live writers referred to in this topic make changes
to object configuration files and run
kubectl apply
to write the changes.
How to create objects
Use kubectl apply
to create all objects, except those that already exist,
defined by configuration files in a specified directory:
kubectl apply -f <directory>
This sets the kubectl.kubernetes.io/last-applied-configuration: '{...}'
annotation on each object. The annotation contains the contents of the object
configuration file that was used to create the object.
Note:
Add the-R
flag to recursively process directories.Here's an example of an object configuration file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
minReadySeconds: 5
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
Run kubectl diff
to print the object that will be created:
kubectl diff -f https://k8s.io/examples/application/simple_deployment.yaml
Note:
diff
uses server-side dry-run,
which needs to be enabled on kube-apiserver
.
Since diff
performs a server-side apply request in dry-run mode,
it requires granting PATCH
, CREATE
, and UPDATE
permissions.
See Dry-Run Authorization
for details.
Create the object using kubectl apply
:
kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml
Print the live configuration using kubectl get
:
kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml
The output shows that the kubectl.kubernetes.io/last-applied-configuration
annotation
was written to the live configuration, and it matches the configuration file:
kind: Deployment
metadata:
annotations:
# ...
# This is the json representation of simple_deployment.yaml
# It was written by kubectl apply when the object was created
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
# ...
minReadySeconds: 5
selector:
matchLabels:
# ...
app: nginx
template:
metadata:
# ...
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
# ...
name: nginx
ports:
- containerPort: 80
# ...
# ...
# ...
# ...
How to update objects
You can also use kubectl apply
to update all objects defined in a directory, even
if those objects already exist. This approach accomplishes the following:
- Sets fields that appear in the configuration file in the live configuration.
- Clears fields removed from the configuration file in the live configuration.
kubectl diff -f <directory>
kubectl apply -f <directory>
Note:
Add the-R
flag to recursively process directories.Here's an example configuration file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
minReadySeconds: 5
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
Create the object using kubectl apply
:
kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml
Note:
For purposes of illustration, the preceding command refers to a single configuration file instead of a directory.Print the live configuration using kubectl get
:
kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml
The output shows that the kubectl.kubernetes.io/last-applied-configuration
annotation
was written to the live configuration, and it matches the configuration file:
kind: Deployment
metadata:
annotations:
# ...
# This is the json representation of simple_deployment.yaml
# It was written by kubectl apply when the object was created
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
# ...
minReadySeconds: 5
selector:
matchLabels:
# ...
app: nginx
template:
metadata:
# ...
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
# ...
name: nginx
ports:
- containerPort: 80
# ...
# ...
# ...
# ...
Directly update the replicas
field in the live configuration by using kubectl scale
.
This does not use kubectl apply
:
kubectl scale deployment/nginx-deployment --replicas=2
Print the live configuration using kubectl get
:
kubectl get deployment nginx-deployment -o yaml
The output shows that the replicas
field has been set to 2, and the last-applied-configuration
annotation does not contain a replicas
field:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
# ...
# note that the annotation does not contain replicas
# because it was not updated through apply
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
replicas: 2 # written by scale
# ...
minReadySeconds: 5
selector:
matchLabels:
# ...
app: nginx
template:
metadata:
# ...
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
# ...
name: nginx
ports:
- containerPort: 80
# ...
Update the simple_deployment.yaml
configuration file to change the image from
nginx:1.14.2
to nginx:1.16.1
, and delete the minReadySeconds
field:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.16.1 # update the image
ports:
- containerPort: 80
Apply the changes made to the configuration file:
kubectl diff -f https://k8s.io/examples/application/update_deployment.yaml
kubectl apply -f https://k8s.io/examples/application/update_deployment.yaml
Print the live configuration using kubectl get
:
kubectl get -f https://k8s.io/examples/application/update_deployment.yaml -o yaml
The output shows the following changes to the live configuration:
- The
replicas
field retains the value of 2 set bykubectl scale
. This is possible because it is omitted from the configuration file. - The
image
field has been updated tonginx:1.16.1
fromnginx:1.14.2
. - The
last-applied-configuration
annotation has been updated with the new image. - The
minReadySeconds
field has been cleared. - The
last-applied-configuration
annotation no longer contains theminReadySeconds
field.
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
# ...
# The annotation contains the updated image to nginx 1.16.1,
# but does not contain the updated replicas to 2
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
replicas: 2 # Set by `kubectl scale`. Ignored by `kubectl apply`.
# minReadySeconds cleared by `kubectl apply`
# ...
selector:
matchLabels:
# ...
app: nginx
template:
metadata:
# ...
labels:
app: nginx
spec:
containers:
- image: nginx:1.16.1 # Set by `kubectl apply`
# ...
name: nginx
ports:
- containerPort: 80
# ...
# ...
# ...
# ...
Warning:
Mixingkubectl apply
with the imperative object configuration commands
create
and replace
is not supported. This is because create
and replace
do not retain the kubectl.kubernetes.io/last-applied-configuration
that kubectl apply
uses to compute updates.How to delete objects
There are two approaches to delete objects managed by kubectl apply
.
Recommended: kubectl delete -f <filename>
Manually deleting objects using the imperative command is the recommended approach, as it is more explicit about what is being deleted, and less likely to result in the user deleting something unintentionally:
kubectl delete -f <filename>
Alternative: kubectl apply -f <directory> --prune
As an alternative to kubectl delete
, you can use kubectl apply
to identify objects to be deleted after
their manifests have been removed from a directory in the local filesystem.
In Kubernetes 1.31, there are two pruning modes available in kubectl apply:
- Allowlist-based pruning: This mode has existed since kubectl v1.5 but is still in alpha due to usability, correctness and performance issues with its design. The ApplySet-based mode is designed to replace it.
- ApplySet-based pruning: An apply set is a server-side object (by default, a Secret) that kubectl can use to accurately and efficiently track set membership across apply operations. This mode was introduced in alpha in kubectl v1.27 as a replacement for allowlist-based pruning.
<div class="feature-state-notice feature-alpha">
<span class="feature-state-name">FEATURE STATE:</span>
<code>Kubernetes v1.5 [alpha]</code>
</div>
Warning:
Take care when using--prune
with kubectl apply
in allow list mode. Which
objects are pruned depends on the values of the --prune-allowlist
, --selector
and --namespace
flags, and relies on dynamic discovery of the objects in scope.
Especially if flag values are changed between invocations, this can lead to objects
being unexpectedly deleted or retained.To use allowlist-based pruning, add the following flags to your kubectl apply
invocation:
--prune
: Delete previously applied objects that are not in the set passed to the current invocation.--prune-allowlist
: A list of group-version-kinds (GVKs) to consider for pruning. This flag is optional but strongly encouraged, as its default value is a partial list of both namespaced and cluster-scoped types, which can lead to surprising results.--selector/-l
: Use a label selector to constrain the set of objects selected for pruning. This flag is optional but strongly encouraged.--all
: use instead of--selector/-l
to explicitly select all previously applied objects of the allowlisted types.
Allowlist-based pruning queries the API server for all objects of the allowlisted GVKs that match the given labels (if any), and attempts to match the returned live object configurations against the object
manifest files. If an object matches the query, and it does not have a
manifest in the directory, and it has a kubectl.kubernetes.io/last-applied-configuration
annotation,
it is deleted.
kubectl apply -f <directory> --prune -l <labels> --prune-allowlist=<gvk-list>
Warning:
Apply with prune should only be run against the root directory containing the object manifests. Running against sub-directories can cause objects to be unintentionally deleted if they were previously applied, have the labels given (if any), and do not appear in the subdirectory.<div class="feature-state-notice feature-alpha">
<span class="feature-state-name">FEATURE STATE:</span>
<code>Kubernetes v1.27 [alpha]</code>
</div>
Caution:
kubectl apply --prune --applyset
is in alpha, and backwards incompatible
changes might be introduced in subsequent releases.To use ApplySet-based pruning, set the KUBECTL_APPLYSET=true
environment variable,
and add the following flags to your kubectl apply
invocation:
--prune
: Delete previously applied objects that are not in the set passed to the current invocation.--applyset
: The name of an object that kubectl can use to accurately and efficiently track set membership acrossapply
operations.
KUBECTL_APPLYSET=true kubectl apply -f <directory> --prune --applyset=<name>
By default, the type of the ApplySet parent object used is a Secret. However,
ConfigMaps can also be used in the format: --applyset=configmaps/<name>
.
When using a Secret or ConfigMap, kubectl will create the object if it does not already exist.
It is also possible to use custom resources as ApplySet parent objects. To enable
this, label the Custom Resource Definition (CRD) that defines the resource you want
to use with the following: applyset.kubernetes.io/is-parent-type: true
. Then, create
the object you want to use as an ApplySet parent (kubectl does not do this automatically
for custom resources). Finally, refer to that object in the applyset flag as follows:
--applyset=<resource>.<group>/<name>
(for example, widgets.custom.example.com/widget-name
).
With ApplySet-based pruning, kubectl adds the applyset.kubernetes.io/part-of=<parentID>
label to each object in the set before they are sent to the server. For performance reasons,
it also collects the list of resource types and namespaces that the set contains and adds
these in annotations on the live parent object. Finally, at the end of the apply operation,
it queries the API server for objects of those types in those namespaces
(or in the cluster scope, as applicable) that belong to the set, as defined by the
applyset.kubernetes.io/part-of=<parentID>
label.
Caveats and restrictions:
- Each object may be a member of at most one set.
- The
--namespace
flag is required when using any namespaced parent, including the default Secret. This means that ApplySets spanning multiple namespaces must use a cluster-scoped custom resource as the parent object. - To safely use ApplySet-based pruning with multiple directories, use a unique ApplySet name for each.
How to view an object
You can use kubectl get
with -o yaml
to view the configuration of a live object:
kubectl get -f <filename|url> -o yaml
How apply calculates differences and merges changes
Caution:
A patch is an update operation that is scoped to specific fields of an object instead of the entire object. This enables updating only a specific set of fields on an object without reading the object first.When kubectl apply
updates the live configuration for an object,
it does so by sending a patch request to the API server. The
patch defines updates scoped to specific fields of the live object
configuration. The kubectl apply
command calculates this patch request
using the configuration file, the live configuration, and the
last-applied-configuration
annotation stored in the live configuration.
Merge patch calculation
The kubectl apply
command writes the contents of the configuration file to the
kubectl.kubernetes.io/last-applied-configuration
annotation. This
is used to identify fields that have been removed from the configuration
file and need to be cleared from the live configuration. Here are the steps used
to calculate which fields should be deleted or set:
- Calculate the fields to delete. These are the fields present in
last-applied-configuration
and missing from the configuration file. - Calculate the fields to add or set. These are the fields present in the configuration file whose values don't match the live configuration.
Here's an example. Suppose this is the configuration file for a Deployment object:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.16.1 # update the image
ports:
- containerPort: 80
Also, suppose this is the live configuration for the same Deployment object:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
# ...
# note that the annotation does not contain replicas
# because it was not updated through apply
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
replicas: 2 # written by scale
# ...
minReadySeconds: 5
selector:
matchLabels:
# ...
app: nginx
template:
metadata:
# ...
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
# ...
name: nginx
ports:
- containerPort: 80
# ...
Here are the merge calculations that would be performed by kubectl apply
:
- Calculate the fields to delete by reading values from
last-applied-configuration
and comparing them to values in the configuration file. Clear fields explicitly set to null in the local object configuration file regardless of whether they appear in thelast-applied-configuration
. In this example,minReadySeconds
appears in thelast-applied-configuration
annotation, but does not appear in the configuration file. Action: ClearminReadySeconds
from the live configuration. - Calculate the fields to set by reading values from the configuration
file and comparing them to values in the live configuration. In this example,
the value of
image
in the configuration file does not match the value in the live configuration. Action: Set the value ofimage
in the live configuration. - Set the
last-applied-configuration
annotation to match the value of the configuration file. - Merge the results from 1, 2, 3 into a single patch request to the API server.
Here is the live configuration that is the result of the merge:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
# ...
# The annotation contains the updated image to nginx 1.16.1,
# but does not contain the updated replicas to 2
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
selector:
matchLabels:
# ...
app: nginx
replicas: 2 # Set by `kubectl scale`. Ignored by `kubectl apply`.
# minReadySeconds cleared by `kubectl apply`
# ...
template:
metadata:
# ...
labels:
app: nginx
spec:
containers:
- image: nginx:1.16.1 # Set by `kubectl apply`
# ...
name: nginx
ports:
- containerPort: 80
# ...
# ...
# ...
# ...
How different types of fields are merged
How a particular field in a configuration file is merged with the live configuration depends on the type of the field. There are several types of fields:
primitive: A field of type string, integer, or boolean. For example,
image
andreplicas
are primitive fields. Action: Replace.map, also called object: A field of type map or a complex type that contains subfields. For example,
labels
,annotations
,spec
andmetadata
are all maps. Action: Merge elements or subfields.list: A field containing a list of items that can be either primitive types or maps. For example,
containers
,ports
, andargs
are lists. Action: Varies.
When kubectl apply
updates a map or list field, it typically does
not replace the entire field, but instead updates the individual subelements.
For instance, when merging the spec
on a Deployment, the entire spec
is
not replaced. Instead the subfields of spec
, such as replicas
, are compared
and merged.
Merging changes to primitive fields
Primitive fields are replaced or cleared.
Note:
-
is used for "not applicable" because the value is not used.Field in object configuration file | Field in live object configuration | Field in last-applied-configuration | Action |
---|---|---|---|
Yes | Yes | - | Set live to configuration file value. |
Yes | No | - | Set live to local configuration. |
No | - | Yes | Clear from live configuration. |
No | - | No | Do nothing. Keep live value. |
Merging changes to map fields
Fields that represent maps are merged by comparing each of the subfields or elements of the map:
Note:
-
is used for "not applicable" because the value is not used.Key in object configuration file | Key in live object configuration | Field in last-applied-configuration | Action |
---|---|---|---|
Yes | Yes | - | Compare sub fields values. |
Yes | No | - | Set live to local configuration. |
No | - | Yes | Delete from live configuration. |
No | - | No | Do nothing. Keep live value. |
Merging changes for fields of type list
Merging changes to a list uses one of three strategies:
- Replace the list if all its elements are primitives.
- Merge individual elements in a list of complex elements.
- Merge a list of primitive elements.
The choice of strategy is made on a per-field basis.
Replace the list if all its elements are primitives
Treat the list the same as a primitive field. Replace or delete the entire list. This preserves ordering.
Example: Use kubectl apply
to update the args
field of a Container in a Pod. This sets
the value of args
in the live configuration to the value in the configuration file.
Any args
elements that had previously been added to the live configuration are lost.
The order of the args
elements defined in the configuration file is
retained in the live configuration.
# last-applied-configuration value
args: ["a", "b"]
# configuration file value
args: ["a", "c"]
# live configuration
args: ["a", "b", "d"]
# result after merge
args: ["a", "c"]
Explanation: The merge used the configuration file value as the new list value.
Merge individual elements of a list of complex elements:
Treat the list as a map, and treat a specific field of each element as a key. Add, delete, or update individual elements. This does not preserve ordering.
This merge strategy uses a special tag on each field called a patchMergeKey
. The
patchMergeKey
is defined for each field in the Kubernetes source code:
types.go
When merging a list of maps, the field specified as the patchMergeKey
for a given element
is used like a map key for that element.
Example: Use kubectl apply
to update the containers
field of a PodSpec.
This merges the list as though it was a map where each element is keyed
by name
.
# last-applied-configuration value
containers:
- name: nginx
image: nginx:1.16
- name: nginx-helper-a # key: nginx-helper-a; will be deleted in result
image: helper:1.3
- name: nginx-helper-b # key: nginx-helper-b; will be retained
image: helper:1.3
# configuration file value
containers:
- name: nginx
image: nginx:1.16
- name: nginx-helper-b
image: helper:1.3
- name: nginx-helper-c # key: nginx-helper-c; will be added in result
image: helper:1.3
# live configuration
containers:
- name: nginx
image: nginx:1.16
- name: nginx-helper-a
image: helper:1.3
- name: nginx-helper-b
image: helper:1.3
args: ["run"] # Field will be retained
- name: nginx-helper-d # key: nginx-helper-d; will be retained
image: helper:1.3
# result after merge
containers:
- name: nginx
image: nginx:1.16
# Element nginx-helper-a was deleted
- name: nginx-helper-b
image: helper:1.3
args: ["run"] # Field was retained
- name: nginx-helper-c # Element was added
image: helper:1.3
- name: nginx-helper-d # Element was ignored
image: helper:1.3
Explanation:
- The container named "nginx-helper-a" was deleted because no container named "nginx-helper-a" appeared in the configuration file.
- The container named "nginx-helper-b" retained the changes to
args
in the live configuration.kubectl apply
was able to identify that "nginx-helper-b" in the live configuration was the same "nginx-helper-b" as in the configuration file, even though their fields had different values (noargs
in the configuration file). This is because thepatchMergeKey
field value (name) was identical in both. - The container named "nginx-helper-c" was added because no container with that name appeared in the live configuration, but one with that name appeared in the configuration file.
- The container named "nginx-helper-d" was retained because no element with that name appeared in the last-applied-configuration.
Merge a list of primitive elements
As of Kubernetes 1.5, merging lists of primitive elements is not supported.
Note:
Which of the above strategies is chosen for a given field is controlled by thepatchStrategy
tag in types.go
If no patchStrategy
is specified for a field of type list, then
the list is replaced.Default field values
The API server sets certain fields to default values in the live configuration if they are not specified when the object is created.
Here's a configuration file for a Deployment. The file does not specify strategy
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
minReadySeconds: 5
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
Create the object using kubectl apply
:
kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml
Print the live configuration using kubectl get
:
kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml
The output shows that the API server set several fields to default values in the live configuration. These fields were not specified in the configuration file.
apiVersion: apps/v1
kind: Deployment
# ...
spec:
selector:
matchLabels:
app: nginx
minReadySeconds: 5
replicas: 1 # defaulted by apiserver
strategy:
rollingUpdate: # defaulted by apiserver - derived from strategy.type
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate # defaulted by apiserver
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
imagePullPolicy: IfNotPresent # defaulted by apiserver
name: nginx
ports:
- containerPort: 80
protocol: TCP # defaulted by apiserver
resources: {} # defaulted by apiserver
terminationMessagePath: /dev/termination-log # defaulted by apiserver
dnsPolicy: ClusterFirst # defaulted by apiserver
restartPolicy: Always # defaulted by apiserver
securityContext: {} # defaulted by apiserver
terminationGracePeriodSeconds: 30 # defaulted by apiserver
# ...
In a patch request, defaulted fields are not re-defaulted unless they are explicitly cleared as part of a patch request. This can cause unexpected behavior for fields that are defaulted based on the values of other fields. When the other fields are later changed, the values defaulted from them will not be updated unless they are explicitly cleared.
For this reason, it is recommended that certain fields defaulted by the server are explicitly defined in the configuration file, even if the desired values match the server defaults. This makes it easier to recognize conflicting values that will not be re-defaulted by the server.
Example:
# last-applied-configuration
spec:
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
# configuration file
spec:
strategy:
type: Recreate # updated value
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
# live configuration
spec:
strategy:
type: RollingUpdate # defaulted value
rollingUpdate: # defaulted value derived from type
maxSurge : 1
maxUnavailable: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
# result after merge - ERROR!
spec:
strategy:
type: Recreate # updated value: incompatible with rollingUpdate
rollingUpdate: # defaulted value: incompatible with "type: Recreate"
maxSurge : 1
maxUnavailable: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
Explanation:
- The user creates a Deployment without defining
strategy.type
. - The server defaults
strategy.type
toRollingUpdate
and defaults thestrategy.rollingUpdate
values. - The user changes
strategy.type
toRecreate
. Thestrategy.rollingUpdate
values remain at their defaulted values, though the server expects them to be cleared. If thestrategy.rollingUpdate
values had been defined initially in the configuration file, it would have been more clear that they needed to be deleted. - Apply fails because
strategy.rollingUpdate
is not cleared. Thestrategy.rollingupdate
field cannot be defined with astrategy.type
ofRecreate
.
Recommendation: These fields should be explicitly defined in the object configuration file:
- Selectors and PodTemplate labels on workloads, such as Deployment, StatefulSet, Job, DaemonSet, ReplicaSet, and ReplicationController
- Deployment rollout strategy
How to clear server-defaulted fields or fields set by other writers
Fields that do not appear in the configuration file can be cleared by
setting their values to null
and then applying the configuration file.
For fields defaulted by the server, this triggers re-defaulting
the values.
How to change ownership of a field between the configuration file and direct imperative writers
These are the only methods you should use to change an individual object field:
- Use
kubectl apply
. - Write directly to the live configuration without modifying the configuration file:
for example, use
kubectl scale
.
Changing the owner from a direct imperative writer to a configuration file
Add the field to the configuration file. For the field, discontinue direct updates to
the live configuration that do not go through kubectl apply
.
Changing the owner from a configuration file to a direct imperative writer
As of Kubernetes 1.5, changing ownership of a field from a configuration file to an imperative writer requires manual steps:
- Remove the field from the configuration file.
- Remove the field from the
kubectl.kubernetes.io/last-applied-configuration
annotation on the live object.
Changing management methods
Kubernetes objects should be managed using only one method at a time. Switching from one method to another is possible, but is a manual process.
Note:
It is OK to use imperative deletion with declarative management.Migrating from imperative command management to declarative object configuration
Migrating from imperative command management to declarative object configuration involves several manual steps:
Export the live object to a local configuration file:
kubectl get <kind>/<name> -o yaml > <kind>_<name>.yaml
Manually remove the
status
field from the configuration file.Note:
This step is optional, askubectl apply
does not update the status field even if it is present in the configuration file.Set the
kubectl.kubernetes.io/last-applied-configuration
annotation on the object:kubectl replace --save-config -f <kind>_<name>.yaml
Change processes to use
kubectl apply
for managing the object exclusively.
Migrating from imperative object configuration to declarative object configuration
Set the
kubectl.kubernetes.io/last-applied-configuration
annotation on the object:kubectl replace --save-config -f <kind>_<name>.yaml
Change processes to use
kubectl apply
for managing the object exclusively.
Defining controller selectors and PodTemplate labels
Warning:
Updating selectors on controllers is strongly discouraged.The recommended approach is to define a single, immutable PodTemplate label used only by the controller selector with no other semantic meaning.
Example:
selector:
matchLabels:
controller-selector: "apps/v1/deployment/nginx"
template:
metadata:
labels:
controller-selector: "apps/v1/deployment/nginx"
What's next
2 - Declarative Management of Kubernetes Objects Using Kustomize
Kustomize is a standalone tool to customize Kubernetes objects through a kustomization file.
Since 1.14, Kubectl also supports the management of Kubernetes objects using a kustomization file. To view Resources found in a directory containing a kustomization file, run the following command:
kubectl kustomize <kustomization_directory>
To apply those Resources, run kubectl apply
with --kustomize
or -k
flag:
kubectl apply -k <kustomization_directory>
Before you begin
Install kubectl
.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
To check the version, enterkubectl version
.Overview of Kustomize
Kustomize is a tool for customizing Kubernetes configurations. It has the following features to manage application configuration files:
- generating resources from other sources
- setting cross-cutting fields for resources
- composing and customizing collections of resources
Generating Resources
ConfigMaps and Secrets hold configuration or sensitive data that are used by other Kubernetes objects, such as Pods. The source of truth of ConfigMaps or Secrets are usually external to a cluster, such as a .properties
file or an SSH keyfile.
Kustomize has secretGenerator
and configMapGenerator
, which generate Secret and ConfigMap from files or literals.
configMapGenerator
To generate a ConfigMap from a file, add an entry to the files
list in configMapGenerator
. Here is an example of generating a ConfigMap with a data item from a .properties
file:
# Create a application.properties file
cat <<EOF >application.properties
FOO=Bar
EOF
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-1
files:
- application.properties
EOF
The generated ConfigMap can be examined with the following command:
kubectl kustomize ./
The generated ConfigMap is:
apiVersion: v1
data:
application.properties: |
FOO=Bar
kind: ConfigMap
metadata:
name: example-configmap-1-8mbdf7882g
To generate a ConfigMap from an env file, add an entry to the envs
list in configMapGenerator
. Here is an example of generating a ConfigMap with a data item from a .env
file:
# Create a .env file
cat <<EOF >.env
FOO=Bar
EOF
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-1
envs:
- .env
EOF
The generated ConfigMap can be examined with the following command:
kubectl kustomize ./
The generated ConfigMap is:
apiVersion: v1
data:
FOO: Bar
kind: ConfigMap
metadata:
name: example-configmap-1-42cfbf598f
Note:
Each variable in the.env
file becomes a separate key in the ConfigMap that you generate. This is different from the previous example which embeds a file named application.properties
(and all its entries) as the value for a single key.ConfigMaps can also be generated from literal key-value pairs. To generate a ConfigMap from a literal key-value pair, add an entry to the literals
list in configMapGenerator. Here is an example of generating a ConfigMap with a data item from a key-value pair:
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-2
literals:
- FOO=Bar
EOF
The generated ConfigMap can be checked by the following command:
kubectl kustomize ./
The generated ConfigMap is:
apiVersion: v1
data:
FOO: Bar
kind: ConfigMap
metadata:
name: example-configmap-2-g2hdhfc6tk
To use a generated ConfigMap in a Deployment, reference it by the name of the configMapGenerator. Kustomize will automatically replace this name with the generated name.
This is an example deployment that uses a generated ConfigMap:
# Create a application.properties file
cat <<EOF >application.properties
FOO=Bar
EOF
cat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: my-app
volumeMounts:
- name: config
mountPath: /config
volumes:
- name: config
configMap:
name: example-configmap-1
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
configMapGenerator:
- name: example-configmap-1
files:
- application.properties
EOF
Generate the ConfigMap and Deployment:
kubectl kustomize ./
The generated Deployment will refer to the generated ConfigMap by name:
apiVersion: v1
data:
application.properties: |
FOO=Bar
kind: ConfigMap
metadata:
name: example-configmap-1-g4hk9g2ff8
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: my-app
name: my-app
spec:
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- image: my-app
name: app
volumeMounts:
- mountPath: /config
name: config
volumes:
- configMap:
name: example-configmap-1-g4hk9g2ff8
name: config
secretGenerator
You can generate Secrets from files or literal key-value pairs. To generate a Secret from a file, add an entry to the files
list in secretGenerator
. Here is an example of generating a Secret with a data item from a file:
# Create a password.txt file
cat <<EOF >./password.txt
username=admin
password=secret
EOF
cat <<EOF >./kustomization.yaml
secretGenerator:
- name: example-secret-1
files:
- password.txt
EOF
The generated Secret is as follows:
apiVersion: v1
data:
password.txt: dXNlcm5hbWU9YWRtaW4KcGFzc3dvcmQ9c2VjcmV0Cg==
kind: Secret
metadata:
name: example-secret-1-t2kt65hgtb
type: Opaque
To generate a Secret from a literal key-value pair, add an entry to literals
list in secretGenerator
. Here is an example of generating a Secret with a data item from a key-value pair:
cat <<EOF >./kustomization.yaml
secretGenerator:
- name: example-secret-2
literals:
- username=admin
- password=secret
EOF
The generated Secret is as follows:
apiVersion: v1
data:
password: c2VjcmV0
username: YWRtaW4=
kind: Secret
metadata:
name: example-secret-2-t52t6g96d8
type: Opaque
Like ConfigMaps, generated Secrets can be used in Deployments by referring to the name of the secretGenerator:
# Create a password.txt file
cat <<EOF >./password.txt
username=admin
password=secret
EOF
cat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: my-app
volumeMounts:
- name: password
mountPath: /secrets
volumes:
- name: password
secret:
secretName: example-secret-1
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
secretGenerator:
- name: example-secret-1
files:
- password.txt
EOF
generatorOptions
The generated ConfigMaps and Secrets have a content hash suffix appended. This ensures that a new ConfigMap or Secret is generated when the contents are changed. To disable the behavior of appending a suffix, one can use generatorOptions
. Besides that, it is also possible to specify cross-cutting options for generated ConfigMaps and Secrets.
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-3
literals:
- FOO=Bar
generatorOptions:
disableNameSuffixHash: true
labels:
type: generated
annotations:
note: generated
EOF
Runkubectl kustomize ./
to view the generated ConfigMap:
apiVersion: v1
data:
FOO: Bar
kind: ConfigMap
metadata:
annotations:
note: generated
labels:
type: generated
name: example-configmap-3
Setting cross-cutting fields
It is quite common to set cross-cutting fields for all Kubernetes resources in a project. Some use cases for setting cross-cutting fields:
- setting the same namespace for all Resources
- adding the same name prefix or suffix
- adding the same set of labels
- adding the same set of annotations
Here is an example:
# Create a deployment.yaml
cat <<EOF >./deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
EOF
cat <<EOF >./kustomization.yaml
namespace: my-namespace
namePrefix: dev-
nameSuffix: "-001"
commonLabels:
app: bingo
commonAnnotations:
oncallPager: 800-555-1212
resources:
- deployment.yaml
EOF
Run kubectl kustomize ./
to view those fields are all set in the Deployment Resource:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
oncallPager: 800-555-1212
labels:
app: bingo
name: dev-nginx-deployment-001
namespace: my-namespace
spec:
selector:
matchLabels:
app: bingo
template:
metadata:
annotations:
oncallPager: 800-555-1212
labels:
app: bingo
spec:
containers:
- image: nginx
name: nginx
Composing and Customizing Resources
It is common to compose a set of Resources in a project and manage them inside the same file or directory. Kustomize offers composing Resources from different files and applying patches or other customization to them.
Composing
Kustomize supports composition of different resources. The resources
field, in the kustomization.yaml
file, defines the list of resources to include in a configuration. Set the path to a resource's configuration file in the resources
list.
Here is an example of an NGINX application comprised of a Deployment and a Service:
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a service.yaml file
cat <<EOF > service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
EOF
# Create a kustomization.yaml composing them
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
- service.yaml
EOF
The Resources from kubectl kustomize ./
contain both the Deployment and the Service objects.
Customizing
Patches can be used to apply different customizations to Resources. Kustomize supports different patching
mechanisms through patchesStrategicMerge
and patchesJson6902
. patchesStrategicMerge
is a list of file paths. Each file should be resolved to a strategic merge patch. The names inside the patches must match Resource names that are already loaded. Small patches that do one thing are recommended. For example, create one patch for increasing the deployment replica number and another patch for setting the memory limit.
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a patch increase_replicas.yaml
cat <<EOF > increase_replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
EOF
# Create another patch set_memory.yaml
cat <<EOF > set_memory.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
template:
spec:
containers:
- name: my-nginx
resources:
limits:
memory: 512Mi
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
patchesStrategicMerge:
- increase_replicas.yaml
- set_memory.yaml
EOF
Run kubectl kustomize ./
to view the Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- image: nginx
name: my-nginx
ports:
- containerPort: 80
resources:
limits:
memory: 512Mi
Not all Resources or fields support strategic merge patches. To support modifying arbitrary fields in arbitrary Resources,
Kustomize offers applying JSON patch through patchesJson6902
.
To find the correct Resource for a Json patch, the group, version, kind and name of that Resource need to be
specified in kustomization.yaml
. For example, increasing the replica number of a Deployment object can also be done
through patchesJson6902
.
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a json patch
cat <<EOF > patch.yaml
- op: replace
path: /spec/replicas
value: 3
EOF
# Create a kustomization.yaml
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
patchesJson6902:
- target:
group: apps
version: v1
kind: Deployment
name: my-nginx
path: patch.yaml
EOF
Run kubectl kustomize ./
to see the replicas
field is updated:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- image: nginx
name: my-nginx
ports:
- containerPort: 80
In addition to patches, Kustomize also offers customizing container images or injecting field values from other objects into containers
without creating patches. For example, you can change the image used inside containers by specifying the new image in images
field in kustomization.yaml
.
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
images:
- name: nginx
newName: my.image.registry/nginx
newTag: 1.4.0
EOF
Run kubectl kustomize ./
to see that the image being used is updated:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- image: my.image.registry/nginx:1.4.0
name: my-nginx
ports:
- containerPort: 80
Sometimes, the application running in a Pod may need to use configuration values from other objects. For example,
a Pod from a Deployment object need to read the corresponding Service name from Env or as a command argument.
Since the Service name may change as namePrefix
or nameSuffix
is added in the kustomization.yaml
file. It is
not recommended to hard code the Service name in the command argument. For this usage, Kustomize can inject the Service name into containers through vars
.
# Create a deployment.yaml file (quoting the here doc delimiter)
cat <<'EOF' > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
command: ["start", "--host", "$(MY_SERVICE_NAME)"]
EOF
# Create a service.yaml file
cat <<EOF > service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
EOF
cat <<EOF >./kustomization.yaml
namePrefix: dev-
nameSuffix: "-001"
resources:
- deployment.yaml
- service.yaml
vars:
- name: MY_SERVICE_NAME
objref:
kind: Service
name: my-nginx
apiVersion: v1
EOF
Run kubectl kustomize ./
to see that the Service name injected into containers is dev-my-nginx-001
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: dev-my-nginx-001
spec:
replicas: 2
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- command:
- start
- --host
- dev-my-nginx-001
image: nginx
name: my-nginx
Bases and Overlays
Kustomize has the concepts of bases and overlays. A base is a directory with a kustomization.yaml
, which contains a
set of resources and associated customization. A base could be either a local directory or a directory from a remote repo,
as long as a kustomization.yaml
is present inside. An overlay is a directory with a kustomization.yaml
that refers to other
kustomization directories as its bases
. A base has no knowledge of an overlay and can be used in multiple overlays.
An overlay may have multiple bases and it composes all resources
from bases and may also have customization on top of them.
Here is an example of a base:
# Create a directory to hold the base
mkdir base
# Create a base/deployment.yaml
cat <<EOF > base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
EOF
# Create a base/service.yaml file
cat <<EOF > base/service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
EOF
# Create a base/kustomization.yaml
cat <<EOF > base/kustomization.yaml
resources:
- deployment.yaml
- service.yaml
EOF
This base can be used in multiple overlays. You can add different namePrefix
or other cross-cutting fields
in different overlays. Here are two overlays using the same base.
mkdir dev
cat <<EOF > dev/kustomization.yaml
resources:
- ../base
namePrefix: dev-
EOF
mkdir prod
cat <<EOF > prod/kustomization.yaml
resources:
- ../base
namePrefix: prod-
EOF
How to apply/view/delete objects using Kustomize
Use --kustomize
or -k
in kubectl
commands to recognize Resources managed by kustomization.yaml
.
Note that -k
should point to a kustomization directory, such as
kubectl apply -k <kustomization directory>/
Given the following kustomization.yaml
,
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a kustomization.yaml
cat <<EOF >./kustomization.yaml
namePrefix: dev-
commonLabels:
app: my-nginx
resources:
- deployment.yaml
EOF
Run the following command to apply the Deployment object dev-my-nginx
:
> kubectl apply -k ./
deployment.apps/dev-my-nginx created
Run one of the following commands to view the Deployment object dev-my-nginx
:
kubectl get -k ./
kubectl describe -k ./
Run the following command to compare the Deployment object dev-my-nginx
against the state that the cluster would be in if the manifest was applied:
kubectl diff -k ./
Run the following command to delete the Deployment object dev-my-nginx
:
> kubectl delete -k ./
deployment.apps "dev-my-nginx" deleted
Kustomize Feature List
Field | Type | Explanation |
---|---|---|
namespace | string | add namespace to all resources |
namePrefix | string | value of this field is prepended to the names of all resources |
nameSuffix | string | value of this field is appended to the names of all resources |
commonLabels | map[string]string | labels to add to all resources and selectors |
commonAnnotations | map[string]string | annotations to add to all resources |
resources | []string | each entry in this list must resolve to an existing resource configuration file |
configMapGenerator | []ConfigMapArgs | Each entry in this list generates a ConfigMap |
secretGenerator | []SecretArgs | Each entry in this list generates a Secret |
generatorOptions | GeneratorOptions | Modify behaviors of all ConfigMap and Secret generator |
bases | []string | Each entry in this list should resolve to a directory containing a kustomization.yaml file |
patchesStrategicMerge | []string | Each entry in this list should resolve a strategic merge patch of a Kubernetes object |
patchesJson6902 | []Patch | Each entry in this list should resolve to a Kubernetes object and a Json Patch |
vars | []Var | Each entry is to capture text from one resource's field |
images | []Image | Each entry is to modify the name, tags and/or digest for one image without creating patches |
configurations | []string | Each entry in this list should resolve to a file containing Kustomize transformer configurations |
crds | []string | Each entry in this list should resolve to an OpenAPI definition file for Kubernetes types |
What's next
3 - Managing Kubernetes Objects Using Imperative Commands
Kubernetes objects can quickly be created, updated, and deleted directly using
imperative commands built into the kubectl
command-line tool. This document
explains how those commands are organized and how to use them to manage live objects.
Before you begin
Install kubectl
.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
To check the version, enterkubectl version
.Trade-offs
The kubectl
tool supports three kinds of object management:
- Imperative commands
- Imperative object configuration
- Declarative object configuration
See Kubernetes Object Management for a discussion of the advantages and disadvantage of each kind of object management.
How to create objects
The kubectl
tool supports verb-driven commands for creating some of the most common
object types. The commands are named to be recognizable to users unfamiliar with
the Kubernetes object types.
run
: Create a new Pod to run a Container.expose
: Create a new Service object to load balance traffic across Pods.autoscale
: Create a new Autoscaler object to automatically horizontally scale a controller, such as a Deployment.
The kubectl
tool also supports creation commands driven by object type.
These commands support more object types and are more explicit about
their intent, but require users to know the type of objects they intend
to create.
create <objecttype> [<subtype>] <instancename>
Some objects types have subtypes that you can specify in the create
command.
For example, the Service object has several subtypes including ClusterIP,
LoadBalancer, and NodePort. Here's an example that creates a Service with
subtype NodePort:
kubectl create service nodeport <myservicename>
In the preceding example, the create service nodeport
command is called
a subcommand of the create service
command.
You can use the -h
flag to find the arguments and flags supported by
a subcommand:
kubectl create service nodeport -h
How to update objects
The kubectl
command supports verb-driven commands for some common update operations.
These commands are named to enable users unfamiliar with Kubernetes
objects to perform updates without knowing the specific fields
that must be set:
scale
: Horizontally scale a controller to add or remove Pods by updating the replica count of the controller.annotate
: Add or remove an annotation from an object.label
: Add or remove a label from an object.
The kubectl
command also supports update commands driven by an aspect of the object.
Setting this aspect may set different fields for different object types:
set
<field>
: Set an aspect of an object.
Note:
In Kubernetes version 1.5, not every verb-driven command has an associated aspect-driven command.The kubectl
tool supports these additional ways to update a live object directly,
however they require a better understanding of the Kubernetes object schema.
edit
: Directly edit the raw configuration of a live object by opening its configuration in an editor.patch
: Directly modify specific fields of a live object by using a patch string. For more details on patch strings, see the patch section in API Conventions.
How to delete objects
You can use the delete
command to delete an object from a cluster:
delete <type>/<name>
Note:
You can usekubectl delete
for both imperative commands and imperative object
configuration. The difference is in the arguments passed to the command. To use
kubectl delete
as an imperative command, pass the object to be deleted as
an argument. Here's an example that passes a Deployment object named nginx:kubectl delete deployment/nginx
How to view an object
There are several commands for printing information about an object:
get
: Prints basic information about matching objects. Useget -h
to see a list of options.describe
: Prints aggregated detailed information about matching objects.logs
: Prints the stdout and stderr for a container running in a Pod.
Using set
commands to modify objects before creation
There are some object fields that don't have a flag you can use
in a create
command. In some of those cases, you can use a combination of
set
and create
to specify a value for the field before object
creation. This is done by piping the output of the create
command to the
set
command, and then back to the create
command. Here's an example:
kubectl create service clusterip my-svc --clusterip="None" -o yaml --dry-run=client | kubectl set selector --local -f - 'environment=qa' -o yaml | kubectl create -f -
- The
kubectl create service -o yaml --dry-run=client
command creates the configuration for the Service, but prints it to stdout as YAML instead of sending it to the Kubernetes API server. - The
kubectl set selector --local -f - -o yaml
command reads the configuration from stdin, and writes the updated configuration to stdout as YAML. - The
kubectl create -f -
command creates the object using the configuration provided via stdin.
Using --edit
to modify objects before creation
You can use kubectl create --edit
to make arbitrary changes to an object
before it is created. Here's an example:
kubectl create service clusterip my-svc --clusterip="None" -o yaml --dry-run=client > /tmp/srv.yaml
kubectl create --edit -f /tmp/srv.yaml
- The
kubectl create service
command creates the configuration for the Service and saves it to/tmp/srv.yaml
. - The
kubectl create --edit
command opens the configuration file for editing before it creates the object.
What's next
4 - Imperative Management of Kubernetes Objects Using Configuration Files
Kubernetes objects can be created, updated, and deleted by using the kubectl
command-line tool along with an object configuration file written in YAML or JSON.
This document explains how to define and manage objects using configuration files.
Before you begin
Install kubectl
.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
To check the version, enterkubectl version
.Trade-offs
The kubectl
tool supports three kinds of object management:
- Imperative commands
- Imperative object configuration
- Declarative object configuration
See Kubernetes Object Management for a discussion of the advantages and disadvantage of each kind of object management.
How to create objects
You can use kubectl create -f
to create an object from a configuration file.
Refer to the kubernetes API reference
for details.
kubectl create -f <filename|url>
How to update objects
Warning:
Updating objects with thereplace
command drops all
parts of the spec not specified in the configuration file. This
should not be used with objects whose specs are partially managed
by the cluster, such as Services of type LoadBalancer
, where
the externalIPs
field is managed independently from the configuration
file. Independently managed fields must be copied to the configuration
file to prevent replace
from dropping them.You can use kubectl replace -f
to update a live object according to a
configuration file.
kubectl replace -f <filename|url>
How to delete objects
You can use kubectl delete -f
to delete an object that is described in a
configuration file.
kubectl delete -f <filename|url>
Note:
If configuration file has specified the generateName
field in the metadata
section instead of the name
field, you cannot delete the object using
kubectl delete -f <filename|url>
.
You will have to use other flags for deleting the object. For example:
kubectl delete <type> <name>
kubectl delete <type> -l <label>
How to view an object
You can use kubectl get -f
to view information about an object that is
described in a configuration file.
kubectl get -f <filename|url> -o yaml
The -o yaml
flag specifies that the full object configuration is printed.
Use kubectl get -h
to see a list of options.
Limitations
The create
, replace
, and delete
commands work well when each object's
configuration is fully defined and recorded in its configuration
file. However when a live object is updated, and the updates are not merged
into its configuration file, the updates will be lost the next time a replace
is executed. This can happen if a controller, such as
a HorizontalPodAutoscaler, makes updates directly to a live object. Here's
an example:
- You create an object from a configuration file.
- Another source updates the object by changing some field.
- You replace the object from the configuration file. Changes made by the other source in step 2 are lost.
If you need to support multiple writers to the same object, you can use
kubectl apply
to manage the object.
Creating and editing an object from a URL without saving the configuration
Suppose you have the URL of an object configuration file. You can use
kubectl create --edit
to make changes to the configuration before the
object is created. This is particularly useful for tutorials and tasks
that point to a configuration file that could be modified by the reader.
kubectl create -f <url> --edit
Migrating from imperative commands to imperative object configuration
Migrating from imperative commands to imperative object configuration involves several manual steps.
Export the live object to a local object configuration file:
kubectl get <kind>/<name> -o yaml > <kind>_<name>.yaml
Manually remove the status field from the object configuration file.
For subsequent object management, use
replace
exclusively.kubectl replace -f <kind>_<name>.yaml
Defining controller selectors and PodTemplate labels
Warning:
Updating selectors on controllers is strongly discouraged.The recommended approach is to define a single, immutable PodTemplate label used only by the controller selector with no other semantic meaning.
Example label:
selector:
matchLabels:
controller-selector: "apps/v1/deployment/nginx"
template:
metadata:
labels:
controller-selector: "apps/v1/deployment/nginx"
What's next
5 - Update API Objects in Place Using kubectl patch
This task shows how to use kubectl patch
to update an API object in place. The exercises
in this task demonstrate a strategic merge patch and a JSON merge patch.
Before you begin
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
To check the version, enterkubectl version
.Use a strategic merge patch to update a Deployment
Here's the configuration file for a Deployment that has two replicas. Each replica is a Pod that has one container:
apiVersion: apps/v1
kind: Deployment
metadata:
name: patch-demo
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: patch-demo-ctr
image: nginx
tolerations:
- effect: NoSchedule
key: dedicated
value: test-team
Create the Deployment:
kubectl apply -f https://k8s.io/examples/application/deployment-patch.yaml
View the Pods associated with your Deployment:
kubectl get pods
The output shows that the Deployment has two Pods. The 1/1
indicates that
each Pod has one container:
NAME READY STATUS RESTARTS AGE
patch-demo-28633765-670qr 1/1 Running 0 23s
patch-demo-28633765-j5qs3 1/1 Running 0 23s
Make a note of the names of the running Pods. Later, you will see that these Pods get terminated and replaced by new ones.
At this point, each Pod has one Container that runs the nginx image. Now suppose you want each Pod to have two containers: one that runs nginx and one that runs redis.
Create a file named patch-file.yaml
that has this content:
spec:
template:
spec:
containers:
- name: patch-demo-ctr-2
image: redis
Patch your Deployment:
kubectl patch deployment patch-demo --patch-file patch-file.yaml
View the patched Deployment:
kubectl get deployment patch-demo --output yaml
The output shows that the PodSpec in the Deployment has two Containers:
containers:
- image: redis
imagePullPolicy: Always
name: patch-demo-ctr-2
...
- image: nginx
imagePullPolicy: Always
name: patch-demo-ctr
...
View the Pods associated with your patched Deployment:
kubectl get pods
The output shows that the running Pods have different names from the Pods that
were running previously. The Deployment terminated the old Pods and created two
new Pods that comply with the updated Deployment spec. The 2/2
indicates that
each Pod has two Containers:
NAME READY STATUS RESTARTS AGE
patch-demo-1081991389-2wrn5 2/2 Running 0 1m
patch-demo-1081991389-jmg7b 2/2 Running 0 1m
Take a closer look at one of the patch-demo Pods:
kubectl get pod <your-pod-name> --output yaml
The output shows that the Pod has two Containers: one running nginx and one running redis:
containers:
- image: redis
...
- image: nginx
...
Notes on the strategic merge patch
The patch you did in the preceding exercise is called a strategic merge patch.
Notice that the patch did not replace the containers
list. Instead it added a new
Container to the list. In other words, the list in the patch was merged with the
existing list. This is not always what happens when you use a strategic merge patch on a list.
In some cases, the list is replaced, not merged.
With a strategic merge patch, a list is either replaced or merged depending on its
patch strategy. The patch strategy is specified by the value of the patchStrategy
key
in a field tag in the Kubernetes source code. For example, the Containers
field of PodSpec
struct has a patchStrategy
of merge
:
type PodSpec struct {
...
Containers []Container `json:"containers" patchStrategy:"merge" patchMergeKey:"name" ...`
...
}
You can also see the patch strategy in the OpenApi spec:
"io.k8s.api.core.v1.PodSpec": {
...,
"containers": {
"description": "List of containers belonging to the pod. ...."
},
"x-kubernetes-patch-merge-key": "name",
"x-kubernetes-patch-strategy": "merge"
}
And you can see the patch strategy in the Kubernetes API documentation.
Create a file named patch-file-tolerations.yaml
that has this content:
spec:
template:
spec:
tolerations:
- effect: NoSchedule
key: disktype
value: ssd
Patch your Deployment:
kubectl patch deployment patch-demo --patch-file patch-file-tolerations.yaml
View the patched Deployment:
kubectl get deployment patch-demo --output yaml
The output shows that the PodSpec in the Deployment has only one Toleration:
tolerations:
- effect: NoSchedule
key: disktype
value: ssd
Notice that the tolerations
list in the PodSpec was replaced, not merged. This is because
the Tolerations field of PodSpec does not have a patchStrategy
key in its field tag. So the
strategic merge patch uses the default patch strategy, which is replace
.
type PodSpec struct {
...
Tolerations []Toleration `json:"tolerations,omitempty" protobuf:"bytes,22,opt,name=tolerations"`
...
}
Use a JSON merge patch to update a Deployment
A strategic merge patch is different from a JSON merge patch. With a JSON merge patch, if you want to update a list, you have to specify the entire new list. And the new list completely replaces the existing list.
The kubectl patch
command has a type
parameter that you can set to one of these values:
Parameter value | Merge type |
---|---|
json | JSON Patch, RFC 6902 |
merge | JSON Merge Patch, RFC 7386 |
strategic | Strategic merge patch |
For a comparison of JSON patch and JSON merge patch, see JSON Patch and JSON Merge Patch.
The default value for the type
parameter is strategic
. So in the preceding exercise, you
did a strategic merge patch.
Next, do a JSON merge patch on your same Deployment. Create a file named patch-file-2.yaml
that has this content:
spec:
template:
spec:
containers:
- name: patch-demo-ctr-3
image: gcr.io/google-samples/hello-app:2.0
In your patch command, set type
to merge
:
kubectl patch deployment patch-demo --type merge --patch-file patch-file-2.yaml
View the patched Deployment:
kubectl get deployment patch-demo --output yaml
The containers
list that you specified in the patch has only one Container.
The output shows that your list of one Container replaced the existing containers
list.
spec:
containers:
- image: gcr.io/google-samples/hello-app:2.0
...
name: patch-demo-ctr-3
List the running Pods:
kubectl get pods
In the output, you can see that the existing Pods were terminated, and new Pods
were created. The 1/1
indicates that each new Pod is running only one Container.
NAME READY STATUS RESTARTS AGE
patch-demo-1307768864-69308 1/1 Running 0 1m
patch-demo-1307768864-c86dc 1/1 Running 0 1m
Use strategic merge patch to update a Deployment using the retainKeys strategy
Here's the configuration file for a Deployment that uses the RollingUpdate
strategy:
apiVersion: apps/v1
kind: Deployment
metadata:
name: retainkeys-demo
spec:
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 30%
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: retainkeys-demo-ctr
image: nginx
Create the deployment:
kubectl apply -f https://k8s.io/examples/application/deployment-retainkeys.yaml
At this point, the deployment is created and is using the RollingUpdate
strategy.
Create a file named patch-file-no-retainkeys.yaml
that has this content:
spec:
strategy:
type: Recreate
Patch your Deployment:
kubectl patch deployment retainkeys-demo --type strategic --patch-file patch-file-no-retainkeys.yaml
In the output, you can see that it is not possible to set type
as Recreate
when a value is defined for spec.strategy.rollingUpdate
:
The Deployment "retainkeys-demo" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'
The way to remove the value for spec.strategy.rollingUpdate
when updating the value for type
is to use the retainKeys
strategy for the strategic merge.
Create another file named patch-file-retainkeys.yaml
that has this content:
spec:
strategy:
$retainKeys:
- type
type: Recreate
With this patch, we indicate that we want to retain only the type
key of the strategy
object. Thus, the rollingUpdate
will be removed during the patch operation.
Patch your Deployment again with this new patch:
kubectl patch deployment retainkeys-demo --type strategic --patch-file patch-file-retainkeys.yaml
Examine the content of the Deployment:
kubectl get deployment retainkeys-demo --output yaml
The output shows that the strategy object in the Deployment does not contain the rollingUpdate
key anymore:
spec:
strategy:
type: Recreate
template:
Notes on the strategic merge patch using the retainKeys strategy
The patch you did in the preceding exercise is called a strategic merge patch with retainKeys strategy. This method introduces a new directive $retainKeys
that has the following strategies:
- It contains a list of strings.
- All fields needing to be preserved must be present in the
$retainKeys
list. - The fields that are present will be merged with live object.
- All of the missing fields will be cleared when patching.
- All fields in the
$retainKeys
list must be a superset or the same as the fields present in the patch.
The retainKeys
strategy does not work for all objects. It only works when the value of the patchStrategy
key in a field tag in the Kubernetes source code contains retainKeys
. For example, the Strategy
field of the DeploymentSpec
struct has a patchStrategy
of retainKeys
:
type DeploymentSpec struct {
...
// +patchStrategy=retainKeys
Strategy DeploymentStrategy `json:"strategy,omitempty" patchStrategy:"retainKeys" ...`
...
}
You can also see the retainKeys
strategy in the OpenApi spec:
"io.k8s.api.apps.v1.DeploymentSpec": {
...,
"strategy": {
"$ref": "#/definitions/io.k8s.api.apps.v1.DeploymentStrategy",
"description": "The deployment strategy to use to replace existing pods with new ones.",
"x-kubernetes-patch-strategy": "retainKeys"
},
....
}
And you can see the retainKeys
strategy in the
Kubernetes API documentation.
Alternate forms of the kubectl patch command
The kubectl patch
command takes YAML or JSON. It can take the patch as a file or
directly on the command line.
Create a file named patch-file.json
that has this content:
{
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "patch-demo-ctr-2",
"image": "redis"
}
]
}
}
}
}
The following commands are equivalent:
kubectl patch deployment patch-demo --patch-file patch-file.yaml
kubectl patch deployment patch-demo --patch 'spec:\n template:\n spec:\n containers:\n - name: patch-demo-ctr-2\n image: redis'
kubectl patch deployment patch-demo --patch-file patch-file.json
kubectl patch deployment patch-demo --patch '{"spec": {"template": {"spec": {"containers": [{"name": "patch-demo-ctr-2","image": "redis"}]}}}}'
Update an object's replica count using kubectl patch
with --subresource
Kubernetes v1.24 [alpha]
The flag --subresource=[subresource-name]
is used with kubectl commands like get, patch,
edit and replace to fetch and update status
and scale
subresources of the resources
(applicable for kubectl version v1.24 or more). This flag is used with all the API resources
(built-in and CRs) that have status
or scale
subresource. Deployment is one of the
examples which supports these subresources.
Here's a manifest for a Deployment that has two replicas:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
Create the Deployment:
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
View the Pods associated with your Deployment:
kubectl get pods -l app=nginx
In the output, you can see that Deployment has two Pods. For example:
NAME READY STATUS RESTARTS AGE
nginx-deployment-7fb96c846b-22567 1/1 Running 0 47s
nginx-deployment-7fb96c846b-mlgns 1/1 Running 0 47s
Now, patch that Deployment with --subresource=[subresource-name]
flag:
kubectl patch deployment nginx-deployment --subresource='scale' --type='merge' -p '{"spec":{"replicas":3}}'
The output is:
scale.autoscaling/nginx-deployment patched
View the Pods associated with your patched Deployment:
kubectl get pods -l app=nginx
In the output, you can see one new pod is created, so now you have 3 running pods.
NAME READY STATUS RESTARTS AGE
nginx-deployment-7fb96c846b-22567 1/1 Running 0 107s
nginx-deployment-7fb96c846b-lxfr2 1/1 Running 0 14s
nginx-deployment-7fb96c846b-mlgns 1/1 Running 0 107s
View the patched Deployment:
kubectl get deployment nginx-deployment -o yaml
...
spec:
replicas: 3
...
status:
...
availableReplicas: 3
readyReplicas: 3
replicas: 3
Note:
If you runkubectl patch
and specify --subresource
flag for resource that doesn't support that
particular subresource, the API server returns a 404 Not Found error.Summary
In this exercise, you used kubectl patch
to change the live configuration
of a Deployment object. You did not change the configuration file that you originally used to
create the Deployment object. Other commands for updating API objects include
kubectl annotate,
kubectl edit,
kubectl replace,
kubectl scale,
and
kubectl apply.
Note:
Strategic merge patch is not supported for custom resources.What's next
6 - Migrate Kubernetes Objects Using Storage Version Migration
Kubernetes v1.30 [alpha]
Kubernetes relies on API data being actively re-written, to support some maintenance activities related to at rest storage. Two prominent examples are the versioned schema of stored resources (that is, the preferred storage schema changing from v1 to v2 for a given resource) and encryption at rest (that is, rewriting stale data based on a change in how the data should be encrypted).
Before you begin
Install kubectl
.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
Your Kubernetes server must be at or later than version v1.30. To check the version, enterkubectl version
.Ensure that your cluster has the StorageVersionMigrator
and InformerResourceVersion
feature gates
enabled. You will need control plane administrator access to make that change.
Enable storage version migration REST api by setting runtime config
storagemigration.k8s.io/v1alpha1
to true
for the API server. For more information on
how to do that,
read enable or disable a Kubernetes API.
Re-encrypt Kubernetes secrets using storage version migration
To begin with, configure KMS provider to encrypt data at rest in etcd using following encryption configuration.
kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ==
Make sure to enable automatic reload of encryption configuration file by setting
--encryption-provider-config-automatic-reload
to true.Create a Secret using kubectl.
kubectl create secret generic my-secret --from-literal=key1=supersecret
Verify the serialized data for that Secret object is prefixed with
k8s:enc:aescbc:v1:key1
.Update the encryption configuration file as follows to rotate the encryption key.
kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key2 secret: c2VjcmV0IGlzIHNlY3VyZSwgaXMgaXQ/ - aescbc: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ==
To ensure that previously created secret
my-secret
is re-encrypted with new keykey2
, you will use Storage Version Migration.Create a StorageVersionMigration manifest named
migrate-secret.yaml
as follows:kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1 metadata: name: secrets-migration spec: resource: group: "" version: v1 resource: secrets
Create the object using kubectl as follows:
kubectl apply -f migrate-secret.yaml
Monitor migration of Secrets by checking the
.status
of the StorageVersionMigration. A successful migration should have itsSucceeded
condition set to true. Get the StorageVersionMigration object as follows:kubectl get storageversionmigration.storagemigration.k8s.io/secrets-migration -o yaml
The output is similar to:
kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1 metadata: name: secrets-migration uid: 628f6922-a9cb-4514-b076-12d3c178967c resourceVersion: "90" creationTimestamp: "2024-03-12T20:29:45Z" spec: resource: group: "" version: v1 resource: secrets status: conditions: - type: Running status: "False" lastUpdateTime: "2024-03-12T20:29:46Z" reason: StorageVersionMigrationInProgress - type: Succeeded status: "True" lastUpdateTime: "2024-03-12T20:29:46Z" reason: StorageVersionMigrationSucceeded resourceVersion: "84"
Verify the stored secret is now prefixed with
k8s:enc:aescbc:v1:key2
.
Update the preferred storage schema of a CRD
Consider a scenario where a CustomResourceDefinition (CRD) is created to serve custom resources (CRs) and is set as the preferred storage schema. When it's time to introduce v2 of the CRD, it can be added for serving only with a conversion webhook. This enables a smoother transition where users can create CRs using either the v1 or v2 schema, with the webhook in place to perform the necessary schema conversion between them. Before setting v2 as the preferred storage schema version, it's important to ensure that all existing CRs stored as v1 are migrated to v2. This migration can be achieved through Storage Version Migration to migrate all CRs from v1 to v2.
Create a manifest for the CRD, named
test-crd.yaml
, as follows:apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: selfierequests.stable.example.com spec: group: stable.example.com names: plural: SelfieRequests singular: SelfieRequest kind: SelfieRequest listKind: SelfieRequestList scope: Namespaced versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: hostPort: type: string conversion: strategy: Webhook webhook: clientConfig: url: "https://127.0.0.1:9443/crdconvert" caBundle: <CABundle info> conversionReviewVersions: - v1 - v2
Create CRD using kubectl:
kubectl apply -f test-crd.yaml
Create a manifest for an example testcrd. Name the manifest
cr1.yaml
and use these contents:apiVersion: stable.example.com/v1 kind: SelfieRequest metadata: name: cr1 namespace: default
Create CR using kubectl:
kubectl apply -f cr1.yaml
Verify that CR is written and stored as v1 by getting the object from etcd.
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C
where
[...]
contains the additional arguments for connecting to the etcd server.Update the CRD
test-crd.yaml
to include v2 version for serving and storage and v1 as serving only, as follows:apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: selfierequests.stable.example.com spec: group: stable.example.com names: plural: SelfieRequests singular: SelfieRequest kind: SelfieRequest listKind: SelfieRequestList scope: Namespaced versions: - name: v2 served: true storage: true schema: openAPIV3Schema: type: object properties: host: type: string port: type: string - name: v1 served: true storage: false schema: openAPIV3Schema: type: object properties: hostPort: type: string conversion: strategy: Webhook webhook: clientConfig: url: "https://127.0.0.1:9443/crdconvert" caBundle: <CABundle info> conversionReviewVersions: - v1 - v2
Update CRD using kubectl:
kubectl apply -f test-crd.yaml
Create CR resource file with name
cr2.yaml
as follows:apiVersion: stable.example.com/v2 kind: SelfieRequest metadata: name: cr2 namespace: default
Create CR using kubectl:
kubectl apply -f cr2.yaml
Verify that CR is written and stored as v2 by getting the object from etcd.
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr2 [...] | hexdump -C
where
[...]
contains the additional arguments for connecting to the etcd server.Create a StorageVersionMigration manifest named
migrate-crd.yaml
, with the contents as follows:kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1 metadata: name: crdsvm spec: resource: group: stable.example.com version: v1 resource: SelfieRequest
Create the object using kubectl as follows:
kubectl apply -f migrate-crd.yaml
Monitor migration of secrets using status. Successful migration should have
Succeeded
condition set to "True" in the status field. Get the migration resource as follows:kubectl get storageversionmigration.storagemigration.k8s.io/crdsvm -o yaml
The output is similar to:
kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1 metadata: name: crdsvm uid: 13062fe4-32d7-47cc-9528-5067fa0c6ac8 resourceVersion: "111" creationTimestamp: "2024-03-12T22:40:01Z" spec: resource: group: stable.example.com version: v1 resource: testcrds status: conditions: - type: Running status: "False" lastUpdateTime: "2024-03-12T22:40:03Z" reason: StorageVersionMigrationInProgress - type: Succeeded status: "True" lastUpdateTime: "2024-03-12T22:40:03Z" reason: StorageVersionMigrationSucceeded resourceVersion: "106"
Verify that previously created cr1 is now written and stored as v2 by getting the object from etcd.
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C
where
[...]
contains the additional arguments for connecting to the etcd server.