Tasks

Step-by-step instructions for performing operations with Kubernetes.

Edit This Page

Debugging a StatefulSet

This task shows you how to debug a StatefulSet.

Before you begin

Debugging a StatefulSet

In order to list all the pods which belong to a StatefulSet, which have a label app=myapp set on them, you can use the following:

kubectl get pods -l app=myapp

If you find that any Pods listed are in Unknown or Terminating state for an extended period of time, refer to the Deleting StatefulSet Pods task for instructions on how to deal with them. You can debug individual Pods in a StatefulSet using the Debugging Pods guide.

StatefulSets provide a debug mechanism to pause all controller operations on Pods using an annotation. Setting the pod.alpha.kubernetes.io/initialized annotation to "false" on any StatefulSet Pod will pause all operations of the StatefulSet. When paused, the StatefulSet will not perform any scaling operations. Once the debug hook is set, you can execute commands within the containers of StatefulSet pods without interference from scaling operations. You can set the annotation to "false" by executing the following:

kubectl annotate pods <pod-name> pod.alpha.kubernetes.io/initialized="false" --overwrite

When the annotation is set to "false", the StatefulSet will not respond to its Pods becoming unhealthy or unavailable. It will not create replacement Pods till the annotation is removed or set to "true" on each StatefulSet Pod.

Step-wise Initialization

You can also use the same annotation to debug race conditions during bootstrapping of the StatefulSet by setting the pod.alpha.kubernetes.io/initialized annotation to "false" in the .spec.template.metadata.annotations field of the StatefulSet prior to creating it.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: my-app
spec:
  serviceName: "my-app"
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        pod.alpha.kubernetes.io/initialized: "false"
...
...
... 

After setting the annotation, if you create the StatefulSet, you can wait for each Pod to come up and verify that it has initialized correctly. The StatefulSet will not create any subsequent Pods till the debug annotation is set to "true" (or removed) on each Pod that has already been created. You can set the annotation to "true" by executing the following:

kubectl annotate pods <pod-name> pod.alpha.kubernetes.io/initialized="true" --overwrite

What’s next

Learn more about debugging an init-container.

Analytics

Create an Issue Edit this Page