This article is more than one year old. Older articles may contain outdated content. Check that the information in the page has not become incorrect since its publication.
Kubernetes 1.24: gRPC container probes in beta
_Update: Since this article was posted, the feature was graduated to GA in v1.27 and doesn't require any feature gates to be enabled.
With Kubernetes 1.24 the gRPC probes functionality entered beta and is available by default. Now you can configure startup, liveness, and readiness probes for your gRPC app without exposing any HTTP endpoint, nor do you need an executable. Kubernetes can natively connect to your workload via gRPC and query its status.
Some history
It's useful to let the system managing your workload check that the app is healthy, has started OK, and whether the app considers itself good to accept traffic. Before the gRPC support was added, Kubernetes already allowed you to check for health based on running an executable from inside the container image, by making an HTTP request, or by checking whether a TCP connection succeeded.
For most apps, those checks are enough. If your app provides a gRPC endpoint
for a health (or readiness) check, it is easy
to repurpose the exec
probe to use it for gRPC health checking.
In the blog article Health checking gRPC servers on Kubernetes,
Ahmet Alp Balkan described how you can do that — a mechanism that still works today.
There is a commonly used tool to enable this that was created on August 21, 2018, and with the first release at Sep 19, 2018.
This approach for gRPC apps health checking is very popular. There are 3,626 Dockerfiles
with the grpc_health_probe
and 6,621 yaml files that are discovered with the
basic search on GitHub (at the moment of writing). This is a good indication of the tool popularity
and the need to support this natively.
Kubernetes v1.23 introduced an alpha-quality implementation of native support for querying a workload status using gRPC. Because it was an alpha feature, this was disabled by default for the v1.23 release.
Using the feature
We built gRPC health checking in similar way with other probes and believe
it will be easy to use
if you are familiar with other probe types in Kubernetes.
The natively supported health probe has many benefits over the workaround involving grpc_health_probe
executable.
With the native gRPC support you don't need to download and carry 10MB
of an additional executable with your image.
Exec probes are generally slower than a gRPC call as they require instantiating a new process to run an executable.
It also makes the checks less sensible for edge cases when the pod is running at maximum resources and has troubles
instantiating new processes.
There are a few limitations though. Since configuring a client certificate for probes is hard, services that require client authentication are not supported. The built-in probes are also not checking the server certificates and ignore related problems.
Built-in checks also cannot be configured to ignore certain types of errors
(grpc_health_probe
returns different exit codes for different errors),
and cannot be "chained" to run the health check on multiple services in a single probe.
But all these limitations are quite standard for gRPC and there are easy workarounds for those.
Try it for yourself
Cluster-level setup
You can try this feature today. To try native gRPC probes, you can spin up a Kubernetes cluster
yourself with the GRPCContainerProbe
feature gate enabled, there are many tools available.
Since the feature gate GRPCContainerProbe
is enabled by default in 1.24,
many vendors will have this functionality working out of the box.
So you may just create an 1.24 cluster on platform of your choice. Some vendors
allow to enable alpha features on 1.23 clusters.
For example, at the moment of writing, you can spin up the test cluster on GKE for a quick test. Other vendors may also have similar capabilities, especially if you are reading this blog post long after the Kubernetes 1.24 release.
On GKE use the following command (note, version is 1.23
and enable-kubernetes-alpha
are specified).
gcloud container clusters create test-grpc \
--enable-kubernetes-alpha \
--no-enable-autorepair \
--no-enable-autoupgrade \
--release-channel=rapid \
--cluster-version=1.23
You will also need to configure kubectl
to access the cluster:
gcloud container clusters get-credentials test-grpc
Trying the feature out
Let's create the pod to test how gRPC probes work. For this test we will use the agnhost
image.
This is a k8s maintained image with that can be used for all sorts of workload testing.
For example, it has a useful grpc-health-checking module
that exposes two ports - one is serving health checking service,
another - http port to react on commands make-serving
and make-not-serving
.
Here is an example pod definition. It starts the grpc-health-checking
module,
exposes ports 5000
and 8080
, and configures gRPC readiness probe:
---
apiVersion: v1
kind: Pod
metadata:
name: test-grpc
spec:
containers:
- name: agnhost
# image changed since publication (previously used registry "k8s.gcr.io")
image: registry.k8s.io/e2e-test-images/agnhost:2.35
command: ["/agnhost", "grpc-health-checking"]
ports:
- containerPort: 5000
- containerPort: 8080
readinessProbe:
grpc:
port: 5000
In the manifest file called test.yaml
, you can create the pod and check its status.
The pod will be in ready state as indicated by the snippet of the output.
kubectl apply -f test.yaml
kubectl describe test-grpc
The output will contain something like this:
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Now let's change the health checking endpoint status to NOT_SERVING. In order to call the http port of the Pod, let's create a port forward:
kubectl port-forward test-grpc 8080:8080
You can curl
to call the command...
curl http://localhost:8080/make-not-serving
... and in a few seconds the port status will switch to not ready.
kubectl describe pod test-grpc
The output now will have:
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
...
Warning Unhealthy 2s (x6 over 42s) kubelet Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
Once it is switched back, in about one second the Pod will get back to ready status:
curl http://localhost:8080/make-serving
kubectl describe test-grpc
The output indicates that the Pod went back to being Ready
:
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
This new built-in gRPC health probing on Kubernetes makes implementing a health-check via gRPC
much easier than the older approach that relied on using a separate exec
probe. Read through
the official
documentation
to learn more and provide feedback before the feature will be promoted to GA.
Summary
Kubernetes is a popular workload orchestration platform and we add features based on feedback and demand. Features like gRPC probes support is a minor improvement that will make life of many app developers easier and apps more resilient. Try it today and give feedback, before the feature went into GA.