configure-probes-in-kubernetes

Configuration of Probes in Kubernetes

In this post I will describe probes in kubernetes and  how to configure these probes in a .NET Core (.NET)  microservice and how to configure them in Kubernetes using Helm charts.

What are Kubernetes probes?

Pod lifecycle

In Kubernetes, each pod has a PodStatus, which not only contains the pod phase (Pending, ContainerCreating, Running) but also contains an array of PodConditions, which are PodScheduled, ContainersReady, Initialized, and Ready. Each condition can be either True, False, or Unknown.

Kubernetes allows us to set up probes, which are health checks that can monitor a container’s status.

Kubelet can periodically run an action, such as an HTTP GET request to a given path, and modify the container’s state based on the result. How could we use probes for our web app? Let’s take a look at what kinds of probes exist.

Three kinds of probes

Kubernetes provides three different probe types, each with its own behavior.

1- Liveness probe

A liveness probe detects if the container is properly running and, if the failure threshold has been reached, kills the container. In most cases, depending on the container’s restart policy, Kubernetes will automatically restart the failed container. We might want to use a liveness probe for a container that can fail with no possible recovery.

2- Readiness probe

A readiness probe checks if the container can respond to new requests. Unlike a liveness probe, a readiness probe doesn’t kill the container. Kubernetes simply hides the container’s Pod from corresponding Services, so that no traffic is redirected to it. As an example, a web service that occasionally runs large, expensive jobs might slow down, so we would want our readiness probe to check if such an operation is in progress.

3- Startup probe

A startup probe checks if the container has properly started. It has higher priority over the two other probe types, which are disabled by Kubernetes until the startup probe is successful, at which point they take over. Startup probes are useful in situations where your app can take a long time to start, or could occasionally fail on startup.

Probes common settings:

• initialDelaySeconds: how many seconds should pass before probes should be active
• periodSeconds: the probing frequency
• timeoutSeconds: the number of seconds after which the probe times out
• successThreshold: how many times should the probe succeed before the container should be seen as healthy
• failureThreshold: how many times should the probe fail before the container should be seen as failing

As we can see here, while there are common settings, various Kubernetes probe types have different behaviors and, as a result, different use cases.

Liveness Probe cofiguration

A liveness probe looks as follows:

livenessProbe:
  httpGet:
    path: /health
    port: http

The code tells Kubernetes to perform the liveness probe on the URL /health on port 80 (port 80 is HTTP). By default, K8s checks every 10 seconds but you can change this value using the periodSeconds parameter (see later).

Readiness Probe configuration

A readiness probe configuration is as follows:

readinessProbe:
  httpGet:
    path: /health
    port: http

This probe is a very simple, checks the /health endpoint. Application should execute some logic like warming up the cache which takes a couple of minutes. Then Kubernetes will wait until this is done and then start routing traffic to the pod.

Configuring Health Checks in Startup.cs

Health checks can be configured in the Configure method of the Startup class. To create a simple health check, you can use the MapHealthCheck extension and provide the name of the endpoint.

app.UseEndpoints(endpoints =>
{
    endpoints.MapHealthChecks("/health");
});

Configuring Health and Readiness Probes with Helm

Open the deployment.yaml file from  in the Helm charts folder (charts:productmicroservices: templates). There you can see the liveness and readiness probe.

{{- if .Values.probes.enabled }}
          livenessProbe:
            httpGet:
              path: /health
              port: http
          readinessProbe:
            httpGet:
              path: /ready
              port: http
          {{- end }}

As you can see, the liveness probe checks the /health endpoint and the readiness probe the /ready endpoint. Since the /ready endpoint doesn’t exist, the pod won’t be able to start. The probes are only added when the probes.enabled value is set to true. by default, this value is already set to true. To change the value, go to the values.yaml file.

probes:
  enabled: true

Testing the Health and Readiness Probes

Deploy the productmicroservice to kubernetes by using the Helm charts (Kubernets locally or Azure AKs).  I am running kubernetes locally here.

Start the Octant dashboard for this deployment and on the dashboard go to the Event section then you can see the following:

From the above image you can see that Readness probe is failed with code 404. This has been caused by Kubernetes checking the liveness of the pod before its port was opened. To prevent Kubernetes from checking too fast (too early), use the initialDelaySeconds parameter to tell K8s to wait a certain amount of time before checking the first time.

You can event test using the CI/CD pipeline and you will see that it fails during the Helm upgrade release task. This task can times out after five minutes the reason is  that pod is not able to start (readiness probe is failed).

Fix of the Readiness Probe failing.

In  the deployment.yaml (under path: charts:productmicroservices: templates), change the path in the readiness probe from /ready to /health and  add the initialDelaySeconds parameter and set it to 15 seconds. This tells Kubernetes to wait 15 seconds before it executes its first check. The finished liveness and readiness probe looks as follows

{{- if .Values.probes.enabled }}
livenessProbe:
  httpGet:
    path: /health
    port: http
  initialDelaySeconds: 15
readinessProbe:
  httpGet:
    path: /health
    port: http
  initialDelaySeconds: 15
{{- end }}

After that you can run the upgrade command:

helm upgrade product productmicroservice

Conclusion

We have talked about probes in Kubernetes and  there are three kind of Probes, Readness, Liveness andSstartup Probes

Readiness probes are used to check if a pod is ready to receive traffic. Only after a successful probe, traffic is routed to the pod. Liveness probes work the same way as readiness probes and check periodically if a pod is still alive. If a pod is not alive anymore, Kubernetes restarts it. We have also described how to configure these probes in Helm Chartes.

In my next post I shall describe How to use Azure Container Registry in Kubernetes

The code can be found on my GitHub.

This post is part of “Kubernetes step by step”.

Back to home page