How Kubernetes probes work

I’m going to show you, really show you, how probes work in Kubernetes. How they can make your application more resilient, and how they can help you prevent avoidable mistakes. Like restart loops that take hours to recover from, and dropping requests during rollouts. Every interactive demo in this post uses webernetes, my partial port of the Kubernetes to TypeScript. It contains more than 100,000 lines of ported Kubernetes Go code to run a simulated cluster right here in your browser. I verified the behaviour of these demos against k3s and managed to find a bug in Kubernetes! More on that later. What you will learn A pod without probes I want to run a pod with a single container. Here’s its manifest, pod-a.yaml: pod-a.yaml1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest" This image, my-app:latest, spends a few seconds initialising before listening on port 8080. You will see this below when you click restart to send the container a signal, causing it to crash and get started back up by Kubernetes. You can pause or reset any demo at any time. node-10/2Restart container Not yet complete. After the first crash, the container restarts straight away. After the second, Kubernetes imposes a CrashLoopBackOff on it before starting it again. By default this delay is 10 seconds, doubling with each crash up to a maximum wait of 5 minutes. I shortened it to 3 seconds for this demo. In both cases, Kubernetes considers the container Ready as soon as it starts, even though we know it’s not. It’s still doing startup work and not listening on port 8080. Next I’ll add pod-b, which sends a request to pod-a every 2 seconds. Throughout the post, you can think of pod-b as any source of client traffic: an ingress controller, a load balancer, inter-service requests, etc. If you restart pod-a in the demo below while a request is on its way, that request will fail. node-1Cause a request to fail Not yet complete. From the moment you restart the container until its startup work finishes, requests will fail, even though the container is considered Ready! This is not what I want. I need Kubernetes know when pod-a is ready to receive traffic. For this, Kubernetes gives us probes. Probes are periodic checks sent to containers to determine their health. They come in three flavours: Startup probes determine whether my application inside the container has started. Readiness probes determine whether my application is ready to receive traffic. Liveness probes determine whether my application needs to be restarted. It sounds like startup probes are best suited to the problem I showed you in the demos above, so let’s start there. Startup probes Below, I’ve added a startup probe to pod-a.yaml: pod-a.yaml1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest"9 startupProbe:10 httpGet:11 path: "/startup"12 port: 808013 periodSeconds: 114 failureThreshold: 5 It’s an httpGet probe that sends a GET /startup request to the pod on port 8080. Status codes 200-399 count as a success. This happens every periodSeconds seconds, and is allowed to fail failureThreshold consecutive times before Kubernetes kills the container. This gives my container ~5 seconds to complete its startup work. Kubernetes also supports tcpSocket, exec, and grpc probes. These establish a TCP connection, run a command inside the container, or call the gRPC health-checking protocol to establish container health. You can read about them in the Kubernetes documentation. I’ll be using httpGet throughout this post. Probes are sent by a process called the kubelet. Each node in the cluster has its own kubelet, and it’s the kubelet’s job to make sure the right pods are running and being probed for each node. When you restart pod-a below, it now shows as NotReady. Kubernetes is now aware that pod-a hasn’t initialised yet. It only becomes Ready after the first startup probe succeeds. node-10/2Restart container Not yet complete.kubelet NotReady is the default for pods with containers that have a startup probe. However, even when not ready, pod-b still sends requests to pod-a and those requests still fail during the container’s startup period. This is because I’ve configured pod-b to send requests directly to pod-a’s IP address, which bypasses the readiness mechanism. I'm lying a bit about NotReadyTechnically Kubernetes doesn’t have a NotReady condition, it has a Ready condition that can be True, False, or Unknown. I’m referring to it as NotReady because it was shorter than having Ready=True or Ready=False in the demos. To fix these failed requests I need to graduate to a more production-grade setup: multiple copies of pod-a with requests load-balanced between them. I’m going to create a ReplicaSet configured to run 2 replicas of pod-a and a Service to load balance between them. replica-set-a.yaml1apiVersion: "apps/v1"2kind: "ReplicaSet"3metadata:4 name: "replica-set-a"5spec:6 # Run 2 copies of the pod defined under `template`.7 replicas: 28 selector:9 matchLabels:10 # Consider pods with this label to be part of this replica set.11 app: "pod-a"12 template:13 metadata:14 labels:15 app: "pod-a"16 spec:17 # The same pod spec from before.18 containers:19 - name: "app"20 image: "my-app:latest"21 startupProbe:22 httpGet:23 path: "/startup"24 port: 808025 periodSeconds: 126 failureThreshold: 5 service-a.yaml1apiVersion: "v1"2kind: "Service"3metadata:4 name: "service-a"5spec:6 selector:7 # Load-balance between pods that have this label.8 app: "pod-a"9 ports:10 # Send requests to this port on the pods.11 - port: 8012 targetPort: 8080 pod-b will from now on send requests to the DNS name Kubernetes creates for the Service, in this case service-a.default.svc.cluster.local, instead of directly to an individual pod. Kubernetes uses a pod’s Ready condition to include or exclude it from Service load balancing. Below you can click the restart button to crash only the top container. Notice that when the top container is starting up, requests are always sent to the bottom container. When a container is NotReady, it marks the whole pod not ready and it won’t get traffic from any Services it is part of. node-10/2Restart top container Not yet complete.kubelet Despite this, requests can still fail if they’re in-flight when you restart the top container. This happens because the restart button crashes the container abruptly. It doesn’t get a chance to finish in-flight requests. The better thing to do here is delete the pod and rely on the ReplicaSet to bring up a new one. This is better for 2 reasons: Kubernetes gives pods a 30-second termination grace period by default, which I’ve configured to 2 seconds in this post so you don’t have to wait. When deleted, pods are considered terminating and Kubernetes removes them from any Services they’re part of. They won’t receive any new requests. ReplicaSets don’t count terminating pods as active replicas, so they create replacements as soon as the deleted pod is terminating. Together, graceful termination and the startup probe keep requests away from containers that are starting or stopping. In this next demo, clicking delete won’t cause any requests from pod-b to fail. node-10/2Wait for containers to be ready Not yet complete.kubelet There’s always a pod ready to service a new request, making it safe to delete pods without interrupting user traffic. How does this grace period actually work? How to misconfigure a startup probe Earlier I mentioned that I’m giving my pod ~5 seconds to complete its startup work by setting failureThreshold to 5 with a periodSeconds of 1. Choose these values on your own containers carefully. Too little time can cause a container to crash-loop. Setting the failureThreshold below will restart the container with the new value. Set it to 1 or 2 and see what happens. node-1Make pod-a crash loop Not yet complete.kubeletfailureThreshold5 After a few restarts, pod-a is put in CrashLoopBackOff. The startup probe never gives the container enough time to start, so this demo crash-loops until you set failureThreshold back to 3 or above. When configuring this for your own containers, choose values that allow for your worst-case startup time. Readiness probes After any startup probe succeeds, readiness probes monitor the container for the rest of its life. Failing a readiness probe marks the container NotReady and removes it from receiving requests for any Service it is part of. I’ve modified pod-a.yaml to have just a readiness probe for now: pod-a.yaml1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest"9 readinessProbe:10 httpGet:11 path: "/ready"12 port: 808013 periodSeconds: 314 failureThreshold: 115 successThreshold: 1 I’m sending it to the /ready endpoint every 3 seconds. After a single failure, the container gets the NotReady condition. Switch /ready in the demo below from 200 to 503 and watch the container become not ready. node-1Wait for pod-a to become ready Not yet complete.kubelet/ready200503 Out-of-band probing The demo above sets failureThreshold and successThreshold to 1, but I don’t want a single transient failure to remove my pods from their Services. Below I’ve set the thresholds to 2. Set /ready to 503 again and notice it now takes 2 failures before the container becomes NotReady. node-1Wait for pod-a to become ready Not yet complete.kubelet/ready200503 You may notice here that when flipping from ready to not ready, an out-of-band probe can be fired.. This is for the same reasons as before. The pod is NotReady and its status just got updated.