Kubectl New Feature: debug

In this post, I will talk about the background of debugging, a new alpha feature in the kubectl.

In Kubernetes, each pod runs as a docker container and these containers are isolated through the Linux namespace isolation feature.

Each container has its own process filesystem, as shown below.

$ docker exec nginx ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1  .... nginx: master process nginx -g daemon off;
nginx 6 .... nginx: worker process
root 18 0.1 0.0 2388 696 pts/0 Ss 13:54 0:00 sh

This is the output of a healthy process and everything looks good and we can attach pods via exec command.

As you can see in the output, Nginx process works as PID 1. If this process crashes, it is impossible to attach to this container via kubectl exec.

So how can we debug our applications when their main process crashes?

Solution of Kubernetes

Kubernetes 1.17 supports ephemeral containers, and this capability makes it possible to share PID namespaces.

Ephemeral containers allow injections to IPC namespaces and you can run debug commands on your crashed pods.

To use this capability, you need an ephemeral container definition, which is shown below:

{
"apiVersion": "v1",
"kind": "EphemeralContainers",
"metadata": {
   "name": "example-pod"
},
"ephemeralContainers": [{
"command": [
   "sh"
],
"image": "busybox",
"imagePullPolicy": "IfNotPresent",
"name”: "debugger",
"stdin": true,
"tty": true,
"terminationMessagePolicy": "File"
   }]
}

Then, you need to attach to the IPC namespace of your container with this command:

kubectl replace — raw /api/v1/namespaces/default/pods/java-app/ephemeralcontainers -f ephemeral.json

What does `kubectl debug` do?

The new debug feature automates ephemeral container attaching and namespace sharing with a single command and without any manifest files. Even if your cluster version is 1.15, 1.16 or 1.17, you should enable the ephemeral container feature. But with version 1.18, this feature comes by default.

I have a created sample cluster configuration file for kubeadm that enables feature-gates on Kubernetes clusters.

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  featureGates:
     EphemeralContainers: true
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.0
apiServer:
  featureGates:
    EphemeralContainers: true
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.0
  controllerManager:
    featureGates:
       EphemeralContainers: true

Summary

As you can see above, you need to enable EphemeralContainer and write another JSON manifest file to use debug containers in previous versions. But on version 1.18 kubectl, you can officially complete this flow with only one CLI argument.

kubectl alpha debug -it java-app-failing — image=busybox — target=failing-java-app

Kubernetes environment is improving itself day by day. These new debug capabilities make debugging easy on your pods. We are excited about the upcoming new features in Kubernetes.

Have a nice day, stay at home and support the Kubernetes community!

Originally published at https://www.kloia.com.