Attacking Kubernetes through Kubelet

The image below shows a high-level overview of Kubernetes' architecture:

The inner workings of Kubernetes are quite involved and this article will not discuss them in detail. As a brief overview of the relevant bits:

• Human administrators talk to the "API Server" on a master node. The canonical name for this service is kube-apiserver and communication is done over a RESTful API (usually abstracted with the CLI tool kubectl).

• Communication between the kube-apiserver on the master and kubelet services on the worker nodes is bi-directional. This communication also happens over REST APIs.

• "Pods" (logical groupings of one or more containers) run on the worker nodes. The kube-apiserver holds the information that allows each kubelet to determine what it should be running.

Authentication between the various components is ideally done over mutual TLS.

However, each pod is assigned a Service Account by default; the Service Account in question and the extent of the privileges are configurable. These service accounts allow various services to interact with the kube-apiserver by using a Bearer token.

Attackers may wish to gain authenticated access to the kube-apiserver. This could allow them to, for example, read secrets or gain access to services in the cluster. This can also lead to code execution on the underlying node machines, facilitating wider lateral movement.

For this scenario we will assume that administrative authentication to the kube-apiserver is properly secured using mutual TLS authentication, but that kubelet was deployed unhardened on the worker nodes. Due to a lack of network segregation in the setup, the kubelet APIs are accessible to an attacker over the network on the default port 10250/TCP.

The Kubernetes documentation states that kubelet defaults to a mode that allows anonymous authentication:

Anonymous authentication provides full access to the kubelet API, the only requirement being network access to the service.

While the kubelet API is not documented, it's straightforward to grep the source for available endpoints. These include:

• /pods - lists running pods

• /exec - runs a command in a container and returns a link to view the output.

Other API endpoints not relevant to this post allow port forwarding, fetching logs and viewing metrics.

A simple GET request to /pods will list pods and their containers:

A template request to run a command in a container is shown below:

It should be noted that the command is passed as an array (split by spaces) and that the above is a GET request.

Target {namespace}, {pod} and {container} values can be obtained from the /pods endpoint as shown in the previous section. For example, to run ls / in the tiller container the request would be:

The request returns 302 redirect with a link to a stream that should be read with a WebSocket.

The author's kubelet-anon-rce script automates issuing a command and streaming the response:

The Kubernetes documentation states that pods are assigned a Service Account by default:

Following the "Accessing the Cluster" link shows that tokens are mounted at the following path:

The token for the tiller Service Account can thus be retrieved by using the kubelet API /exec endpoint to print it out:

The token can be used to authenticate to the kube-apiserver API. This service will usually listen on 6443/TCP on master nodes.

Interaction with the API through curl is straightforward:

A more elegant solution is to use kubectl directly:

The example above fetches Base64 encoded secrets configured in the cluster.

Note that because of differences in HTTP headers that the kube-apiserver expects to see, it may be necessary to download a version of kubectl that matches the target Kubernetes cluster. The cluster's version can be obtained by hitting the /version endpoint. Corresponding kubectl binaries can then be fetched by modifying the version in the request below:

Access to an underlying node's filesystem can be obtained by mounting the node's root directory into a container deployed in a pod.

The following deployment, node-access.yaml, mounts the host node's filesystem to /host in a container that spawns a reverse shell back to an attacker:

This can be deployed with the following command:

While not technically RCE on the node, a remote containerized shell with access to the filesystem will in many cases lead to RCE.

Access to the host filesystem could also be used to read private keys used to communicate with other Kubernetes components:

It should be noted that with the method above, the attacker has no direct control over which node in the cluster he will gain access to: the Kubernetes Scheduler will allocate the pod to a node based on resource availability in the cluster at that point.

The tiller pod attacked in this scenario is a good target to get a token with high privileges on the kube-apiserver as Tiller is a component of Helm, a package manager for Kubernetes.

Given the nature of the service, Tiller requires high privileges and the Helm docs suggest assigning the cluster-admin role to its Service Account.

If a high-privileged Service Account is not available, an attacker may consider obtaining any token with "create pod" privileges in a given namespace. The attacker could then proceed to create pods with any other target Service Account token from the namespace mounted, thus gaining those privileges. Alternatively tokens with the desired privileges, such as "read secrets", may be readily available.

The rbac.authorization.k8s.io API can provide a lot of information about roles and service accounts available in given namespaces:

The kubelet service should be run with --anonymous-auth false and the service should be segregated at the network level.

It is also recommended to ensure that all Service Accounts have the least privileges needed for their tasks.