My last post, IPv6 Finally Arriving on SWITCHengines, described what users of our IaaS offering can expect from our newly introduced IPv6 support: Instances using the shared default network (“private”) will get publicly routable IPv6 addresses.
This post explains how we set this up, and why we decided to go this route. We hope that this is interesting to curious SWITCHengines users, and useful for other operators of OpenStack infrastructure.
Before IPv6: Neutron, Tenant Networks and Floating IP
[Feel free to skip this section if you are familiar with Tenant Networks and Floating IPs.]
SWITCHengines uses Neutron, the current generation of OpenStack Networking. Neutron supports user-definable networks, routers and additional “service” functions such as Load Balancers or VPN gateways. In principle, every user can build her own network (or several) isolated from the other tenants. There is a default network accessible to all tenants. It is called private, which I find quite confusing because it is totally not private, but shared between all the tenants. But it has a range of private (in the sense of RFC 1918) IPv4 addresses—a subnet in OpenStack terminology—that is used to assign “fixed” addresses to instances.
There is another network called public, which provides global connectivity. Users cannot connect instances to it directly, but they can use Neutron routers (which include NAT functionality) to route between the private (RFC 1918) addresses of their instances on tenant networks (whether the shared “private” or their own) and the public network, and by extension, the Internet. By default, they get “outbound-only” connectivity using 1:N NAT, like users behind a typical broadband router. But they can also request a Floating IP, which can be associated with a particular instance port. In this case, a 1:1 NAT provides both outbound and inbound connectivity to the instance.
The router between the shared private network and the external public network was provisioned by us; it is called private-router. Users who build their own tenant networks and want to connect them with the outside world need to set up their own routers.
This is a fairly standard setup for OpenStack installations, although some operators, especially in the “public cloud” business, forgo private addresses and NAT, and let customers connect their VMs directly to a network with publicly routable addresses. (Sometimes I wish we’d have done that when we built SWITCHengines—but IPv4 address conservation arguments were strong in our minds at the time. Now it seems hard to move to such a model for IPv4. But let’s assume that IPv6 will eventually displace IPv4, so this will become moot.)
Adding IPv6: Subnet, router port, return route—that’s it
So at the outset, we have
- a shared internal network called private
- a provider network with Internet connectivity called public
- a shared router between private and public called private-router
We use the “Kilo” (2015.1) version of OpenStack.
As another requirement, the “real” network underlying the public network (in our case a VLAN) needs connectivity to the IPv6 Internet.
Create an IPv6 Subnet with the appropriate options
And of course we need a fresh range of IPv6 addresses that we can route on the Internet. A single /64 will be sufficient. We use this to define a new Subnet in Neutron:
neutron subnet-create --ip-version 6 --name private-ipv6 \ --ipv6-ra-mode dhcpv6-stateless --ipv6-address-mode dhcpv6-stateless \ private 2001:620:5ca1:80f0::/64
Note that we use dhcpv6-stateless for both ra-mode and address-mode. This will actually use SLAAC (stateless address autoconfiguration) and router advertisements to configure IPv6 on the instance. Stateless DHCPv6 could be used to convey information such as name server addresses, but I don’t think we’re actively using that now.
We should now see a radvd process running in an appropriate namespace on the network node. And instances—both new and pre-existing!—will start to get IPv6 addresses if they are configured to use SLAAC, as is the default for most modern OSes.
Create a new port on the shared router to connect the IPv6 Subnet
Next, we need to add a port to the shared private-router that connects this new subnet with the outside world via the public network:
neutron router-interface-add private-router private-ipv6
Configure a return route on each upstream router
Now the outside world also needs a route back to our IPv6 subnet. The subnet is already part of a larger aggregate that is routed toward our two upstream routers. It is sufficient to add a static route for our subnet on each of them. But where do we point that route to, i.e. what should be the “next hop”? We use the link-local address of the external (gateway) port of our Neutron router, which we can find out by looking inside the namespace for the router on the network node. Our router private-router has UUID 2b8d1b4f-1df1-476a-ab77-f69bb0db3a59. So we can run the following command on the network node:
$ sudo ip netns exec qrouter-2b8d1b4f-1df1-476a-ab77-f69bb0db3a59 ip -6 addr list 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 55: qg-2d73d3fb-f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 inet6 2001:620:5ca1:80fd:f816:3eff:fe00:30d7/64 scope global dynamic valid_lft 2591876sec preferred_lft 604676sec inet6 fe80::f816:3eff:fe00:30d7/64 scope link valid_lft forever preferred_lft forever 93: qr-02b9a67d-24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 inet6 2001:620:5ca1:80f0::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7d:755b/64 scope link valid_lft forever preferred_lft forever 98: qr-6aaf629f-19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 inet6 fe80::f816:3eff:feb6:85f4/64 scope link valid_lft forever preferred_lft forever
The port we’re looking for is the one whose name starts with qr-, the gateway port. The address we’re looking for is the one starting with fe80:, the link-local address.
The “internal” subnet has address 2001:620:5ca1:80f0::/64, and VLAN 908 (ve908 in router-ese) is the VLAN that connects our network node to the upstream router. So this is what we configure on each of our routers using the “industry-standard CLI”:
ipv6 route 2001:620:5ca1:80f0::/64 ve 908 fe80::f816:3eff:fe00:30d7
And we’re done! IPv6 packets can flow between instances on our private network and the Internet.
Of course this is not the end of the story. While our customers were mostly happy that they suddenly got IPv6, there are a few surprises that came up. In a future episode, we’ll tell you more about them and how they can be addressed.