Wednesday, December 7, 2016

Excuse me, your MAC is showing

by Craig Miller 

MACs exposed*
Stateless Address AutoConfiguration (SLAAC) is an easy way to get an IPv6 network up and running quickly. But  RFC 4291 states that the  the interface MAC address should be used when generating a Global Unique Address (GUA).

For the most part, this works quite well. Most MAC addresses are unique (although they only need to be unique within the same broadcast domain). But it has problems when it comes to hardware repair. What happens when you need to replace a NIC card, the GUA changes. Not good.

I have argued in the past that this was a small issue, since servers are probably going to have static IPv6 addresses, which get mapped in DNS to names. Therefore changing a NIC card won't change your GUA.

Exposing information

Other's have expressed concerns that using the MAC address in the GUA exposes that information to the outside world. But on the far side of a router, the MAC address is no longer significant, and therefore pretty useless for network attacks.

But sharing the MAC address globally does reveal the hardware vendor of your NIC, since the first 3 bytes of a MAC address is registered to a manufacturer. This may lead some try to infer an installed OS, and limit their attacks based on this assumption. But the privacy extensions (RFC 4941) have all but eliminated this problem, since temporary GUAs are used for outbound connections.

When infinite is only 16 million

But in-bound connections are still received on the SLAAC MAC-based GUAs.  Reading RFC 7707 (Network Reconnaissance in IPv6 Networks) has convinced me that it is time to move beyond the old MAC-based GUA. RFC 7707 discusses how an attacker can limit the number of addresses being probed from the 2^64 address space by selecting a common NIC vendor (such as Apple, 10:9a:dd:xx:yy:zz) thus reducing the address pool to 2^24. Or in more humanly understandable numbers reducing the potential address from 1 in 18,446,744,073,709,551,616 to 1 in 16,777,216 (or 16 million). Suddenly, the seemingly infinite address space of 2^64 is quite manageable to probe.

A Standard to the rescue

Fortunately, there's a new standard to disconnect the MAC address from the GUA. RFC 7217 (Semantically Opaque Address Generation). Rather than use the MAC address, the RFC recommends using a cryptographic method to generate the lower 64 bits of an address. This opens the entire 2^64 address space again for a GUA on a device. Making it very difficult to guess by an attacker.

The latest version of Mac OSX (10.12) and Windows (10) support this method of GUA generation. Unfortunately, Linux a former leader in the realm of IPv6 support, is behind. A feature enhancement (#4625) has been submitted to systemd for RFC 7217 support.

SLAAC, letting you slack off

Where does this leave us? IPv6 is improving, clever people are fixing the problems of the earlier implementations. With RDNSS (see IPv6 RA, RA, RA) and Semantically Opaque addresses, a SLAAC environment can be viable option to run IPv6 networks with the minimum of management hassle**.

* Macadamia Nuts in and out of the shell
** OK, if you have Windows machines on the network, it may not be enough (see Dual Stack, the good, bad, and ugly)

Monday, November 21, 2016

IPv6 & Systemd, another look

by Craig Miller

Lennart Poettering on systemd*

Systemd is coming to a computer near you. All the major linux distributions have adopted it. Systemd is a replacement for init, PID 1, and if the developers had stopped their, there would be no real impact. But the developers have expanded the influence of systemd to include many other aspects of how your linux workstation operates.
  • networkd - Controls the aspects of networking, including DHCPv6 Client, RA processing, SLAAC address assignment, IPv6 route insertion, etc
  • resolved - Acts like DNSMasq (but more limited) in resolving Domain Names. It gets pointed to DNS servers via networkd.
  • Other aspects of the linux host, including logind, hostnamed, locald, timedated, machined, importd to name a few.
The systemd daemons will also process IPv4 information, but that is beyond the scope of this blog.

Another Look at systemd

It has been six months since I took a good look at systemd and IPv6. You may remember that there were some clear issues which may cause one to pause before deploying a systemd host in an IPv6 production network.

A recap of IPv6 issues with systemd

The devs at systemd (and Redhat) have decided to re-implement functionality already in the kernel code. Therefore there are a few things which worked just fine in a non-systemd system, but do not in a modern system. Retesting with systemd version 231, we see there are improvements (below).
Description Issue Status
IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases#2977FIXED
Temporary addresses (RFC 4941) are broken from version 224 to 228#2242FIXED
nterface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd#2912Still broken
Fails to send Router Solicitation#2365Still broken
Unable to view DUID (DHCPv6 Identifier) on host#2952Closed but Still broken
Bridged Interfaces get IPv6 SLAAC addresses#2572FIXED
Systemd in a VM failed to start due to RA parsing error#2228FIXED
IPv6 incorrectly not enabled on Virtuozzo containers#2059FIXED
IPv6cceptRouterAdvertisements=yes or unset accepts too many prefixes#2004FIXED
Does not support DHCPV6-PD#1080Still broken
Does not support SLAAC RDNSS#1079FIXED

A few more issues have been added in the past 6 months

networkd does not support DNS List option in RA#4590            Open
Semantically Opaque Interface Identifiers with IPv6 SLAAC RFC 7217#4625Open
networkd: handle MTU field in IPv6 RA#4464Open
RFC 7084 support in networkd (automatic ipv6 prefix delegation)#4073Open

Time will improve systemd

It is unfortunate that the systemd team has decided to re-implement existing functionality (e.g. in the kernel, dnsmasq, odhcpd). But like fine wine, it is improving. Be sure to check the details of the issues which may impact your testing/deployment and save yourself a bunch of time. As always, taking the time to plan your IPv6 deployment will save you time in the end.

*Lennart Poettering T Shirt 

Tuesday, November 1, 2016

IPv6 in the palm of your hand

by Craig Miller

With the all of the major wireless providers in the US, and large service providers, like Sky, in Europe now offering native IPv6 service (see Glass Half Full), there is no time like the present to learn the new protocol, IPv6.

No IPv6 here
And yet even with all this new IPv6 access available, there are still many who's first reaction to a problem is to turn off IPv6. Like ostriches, it seems like they would rather stick their head in the sand, and hope that IPv6 will go away.

A new IPv6 device

I recently bought a new Android cell phone. Alas, in Canada there is only one provider, Telus, who offers IPv6 wireless service. And I am not their customer. Fortunately, I do have an IPv6 WLAN. While re-installing my useful apps on the new phone, I took a look at the Android IPv6 apps that are out there. And I was surprised and disappointed  to see the abundance of "disable IPv6" apps out there. Seriously, there has got to be a better way.
Step into the future, rather than disable IPv6

Helpful apps

But there were other apps as well, to help one learn IPv6. There is an excellent Hurricane Electric* IPv6 Network Tools, which can tell you a lot about your network.

There are even an IPv6 subnet calculators, which of course, aren't really required, since IPv6 does not have variable subnet masking (see Simplifying Subnetting), making things much easier than IPv4.

IPv6 in the palm of your hand

With Apple pushing in iOS 10 for full IPv6-only networking in their apps, and Android IPv6 helpful apps, you can now learn IPv6 on your phone while waiting for your coffee at Starbucks. Ah, good coffee, and the future of the internet in your hand,  it is a good time to explore IPv6.

*Hurricane Electric is an excellent IPv6 service provider, and offers free IPv6 tunnels for those who do not have native IPv6 service.

Monday, September 26, 2016

Extending a /64 Prefix

by Craig Miller

Extending a /64 Prefix
It would appear that ISPs are still learning how to deploy IPv6, and the old "conserve address space" thinking continues. Amazingly there are many ISPs still using a single /64 for their customers, with no Prefix Delegation (PD).

Some math, how far will a /32 go?

ISPs with only a /32, and not giving their customers more than a /64, have failed to understand just how big a /32 is. With a /32, there are only 4 billion subnets to give out to their customers. If the ISP gives out an additional /64 (using DHPCv6 PD) to each customer, that is still 2 billion customers, each with a /64 in their home network.

But what if the ISP was a little more generous, and gave out a /56, allowing customers to subnet within their home (255 subnets)? That /32 would be exhausted only after 16 million customers (or about the total number of internet customers of AT&T in 2016 Q2. That is just one /32, if an ISP runs out, they can apply for another. As I have stated before, we need to shed the "conserve address space" thinking of IPv4 when doing network planning in IPv6.

Wireless 3GPP and an IPv6 Hotspot

But what if your ISP only provides a /64 for the end device with NO additional subnets. Telstra does this with their wireless network. Phones only need to attach to the IPv6 wireless network, right?

What if the customer wanted to run a hotspot? Since there is no additional /64 for the "LAN" network what is one to do? Fortunately there are smart minds suggesting solutions. OpenWRT has a "relay-mode" which relays RAs and NDP traffic from upstream, thus extending the WAN-side /64 prefix into the LAN-side. There is also the v6brouter project which uses bridging for IPv6 while maintaining IPv4 NAT.

Standards to the Rescue

And there is RFC 7278 Extending an IPv6 /64 prefix from a 3GPP mobile interface to a LAN link. The RFC describes processing RA and NDP packets which sounds very similar in concept to the OpenWRT/odhcpd implementation.

The good news is that ISPs are learning more about IPv6 deployment. If you are one of the unlucky ones with only a /64 or less (no PD), comment to your ISP, and let them know there is plenty of room. And if you want to set up that IPv6 enabled hotspot on your phone, go ahead, and thank the authors of RFC 7278.

* extension cord, creative commons

Tuesday, August 23, 2016

A glass half full

by Craig Miller

This month, the Internet Society (ISOC) reports that the four major mobile carriers in the US passed the milestone of  50% IPv6 traffic to Facebook. This sounds like great news, and it is.

But looking a little deeper yields more information.  Why the US Mobile Carriers? Why is this metric highlighting Facebook?

The good news

Where is there the greatest need for expanded IP addresses since ARIN ran out of IPv4 addresses last fall (2015)? The explosion of smartphones and LTE service in the US, has led to the rapid adoption of IPv6 by even the most conservative of carriers, like AT&T. The fact is that if you need more address space as an ISP (Mobile Carriers are just wireless ISPs), you must turn to IPv6. This is good news for IPv6 advocates.

And in Europe, Sky has recently enabled IPv6 achieving 80% of its customer-base now with IPv6. BT, hopes to follow in 2017.

Finally after waiting 20 years, serious IPv6 adoption is actually happening.

The less good news

Facebook is the metric for two reasons: it is a big content provider, and it is one of the few content providers which support native IPv6 connectivity. Unfortunately other content providers have not woken up and smelled the coffee, and remain stuck in the IPv4-only world.

Hopefully everyone enjoyed the recent Olympics in Rio, which seemed to be a success. Unfortunately, CBC in Canada has zero IPv6 presence on the web. Kudos to NBC in the US which setup a separate domain which does support IPv6. Unfortunately all that streaming of events north of the border, whether it be the 100 meter sprint, or waterpolo was all over IPv4.

The reality is that content providers see few business reasons to move to IPv6. After all there are transition technologies such as dual-stack or NAT64 which allow them (and many others, CBS, ABC, Twitter, Reddit) to continue serving IPv4-only content. Sadly, even some content providers still see IPv6 as an inferior transport.

Is it half full or half empty?

Like a glass of water that is half full, it is good news that  more than half of the mobile traffic is IPv6 capable, but the glass is is still  half empty when it comes to content providers supporting IPv6. It looks like us IPv6 advocates will have to wait a little longer for the content providers to move into the future of the internet.

* Glass courtesy of creative commons

Sunday, July 10, 2016

Transitions, getting from here (IPv6) to there (IPv4)

by Craig Miller

Parlez vous IPv6?
There will be no IPv6 flash-cut-over. This is not news, but the methods of transition might be. We will be living with IPv4 for some years to come, and to help us stay connected are several transition mechanisms.

Bolstered by the news (from Google and others) that IPv6 traffic on the internet was up to 10%, I set up an IPv6-only network in my house. Initially the network was only connected to the internet via IPv6. I found it is a dark world out there. Although many ISPs are moving to IPv6 with ARIN finally running out of IPv4 addresses last fall (of 2015), the content providers are still slow to enable IPv6. I was surprised and disheartened that most of my news providers (CBC, BBC, NBC, ABC, CBS, FoxNews) were all only running the legacy IP protocol, and therefore inaccessible from my IPv6-only network.

Transitions, how to get there

The easiest transition mechanism to use is dual-stack. And I have discussed this in previous posts (see Dual Stack, the good, the bad, the ugly). But there is a cost to dual-stack, specifically to network administration and maintenance. Now routers have to be configured for IPv4 and IPv6, the same with firewalls and DNS, and other networking elements.

But the good news is that once the IPv6 pipes are turned up, dual-stack just kind of works. Mostly because if there is an IPv4-only service, the host (e.g. your laptop) will just use IPv4 to connect, and the lack of IPv6 will be transparent to the end user.

Simplify the Network

We had it good after IP won the protocol wars. We didn't have to run multi-protocol routers (of the 90s) which supported not only IP but Novel/IPX, AppleTalk, Vines, and NetbEUI. We only had to support one protocol, IP(v4), and that was much easier.

Companies such as Facebook, and T-Mobile have figured out that it is easier to support just one networking protocol as well, and they have chosen IPv6. But how does one prevent seeing the dark world I did in my little IPv6-only network?

DNS64 returning false addresses

Enter NAT64/DNS64, RFC 6146. Yes, in the past I said that NAT is bad, and it is, but NAT64 is more like protocol translation, translating from IPv6 to IPv4 (and back). NAT64 relies on two things, 1) the large address space of IPv6, and 2) DNS64 resolving IPv4-only hosts into a special IPv6 address.

When the user wants to go to a IPv4-only service, DNS64 will return a special IP address starting with the prefix* 64:ff9b::/96 (e.g. 64:ff9b::c73b:9607). The connection will be opened to this address, which happens to be the NAT64 box. Since the IPv4 address is encoded in the last 32 bits of the special IPv6 address, the NAT64 opens an IPv4 connection to the legacy service acting as a protocol translation proxy service. Like IPv4 NAT, it keeps a session table, and can return packets from the legacy service to IPv6-only user's laptop.
A nice feature of this transition mechanism, besides a simpler internal IPv6 network, is that the DNS64 service does not have to be located inside your network. Google has made a public DNS64 service available at 2001:4860:4860::64

Testing NAT64 with OpenWRT

Letting Google's DNS64 do the hard work, means that one only needs setup a NAT64 service. On OpenWRT, it is possible to setup NAT64 service, although it is not a simple install and go, as the NAT64 package tayga has been removed from the most recent release repository. However the package can be found in the 12.09 release. The UCI (Universal Configuration Interface) does not work on the latest release (15.05.1), but following the quick setup instructions on the tayga website should get you going.

Although NAT64 on OpenWRT is great for a test network to give you some experience in using the transition technology of NAT64/DNS64, it is not recommended for a production network. Tayga runs in user space, and is somewhat slow.

Production quality vendors such as Cisco support NAT64 as well.

Industry moving to NAT64

Apple has made a big push in iOS 10 to have all new/updated apps in the app store support NAT64 (or IPv6-only) networks. Google with Android 6, while not making quite the commitment Apple has made, will support NAT64 (and IPv6-only) networks if the app supports it (alas there are still too many apps which are v4-only). ChromeOS works well in a NAT64/IPv6-only environment, but like Android, there are applications which are IPv4-only.

With the simplicity of only supporting one protocol on the network, and a transition mechanism the industry is behind, there is no time like the present to start gaining some experience with NAT64, and shine the light into your IPv6-only network.

*although it possible to use other prefixes for NAT64 (than 64:ff9b::/96 RFC 6052), Google's DNS64 returns addresses with this prefix, which makes life easier.

Monday, June 27, 2016

Feature Parity!

by Craig Miller

Bridging Feature Parity for all wheeled vehicles
We have solved problems for the 25 years or so that the internet has become a foundation of computer networking in our society, with many clever tricks, or even hacks. NAT (Network Address Translation) is probably the most well known. But there are others as well.

If we are going to move forward into the brave new world of IPv6, we need Parity with the features and capabilities of the legacy protocol, IPv4. That doesn't mean that we need similar tricks, but we do need similar solutions.

Wireless Ethernet Bridging

Case in point, I wanted to setup a wireless Ethernet bridge in my home to bring wired connectivity to a corner of the house where it is difficult to run wires. This seemed like a simple task of running a second router as an Ethernet bridge, acting as a wireless client of the first wireless router*.

Oops, lost the Ethernet MAC header

And that is where the fun began. Because, when one configures a second router as a wireless client, there is more going on that meets the eye. From a high level, the packet starts as Ethernet (802.3), but then changes to 802.11 (Wireless Layer 2 protocol), and then must change back to Ethernet. The last step is where the problem begins, what Ethernet MAC address to use, since it was stripped off when the 802.11 header was put on the packet?

Using Scapy notation, the packet would look like:
/Ethernet/payload (original packet)
/802.11/payload   (packet of the air)
/??/payload       (The Ethernet MAC address has been lost)

There are a couple of ways to deal with this, one could have been encapsulation of the bridged frame in 802.11 header, but instead a trick was used which only works for IPv4. Alas, no IPv6 frames will be bridged across the wireless link. Which in my home, kind of defeats the point.

WDS to the rescue, dang no standard

But wait, what about WDS (Wireless Distribution System)? After all, it was made for wireless bridging. Yes, it handles the problem of the the unknown Ethernet MAC address, and should be the solution. However, WDS was never standardized, and as such, is incompatible between different vendors and even different products (think chip sets) of the same vendor.

Still need solutions in an IPv6 world

We must demand feature IPv6 Parity from vendors and ask for the same from the open source community The networking problems that we have solved over the past 25+ years, still need solutions in an IPv6 world.

*Over the 5Ghz band, as the 2.4 Ghz band is way to crowded

Tuesday, June 14, 2016

Breaking the NAT frame of mind

by Craig Miller

Let's face it, we have been living with NAT (Network Address Translation) for so long, NAT could vote, drink, even have a kid. NAT has been around for 23 years (since RFC 1621 was written in 1993). An interesting concept in the RFC is that NAT would be a temporary measure:

If nothing else, this solution can serve to provide temporarily relief while other, more complex and far-reaching solutions are worked out.
Note, that far-reaching solution is IPv6, which was standardized two years later (RFC 1883).

NAT helped with Address Exhaustion, but...

NAT was the solution to the problem of IP address space exhaustion. The authors of RFC 1621 saw security issues with NAT, such as passing encrypted traffic (ESP),  and debugging problems, since IP addresses change.

But there was that address problem, and a generation of IT people have grown up with NAT, thinking it is a feature rather than breaking end-to-end connectivity, which existed before NAT.

Fast forward to Today

So fast forward to today, and although addressing should not be a problem in IPv6, there are those who struggle with it. As I have written in previous posts, some ISPs are only handing out /64 addresses. When faced with a single /64, what is a router to do? Routing by definition is connecting two or more networks (in IPv6, this means two ore more /64 networks).

Unfortunately, some have discovered that the Linux kernel can mangle IPv6 addresses as well as it can IPv4 addresses, and have written how-to's on how to implement NAT6 to solve the single /64 problem. In my mind this is the wrong thing to do. NAT whether it is IPv4 or IPv6 incur res the same set of problems: difficulty in debugging, no improved security, multiple users sharing a single address, breaking end-to-end connectivity, increased latency, overlapping address space to name a few. Using NAT6 is IPv4-thinking. We must break out of this kind of thinking, and apply better solutions to the problem.

  1. Encourage our ISP to support Prefix Delegation, providing more than a /64 to the customer. (to be fair many ISPs who have a good understanding of IPv6 are already doing this)
  2. Use a v6brouter approach, which bridges IPv6 with a firewall, avoiding NAT.  (see NAT-iness and the v6Brouter) And continue to push your ISP for more than a single /64

Restoring end-to-end Connectivity

There may be situations in the future where NAT6 will be a reasonable solution. Not sure what they are today, but I want to keep an open mind. But for now, I would suggest we leave our IPv4 thinking behind, and try to restore true end-to-end network connectivity. Something we haven't enjoyed for 20 years.

* Router Image

Sunday, May 15, 2016

Fighting FUD

by Craig Miller

FUD is Fear, Uncertainty and Doubt. There's a lot of mis-information about IPv6 circulating on the internet. Much of it is old, out-of-date, and just plain wrong.


The fear is that since IPv6 uses global addresses, external hosts can make inbound connections to your machine.

  • This is IPv4/NAT thinking. That somehow NAT is considered a security blanket that protects us from the nasty outside world. Before there was NAT (see Breaking the NAT Frame of Mind) there were firewalls to keep the bad guys out. Firewalls have improved immensely in the past 20 years. They are stateful, and do connection tracking. Modern firewalls will only allow outbound connections (by default), thus keeping the bad guys on the outside. Even your IPv6 enabled Home Router comes with an IPv6 connection tracking firewall.


Another is the uncertainty that IPv6 leaks your MAC address to the internet. I find several things wrong with this:

  • This applied to SLAAC (Stateless Address Auto Config) initially, where the later 64 bits of the host address are formed from an EUI-64 (a 64 bit MAC address). However since RFC 4941 and IPv6 privacy extensions, this is no longer the case. A temporary, randomized, IPv6 is used for outbound connections. The temporary address is changed every 24 hours (by default). All major OSs support IPv6 privacy extensions.
  • DHCPv6 does not use the MAC address to create an IPv6 address, and therefore there is nothing to worry about in most Enterprise networks (which tend to use DHCPv6).
  • So what? What is someone going to do with your MAC address on the internet? MAC address by default is locally significant. That means once the packet crosses a router, the sender's MAC address is gone, replaced by the router's MAC address. Knowing the sender's MAC address has no value to the receiver.


Last piece of FUD I want to cover, is that we are wasting address space, and the doubt that we'll have to go to yet another version of IP in a few years.

  • While it is true a /64 for every subnet seems like a lot of IP addresses per subnet, it is a change in the paradigm we have been living with in IPv4. Rather than tightly packed subnets, where an attack is easy, since there is a good chance when guessing an IPv4 address that there is a host there, the /64 creates a sparsely populated subnet, where it is hard to guess an IP address, and therefore harder to launch an attack (assuming one gets past the firewall, or it is an inside job).
  • There are enough addresses to give out a /56 to everyone. Yes, the IPv6 address space is that big. RFC 7421 is a good read on IPv6 address space. With the currently allocated 2000::/3 there are over 10 trillion /48 to be allocated. And that is less than 1% of the total IPv6 address space.

IPv6 has matured, and is ready

IPv6 has matured a lot since 1995 (RFC 1883). Over 10% of internet traffic is over IPv6 today. Large deployments, such as the huge server farms at Facebook at using IPv6. Lessons have been learned, the protocol has been improved. It is time to move past the IPv4 thinking, and start learning, deploying, and using IPv6. After all, it is the future of the Internet.

Monday, April 25, 2016

systemd: Ubuntu 16.04 better, not best

by Craig Miller

Sure other Linux distributions have already released versions with systemd. But just last week, Ubuntu, one of the most popular distros released XenialXerus aka 16.04 LTS with systemd integrated. Actually Canonical has been phasing in pieces of systemd for some time. But with the release of 16.04 LTS, the networking component (systemd-networkd) has been enabled.

A quick look of the new releases reveals that Canonical has been following the evolution of systemd, and made sure they had a recent version (229). That is only one behind the current version of 230. Considering they did a code freeze two months ago, that is pretty good.

What IPv6 issues remain for this long term release of Ubuntu?

Referring to the list from systemd: oh, you wanted to run IPv6? post, the folks at Canonical have fixed:
  • Temporary addresses (RFC 4941) are broken from version 224 to 228 (systemdissue#2242) this was actually fixed in v229, and I suspect this is why Ubuntu went with v229.
  • Interface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd (systemd issue #2912)
  • SLAAC address does not conform to RFC 4862, using EUI-64 (fixed in version 230)

What is NOT fixed?
  • IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases (systemd issue#2977)
The behaviour of the RA flood is better than the current version 230, as Ubuntu server isn't disconnected, but the route table is flooded, and the "real" default route is lost, causing the server to lose IPv6 connectivity to anything off-net (like the internet). Even after the RA flood is stopped, and real RAs are heard from the real router, the default route is not inserted, and the server remains crippled. The only method to recover, is to reboot the server. Not the nicest thing to do, especially if the server is in production.

I haven't tested the virtualization issues, but I am hopeful that they have been improved by Canonical as well.

So where does this leave us?

I stand by my recommendation. systemd, and specifically systemd-networkd is not ready for IPv6 production systems. That said, it is certainly good enough for testing and development systems, giving you valuable experience towards your IPv6 deployment.

Sunday, April 10, 2016

systemd: oh, you wanted to run IPv6?

by Craig Miller

Oh, you wanted to run IPv6?
Regardless of what you think of systemd, it is coming to a Linux computer near you. But what has systemd done for IPv6? The short answer is that it has replaced, superseded, supplanted, the mature kernel IPv6 code with new less mature, more buggy application layer code. For reasons unknown to the author, systemd has decide to take over mundane operations, such as setting up SLAAC addresses, Temporary addresses, and IPv6 routes, rather than allowing the kernel to continue these operations.

That said, since your Linux system will be (or is) running systemd, it is good to know the issues, and how to test them.

How do you get your IPv6 prefix?

Smaller organizations and home users will use PA (Provider Assigned) addressing. Depending on how much you pay your ISP, your prefix block could be changing dynamically. As a new prefix is delegated, SLAAC (Stateless Auto Address Config) will automagically renumber the hosts on your network.

Larger organizations will apply to their RIR (Regional Internet Registry) such as APNIC (Asia), RIPE (Europe), or ARIN (North America) for a PI (Provider Independent) block of addresses.

Know your environment

The first place is to understand your network environment. Large Enterprise has embarked on the road of using DHCPv6 in an attempt to limit the number of IPv6 addresses assigned to each host. In order to increase the ability to track users, and make building ACLs (Access Control Lists) more easily. DHCPv6 is also the preferred method used by IPAM (Internet Protocol Address Management) software, as it makes host discovery easier.

Smaller organizations may opt to use SLAAC since it is pretty much plug and play. Unless you have Windows machines on your network, then you will need to run DHCPv6 at a minimum for DNS server information distribution (see Dual Stack: the good, bad, and the ugly).

Write a Test Plan

What ever environment you are using for address distribution to your hosts, a test plan in a lab setup is always a good idea. A sample set of tests you may want to run:

  • Basic Testing
    • disable/enable interface
    • Restart systemd-networkd, are IPv6 addresses restored?
  • Virtual Machine (VM) and Container Testing (if you are using virtualization)
  • Negative testing/network attacks, such as RA flood (THC IPv6 is an excellent tool)
    • Rogue RA Testing (no, RAGuard will not stop this attack)

A few IPv6 issues with systemd

The devs at systemd (and Redhat) have decided to re-implement functionality already in the kernel code. Therefore there are a few things which worked just fine in a non-systemd system, but do not in a modern system (systemd version 229).

  • IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases (systemd issue#2977)
  • Temporary addresses (RFC 4941) are broken from version 224 to 228 (systemd issue#2242)
  • Interface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd (systemd issue #2912)
  • Fails to send Router Solicitation (systemd issue #2365)
  • Unable to view DUID (DHCPv6 Identifier) on host (systemd issue #2952)
  • Bridged Interfaces get IPv6 SLAAC addresses (systemd issue#2572)
  • Systemd in a VM failed to start due to RA parsing error (systemd issue#228)
  • IPv6 incorrectly not enabled on Virtuozzo containers (systemd issue#2059)
  • IPv6cceptRouterAdvertisements=yes or unset accepts too many prefixes (systemd issue#2004)
  • Does not support DHCPV6-PD (systemd issue#1080)
  • Does not support SLAAC RDNSS (Recursive DNS Server) option (systemd issue#1079)

Upgrading to systemd distros

Be careful when upgrading your systems which may have not been running systemd before, when all the issues above worked just fine since they were implemented in the kernel*. Examples of OS changes which introduce systemd are from RHEL 6 to 7, CentOS6 to 7, Ubuntu 14.04LTS to 16.04LTS, and Debian 7 to 8 (aka Jessie). More systemd adoption info can be found on Wikipedia.

Time will improve systemd

Like fine wine, I am sure that systemd will improve IPv6 support. After all, it took years to get where we are with the Linux kernel support today. Reviewing the issues above to see if they apply, and having a test plan, will allow you to go into IPv6/systemd with your eyes wide open.

Given the number of IPv6 issues, I would not recommend putting a systemd machine into an IPv6 production environment at this time. But perhaps you can lean on your distro provider to correct the issues which apply to your network, and we will all benefit.

Update Nov 2016 (see IPv6 & Systemd another look)

*DHCPv6 has not been implemented in the kernel

Sunday, March 27, 2016

Dual Stack, the good, bad, and the ugly

by Craig Miller

Dual Stack: a foot in the past and the future
There have been several transitional methods developed to help us through the transitional phase of migrating from IPv4 to IPv6. Remember that although IPv6 has the letters I and P in it, it really is a different networking protocol, and therefore requires additional expertise to really know the new Internet Protocol.

Fortunately, the common operating systems, Windows, MacOS X, Linux, and BSD have excellent support for IPv6. And can run both IPv6 and IPv4 at the same time. This is called Dual Stack.

The Good

Because OSs are bilingual (think Canada:  English and French) there is little to be done on end stations or hosts, to start communicating via IPv6. Assuming the network is correctly configured with RAs (Router Advertisements), the host will pick up and address either via SLAAC (Stateless Address Auto Config) or DHCPv6 (yet another different protocol) and off you go.

Dual Stack has been suggested as the easiest transitional mechanism, since it is already built into almost every machine out there. Of course some applications may have to be rewritten to take advantage of the IPv6 stack.

The Bad

Because hosts are now dual stack does not mean that all applications which run on the hosts are instantly dual stack. There are a lot of applications which were either written a while ago, or by someone who didn't give thought to IPv6.

For example, sshd has supported IPv6 for more than a decade. But recently when I went looking for a sftp client for my Chromebook, I was disappointed to discover that 'SFTP File System' was IPv4 only.

Dual Stack also means double the work for Network Support people. Because although the end station or host, automagically configures itself for IPv6, the routers and firewalls do not. Routers need to be configured for two protocols, which may not have a direct mapping between IPv4 and IPv6. Firewalls, must have two sets of ACLs (Access Control List) and because IPv6 is a different protocol, one can not just copy and paste the IPv4 ACL entries into the IPv6 ACL.

For this reason large service providers such as T-Mobile are dumping dual stack in favour of going IPv6 only, and using technologies like 464XLAT ( RFC 6877) to accommodate those IPv4 only applications (such as Skype and Twitter).

The Ugly

Although I said that end stations or hosts automagically get addresses, and they do, there are apparently different interpretations on how the end station receives DNS (Domain Name Service) server information. You will remember that I mentioned in a previous post, Quad Eh? that DNS was going to be even more useful with IPv6.

There are two ways for a host to receive DNS server information:
  1. RA via the RDNSS field (Recursive DNS Server RFC 6106)
  2. DHCPv6 via option 23 (IANA option numbers)

Unfortunately, not everyone agrees on which method to use*. Google (read: all android phones) is using the first  method and Microsoft (read: all windows machines) is using the second method. Which means that network operators must run a DHCPv6 server even if they don't want to manage addresses (think: the O-bit in the RA). Although larger networks will probably already be using (or plan to use) DHCPv6, for smaller networks, this is just additional overhead.

A Path to the Future

As you can see Dual Stack solves many problems in transitioning from IPv4 to IPv6, but it is not a panacea. As I mentioned in an earlier post Not your Grandmothers IP, there is no time like the present to start learning the new Internet Protocol, and putting a foot towards the future.

* more good info on Dual Stack

Sunday, March 13, 2016

v6Brouter Part 2: v6Bridge Firewall

by Craig Miller

In the last post (see NATiness and the v6Brouter), I introduced the concept of a v6Brouter, part bridge, and part router. Bridging IPv6 is a good way to extend already allocated IPv6 networks (e.g. a /64 already allocated by an upstream ISP) to overcome networks segmented by  IPv4 NAT. But typically bridging does not include any kind of firewall. After all, bridging is simple, just doing a MAC DA lookup, and sending the packet on its way.

But although you may want to extend the IPv6 network, you may not trust all those unknown hosts upstream. It would be nice to have a v6Bridge Firewall.

Netfilter to the rescue

Fortunately the creators of Netfilter (better known as iptables, and ip6tables on Linux) created a layer 2 filtering tool called ebtables. With ebtables, it is possible to inspect bridged packets and forward them as we saw with the v6brouter, or send them up the stack to be inspected by ip6tables.

Challenges to ip6tables when bridging

A challenge of ip6tables when bridging (IPv6) is that it thinks all packets come from the bridge interface (br-lan on OpenWRT). This makes it difficult to filter upstream packets, but allow your trusted downstream packets to pass unfiltered.

The concept of Mark

Fortunately Netfilter created the concept of mark, where a packet is marked with a number (uint32). It is possible to mark a packet (with set-mark) and later filter on that marked value. Fortunately, ebtables and ip6tables share the same mark value.

Solving ip6tables challenge

Since ip6tables can't distinguish which port a bridged packet came from, and ebtables can, it is a simple task of having ebtables mark a packet that comes from the upstream port, and then later having ip6ables filter on that marked value of the packet.

Tying it all together

Now that is is possible to mark a packet at L2 ingress port, and then read the marked value at L3, it is possible to create a v6Bridge Firewall. Here's a diagram which explains the path of a packet in a v6Bridge Firewall.
v6Brouter with v6Bridge Firewall

For example, let's say you want to filter SSH from the outside, but permit the inside network (the extended trusted network) to be able to SSH to the outside and the internet. Or to put it another way, block incoming SSH, while allowing outbound SSH.

The v6Bridge Firewall inspects the packet at ISO layers 2, 3, and 4.

  • Layer 2 - Check ethertype for IPv6, check ingress port, if outside network, mark packet with the value of 16
  • Layer 3 - Read marked value, if 16, go to L4
  • Layer 4 - Check if the destination port = 22 (SSH), if so drop, if not forward to L2
  • Back at Layer 2, Check MAC Destination Address, and send out correct port

All of this happens in the blink of an eye*. Netfilter is highly optimized code running in kernel space. Without hardware support, it would be difficult to find faster code.

v6Brouter and your Firewall too

The v6Brouter project (on github) now includes v6bridge firewall. When extending your IPv6 network across all those cascaded IPv4 NAT situations, you can now have your bridge and firewall it too.

* Running thruput tests on OpenWRT with and without firewall enabled using netcat over IPv6/TCP transferring a 100MB file  yielded no appreciable difference at 100Mbit speeds.

Sunday, February 28, 2016

NAT-tiness and the v6Brouter

by Craig Miller

OpenWRT supports over 1000 routers*
IPv4 NAT (Net Address Translation) is everywhere. It has become the default small network deployment method. Want to cascade a second router on your network, use NAT. Want to create a hotspot on your phone to share your internet, use NAT.

How can IPv6 be widely deployed when there are so many niche deployments of NAT breaking end-to-end network connectivity everywhere? In large scale networks, there are plenty of IPv6 addresses. Prefixes are delegated or assigned, and standard routing is performed. But in smaller networks, where only a /64 was allocated, how do you provide connectivity to the downstream cascaded network, or the smart phone hotspot?

Brouters 101

A solution to the lack of IPv6 prefixes is to use a brouter. A brouter is a part bridge (operating at layer 2 of the OSI model) and part router (routing at layer 3).

In a traditional IPv4 network, a cascaded router would look like this:
IPv4 NAT-ed networks
Packets from the laptop must flow across double NAT to reach the internet. The green networks (dark and light) may be wired or wireless. I see this a lot.

By changing the configuration of Router B, to be a brouter, we see that the IPv6 topology no longer directly maps to the IPv4 topology.
Brouted network with a single (blue) IPv6 prefix
By using a Brouter, IPv6 traffic (including RAs, NDP, etc) is bridged on Router B, while IPv4 traffic continues to be NAT-ed, maintaining the IPv4 topology. Of course there is no reason why IPv4 needs to be NAT-ed at this point, but there are situations (think: smart phone hotspot) where maintaining the existing IPv4 topology will be desired. v6Brouting allows the maintaining of the IPv4 topology, while providing IPv6 access to the downstream networks.

Brouter Advantages

The advantage of using and IPv6 brouter is that it does not require any special protocols from the upstream router (think: prefix delegation), nor does it require the upstream ISP to provide anything more than a /64. As we saw in a previous post, Think Networks Think Big, a /64 has plenty of addresses to extend to small scale subtended networks.

OpenWRT the open source router

OpenWRT is open source software supporting over a 1000 different routers. It includes a daemon, odhcpd, which handles DHCPv6 services on the LAN side. It also has a RA and NDP relay mode which essentially bridges IPv6 traffic across the router. However this occurs at the application layer, and performance suffers because of it.

Another solution to extend the IPv6 /64, is to use the optimized linux bridge code in the kernel to bridge the IPv6 packets. This code is part of the widely deployed netfilter code (think: ip6ables on Linux). Netfilter not only has layer 3, or network layer filtering capability via iptabes and ip6tables, but also layer 2 filtering via ebtables.

v6Brouter config tool

I have written an open source script which leverages netfilter and configures a v6brouter for OpenWRT (v15.05) which can be found on github.

Bringing IPv6 to the isolated masses

In order to have ubiquitous IPv6, we must get through a nasty transition phase, which includes getting past the many, many cascaded NAT deployments. The v6Brouter can bring IPv6 to many of those isolated small networks.

* WRT54G image: By G sintornillos - User generated content., Public Domain,

Monday, February 15, 2016

Think Networks, Think Big

by Craig Miller

Break with the Old Thinking

IPv6 is not only a different protocol than IPv4, it is a different mindset. We have been trained (in the IPv4) world to conserve address space, change subnet masks to the be the smallest amount of addresses for a given subnet, make point to point links /30, and so forth. We have become ingrained with a subconscious frugality about address space.

Grains of wheat on a chessboard

The designers of IPv6 knew the power of 2. Just like the "wheat and chessboard problem" where the number of grains of wheat (some say rice) double with each square on the chess board, the amount of wheat becomes enormous rather quickly, so does the IPv6 address space.

How big is the IPv6 address space? One of the better comparisons I have heard is that if the entire 4 billion IPv4 addresses were equal to a meter in length (think: yard stick), then then the IPv6 address space would be 35,970,651,894,390,958,082,809 light years long! it is a really, really, really big address space.

Sparsely Populated Networks

IPv6 has plenty of addresses. So many in fact, that no one living today will ever see us run out. There are 
18,446,744,073,709,551,616 per /64 subnet. I am not suggesting you try to put 18 quintilian hosts into a single subnet, but part of the transition to IPv6 is from densely packed subnets (IPv4) to sparsely populated ones.

This has an immediate advantage that port scanners, like nmap, will take weeks (or longer) to scan a single subnet. New methods of attack will be developed, but sparsely populated subnets will be the rule in IPv6, not the exception.

Think Networks, think big

We must change our mindset from conserving address space, to how many networks do we need. In a Class B network, where all subnets are /24, it is possible to have 255 subnets. When I was working for the University of Hawaii in the 1990s, they ran the entire University off of a single Class B network where even the point to point links were /24.

With IPv6, the standard subnet is /64. Change this at your peril. Many things stop working, like SLAAC when the prefix is not /64. So taking the /64 as the base line, how many /64's do you need for your office/enterprise/university campus?

It is not uncommon to get a /56 from your ISP. Shoot, I have a /56 at my house. Yes, that is 255 /64 networks. I don't currently need that many networks at my house, but it certainly gives me room to maneuver. 

When applying for a Provider Independent prefix (say from ARIN or RIPE), the RIR (Regional Internet Registry) will typically give you a /48, /40, or even a /32 if you can justify it. How big is a /48? That is 2^16, or 65,535 /64 networks.

Address Planning

So remember, that when creating your address plan for IPv6, don't just overlay it on top of your IPv4 network, laying down subnet over subnet. When creating the IPv4 network, you were in a different mindset, conserve addresses, that no longer applies. But that said, how many subnets do you have in your network today? 200, 2000, 100,000? You can use this a starting point of how many IPv6 networks to request from your RIR.

Count networks, not hosts. You won't have enough hosts to exhaust a single /64. But there are very good reasons to have multiple networks, DMZ, segregation of departmental data, internal cloud, etc.

Wasting addresses

You wouldn't waste things of value (think: dollars, or a Rembrandt). But if the item has little or no value, then it isn't considered a waste. Would you pick up a penny on the street? If so, then you are still thinking conservation of pennies (or addresses).

If you think using a /64 on a point to point link is a waste, then you haven't completed the mental transition from conservation of address space to thinking of networks. Using /64s everywhere, even on point to point links make everything simpler.

Getting back to simpler times

If you are old enough to remember running RIP (Routing Information Protocol RFC 1058) networks, you will remember that we ran /24 networks everywhere (since RIP1 didn't support variable subnet masking). It was a simpler world, and we have an opportunity with IPv6 to get back to that simplicity.

Sunday, January 24, 2016

Network X-Ray Vision

by Craig Miller

Seeing packets on the wire

Last month I wrote of tools which are useful in discovering IPv6 hosts, and prefixes on your network (see Tools of the Trade). In this post, I'd like to cover two more tools in more detail. I think of them as the X-Ray Vision tools for your network, tcpdump and wireshark.

What is there to see?

Both of these tools will allow you to see the individual packets that are flowing on your network. Each has a protocol decoder which will present the packet in an easily human readable format. Through the decoding, you will see the ethernet header followed by the IP header, followed by the UDP or TCP header (in Scapy format: /Ethernet/IPv6/TCP).

With this information, you will be able to see check the EtherType field is correct (86DD for IPv6) or the actual source IPv6 address of a packet. For example, remember that IPv6 hosts have multiple IPv6 addresses, and it may not be clear from the outset which address the stack has selected as its source address.

Overview of tcpdump and wireshark

tcpdump is the oldest, and dates back to 1987. Therefore it takes a little learning to get used to it, but the efforts will quickly pay off. Although tcpdump was initially just for troubleshooting TCP packets, it has long since been expanded to be a full fledged protocol decoder.

It is CLI-based, and therefore can be run on the far end of a ssh connection, making long distance troubleshooting possible, by actually seeing the packets flowing in a remote network half a globe away.

wireshark is a GUI (Graphical User Interface) tool which grew out of the need for an open source alternative to the very expensive Network Associates Sniffer products in the 1990s. It runs on most OSs such as Windows, Linux, and MacOS X. For those who have used wireshark for sometime, you will remember than the original name was ethereal. It has virtually replaced all the closed source protocol decoder tools, and has a very active development community.

As with any GUI-based tool, it is easier to use, with familiar menus, and easy capture start and stop buttons. Drilling down on individual packets is also quite straight forward as it uses a familiar hierarchical approach to headers (e.g. clicking the '+' displays the detail of the header).

But being GUI-based is a double-edged sword, easier to use, but harder to run when debugging a network on the other side of the globe.

Tying the two together

But one doesn't have to choose between the two. Each will read a previously saved capture file, typically in pcap format. Therefore it is possible to use tcpdump to capture on a distant network, and send the pcap file back to a local machine to be displayed with wireshark. Of course, this can get tedious if repeated more than a few times.

To have tcpdump save the capture to a file use the -w option. The following will capture 100 packets from interface eth0 to the file mycap.pcap
tcpdump -c 100 -i eth0 -w mycap.pcap

After transferring mycap.pcap file back to your local machine, you can view it with wireshark by just specifying the filename on the command line (or use the File->Open menu).
wireshark mycap.pcap

And there your remote network traffic will be displayed in all its protocol decoded glory!

Drinking from a firehose

You will quickly discover that unless you are on a small test network, you will quickly be overrun by the shear amount of packets. Like trying to find a needle in a haystack, without some sort of filter, finding the real packets of interest will be a challenge.

Capture filters to the rescue. wireshark borrowed the concept, and syntax of capture filters from tcpdump. This is good, as you only have to learn one syntax for both tools.

Since this is a blog about IPv6,the two must useful capture filters are:
  1. ip6
  2. icmp6
Continuing on the example above, but only capturing IPv6 traffic place the capture filter last on the command line.
tcpdump -c 100 -i eth0 -w mycap.pcap ip6

Now when you drag the pcap file back to your local machine, only the IPv6 traffic will be displayed.
$ tcpdump -r mycap.pcap
11:21:59.728933 IP6 2607:c000:8100:5600:3003:203f:53b5:37f2 > 2607:f8b0:400a:804::1005: ICMP6, echo request, seq 1, length 64
11:21:59.743360 IP6 2607:f8b0:400a:804::1005 > 2607:c000:8100:5600:3003:203f:53b5:37f2: ICMP6, echo reply, seq 1, length 64

Another capture filter useful in debugging the mechanics of IPv6 on your network is icmp6. You will remember that IPv6 uses ICMPv6 for many things including arp-like MAC address resolution, and Router Advertisements (RAs). To capture only the ICMPv6 packets on your remote network.
tcpdump -c 100 -i eth0 -w mycap.pcap icmp6

As I mentioned wireshark uses the same syntax for capture filters. In order to use a capture filter with wireshark one has to start a capture with options. This will open a new dialog box with a Capture Filter button. You can type the capture filter in the blank on the right, or click on the button for a selection of common capture filters.

Is anyone out there?

Of course, sniffing a switched network (or even modern wireless networks), you will quickly discover you can only see the packets destined to the host where you are running tcpdump (or wireshark). Ethernet switches conveniently switch other traffic off of your LAN connection. This is good for reducing traffic congestion, but less good if you were interested in using your X-Ray glasses on the network.

Picking a centralized location on your network, such as your router, is often the best place for sniffing the network, since the router will see any packet that is destined to be forwarded off the network.

Because the router is such a good location to debug network issues, you can often find a packet capture application for the device. OpenWRT a popular open source distribution for home routers, and conveniently has tcpdump as a installable package.

The Internet is your friend

Because tcpdump and wireshark are popular tools, there is a lot of good information on the internet. This post is not intended to be a definitive tutorial on the tools, but to help you get started looking in the right direction for debugging IPv6 networks with your X-Ray glasses. Here is a couple of good starting points.

Using your X-Ray Glasses

With these packet sniffer tools, you will see in great detail what is on your network. Using the capture filters, you can quickly debug or just explore how IPv6 is working on your network. Capture filters help you use your X-Ray Glasses wisely.