systemd: Ubuntu 16.04 better, not best

by Craig Miller

Sure other Linux distributions have already released versions with systemd. But just last week, Ubuntu, one of the most popular distros released XenialXerus aka 16.04 LTS with systemd integrated. Actually Canonical has been phasing in pieces of systemd for some time. But with the release of 16.04 LTS, the networking component (systemd-networkd) has been enabled.

A quick look of the new releases reveals that Canonical has been following the evolution of systemd, and made sure they had a recent version (229). That is only one behind the current version of 230. Considering they did a code freeze two months ago, that is pretty good.

What IPv6 issues remain for this long term release of Ubuntu?

Referring to the list from systemd: oh, you wanted to run IPv6? post, the folks at Canonical have fixed:

• Temporary addresses (RFC 4941) are broken from version 224 to 228 (systemdissue#2242) this was actually fixed in v229, and I suspect this is why Ubuntu went with v229.
• Interface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd (systemd issue #2912)
• SLAAC address does not conform to RFC 4862, using EUI-64 (fixed in version 230)

What is NOT fixed?

• IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases (systemd issue#2977)

The behaviour of the RA flood is better than the current version 230, as Ubuntu server isn't disconnected, but the route table is flooded, and the "real" default route is lost, causing the server to lose IPv6 connectivity to anything off-net (like the internet). Even after the RA flood is stopped, and real RAs are heard from the real router, the default route is not inserted, and the server remains crippled. The only method to recover, is to reboot the server. Not the nicest thing to do, especially if the server is in production.

I haven't tested the virtualization issues, but I am hopeful that they have been improved by Canonical as well.

So where does this leave us?

I stand by my recommendation. systemd, and specifically systemd-networkd is not ready for IPv6 production systems. That said, it is certainly good enough for testing and development systems, giving you valuable experience towards your IPv6 deployment.

systemd: oh, you wanted to run IPv6?

by Craig Miller

Oh, you wanted to run IPv6?

Regardless of what you think of systemd, it is coming to a Linux computer near you. But what has systemd done for IPv6? The short answer is that it has replaced, superseded, supplanted, the mature kernel IPv6 code with new less mature, more buggy application layer code. For reasons unknown to the author, systemd has decide to take over mundane operations, such as setting up SLAAC addresses, Temporary addresses, and IPv6 routes, rather than allowing the kernel to continue these operations.

That said, since your Linux system will be (or is) running systemd, it is good to know the issues, and how to test them.

How do you get your IPv6 prefix?

Smaller organizations and home users will use PA (Provider Assigned) addressing. Depending on how much you pay your ISP, your prefix block could be changing dynamically. As a new prefix is delegated, SLAAC (Stateless Auto Address Config) will automagically renumber the hosts on your network.

Larger organizations will apply to their RIR (Regional Internet Registry) such as APNIC (Asia), RIPE (Europe), or ARIN (North America) for a PI (Provider Independent) block of addresses.

Know your environment

The first place is to understand your network environment. Large Enterprise has embarked on the road of using DHCPv6 in an attempt to limit the number of IPv6 addresses assigned to each host. In order to increase the ability to track users, and make building ACLs (Access Control Lists) more easily. DHCPv6 is also the preferred method used by IPAM (Internet Protocol Address Management) software, as it makes host discovery easier.

Smaller organizations may opt to use SLAAC since it is pretty much plug and play. Unless you have Windows machines on your network, then you will need to run DHCPv6 at a minimum for DNS server information distribution (see Dual Stack: the good, bad, and the ugly).

Write a Test Plan

What ever environment you are using for address distribution to your hosts, a test plan in a lab setup is always a good idea. A sample set of tests you may want to run:

• Basic Testing
• disable/enable interface
• Restart systemd-networkd, are IPv6 addresses restored?
• Virtual Machine (VM) and Container Testing (if you are using virtualization)
• Negative testing/network attacks, such as RA flood (THC IPv6 is an excellent tool)
• Rogue RA Testing (no, RAGuard will not stop this attack)

A few IPv6 issues with systemd

The devs at systemd (and Redhat) have decided to re-implement functionality already in the kernel code. Therefore there are a few things which worked just fine in a non-systemd system, but do not in a modern system (systemd version 229).

• IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases (systemd issue#2977)
• Temporary addresses (RFC 4941) are broken from version 224 to 228 (systemd issue#2242)
• Interface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd (systemd issue #2912)
• Fails to send Router Solicitation (systemd issue #2365)
• Unable to view DUID (DHCPv6 Identifier) on host (systemd issue #2952)
• Bridged Interfaces get IPv6 SLAAC addresses (systemd issue#2572)
• Systemd in a VM failed to start due to RA parsing error (systemd issue#228)
• IPv6 incorrectly not enabled on Virtuozzo containers (systemd issue#2059)
• IPv6cceptRouterAdvertisements=yes or unset accepts too many prefixes (systemd issue#2004)
• Does not support DHCPV6-PD (systemd issue#1080)
• Does not support SLAAC RDNSS (Recursive DNS Server) option (systemd issue#1079)

Upgrading to systemd distros

Be careful when upgrading your systems which may have not been running systemd before, when all the issues above worked just fine since they were implemented in the kernel*. Examples of OS changes which introduce systemd are from RHEL 6 to 7, CentOS6 to 7, Ubuntu 14.04LTS to 16.04LTS, and Debian 7 to 8 (aka Jessie). More systemd adoption info can be found on Wikipedia.

Time will improve systemd

Like fine wine, I am sure that systemd will improve IPv6 support. After all, it took years to get where we are with the Linux kernel support today. Reviewing the issues above to see if they apply, and having a test plan, will allow you to go into IPv6/systemd with your eyes wide open.

Given the number of IPv6 issues, I would not recommend putting a systemd machine into an IPv6 production environment at this time. But perhaps you can lean on your distro provider to correct the issues which apply to your network, and we will all benefit.

Update Nov 2016 (see IPv6 & Systemd another look)

*DHCPv6 has not been implemented in the kernel

Dual Stack, the good, bad, and the ugly

by Craig Miller

Dual Stack: a foot in the past and the future

There have been several transitional methods developed to help us through the transitional phase of migrating from IPv4 to IPv6. Remember that although IPv6 has the letters I and P in it, it really is a different networking protocol, and therefore requires additional expertise to really know the new Internet Protocol.

Fortunately, the common operating systems, Windows, MacOS X, Linux, and BSD have excellent support for IPv6. And can run both IPv6 and IPv4 at the same time. This is called Dual Stack.

The Good

Because OSs are bilingual (think Canada:  English and French) there is little to be done on end stations or hosts, to start communicating via IPv6. Assuming the network is correctly configured with RAs (Router Advertisements), the host will pick up and address either via SLAAC (Stateless Address Auto Config) or DHCPv6 (yet another different protocol) and off you go.

Dual Stack has been suggested as the easiest transitional mechanism, since it is already built into almost every machine out there. Of course some applications may have to be rewritten to take advantage of the IPv6 stack.

The Bad

Because hosts are now dual stack does not mean that all applications which run on the hosts are instantly dual stack. There are a lot of applications which were either written a while ago, or by someone who didn't give thought to IPv6.

For example, sshd has supported IPv6 for more than a decade. But recently when I went looking for a sftp client for my Chromebook, I was disappointed to discover that 'SFTP File System' was IPv4 only.

Dual Stack also means double the work for Network Support people. Because although the end station or host, automagically configures itself for IPv6, the routers and firewalls do not. Routers need to be configured for two protocols, which may not have a direct mapping between IPv4 and IPv6. Firewalls, must have two sets of ACLs (Access Control List) and because IPv6 is a different protocol, one can not just copy and paste the IPv4 ACL entries into the IPv6 ACL.

For this reason large service providers such as T-Mobile are dumping dual stack in favour of going IPv6 only, and using technologies like 464XLAT ( RFC 6877) to accommodate those IPv4 only applications (such as Skype and Twitter).

The Ugly

Although I said that end stations or hosts automagically get addresses, and they do, there are apparently different interpretations on how the end station receives DNS (Domain Name Service) server information. You will remember that I mentioned in a previous post, Quad Eh? that DNS was going to be even more useful with IPv6.

There are two ways for a host to receive DNS server information:

1. RA via the RDNSS field (Recursive DNS Server RFC 6106)
2. DHCPv6 via option 23 (IANA option numbers)

Unfortunately, not everyone agrees on which method to use*. Google (read: all android phones) is using the first  method and Microsoft (read: all windows machines) is using the second method. Which means that network operators must run a DHCPv6 server even if they don't want to manage addresses (think: the O-bit in the RA). Although larger networks will probably already be using (or plan to use) DHCPv6, for smaller networks, this is just additional overhead.

A Path to the Future

As you can see Dual Stack solves many problems in transitioning from IPv4 to IPv6, but it is not a panacea. As I mentioned in an earlier post Not your Grandmothers IP, there is no time like the present to start learning the new Internet Protocol, and putting a foot towards the future.


* more good info on Dual Stack https://ttlexpired.co.uk/author/richardpattersonnz/

v6Brouter Part 2: v6Bridge Firewall

by Craig Miller

In the last post (see NATiness and the v6Brouter), I introduced the concept of a v6Brouter, part bridge, and part router. Bridging IPv6 is a good way to extend already allocated IPv6 networks (e.g. a /64 already allocated by an upstream ISP) to overcome networks segmented by  IPv4 NAT. But typically bridging does not include any kind of firewall. After all, bridging is simple, just doing a MAC DA lookup, and sending the packet on its way.

But although you may want to extend the IPv6 network, you may not trust all those unknown hosts upstream. It would be nice to have a v6Bridge Firewall.

Netfilter to the rescue

Fortunately the creators of Netfilter (better known as iptables, and ip6tables on Linux) created a layer 2 filtering tool called ebtables. With ebtables, it is possible to inspect bridged packets and forward them as we saw with the v6brouter, or send them up the stack to be inspected by ip6tables.

Challenges to ip6tables when bridging

A challenge of ip6tables when bridging (IPv6) is that it thinks all packets come from the bridge interface (br-lan on OpenWRT). This makes it difficult to filter upstream packets, but allow your trusted downstream packets to pass unfiltered.

The concept of Mark

Fortunately Netfilter created the concept of mark, where a packet is marked with a number (uint32). It is possible to mark a packet (with set-mark) and later filter on that marked value. Fortunately, ebtables and ip6tables share the same mark value.

Solving ip6tables challenge

Since ip6tables can't distinguish which port a bridged packet came from, and ebtables can, it is a simple task of having ebtables mark a packet that comes from the upstream port, and then later having ip6ables filter on that marked value of the packet.

Tying it all together

Now that is is possible to mark a packet at L2 ingress port, and then read the marked value at L3, it is possible to create a v6Bridge Firewall. Here's a diagram which explains the path of a packet in a v6Bridge Firewall.

v6Brouter with v6Bridge Firewall

For example, let's say you want to filter SSH from the outside, but permit the inside network (the extended trusted network) to be able to SSH to the outside and the internet. Or to put it another way, block incoming SSH, while allowing outbound SSH.

The v6Bridge Firewall inspects the packet at ISO layers 2, 3, and 4.

• Layer 2 - Check ethertype for IPv6, check ingress port, if outside network, mark packet with the value of 16
• Layer 3 - Read marked value, if 16, go to L4
• Layer 4 - Check if the destination port = 22 (SSH), if so drop, if not forward to L2
• Back at Layer 2, Check MAC Destination Address, and send out correct port

All of this happens in the blink of an eye*. Netfilter is highly optimized code running in kernel space. Without hardware support, it would be difficult to find faster code.

v6Brouter and your Firewall too

The v6Brouter project (on github) now includes v6bridge firewall. When extending your IPv6 network across all those cascaded IPv4 NAT situations, you can now have your bridge and firewall it too.


* Running thruput tests on OpenWRT with and without firewall enabled using netcat over IPv6/TCP transferring a 100MB file  yielded no appreciable difference at 100Mbit speeds.

NAT-tiness and the v6Brouter

by Craig Miller

OpenWRT supports over 1000 routers*

IPv4 NAT (Net Address Translation) is everywhere. It has become the default small network deployment method. Want to cascade a second router on your network, use NAT. Want to create a hotspot on your phone to share your internet, use NAT.

How can IPv6 be widely deployed when there are so many niche deployments of NAT breaking end-to-end network connectivity everywhere? In large scale networks, there are plenty of IPv6 addresses. Prefixes are delegated or assigned, and standard routing is performed. But in smaller networks, where only a /64 was allocated, how do you provide connectivity to the downstream cascaded network, or the smart phone hotspot?

Brouters 101

A solution to the lack of IPv6 prefixes is to use a brouter. A brouter is a part bridge (operating at layer 2 of the OSI model) and part router (routing at layer 3).

In a traditional IPv4 network, a cascaded router would look like this:

IPv4 NAT-ed networks

Packets from the laptop must flow across double NAT to reach the internet. The green networks (dark and light) may be wired or wireless. I see this a lot.

By changing the configuration of Router B, to be a brouter, we see that the IPv6 topology no longer directly maps to the IPv4 topology.

Brouted network with a single (blue) IPv6 prefix

By using a Brouter, IPv6 traffic (including RAs, NDP, etc) is bridged on Router B, while IPv4 traffic continues to be NAT-ed, maintaining the IPv4 topology. Of course there is no reason why IPv4 needs to be NAT-ed at this point, but there are situations (think: smart phone hotspot) where maintaining the existing IPv4 topology will be desired. v6Brouting allows the maintaining of the IPv4 topology, while providing IPv6 access to the downstream networks.

Brouter Advantages

The advantage of using and IPv6 brouter is that it does not require any special protocols from the upstream router (think: prefix delegation), nor does it require the upstream ISP to provide anything more than a /64. As we saw in a previous post, Think Networks Think Big, a /64 has plenty of addresses to extend to small scale subtended networks.

OpenWRT the open source router

OpenWRT is open source software supporting over a 1000 different routers. It includes a daemon, odhcpd, which handles DHCPv6 services on the LAN side. It also has a RA and NDP relay mode which essentially bridges IPv6 traffic across the router. However this occurs at the application layer, and performance suffers because of it.

Another solution to extend the IPv6 /64, is to use the optimized linux bridge code in the kernel to bridge the IPv6 packets. This code is part of the widely deployed netfilter code (think: ip6ables on Linux). Netfilter not only has layer 3, or network layer filtering capability via iptabes and ip6tables, but also layer 2 filtering via ebtables.

v6Brouter config tool

I have written an open source script which leverages netfilter and configures a v6brouter for OpenWRT (v15.05) which can be found on github.

Bringing IPv6 to the isolated masses

In order to have ubiquitous IPv6, we must get through a nasty transition phase, which includes getting past the many, many cascaded NAT deployments. The v6Brouter can bring IPv6 to many of those isolated small networks.




* WRT54G image: By G sintornillos - User generated content., Public Domain, https://commons.wikimedia.org/w/index.php?curid=11231198

Think Networks, Think Big

by Craig Miller

Break with the Old Thinking

IPv6 is not only a different protocol than IPv4, it is a different mindset. We have been trained (in the IPv4) world to conserve address space, change subnet masks to the be the smallest amount of addresses for a given subnet, make point to point links /30, and so forth. We have become ingrained with a subconscious frugality about address space.

Grains of wheat on a chessboard

The designers of IPv6 knew the power of 2. Just like the "wheat and chessboard problem" where the number of grains of wheat (some say rice) double with each square on the chess board, the amount of wheat becomes enormous rather quickly, so does the IPv6 address space.

How big is the IPv6 address space? One of the better comparisons I have heard is that if the entire 4 billion IPv4 addresses were equal to a meter in length (think: yard stick), then then the IPv6 address space would be 35,970,651,894,390,958,082,809 light years long! it is a really, really, really big address space.

Sparsely Populated Networks

IPv6 has plenty of addresses. So many in fact, that no one living today will ever see us run out. There are 
18,446,744,073,709,551,616 per /64 subnet. I am not suggesting you try to put 18 quintilian hosts into a single subnet, but part of the transition to IPv6 is from densely packed subnets (IPv4) to sparsely populated ones.

This has an immediate advantage that port scanners, like nmap, will take weeks (or longer) to scan a single subnet. New methods of attack will be developed, but sparsely populated subnets will be the rule in IPv6, not the exception.

Think Networks, think big

We must change our mindset from conserving address space, to how many networks do we need. In a Class B network, where all subnets are /24, it is possible to have 255 subnets. When I was working for the University of Hawaii in the 1990s, they ran the entire University off of a single Class B network where even the point to point links were /24.

With IPv6, the standard subnet is /64. Change this at your peril. Many things stop working, like SLAAC when the prefix is not /64. So taking the /64 as the base line, how many /64's do you need for your office/enterprise/university campus?

It is not uncommon to get a /56 from your ISP. Shoot, I have a /56 at my house. Yes, that is 255 /64 networks. I don't currently need that many networks at my house, but it certainly gives me room to maneuver. 

When applying for a Provider Independent prefix (say from ARIN or RIPE), the RIR (Regional Internet Registry) will typically give you a /48, /40, or even a /32 if you can justify it. How big is a /48? That is 2^16, or 65,535 /64 networks.

Address Planning

So remember, that when creating your address plan for IPv6, don't just overlay it on top of your IPv4 network, laying down subnet over subnet. When creating the IPv4 network, you were in a different mindset, conserve addresses, that no longer applies. But that said, how many subnets do you have in your network today? 200, 2000, 100,000? You can use this a starting point of how many IPv6 networks to request from your RIR.

Count networks, not hosts. You won't have enough hosts to exhaust a single /64. But there are very good reasons to have multiple networks, DMZ, segregation of departmental data, internal cloud, etc.

Wasting addresses

You wouldn't waste things of value (think: dollars, or a Rembrandt). But if the item has little or no value, then it isn't considered a waste. Would you pick up a penny on the street? If so, then you are still thinking conservation of pennies (or addresses).

If you think using a /64 on a point to point link is a waste, then you haven't completed the mental transition from conservation of address space to thinking of networks. Using /64s everywhere, even on point to point links make everything simpler.

Getting back to simpler times

If you are old enough to remember running RIP (Routing Information Protocol RFC 1058) networks, you will remember that we ran /24 networks everywhere (since RIP1 didn't support variable subnet masking). It was a simpler world, and we have an opportunity with IPv6 to get back to that simplicity.

Network X-Ray Vision

by Craig Miller

Seeing packets on the wire

Last month I wrote of tools which are useful in discovering IPv6 hosts, and prefixes on your network (see Tools of the Trade). In this post, I'd like to cover two more tools in more detail. I think of them as the X-Ray Vision tools for your network, tcpdump and wireshark.

What is there to see?

Both of these tools will allow you to see the individual packets that are flowing on your network. Each has a protocol decoder which will present the packet in an easily human readable format. Through the decoding, you will see the ethernet header followed by the IP header, followed by the UDP or TCP header (in Scapy format: /Ethernet/IPv6/TCP).

With this information, you will be able to see check the EtherType field is correct (86DD for IPv6) or the actual source IPv6 address of a packet. For example, remember that IPv6 hosts have multiple IPv6 addresses, and it may not be clear from the outset which address the stack has selected as its source address.

Overview of tcpdump and wireshark

tcpdump is the oldest, and dates back to 1987. Therefore it takes a little learning to get used to it, but the efforts will quickly pay off. Although tcpdump was initially just for troubleshooting TCP packets, it has long since been expanded to be a full fledged protocol decoder.

It is CLI-based, and therefore can be run on the far end of a ssh connection, making long distance troubleshooting possible, by actually seeing the packets flowing in a remote network half a globe away.

wireshark is a GUI (Graphical User Interface) tool which grew out of the need for an open source alternative to the very expensive Network Associates Sniffer products in the 1990s. It runs on most OSs such as Windows, Linux, and MacOS X. For those who have used wireshark for sometime, you will remember than the original name was ethereal. It has virtually replaced all the closed source protocol decoder tools, and has a very active development community.

As with any GUI-based tool, it is easier to use, with familiar menus, and easy capture start and stop buttons. Drilling down on individual packets is also quite straight forward as it uses a familiar hierarchical approach to headers (e.g. clicking the '+' displays the detail of the header).

But being GUI-based is a double-edged sword, easier to use, but harder to run when debugging a network on the other side of the globe.

Tying the two together

But one doesn't have to choose between the two. Each will read a previously saved capture file, typically in pcap format. Therefore it is possible to use tcpdump to capture on a distant network, and send the pcap file back to a local machine to be displayed with wireshark. Of course, this can get tedious if repeated more than a few times.

To have tcpdump save the capture to a file use the -w option. The following will capture 100 packets from interface eth0 to the file mycap.pcap

tcpdump -c 100 -i eth0 -w mycap.pcap

After transferring mycap.pcap file back to your local machine, you can view it with wireshark by just specifying the filename on the command line (or use the File->Open menu).

wireshark mycap.pcap

And there your remote network traffic will be displayed in all its protocol decoded glory!

Drinking from a firehose

You will quickly discover that unless you are on a small test network, you will quickly be overrun by the shear amount of packets. Like trying to find a needle in a haystack, without some sort of filter, finding the real packets of interest will be a challenge.

Capture filters to the rescue. wireshark borrowed the concept, and syntax of capture filters from tcpdump. This is good, as you only have to learn one syntax for both tools.

Since this is a blog about IPv6,the two must useful capture filters are:

1. ip6
2. icmp6

Continuing on the example above, but only capturing IPv6 traffic place the capture filter last on the command line.

tcpdump -c 100 -i eth0 -w mycap.pcap ip6

Now when you drag the pcap file back to your local machine, only the IPv6 traffic will be displayed.

$ tcpdump -r mycap.pcap
11:21:59.728933 IP6 2607:c000:8100:5600:3003:203f:53b5:37f2 > 2607:f8b0:400a:804::1005: ICMP6, echo request, seq 1, length 64
11:21:59.743360 IP6 2607:f8b0:400a:804::1005 > 2607:c000:8100:5600:3003:203f:53b5:37f2: ICMP6, echo reply, seq 1, length 64

Another capture filter useful in debugging the mechanics of IPv6 on your network is icmp6. You will remember that IPv6 uses ICMPv6 for many things including arp-like MAC address resolution, and Router Advertisements (RAs). To capture only the ICMPv6 packets on your remote network.

tcpdump -c 100 -i eth0 -w mycap.pcap icmp6

As I mentioned wireshark uses the same syntax for capture filters. In order to use a capture filter with wireshark one has to start a capture with options. This will open a new dialog box with a Capture Filter button. You can type the capture filter in the blank on the right, or click on the button for a selection of common capture filters.

Is anyone out there?

Of course, sniffing a switched network (or even modern wireless networks), you will quickly discover you can only see the packets destined to the host where you are running tcpdump (or wireshark). Ethernet switches conveniently switch other traffic off of your LAN connection. This is good for reducing traffic congestion, but less good if you were interested in using your X-Ray glasses on the network.

Picking a centralized location on your network, such as your router, is often the best place for sniffing the network, since the router will see any packet that is destined to be forwarded off the network.

Because the router is such a good location to debug network issues, you can often find a packet capture application for the device. OpenWRT a popular open source distribution for home routers, and conveniently has tcpdump as a installable package.

The Internet is your friend

Because tcpdump and wireshark are popular tools, there is a lot of good information on the internet. This post is not intended to be a definitive tutorial on the tools, but to help you get started looking in the right direction for debugging IPv6 networks with your X-Ray glasses. Here is a couple of good starting points.

• tcpdump filters (a pdf file)
• wireshark capture filters
• wireshark tutorial

Using your X-Ray Glasses

With these packet sniffer tools, you will see in great detail what is on your network. Using the capture filters, you can quickly debug or just explore how IPv6 is working on your network. Capture filters help you use your X-Ray Glasses wisely.

IPv6 is not coming, it is here!

Where are we at?

by Craig Miller

The end of the year is full retrospectives to remind us how far we have come, since we last passed this spot on the earth's orbit around the sun. Why should IPv6 be any different?

For years there has been a Mexican Standoff of sorts with regards to IPv6 adoption. The ISPs didn't want to offer IPv6, because the Content Providers didn't have any IPv6 enabled sites. The Content Providers didn't want to offer IPv6 content, since there was no one to come see it, and so on.

Breaking the log jam

Slowly we are breaking that log jam. But one of the most significant examples of change is Telus, a national ISP here in Canada. This fall they turned on IPv6, and within a short amount of time, 1 out of 3 users on their network is using IPv6.

Telus IPv6 Adoption for 2015

I apologize for the small numbers. But as you can see IPv6 on the Telus network went from essentially Zero, to 37% in the last six months, and really most of it was since the end of September. The State of the Internet site contains the full size graph, as well as many other providers whose names you will recognize, like AT&T, Comcast, and SurfNet (in Europe).

Applications need IPv6 too

But having content and ISPs providing IPv6 connectivity isn't the end of the IPv6 adoption story. There are also the applications. There are still way too many IPv4-only-applications in use today. I ran into one the other day on my Chromebook, SFTP File System. A nice application, allows the mounting of a remote file system over SFTP on the Chromebook. But when I tried to connect to a IPv6 address (or more correctly a DNS name mapped to only an IPv6 address), the application failed, giving a fairly cryptic error. Fortunately, this application is open source, and after inspection, it is easy to spot the IPv4-only socket call.

Help is on the way

The IPv6 Forum has published royalty free sample code (tested on BSD and Linux) to help developers, not only support IPv6 in their applications, but to do so intelligently (using non-blocking socket calls, rather than blocking and making the user wait 30 seconds for the call to fail). The sample code includes client and server examples.

Don't be left behind

With millions of new IPv6 users (who probably don't know they are running IPv6) suddenly online, there is no time like the present to start thinking about your implementation, and how you can support IPv6. Don't be left behind.

Quibbling in IPv6

Looking at the Language of IPv6

by Craig Miller

Trouble with Quibbles

Perhaps I should back up the train a bit, and talk about the IPv6 address and what the parts are called. An example IPv6 address is 2001:0DB8:ABCD:EF01:0001:002:0003:0004 * The address is made of of 8 groups of hexadecimal groups representing 16 bits, separated by colons.

Blowing Chunks

But what is each group of 4 hexidecimal letters called? Apparently it has taken the IETF 13 years to realize a name was needed. Back in 2011, in true IETF style, a draft RFC was created to foster discussion. It is interesting to look at the evolution of the draft RFC proposal (Naming IPv6 address parts). In the earlier drafts (e.g. draft-2), you can see the following suggestions:

1. Chazwazza 
2. Chunk 
3. Column 
4. Colonade, Colonnade 
5. Doctet 
6. Field 
7. Hexadectet 
8. Hit 
9. Orone 
10. Part 
11. Provider number, customer number, network number 
12. Quad nibble, qibble, quibble 
13. Segment 
14. Tuple 
15. Word 

All had reasons for why they represented 16 bits of information, and would not be confusing with other networking terms.

And in light conversation...

As you progress to the 4th revision of the RFC (Naming IPv6 address parts) you will find that they paired the list down to two.

1. Hextet
2. Quibble

Hextet is the official name based on revision 4, with Quibble to allowed in informal conversation. However the RFC was never standardized (there is no RFC number assigned), so it appears to be still up for grabs.

May the chazwazza be with you!

* There is an official documentation prefix defined by RFC 3849, 2001:db8::/32
** ST TroubleWithTribbles" by Source. Licensed under Fair use via Wikipedia

Tools of the trade

IPv6 Tools

by Craig Miller

IPv6 Tools

Just like the old saying, "with a hammer, everything looks like a nail"  we tend to over use ping or ping6 to troubleshoot our networks. In this post, I wanted to share some other tools which I use in debugging networks.

IP

ip is the successor to the venerable ifonfig. And with good reason, as ip can tell you much more about your configuration. It is installed by default on most distros, and usually lives at /sbin/ip.

link status

ip can display the status of the link (Layer 2 in the OSI model), as well as allow configuration of a VLAN based interface. To display the the link status use:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 10:9a:dd:54:f6:34 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DORMANT qlen 1000
    link/ether 10:9a:dd:ae:81:77 brd ff:ff:ff:ff:ff:ff

As you can see, it reports that status of link (is that cable plugged in?), and the MAC (Media Access Control, aka Ethernet) address.

ip addresses

ip can also show IPv4 and IPv6 addressing (using -4 and -6 respectively).

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 10:9a:dd:54:f6:34 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.15/24 brd 10.1.1.255 scope global eth0
    inet6 2607:c000:815f:5600:e4a1:4d5f:961a:4973/64 scope global temporary dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 2001:470:1d:489:fd2f:ea14:d171:c541/64 scope global temporary dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 2607:c000:815f:5600:fd2f:ea14:d171:c541/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:487d:35e:3834:c9a/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2607:c000:815f:5600:487d:35e:3834:c9a/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:a0f0:7c93:4135:b344/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2607:c000:815f:5600:a0f0:7c93:4135:b344/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2607:c000:815f:5600:129a:ddff:fe54:f634/64 scope global dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 2001:470:1d:489:4d85:44b3:3b87:1513/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:5cd0:431a:b989:4517/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:a121:bf93:87b8:c125/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:c8c8:e6c4:ed49:e502/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:129a:ddff:fe54:f634/64 scope global dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 fe80::129a:ddff:fe54:b634/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DORMANT qlen 1000
    link/ether 10:9a:dd:ae:81:77 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::129a:ddff:feae:8177/64 scope link 
       valid_lft forever preferred_lft forever

As you can see, there are many SLAAC temporary addresses (RFC 4941) on the eth0 interface.

But wait, there's more!

routing

ip can also display IPv4 and IPv6 routing. Remember that IPv6 is a different network protocol (Layer 3), and packets can flow differently than their IPv4 counter parts. No surprises looking at the IPv4 routes:

$ ip -4  route
default via 10.1.1.1 dev eth0  metric 100 
10.1.1.0/24 dev eth0  proto kernel  scope link  src 10.1.1.15 

The IPv6 route table displays the default route with a link-local next hop

$ ip -6  route
2001:470:1d:489::/64 dev eth0  proto kernel  metric 256  expires 6865sec mtu 1280
2607:c000:815f:5600::/64 dev eth0  proto kernel  metric 256  expires 6865sec
fe80::/64 dev eth0  proto kernel  metric 256  mtu 1280
fe80::/64 dev eth1  proto kernel  metric 256 
default via fe80::224:a5ff:fee1:7ca dev eth0  proto kernel  metric 1024  expires 1464sec mtu 1280 hoplimit 64

rdisc6

While ip will tell you the configuration of the host, rdisc6 will tell you the configuration of your router, or at least what it is sending out as Router Advertisements (RAs). RAs send out prefixes, and controls whether clients will start DHCPv6 clients (RFC 3315), with the A, M, and O flags. rdisc6 will make a router solicitation (RS), and print out the RA in response.

$ rdisc6 eth0
Soliciting ff02::2 (ff02::2) on eth0...

Hop limit                 :           64 (      0x40)
Stateful address conf.    :           No
Stateful other conf.      :          Yes
Router preference         :       medium
Router lifetime           :         1800 (0x00000708) seconds
Reachable time            :  unspecified (0x00000000)
Retransmit time           :  unspecified (0x00000000)
 Source link-layer address: 00:24:A5:E1:07:CA
 MTU                      :         1280 bytes (valid)
 Prefix                   : 2607:c000:815f:5600::/64
  Valid time              :         7200 (0x00001c20) seconds
  Pref. time              :         1800 (0x00000708) seconds
 Prefix                   : 2001:470:1d:489::/64
  Valid time              :         7200 (0x00001c20) seconds
  Pref. time              :         1800 (0x00000708) seconds
 Route                    : 2607:c000:815f:5600::/56
  Route preference        :       medium
  Route lifetime          :         7200 (0x00001c20) seconds
 Recursive DNS server     : 2607:c000:815f:5600::1
  DNS server lifetime     :         1800 (0x00000708) seconds
 from fe80::224:a5ff:fee1:7ca

The Stateful address is the M flag, and Stateful other config, is the O flag. You can also see that two prefixes are being advertised into this network (the prefix advertised by my ISP is dynamic, where as my Hurricane Electric tunnel prefix is static).

There is a companion utility ndisc6 which will generate neighbour solicitations (NS). I don't use it much, but in order to install rdisc6, you will most often install the ndisc6 package.

v6disc

In addition to the tools above, I have written an IPv6 automatic discovery tool which you can find on github. v6disc.sh will detect which interfaces are up, and query all IPv6 nodes. if you have been wondering when nmap my scan your IPv6 networks, wait no longer. v6disc also has a Dual Stack option which will correlate IPv6 and IPv4 addresses, making your transition to IPv6 easier.

$ ./v6disc.sh -D
-- Searching for interface(s)
Found interface(s): eth0
-- INT:eth0 prefixs: 2607:c000:815f:5600 2001:470:1d:489
-- Detecting hosts on eth0 link
fe80::129a:ddff:fe54:b634 10.1.1.15
fe80::203:93ff:fe67:4362 10.1.1.18
fe80::211:24ff:fece:f1a 10.1.1.12
fe80::211:24ff:fee1:dbc8 10.1.1.14
fe80::224:a5ff:fef1:7ca 10.1.1.1
fe80::225:31ff:fe02:aecb 10.1.1.9
fe80::226:bbff:fe1e:7e15 10.1.1.23
fe80::256:b3ff:fe04:cbe5 10.1.1.122
fe80::280:77ff:feeb:1dde 10.1.1.13
fe80::a00:27ff:fe21:e445 10.1.1.123
-- Discovered hosts
2607:c000:815f:5600:129a:ddff:fe54:b634 10.1.1.15
2607:c000:815f:5600:203:93ff:fe67:4362 10.1.1.18
2607:c000:815f:5600:211:24ff:fece:f1a 10.1.1.12
2607:c000:815f:5600:211:24ff:fee1:dbc8 10.1.1.14
2607:c000:815f:5600::1 10.1.1.1
2607:c000:815f:5600:225:31ff:fe02:aecb 10.1.1.9
2607:c000:815f:5600:226:bbff:fe1e:7e15 10.1.1.23
2607:c000:815f:5600:256:b3ff:fe04:cbe5 10.1.1.122
2607:c000:815f:5600:280:77ff:feeb:1dde 10.1.1.13
2607:c000:815f:5600:a00:27ff:fe21:e445 10.1.1.123
2001:470:1d:489:129a:ddff:fe54:b634 10.1.1.15
2001:470:1d:489:203:93ff:fe67:4362 10.1.1.18
2001:470:1d:489:211:24ff:fece:f1a 10.1.1.12
2001:470:1d:489:211:24ff:fee1:dbc8 10.1.1.14
2001:470:1d:489::1 10.1.1.1
2001:470:1d:489:225:31ff:fe02:aecb 10.1.1.9
2001:470:1d:489:226:bbff:fe1e:7e15 10.1.1.23
2001:470:1d:489:256:b3ff:fe04:cbe5 10.1.1.122
2001:470:1d:489:280:77ff:feeb:1dde 10.1.1.13
2001:470:1d:489:a00:27ff:fe21:e445 10.1.1.123
-- Pau

There's even a quiet mode which just returns the discovered hosts addresses without all the chatter (good for scripting). v6disc is open source (GPL) and can be found on github at https://github.com/cvmiller/v6disc

Other Tools

Of course there are the network x-ray tools, tcpdump and wireshark.But are too big to properly cover in this post, so I'll cover in another post.

Happy Network

The keys to troubleshooting are know where you are at (ip addr), know where you are going (ip route), and what is out there (rdisc6 & v6disc). With these powerful, yet easy to use tools, your IPv6 network will be humming along in no time.

* Tools image from Creative Commons