What makes a good IPv6 implementation?

Bad IPv6 support costs Money

I have been working with Docker lately, and as cool as the container technology is, it was originally built without consideration for IPv6, and then IPv6 was bolted on later. Making supporting IPv6 full of expensive work-a-rounds.

But that got me thinking what makes a good IPv6 implementation? Of course this is my opinion, and you are free to toss in other criteria, so think of this as a thought starter.

Why is this important?

With 25% of the internet carried over IPv6 as of this writing, if you are developing a product which has a lifetime of 5 to 10 years, and you aren't giving thought as to how you will support IPv6, then your product will:

• A) fail, or
• B) you will try to bolt on IPv6 on the side, or
• C) have to be completely rewritten.

All of that costs money.

A good IPv6 device implementation

There are broad areas where IPv6 should work well.

Addressing

As much as I like the simplicity of SLAAC (Stateless Address Auto Config), there are certainly use cases where DHCPv6 is a better choice. A good implementation should:

• Support both addressing methods, SLAAC, and DHCPv6
• Be able to reestablish IPv6 GUA (Global Unique Address) once the device comes out of sleep/suspend or link down/up (systemd suffers this problem)
• Play well with DNS. Very few of us enjoy typing IPv6 addresses, the implementation should have a stable IPv6 address which can be entered into DNS without requiring a lot of DNS churn.

Routing

IPv6 is not IPv4 with colons. There are somethings which are different for good reason.

• Default routes are link-local addresses (Docker fails on this one big time). GUAs may change, link-locals shouldn't.
• Supports RA (Router Advertisement) fields, RDNSS (DNS server), and DNSSL (DNS domain search list). Not much use having an address if the host can't resolve names
• If the device is routing (such as Docker) then support DHCPv6-PD, and provide the option of prefix delegation into the container/downstream network.

Resiliency

Basic protection from network misconfiguration, or out right attacks makes the IPv6 device better prepared for production use.

• Rational limit on the number of IPv6 addresses an interface may have. Before systemd, the Linux kernel defaulted to 16. This seemed like a good compromise. Back in systemd v232, it was possible to exhaust memory on an IPv6 host by feeding it Random RA addresses, creating a denial of service. FreeBSD v11.5 has a similar problem, where the system will add over 3000 IPv6 addresses, and the system will slow to a crawl.
• Rational limit on the number of neighbours. IPv6 /64 networks are sparsely populated and therefore one shouldn't have to expect to support all 16 Quintilian (2^64) neighbours. Something like 1000, or even 256 should be enough.
• Don't assume that the Linux Stack has your back. Since systemd has become widespread, there are many IPv6 systemd bugs, which weren't there in the pre-systemd kernel days. IPv6 is a different stack, be sure to test it.

Summary

I am sure I missing a few, but this is a start. When developing a product, the business case for supporting IPv6 well, is that it will save you money in the long run, by not having to go back and try to bolt IPv6 on, or rewrite your network stack later.


P.S I wouldn't recommend putting Docker into production because of the severe IPv6 limitations.

Yachts colliding: Creative Commons/ Mark Pilbeam

IPv6 Printer Support: Finally getting there

by Craig Miller

I recently replaced my printer with a Brother laser printer. I was pleasantly surprised with the level of IPv6 support. Although the printer includes WLAN support, I already had an ethernet cable in place, and it was a snap to connect to the network.

The UI designers of printers long ago realized that the small 20 character display is too limited to provide useful information, and instead there are options to print out full pages of info, including the Network Info. On this, I could not only see the usual IPv4 info, but also the SLAAC address that the printer had picked up.

Typing in the SLAAC address only once

Armed with the SLAAC address, I updated my local DNS server, since I really only wanted to type the IPv6 address once. Once that task was done, it was a snap to log into the printer's management web page over IPv6 using the DNS name.

IPv6 info

Of course, IPv6 support isn't perfect. There is no DHCPv6 support, and the IP Filtering feature is still  IPv4-only. So the next step for Brother is feature parity, but it is a good start.

Investigating IPv6 Printer Services

Even without feature parity, a quick scan by nmap reveals that the printing services are also available over IPv6.

Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-05 09:16 PST
Nmap scan report for 6kunane.hoomaha.net (2001:470:ebbd:0:3e2a:f4ff:fe37:dac4)
Host is up (0.010s latency).
Not shown: 995 closed ports
PORT     STATE SERVICE
80/tcp   open  http
443/tcp  open  https
515/tcp  open  printer
631/tcp  open  ipp
9100/tcp open  jetdirect

Printing from an IPv6-only network

And I was able to successfully print to the Brother printer which sits on my dual-stack network from my IPv6-only network. Kudos to Brother.

Windows 10 now runs in SLAAC Networks

by Craig Miller

Microsoft released the Creator Update last month (11 April 2017) with lots of interesting stuff. But the most interesting for IPv6 is support for the RDNSS field in the RA (Router Advertisement). The RDNSS field is the one that carries DNS server information in the RA.

In order to run an IPv6-only SLAAC-based network the host must need 2 things: an address, and the address of a DNS server. Without DNS, IPv4 or IPv6 is pretty useless these days.

Windows 10 and SLAAC Requirements

In order to see the new feature in action, the Windows 10 machine must:

• Be in a IPv6-only network (no IPv4) 
• Hear a RA (Router Advertisement) without the M-bit set (or DHCPv6 disabled). 

Of course, it would be good if your router was sending RDNSS in the RA. 

Windows 10 SLAAC-only Details

In this environment, the output of ipconfig is still a little misleading:

C:\Users\Craig>ipconfig /all

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : lan

   Description . . . . . . . . . . . : Realtek PCIe GBE 

   ...
   DHCPv6 IAID . . . . . . . . . . . : 75761763
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-1F-A3-2E-82-80-FA-5B-96-37-56

   DNS Servers . . . . . . . . . . . : fdf7:56a9:b7af:1101::1

   Connection-specific DNS Suffix Search List :

                                       lan

The DNS Server field is now showing my RDNSS address (the ULA address of my router) and DNSSL (DNS Search List)!

Another way to confirm the DNS servers that Windows 10 knows about is with a netsh command:
C:\Users\Craig>netsh int ipv6 show dnsservers

Configuration for interface "Ethernet"
    DNS servers configured through DHCP:  fdf7:56a9:b7af:1101::1
    Register with which suffix:           Primary only

Running a quick check to see if it can actually resolve an address using only RDNSS:

C:\Users\Craig>nslookup www.google.com

Server:  OpenWrt.lan

Address:  fdf7:56a9:b7af:1101::1

Non-authoritative answer:

Name:    www.google.com

Addresses:  2604:470:4001:806::2004

          172.217.29.164

Now it is possible to run simplified (SLAAC) networks

The fact that MS is now supporting SLAAC-only networks is a huge shift from their previous DHCPv6 only stance. Why is this important? Because there are use-cases for SLAAC-only networks, and now not only can you use your Android devices (which don’t do DHCPv6) but also your Windows 10 machines as well.

Windows continues to dominate the PC market with about 85%. Now with Windows 10 Creator Update, there is no excuse to not deploy IPv6 in your network now.

* adapted from my ipv6hawaii.org article
** Win10 details from André Lange, author of ip6neigh

North American IPv6 Summit

by Craig Miller

The North American IPv6 Summit was held in Sunnyvale last week. It is always a pleasure to be in a large room with people who get it. There is no convincing that we need to give up our comfortable Linus-blanket of IPv4 for something new and different. No, everyone in the room is a convert, and many are outspoken advocates.

The conference was organized by the regional IPv6 Task Forces: California IPv6 Task Force, Rocky Mountain IPv6 Task Force, Texas IPv6 Task Force, and Mexico IPv6 Task Force.

Speakers, shakers and movers

Some of the speakers were:

• Tony Scott, the former CIO of the Unitied States of America
• John Curran, the President and CEO of ARIN (American Registry for Internet Numbers)
• Kevin Jones, Chair for IPv6 transition at NASA
• John Brzozowski, Chief Architect, IPv6 and Fellow,  at Comcast

Major Points

So if everyone is a convert, there's nothing to talk about, right? Actually there are quite a few things. Some of the key points made at this year's conference were:

• Dual-stack is only half way. We need to start moving to IPv6-only networks. There were presentations on how Cisco, Microsoft, and Comcast are doing just that.
• IPv6 impacts on Cloud Computing, and IoT. A case study of BC Hydro operating 2 million smart meters (IoT) all on IPv6.
• Content is being delivered over IPv6, thanks to CDN (Content Delivery Networks), like Akamai and Cloudfare, fronting IPv4-only legacy sites.
• Microsoft adds SLAAC capability to Windows 10, Creator Update (11 April 2017). Now it is possible to have Windows and Android on the same SLAAC (Stateless Address Auto Config) IPv6-only network!

Missed it? Here's the presentations

Thanks to all the volunteers of the regional task forces, and Linked-in for hosting the conference. The presentations are posted online, in case you didn't make it down to Sunnyvale last week. Hope to see you there next time.

Breath deeply, a /64 is a good thing

by Craig Miller

Earth's atmosphere is vast

IPv6 is a different networking protocol from IPv4. To illustrate, just look at how a host can get a globally unique IP address (GUA) without requiring any servers, without sending a packet*.  This is made possible by the fact that IPv6 does not have variable subnet-masking. All end user LANs are a /64.

There are some who say that it is a waste of address space to use an entire /64 (4 billion * 4 billion) on a single subnet. I would suggest that those who do are still in a IPv4-frame-of-mind. The IPv6 address space is vast (see Simplifying Subnetting).

Take a breath

Think of it this way, how many breaths have your taken since getting up this morning? I have no idea how many I have taken. But if you are scuba diving, and are breathing off of a tank of compressed air, you pay close attention to how much air you have (usually measured in minutes, but when diving, that can change depending on how deep you are, and what kind of effort you are expending). We don't think about how many breaths we take driving into work, because Earth's atmosphere is vast.

A prefix longer than /64?

There is the additional problem, that if the LAN subnet is defined as something other than /64, many things will break, much more than just SLAAC (Stateless Address Auto Config). The authors of RFC 7421 have exhaustively gone through the RFCs to examine what assumes the end user LAN is a /64.

Some failure modes highlighted by RFC 7241:

• Routers may drop packets on interfaces /65 to /126 (inclusive)
• Specific Multicast Addresses will fail (resulting in NDP failures)
• The Cryptographically Generated Address format [RFC3972] relies on /64
• Many Transition mechanisms, such as NAT64, XLAT464 
• Duplicate Address risk, should SLAAC be modified to work with more than /64
• Link-Local, defines the Interface ID (IID) as 64 bits wide
• IP Address Management (IPAM) systems assume /64
• Firewall look up issues (where there are not enough content addressable memory bits to include longer prefixes + L4 port numbers)

Although not specifically a failure mode, a smaller subnet space takes less time for attackers to scan.

Think Different, Think Vast

Think about how many breaths you take in a day, a month, a year. Compared to how much air is in the world, what you breath is insignificant. So take a deep breath, and remember IPv6 address space is vast.


* Duplicate Address Detection (DAD) is performed, but only after a GUA has been selected.
** Photo licences under Creative Commons

Excuse me, your MAC is showing

by Craig Miller 

MACs exposed*

Stateless Address AutoConfiguration (SLAAC) is an easy way to get an IPv6 network up and running quickly. But  RFC 4291 states that the  the interface MAC address should be used when generating a Global Unique Address (GUA).

For the most part, this works quite well. Most MAC addresses are unique (although they only need to be unique within the same broadcast domain). But it has problems when it comes to hardware repair. What happens when you need to replace a NIC card, the GUA changes. Not good.

I have argued in the past that this was a small issue, since servers are probably going to have static IPv6 addresses, which get mapped in DNS to names. Therefore changing a NIC card won't change your GUA.

Exposing information

Other's have expressed concerns that using the MAC address in the GUA exposes that information to the outside world. But on the far side of a router, the MAC address is no longer significant, and therefore pretty useless for network attacks.

But sharing the MAC address globally does reveal the hardware vendor of your NIC, since the first 3 bytes of a MAC address is registered to a manufacturer. This may lead some try to infer an installed OS, and limit their attacks based on this assumption. But the privacy extensions (RFC 4941) have all but eliminated this problem, since temporary GUAs are used for outbound connections.

When infinite is only 16 million

But in-bound connections are still received on the SLAAC MAC-based GUAs.  Reading RFC 7707 (Network Reconnaissance in IPv6 Networks) has convinced me that it is time to move beyond the old MAC-based GUA. RFC 7707 discusses how an attacker can limit the number of addresses being probed from the 2^64 address space by selecting a common NIC vendor (such as Apple, 10:9a:dd:xx:yy:zz) thus reducing the address pool to 2^24. Or in more humanly understandable numbers reducing the potential address from 1 in 18,446,744,073,709,551,616 to 1 in 16,777,216 (or 16 million). Suddenly, the seemingly infinite address space of 2^64 is quite manageable to probe.

A Standard to the rescue

Fortunately, there's a new standard to disconnect the MAC address from the GUA. RFC 7217 (Semantically Opaque Address Generation). Rather than use the MAC address, the RFC recommends using a cryptographic method to generate the lower 64 bits of an address. This opens the entire 2^64 address space again for a GUA on a device. Making it very difficult to guess by an attacker.

The latest version of Mac OSX (10.12) and Windows (10) support this method of GUA generation. Unfortunately, Linux a former leader in the realm of IPv6 support, is behind. A feature enhancement (#4625) has been submitted to systemd for RFC 7217 support.

SLAAC, letting you slack off

Where does this leave us? IPv6 is improving, clever people are fixing the problems of the earlier implementations. With RDNSS (see IPv6 RA, RA, RA) and Semantically Opaque addresses, a SLAAC environment can be viable option to run IPv6 networks with the minimum of management hassle**.


* Macadamia Nuts in and out of the shell
** OK, if you have Windows machines on the network, it may not be enough (see Dual Stack, the good, bad, and ugly)

IPv6 & Systemd, another look

by Craig Miller

Lennart Poettering on systemd*

Systemd is coming to a computer near you. All the major linux distributions have adopted it. Systemd is a replacement for init, PID 1, and if the developers had stopped their, there would be no real impact. But the developers have expanded the influence of systemd to include many other aspects of how your linux workstation operates.

• networkd - Controls the aspects of networking, including DHCPv6 Client, RA processing, SLAAC address assignment, IPv6 route insertion, etc
• resolved - Acts like DNSMasq (but more limited) in resolving Domain Names. It gets pointed to DNS servers via networkd.
• Other aspects of the linux host, including logind, hostnamed, locald, timedated, machined, importd to name a few.

The systemd daemons will also process IPv4 information, but that is beyond the scope of this blog.

Another Look at systemd

It has been six months since I took a good look at systemd and IPv6. You may remember that there were some clear issues which may cause one to pause before deploying a systemd host in an IPv6 production network.

A recap of IPv6 issues with systemd

The devs at systemd (and Redhat) have decided to re-implement functionality already in the kernel code. Therefore there are a few things which worked just fine in a non-systemd system, but do not in a modern system. Retesting with systemd version 231, we see there are improvements (below).

Description	Issue	Status
IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases	#2977	FIXED
Temporary addresses (RFC 4941) are broken from version 224 to 228	#2242	FIXED
nterface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd	#2912	Still broken
Fails to send Router Solicitation	#2365	Still broken
Unable to view DUID (DHCPv6 Identifier) on host	#2952	Closed but Still broken
Bridged Interfaces get IPv6 SLAAC addresses	#2572	FIXED
Systemd in a VM failed to start due to RA parsing error	#2228	FIXED
IPv6 incorrectly not enabled on Virtuozzo containers	#2059	FIXED
IPv6cceptRouterAdvertisements=yes or unset accepts too many prefixes	#2004	FIXED
Does not support DHCPV6-PD	#1080	Still broken
Does not support SLAAC RDNSS	#1079	FIXED

A few more issues have been added in the past 6 months

Description	Issue	Status
networkd does not support DNS List option in RA	#4590	Open
Semantically Opaque Interface Identifiers with IPv6 SLAAC RFC 7217	#4625	Open
networkd: handle MTU field in IPv6 RA	#4464	Open
RFC 7084 support in networkd (automatic ipv6 prefix delegation)	#4073	Open

Time will improve systemd

It is unfortunate that the systemd team has decided to re-implement existing functionality (e.g. in the kernel, dnsmasq, odhcpd). But like fine wine, it is improving. Be sure to check the details of the issues which may impact your testing/deployment and save yourself a bunch of time. As always, taking the time to plan your IPv6 deployment will save you time in the end.

*Lennart Poettering T Shirt 

Fighting FUD

by Craig Miller

FUD is Fear, Uncertainty and Doubt. There's a lot of mis-information about IPv6 circulating on the internet. Much of it is old, out-of-date, and just plain wrong.

Fear 

The fear is that since IPv6 uses global addresses, external hosts can make inbound connections to your machine.

• This is IPv4/NAT thinking. That somehow NAT is considered a security blanket that protects us from the nasty outside world. Before there was NAT (see Breaking the NAT Frame of Mind) there were firewalls to keep the bad guys out. Firewalls have improved immensely in the past 20 years. They are stateful, and do connection tracking. Modern firewalls will only allow outbound connections (by default), thus keeping the bad guys on the outside. Even your IPv6 enabled Home Router comes with an IPv6 connection tracking firewall.

Uncertainty

Another is the uncertainty that IPv6 leaks your MAC address to the internet. I find several things wrong with this:

• This applied to SLAAC (Stateless Address Auto Config) initially, where the later 64 bits of the host address are formed from an EUI-64 (a 64 bit MAC address). However since RFC 4941 and IPv6 privacy extensions, this is no longer the case. A temporary, randomized, IPv6 is used for outbound connections. The temporary address is changed every 24 hours (by default). All major OSs support IPv6 privacy extensions.
• DHCPv6 does not use the MAC address to create an IPv6 address, and therefore there is nothing to worry about in most Enterprise networks (which tend to use DHCPv6).
• So what? What is someone going to do with your MAC address on the internet? MAC address by default is locally significant. That means once the packet crosses a router, the sender's MAC address is gone, replaced by the router's MAC address. Knowing the sender's MAC address has no value to the receiver.

Doubt

Last piece of FUD I want to cover, is that we are wasting address space, and the doubt that we'll have to go to yet another version of IP in a few years.

• While it is true a /64 for every subnet seems like a lot of IP addresses per subnet, it is a change in the paradigm we have been living with in IPv4. Rather than tightly packed subnets, where an attack is easy, since there is a good chance when guessing an IPv4 address that there is a host there, the /64 creates a sparsely populated subnet, where it is hard to guess an IP address, and therefore harder to launch an attack (assuming one gets past the firewall, or it is an inside job).
• There are enough addresses to give out a /56 to everyone. Yes, the IPv6 address space is that big. RFC 7421 is a good read on IPv6 address space. With the currently allocated 2000::/3 there are over 10 trillion /48 to be allocated. And that is less than 1% of the total IPv6 address space.

IPv6 has matured, and is ready

IPv6 has matured a lot since 1995 (RFC 1883). Over 10% of internet traffic is over IPv6 today. Large deployments, such as the huge server farms at Facebook at using IPv6. Lessons have been learned, the protocol has been improved. It is time to move past the IPv4 thinking, and start learning, deploying, and using IPv6. After all, it is the future of the Internet.

systemd: Ubuntu 16.04 better, not best

by Craig Miller

Sure other Linux distributions have already released versions with systemd. But just last week, Ubuntu, one of the most popular distros released XenialXerus aka 16.04 LTS with systemd integrated. Actually Canonical has been phasing in pieces of systemd for some time. But with the release of 16.04 LTS, the networking component (systemd-networkd) has been enabled.

A quick look of the new releases reveals that Canonical has been following the evolution of systemd, and made sure they had a recent version (229). That is only one behind the current version of 230. Considering they did a code freeze two months ago, that is pretty good.

What IPv6 issues remain for this long term release of Ubuntu?

Referring to the list from systemd: oh, you wanted to run IPv6? post, the folks at Canonical have fixed:

• Temporary addresses (RFC 4941) are broken from version 224 to 228 (systemdissue#2242) this was actually fixed in v229, and I suspect this is why Ubuntu went with v229.
• Interface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd (systemd issue #2912)
• SLAAC address does not conform to RFC 4862, using EUI-64 (fixed in version 230)

What is NOT fixed?

• IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases (systemd issue#2977)

The behaviour of the RA flood is better than the current version 230, as Ubuntu server isn't disconnected, but the route table is flooded, and the "real" default route is lost, causing the server to lose IPv6 connectivity to anything off-net (like the internet). Even after the RA flood is stopped, and real RAs are heard from the real router, the default route is not inserted, and the server remains crippled. The only method to recover, is to reboot the server. Not the nicest thing to do, especially if the server is in production.

I haven't tested the virtualization issues, but I am hopeful that they have been improved by Canonical as well.

So where does this leave us?

I stand by my recommendation. systemd, and specifically systemd-networkd is not ready for IPv6 production systems. That said, it is certainly good enough for testing and development systems, giving you valuable experience towards your IPv6 deployment.

systemd: oh, you wanted to run IPv6?

by Craig Miller

Oh, you wanted to run IPv6?

Regardless of what you think of systemd, it is coming to a Linux computer near you. But what has systemd done for IPv6? The short answer is that it has replaced, superseded, supplanted, the mature kernel IPv6 code with new less mature, more buggy application layer code. For reasons unknown to the author, systemd has decide to take over mundane operations, such as setting up SLAAC addresses, Temporary addresses, and IPv6 routes, rather than allowing the kernel to continue these operations.

That said, since your Linux system will be (or is) running systemd, it is good to know the issues, and how to test them.

How do you get your IPv6 prefix?

Smaller organizations and home users will use PA (Provider Assigned) addressing. Depending on how much you pay your ISP, your prefix block could be changing dynamically. As a new prefix is delegated, SLAAC (Stateless Auto Address Config) will automagically renumber the hosts on your network.

Larger organizations will apply to their RIR (Regional Internet Registry) such as APNIC (Asia), RIPE (Europe), or ARIN (North America) for a PI (Provider Independent) block of addresses.

Know your environment

The first place is to understand your network environment. Large Enterprise has embarked on the road of using DHCPv6 in an attempt to limit the number of IPv6 addresses assigned to each host. In order to increase the ability to track users, and make building ACLs (Access Control Lists) more easily. DHCPv6 is also the preferred method used by IPAM (Internet Protocol Address Management) software, as it makes host discovery easier.

Smaller organizations may opt to use SLAAC since it is pretty much plug and play. Unless you have Windows machines on your network, then you will need to run DHCPv6 at a minimum for DNS server information distribution (see Dual Stack: the good, bad, and the ugly).

Write a Test Plan

What ever environment you are using for address distribution to your hosts, a test plan in a lab setup is always a good idea. A sample set of tests you may want to run:

• Basic Testing
• disable/enable interface
• Restart systemd-networkd, are IPv6 addresses restored?
• Virtual Machine (VM) and Container Testing (if you are using virtualization)
• Negative testing/network attacks, such as RA flood (THC IPv6 is an excellent tool)
• Rogue RA Testing (no, RAGuard will not stop this attack)

A few IPv6 issues with systemd

The devs at systemd (and Redhat) have decided to re-implement functionality already in the kernel code. Therefore there are a few things which worked just fine in a non-systemd system, but do not in a modern system (systemd version 229).

• IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases (systemd issue#2977)
• Temporary addresses (RFC 4941) are broken from version 224 to 228 (systemd issue#2242)
• Interface disable/enable IPv4 will reaquire and address, but IPv6 will not (other than link-local), and will remain address-less until restarting networkd (systemd issue #2912)
• Fails to send Router Solicitation (systemd issue #2365)
• Unable to view DUID (DHCPv6 Identifier) on host (systemd issue #2952)
• Bridged Interfaces get IPv6 SLAAC addresses (systemd issue#2572)
• Systemd in a VM failed to start due to RA parsing error (systemd issue#228)
• IPv6 incorrectly not enabled on Virtuozzo containers (systemd issue#2059)
• IPv6cceptRouterAdvertisements=yes or unset accepts too many prefixes (systemd issue#2004)
• Does not support DHCPV6-PD (systemd issue#1080)
• Does not support SLAAC RDNSS (Recursive DNS Server) option (systemd issue#1079)

Upgrading to systemd distros

Be careful when upgrading your systems which may have not been running systemd before, when all the issues above worked just fine since they were implemented in the kernel*. Examples of OS changes which introduce systemd are from RHEL 6 to 7, CentOS6 to 7, Ubuntu 14.04LTS to 16.04LTS, and Debian 7 to 8 (aka Jessie). More systemd adoption info can be found on Wikipedia.

Time will improve systemd

Like fine wine, I am sure that systemd will improve IPv6 support. After all, it took years to get where we are with the Linux kernel support today. Reviewing the issues above to see if they apply, and having a test plan, will allow you to go into IPv6/systemd with your eyes wide open.

Given the number of IPv6 issues, I would not recommend putting a systemd machine into an IPv6 production environment at this time. But perhaps you can lean on your distro provider to correct the issues which apply to your network, and we will all benefit.

Update Nov 2016 (see IPv6 & Systemd another look)

*DHCPv6 has not been implemented in the kernel

Dual Stack, the good, bad, and the ugly

by Craig Miller

Dual Stack: a foot in the past and the future

There have been several transitional methods developed to help us through the transitional phase of migrating from IPv4 to IPv6. Remember that although IPv6 has the letters I and P in it, it really is a different networking protocol, and therefore requires additional expertise to really know the new Internet Protocol.

Fortunately, the common operating systems, Windows, MacOS X, Linux, and BSD have excellent support for IPv6. And can run both IPv6 and IPv4 at the same time. This is called Dual Stack.

The Good

Because OSs are bilingual (think Canada:  English and French) there is little to be done on end stations or hosts, to start communicating via IPv6. Assuming the network is correctly configured with RAs (Router Advertisements), the host will pick up and address either via SLAAC (Stateless Address Auto Config) or DHCPv6 (yet another different protocol) and off you go.

Dual Stack has been suggested as the easiest transitional mechanism, since it is already built into almost every machine out there. Of course some applications may have to be rewritten to take advantage of the IPv6 stack.

The Bad

Because hosts are now dual stack does not mean that all applications which run on the hosts are instantly dual stack. There are a lot of applications which were either written a while ago, or by someone who didn't give thought to IPv6.

For example, sshd has supported IPv6 for more than a decade. But recently when I went looking for a sftp client for my Chromebook, I was disappointed to discover that 'SFTP File System' was IPv4 only.

Dual Stack also means double the work for Network Support people. Because although the end station or host, automagically configures itself for IPv6, the routers and firewalls do not. Routers need to be configured for two protocols, which may not have a direct mapping between IPv4 and IPv6. Firewalls, must have two sets of ACLs (Access Control List) and because IPv6 is a different protocol, one can not just copy and paste the IPv4 ACL entries into the IPv6 ACL.

For this reason large service providers such as T-Mobile are dumping dual stack in favour of going IPv6 only, and using technologies like 464XLAT ( RFC 6877) to accommodate those IPv4 only applications (such as Skype and Twitter).

The Ugly

Although I said that end stations or hosts automagically get addresses, and they do, there are apparently different interpretations on how the end station receives DNS (Domain Name Service) server information. You will remember that I mentioned in a previous post, Quad Eh? that DNS was going to be even more useful with IPv6.

There are two ways for a host to receive DNS server information:

1. RA via the RDNSS field (Recursive DNS Server RFC 6106)
2. DHCPv6 via option 23 (IANA option numbers)

Unfortunately, not everyone agrees on which method to use*. Google (read: all android phones) is using the first  method and Microsoft (read: all windows machines) is using the second method. Which means that network operators must run a DHCPv6 server even if they don't want to manage addresses (think: the O-bit in the RA). Although larger networks will probably already be using (or plan to use) DHCPv6, for smaller networks, this is just additional overhead.

A Path to the Future

As you can see Dual Stack solves many problems in transitioning from IPv4 to IPv6, but it is not a panacea. As I mentioned in an earlier post Not your Grandmothers IP, there is no time like the present to start learning the new Internet Protocol, and putting a foot towards the future.


* more good info on Dual Stack https://ttlexpired.co.uk/author/richardpattersonnz/

Think Networks, Think Big

by Craig Miller

Break with the Old Thinking

IPv6 is not only a different protocol than IPv4, it is a different mindset. We have been trained (in the IPv4) world to conserve address space, change subnet masks to the be the smallest amount of addresses for a given subnet, make point to point links /30, and so forth. We have become ingrained with a subconscious frugality about address space.

Grains of wheat on a chessboard

The designers of IPv6 knew the power of 2. Just like the "wheat and chessboard problem" where the number of grains of wheat (some say rice) double with each square on the chess board, the amount of wheat becomes enormous rather quickly, so does the IPv6 address space.

How big is the IPv6 address space? One of the better comparisons I have heard is that if the entire 4 billion IPv4 addresses were equal to a meter in length (think: yard stick), then then the IPv6 address space would be 35,970,651,894,390,958,082,809 light years long! it is a really, really, really big address space.

Sparsely Populated Networks

IPv6 has plenty of addresses. So many in fact, that no one living today will ever see us run out. There are 
18,446,744,073,709,551,616 per /64 subnet. I am not suggesting you try to put 18 quintilian hosts into a single subnet, but part of the transition to IPv6 is from densely packed subnets (IPv4) to sparsely populated ones.

This has an immediate advantage that port scanners, like nmap, will take weeks (or longer) to scan a single subnet. New methods of attack will be developed, but sparsely populated subnets will be the rule in IPv6, not the exception.

Think Networks, think big

We must change our mindset from conserving address space, to how many networks do we need. In a Class B network, where all subnets are /24, it is possible to have 255 subnets. When I was working for the University of Hawaii in the 1990s, they ran the entire University off of a single Class B network where even the point to point links were /24.

With IPv6, the standard subnet is /64. Change this at your peril. Many things stop working, like SLAAC when the prefix is not /64. So taking the /64 as the base line, how many /64's do you need for your office/enterprise/university campus?

It is not uncommon to get a /56 from your ISP. Shoot, I have a /56 at my house. Yes, that is 255 /64 networks. I don't currently need that many networks at my house, but it certainly gives me room to maneuver. 

When applying for a Provider Independent prefix (say from ARIN or RIPE), the RIR (Regional Internet Registry) will typically give you a /48, /40, or even a /32 if you can justify it. How big is a /48? That is 2^16, or 65,535 /64 networks.

Address Planning

So remember, that when creating your address plan for IPv6, don't just overlay it on top of your IPv4 network, laying down subnet over subnet. When creating the IPv4 network, you were in a different mindset, conserve addresses, that no longer applies. But that said, how many subnets do you have in your network today? 200, 2000, 100,000? You can use this a starting point of how many IPv6 networks to request from your RIR.

Count networks, not hosts. You won't have enough hosts to exhaust a single /64. But there are very good reasons to have multiple networks, DMZ, segregation of departmental data, internal cloud, etc.

Wasting addresses

You wouldn't waste things of value (think: dollars, or a Rembrandt). But if the item has little or no value, then it isn't considered a waste. Would you pick up a penny on the street? If so, then you are still thinking conservation of pennies (or addresses).

If you think using a /64 on a point to point link is a waste, then you haven't completed the mental transition from conservation of address space to thinking of networks. Using /64s everywhere, even on point to point links make everything simpler.

Getting back to simpler times

If you are old enough to remember running RIP (Routing Information Protocol RFC 1058) networks, you will remember that we ran /24 networks everywhere (since RIP1 didn't support variable subnet masking). It was a simpler world, and we have an opportunity with IPv6 to get back to that simplicity.

Tools of the trade

IPv6 Tools

by Craig Miller

IPv6 Tools

Just like the old saying, "with a hammer, everything looks like a nail"  we tend to over use ping or ping6 to troubleshoot our networks. In this post, I wanted to share some other tools which I use in debugging networks.

IP

ip is the successor to the venerable ifonfig. And with good reason, as ip can tell you much more about your configuration. It is installed by default on most distros, and usually lives at /sbin/ip.

link status

ip can display the status of the link (Layer 2 in the OSI model), as well as allow configuration of a VLAN based interface. To display the the link status use:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 10:9a:dd:54:f6:34 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DORMANT qlen 1000
    link/ether 10:9a:dd:ae:81:77 brd ff:ff:ff:ff:ff:ff

As you can see, it reports that status of link (is that cable plugged in?), and the MAC (Media Access Control, aka Ethernet) address.

ip addresses

ip can also show IPv4 and IPv6 addressing (using -4 and -6 respectively).

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 10:9a:dd:54:f6:34 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.15/24 brd 10.1.1.255 scope global eth0
    inet6 2607:c000:815f:5600:e4a1:4d5f:961a:4973/64 scope global temporary dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 2001:470:1d:489:fd2f:ea14:d171:c541/64 scope global temporary dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 2607:c000:815f:5600:fd2f:ea14:d171:c541/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:487d:35e:3834:c9a/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2607:c000:815f:5600:487d:35e:3834:c9a/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:a0f0:7c93:4135:b344/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2607:c000:815f:5600:a0f0:7c93:4135:b344/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2607:c000:815f:5600:129a:ddff:fe54:f634/64 scope global dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 2001:470:1d:489:4d85:44b3:3b87:1513/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:5cd0:431a:b989:4517/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:a121:bf93:87b8:c125/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:c8c8:e6c4:ed49:e502/64 scope global temporary deprecated dynamic 
       valid_lft 6951sec preferred_lft 0sec
    inet6 2001:470:1d:489:129a:ddff:fe54:f634/64 scope global dynamic 
       valid_lft 6951sec preferred_lft 1551sec
    inet6 fe80::129a:ddff:fe54:b634/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DORMANT qlen 1000
    link/ether 10:9a:dd:ae:81:77 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::129a:ddff:feae:8177/64 scope link 
       valid_lft forever preferred_lft forever

As you can see, there are many SLAAC temporary addresses (RFC 4941) on the eth0 interface.

But wait, there's more!

routing

ip can also display IPv4 and IPv6 routing. Remember that IPv6 is a different network protocol (Layer 3), and packets can flow differently than their IPv4 counter parts. No surprises looking at the IPv4 routes:

$ ip -4  route
default via 10.1.1.1 dev eth0  metric 100 
10.1.1.0/24 dev eth0  proto kernel  scope link  src 10.1.1.15 

The IPv6 route table displays the default route with a link-local next hop

$ ip -6  route
2001:470:1d:489::/64 dev eth0  proto kernel  metric 256  expires 6865sec mtu 1280
2607:c000:815f:5600::/64 dev eth0  proto kernel  metric 256  expires 6865sec
fe80::/64 dev eth0  proto kernel  metric 256  mtu 1280
fe80::/64 dev eth1  proto kernel  metric 256 
default via fe80::224:a5ff:fee1:7ca dev eth0  proto kernel  metric 1024  expires 1464sec mtu 1280 hoplimit 64

rdisc6

While ip will tell you the configuration of the host, rdisc6 will tell you the configuration of your router, or at least what it is sending out as Router Advertisements (RAs). RAs send out prefixes, and controls whether clients will start DHCPv6 clients (RFC 3315), with the A, M, and O flags. rdisc6 will make a router solicitation (RS), and print out the RA in response.

$ rdisc6 eth0
Soliciting ff02::2 (ff02::2) on eth0...

Hop limit                 :           64 (      0x40)
Stateful address conf.    :           No
Stateful other conf.      :          Yes
Router preference         :       medium
Router lifetime           :         1800 (0x00000708) seconds
Reachable time            :  unspecified (0x00000000)
Retransmit time           :  unspecified (0x00000000)
 Source link-layer address: 00:24:A5:E1:07:CA
 MTU                      :         1280 bytes (valid)
 Prefix                   : 2607:c000:815f:5600::/64
  Valid time              :         7200 (0x00001c20) seconds
  Pref. time              :         1800 (0x00000708) seconds
 Prefix                   : 2001:470:1d:489::/64
  Valid time              :         7200 (0x00001c20) seconds
  Pref. time              :         1800 (0x00000708) seconds
 Route                    : 2607:c000:815f:5600::/56
  Route preference        :       medium
  Route lifetime          :         7200 (0x00001c20) seconds
 Recursive DNS server     : 2607:c000:815f:5600::1
  DNS server lifetime     :         1800 (0x00000708) seconds
 from fe80::224:a5ff:fee1:7ca

The Stateful address is the M flag, and Stateful other config, is the O flag. You can also see that two prefixes are being advertised into this network (the prefix advertised by my ISP is dynamic, where as my Hurricane Electric tunnel prefix is static).

There is a companion utility ndisc6 which will generate neighbour solicitations (NS). I don't use it much, but in order to install rdisc6, you will most often install the ndisc6 package.

v6disc

In addition to the tools above, I have written an IPv6 automatic discovery tool which you can find on github. v6disc.sh will detect which interfaces are up, and query all IPv6 nodes. if you have been wondering when nmap my scan your IPv6 networks, wait no longer. v6disc also has a Dual Stack option which will correlate IPv6 and IPv4 addresses, making your transition to IPv6 easier.

$ ./v6disc.sh -D
-- Searching for interface(s)
Found interface(s): eth0
-- INT:eth0 prefixs: 2607:c000:815f:5600 2001:470:1d:489
-- Detecting hosts on eth0 link
fe80::129a:ddff:fe54:b634 10.1.1.15
fe80::203:93ff:fe67:4362 10.1.1.18
fe80::211:24ff:fece:f1a 10.1.1.12
fe80::211:24ff:fee1:dbc8 10.1.1.14
fe80::224:a5ff:fef1:7ca 10.1.1.1
fe80::225:31ff:fe02:aecb 10.1.1.9
fe80::226:bbff:fe1e:7e15 10.1.1.23
fe80::256:b3ff:fe04:cbe5 10.1.1.122
fe80::280:77ff:feeb:1dde 10.1.1.13
fe80::a00:27ff:fe21:e445 10.1.1.123
-- Discovered hosts
2607:c000:815f:5600:129a:ddff:fe54:b634 10.1.1.15
2607:c000:815f:5600:203:93ff:fe67:4362 10.1.1.18
2607:c000:815f:5600:211:24ff:fece:f1a 10.1.1.12
2607:c000:815f:5600:211:24ff:fee1:dbc8 10.1.1.14
2607:c000:815f:5600::1 10.1.1.1
2607:c000:815f:5600:225:31ff:fe02:aecb 10.1.1.9
2607:c000:815f:5600:226:bbff:fe1e:7e15 10.1.1.23
2607:c000:815f:5600:256:b3ff:fe04:cbe5 10.1.1.122
2607:c000:815f:5600:280:77ff:feeb:1dde 10.1.1.13
2607:c000:815f:5600:a00:27ff:fe21:e445 10.1.1.123
2001:470:1d:489:129a:ddff:fe54:b634 10.1.1.15
2001:470:1d:489:203:93ff:fe67:4362 10.1.1.18
2001:470:1d:489:211:24ff:fece:f1a 10.1.1.12
2001:470:1d:489:211:24ff:fee1:dbc8 10.1.1.14
2001:470:1d:489::1 10.1.1.1
2001:470:1d:489:225:31ff:fe02:aecb 10.1.1.9
2001:470:1d:489:226:bbff:fe1e:7e15 10.1.1.23
2001:470:1d:489:256:b3ff:fe04:cbe5 10.1.1.122
2001:470:1d:489:280:77ff:feeb:1dde 10.1.1.13
2001:470:1d:489:a00:27ff:fe21:e445 10.1.1.123
-- Pau

There's even a quiet mode which just returns the discovered hosts addresses without all the chatter (good for scripting). v6disc is open source (GPL) and can be found on github at https://github.com/cvmiller/v6disc

Other Tools

Of course there are the network x-ray tools, tcpdump and wireshark.But are too big to properly cover in this post, so I'll cover in another post.

Happy Network

The keys to troubleshooting are know where you are at (ip addr), know where you are going (ip route), and what is out there (rdisc6 & v6disc). With these powerful, yet easy to use tools, your IPv6 network will be humming along in no time.

* Tools image from Creative Commons