Friday, November 23, 2018

What makes a good IPv6 implementation?

Bad IPv6 support costs Money

I have been working with Docker lately, and as cool as the container technology is, it was originally built without consideration for IPv6, and then IPv6 was bolted on later. Making supporting IPv6 full of expensive work-a-rounds.

But that got me thinking what makes a good IPv6 implementation? Of course this is my opinion, and you are free to toss in other criteria, so think of this as a thought starter.

Why is this important?


With 25% of the internet carried over IPv6 as of this writing, if you are developing a product which has a lifetime of 5 to 10 years, and you aren't giving thought as to how you will support IPv6, then your product will:
  • A) fail, or
  • B) you will try to bolt on IPv6 on the side, or
  • C) have to be completely rewritten.
All of that costs money.

A good IPv6 device implementation


There are broad areas where IPv6 should work well.

Addressing


As much as I like the simplicity of SLAAC (Stateless Address Auto Config), there are certainly use cases where DHCPv6 is a better choice. A good implementation should:
  • Support both addressing methods, SLAAC, and DHCPv6
  • Be able to reestablish IPv6 GUA (Global Unique Address) once the device comes out of sleep/suspend or link down/up (systemd suffers this problem)
  • Play well with DNS. Very few of us enjoy typing IPv6 addresses, the implementation should have a stable IPv6 address which can be entered into DNS without requiring a lot of DNS churn.

Routing


IPv6 is not IPv4 with colons. There are somethings which are different for good reason.
  • Default routes are link-local addresses (Docker fails on this one big time). GUAs may change, link-locals shouldn't.
  • Supports RA (Router Advertisement) fields, RDNSS (DNS server), and DNSSL (DNS domain search list). Not much use having an address if the host can't resolve names
  • If the device is routing (such as Docker) then support DHCPv6-PD, and provide the option of prefix delegation into the container/downstream network.

Resiliency


Basic protection from network misconfiguration, or out right attacks makes the IPv6 device better prepared for production use.
  • Rational limit on the number of IPv6 addresses an interface may have. Before systemd, the Linux kernel defaulted to 16. This seemed like a good compromise. Back in systemd v232, it was possible to exhaust memory on an IPv6 host by feeding it Random RA addresses, creating a denial of service. FreeBSD v11.5 has a similar problem, where the system will add over 3000 IPv6 addresses, and the system will slow to a crawl.
  • Rational limit on the number of neighbours. IPv6 /64 networks are sparsely populated and therefore one shouldn't have to expect to support all 16 Quintilian (2^64) neighbours. Something like 1000, or even 256 should be enough.
  • Don't assume that the Linux Stack has your back. Since systemd has become widespread, there are many IPv6 systemd bugs, which weren't there in the pre-systemd kernel days. IPv6 is a different stack, be sure to test it.

Summary


I am sure I missing a few, but this is a start. When developing a product, the business case for supporting IPv6 well, is that it will save you money in the long run, by not having to go back and try to bolt IPv6 on, or rewrite your network stack later.


P.S I wouldn't recommend putting Docker into production because of the severe IPv6 limitations.

Yachts colliding: Creative Commons/ Mark Pilbeam

Wednesday, August 8, 2018

Babel: a routing protocol with wireless support

by Craig Miller


Traffic
Previously I wrote about resurrecting the old forgotten routing protocol, RIPng. In a small network of more than one router, you need a routing protocol to share information between the routers. I used RIPng for about six months, turned it on, and pretty much forgot that it was running. Worked like a charm in my wired network.
I moved to a new (to me) house this summer, and thought it was a good opportunity to try out a routing protocol which not only handles wired networks but also wireless. Babel seemed just the thing for this environment.

Enter Babel


Babel is a loop-avoiding distance-vector routing protocol that is robust and efficient both in ordinary wired networks and in wireless mesh networks. Based on the loss of hellos the cost of wireless links can be increased, making sketchy wireless links less preferred.
RFC 6126 standardizes the routing protocol.There are two implementations which are supported on OpenWrt routers, babeld and bird


Creating a network with redundant paths


Like anything in networking, it starts with the physical layer (wireless is a form of physical layer). I attached the wireless links of the backup link router to the production and test routers. Thus creating redundant path of connectivity within my house.
Network Diagram

Running BIRD with Babel


I chose bird6 (the IPv6 version of bird on OpenWrt) because I already had it installed on the routers for RIPng. It was merely a matter of commenting out the RIP section in the /etc/bird6.conf file, and enabling Babel.
The Bird Documentation provides an example. Add the following to /etc/bird6.conf get Babel running in bird6
protocol babel {
    interface "wlan0", "wlan1" {
        type wireless;
        hello interval 1;
        rxcost 512;
    };
    interface "br-lan" {
        type wired;
    };
    import all;
    export all;
}
In the example above, wlan0 is the 2.4 Ghz radio, and wlan1 is the 5 Ghz radio.

Checking the path of connectivity


When determining the connectivity path, traceroute6 (the IPv6 version) is your friend. Checking between the laptop and the DNS server, the path is:
$ traceroute6 6dns
traceroute to 6lilikoi.hoomaha.net (2001:db8:ebbd:4118::1) from 2001:db8:ebbd:bac0:d999:cd8a:cd9b:2037, port 33434, from port 49819, 30 hops max, 60 bytes packets
 1  2001:db8:ebbd:bac0::1 (2001:db8:ebbd:bac0::1)  4.561 ms  0.510 ms  0.487 ms 
 2  2001:db8:ebbd:4118::1 (2001:db8:ebbd:4118::1)  2.562 ms  2.193 ms  1.927 ms 
$ 
The traceroute is showing the path going clockwise through the 2.4 Ghz wireless link.

Network Failure!


To test how well Babel can automatically route around failed links, I started a ping to the DNS server from the laptop and disabled the 2.4 Ghz radio, thus blocking the link the pings were using, and waited...
$ ping6 6dns
PING 6dns(2001:db8:ebbd:4118::1) 56 data bytes
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=1 ttl=63 time=3.54 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=2 ttl=63 time=1.64 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=3 ttl=63 time=2.02 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=4 ttl=63 time=1.64 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=5 ttl=63 time=1.51 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=6 ttl=63 time=1.65 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=7 ttl=63 time=1.58 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=8 ttl=63 time=5.80 ms
From 2001:db8:ebbd:bac0::1 icmp_seq=33 Destination unreachable: No route
From 2001:db8:ebbd:bac0::1 icmp_seq=34 Destination unreachable: No route
...
From 2001:db8:ebbd:bac0::1 icmp_seq=48 Destination unreachable: No route
From 2001:db8:ebbd:bac0::1 icmp_seq=49 Destination unreachable: No route
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=101 ttl=61 time=2.12 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=102 ttl=61 time=3.42 ms
64 bytes from 2001:db8:ebbd:4118::1: icmp_seq=103 ttl=61 time=3.16 ms

As you can see the outage was 93 seconds (101 - 8). Not a record time, OSPF would converge much faster, but still it did fix itself without human intervention.
Checking the connectivity path with traceroute6:
$ traceroute6 6dns
traceroute to 6lilikoi.hoomaha.net (2001:db8:ebbd:4118::1) from 2001:db8:ebbd:bac0:d999:cd8a:cd9b:2037, port 33434, from port 47725, 30 hops max, 60 bytes packets
 1  2001:db8:ebbd:bac0::1 (2001:db8:ebbd:bac0::1)  0.541 ms  0.445 ms  0.437 ms 
 2  2001:db8:ebbd:2080::1 (2001:db8:ebbd:2080::1)  1.705 ms  1.832 ms  1.817 ms 
 3  2001:db8:ebbd:2000::1 (2001:db8:ebbd:2000::1)  2.273 ms  1.891 ms  2.584 ms 
 4  2001:db8:ebbd:4118::1 (2001:db8:ebbd:4118::1)  2.348 ms  2.822 ms  2.289 ms 
$
The path can now be seen to be traveling counter-clockwise around the circle via the 5 Ghz link. The Babel routing protocol is routing packets around the failure.

Wireless is great, except ...


As more and more things come online using wireless there will be more interference and contention for bandwidth, especially in the 2.4 Ghz band. Babel can enables routing of packets around sketchy wireless links due to interference in a crowded wifi environment.

Your Metric may vary


Because wireless is variable, Babel applies differing metrics to routes as the wireless signal changes. An unfortunate side effect of this is that the network is continuously converging (or changing). The route that may have been used last minute to the remote host, my be invalid the next minute.
I noticed this as my previously very stable IPv6-only servers were now disconnecting, or worse, not reachable.

Route Flapping!


As I looked at the OpenWrt syslog (using the logread command) I could see that the routes were continually changing.
Tue Jul 24 14:46:45 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:46:45 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:46:46 2018 daemon.info odhcpd[778]: Using a RA lifetime of 1800 seconds on br-lan
Tue Jul 24 14:47:01 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:47:01 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:47:02 2018 daemon.info odhcpd[778]: Using a RA lifetime of 1800 seconds on br-lan
Tue Jul 24 14:47:33 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:47:33 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:47:34 2018 daemon.info odhcpd[778]: Using a RA lifetime of 1800 seconds on br-lan
Tue Jul 24 14:47:49 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:47:49 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:47:50 2018 daemon.info odhcpd[778]: Using a RA lifetime of 1800 seconds on br-lan
Tue Jul 24 14:48:53 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:48:53 2018 daemon.info odhcpd[778]: Raising SIGUSR1 due to default route change
Tue Jul 24 14:48:54 2018 daemon.info odhcpd[778]: Using a RA lifetime of 1800 seconds on br-lan
...

The problem with this route flapping is that it was being propagated to the other routers which were busy adding and removing routes, causing unreachable to parts of my network. Not a desired behaviour.

Settling things down


To rid my network of the route churn, I changed the Babel wireless interfaces to wired, giving them a stable metric, no longer tied to the variability of the wireless signal quality (signal to noise).
The /etc/bird6.conf now looks like:
protocol babel {
    interface "wlan0", "wlan1" {
        type wired;
        hello interval 5;
    };
    interface "br-lan" {
        type wired;
    };
    import all;
    export all;
}
Restarting bird6, and looking at the syslog, a brief activity can be seen, then the route churn stops, and the network is stable.

My ssh connection was dropped as the network did an initial reconverge, and then I was able log back in and examine the syslog.

Babel, still a work in progress


Babel is still being actively developed, and has a more modern approach to wireless links (something that was near non-existent when RIPng was being standardized back in 1997). Like RIPng, it is easy to set up without having to understand the complexities of OSPF. It is easy to setup on OpenWrt routers and provides redundancy in your network. That said the wireless functionality as implemented by Bird (v 1.63) is not quite there. Fortunately, there is Bird v2.0 out, and I look forward to giving it a try when it comes to OpenWrt.

Postscript


Although the route churn has subsided, I re-measured the convergence time for Babel, and it was quite long, 317 seconds, probably due to the hello timer being set to 5 seconds.
In the end, I reverted my house network to RIPng. Running the same convergence test yielded an outage of only 11 seconds with no route churn.
Perhaps many of the Babel issues are just Bird's implementation. And there may be tweaks to reduce network converge times. I'd happily give Babel another chance, but for now, I'll stick with good ol' RIPng.



** if you are running a firewall, the default on OpenWrt/LEDE, you will need to put in a rule to accept IPv6 UDP port 6696

Originally posted (with more detail) on www.makikiweb.com

Monday, July 9, 2018

Running an IPv6-only webserver

by Craig Miller


There are times in the long struggle for IPv6 adoption that we forget that dual-stack is not the end goal, but merely a transition mechanism. If we are to get to the end goal, IPv6 Everywhere then it isn't enough to have IPv6-only clients on the internet, but also IPv6-only attached content servers.

IPv6-only webserver, easy right?


So how hard can it be? Find a hoster which offers IPv6-only attached servers, install apache, or your favourite webserver, and run, right?
Works great for all 20% that are on the IPv6 internet. But for the other 80% of traffic that is on the internet your server is invisible. Great if you are running a stealth server, not so great if it represents your business and you wanted your customers to reach your website.

Oh, that legacy protocol, IPv4


So not only do you want to have the cool IPv6-ers, but also those legacy IPv4-ers to get to your IPv6-only attached webserver.
One of the techniques to permit IPv4 access to your IPv6-only webserver is to us a HTTP Proxy server. The HTTP Proxy server listens to IPv4 (with a real routable IPv4 address) and then directs traffic to the correct IPv6-only server sitting behind it. Folks have been doing this kind of thing with IPv4 for nearly two decades, the proxy server is also called a load balancer.
Web Proxy Network

Gosh, all my "hits" are from the Proxy Server

So the Proxy Server does the conversion from IPv4 to IPv6, but now all the webserver log entries are from the Proxy Server.
rproxy46-sov-a.mythic-beasts.com - - [15/Jun/2018:01:36:17 +0000] "GET /...
rproxy46-sov-a.mythic-beasts.com - - [15/Jun/2018:01:38:28 +0000] "POST ...
rproxy46-sov-a.mythic-beasts.com - - [15/Jun/2018:01:38:32 +0000] "GET /...
rproxy46-hex-a.mythic-beasts.com - - [15/Jun/2018:01:45:15 +0000] "POST ...
rproxy46-hex-a.mythic-beasts.com - - [15/Jun/2018:01:45:19 +0000] "GET /...
rproxy46-hex-a.mythic-beasts.com - - [15/Jun/2018:01:45:24 +0000] "POST ...
rproxy46-hex-a.mythic-beasts.com - - [16/Jun/2018:03:09:30 +0000] "GET /...
Not all that useful for traffic analysis1. Fortunately, there is a Proxy Protocol which allows the Proxy Server to insert a X-Forwarded-For: line into the HTTP header. This line contains the hostname, or IP address of the original requester.

The Proxy Protocol


In apache, there is a mod-proxy-protocol module which interprets the X-Forwarded-For: line as the original requester, and the logs are now restored to what one would expect.
The downside of the mod-proxy-protocol module is that once it is enabled, the server will not respond correctly to a request without the X-Forwarded-For: line. It returns a 502 server error.
Unfortunately, this means that even though the server is IPv6-only connected, it can not receive requests that do not go through the proxy. The Proxy Server must be in the data path for both IPv4 and IPv6. For load balancer applications, this is understandable.
However, as a migration tool to enable the IPv6-only server to serve both IPv4 and IPv6 it is an unnecessary over-complication.
Even the improved apache module, mod_remoteip which also implements the Proxy Protocol (as of version 2.4.30), does not permit the Proxy Server to forward only IPv4 traffic, and accept IPv6 traffic natively (e.g. without the proxy server).

Controlling access to the server


The old apache 2.2 version of Access Control, using deny from ..., does work in version 2.4 but not with either of modules supporting the Proxy Protocol. To be fair, apache has been stating in their documentation that the old v2.2 way would be deprecated. One must now use the Require not ip ... to work with the mod-proxy-protocol module.
So, it was time to convert my Access List for my server, no time like the present, I guess.

The Downside of using a Proxy server


Other transition mechanisms, such as NAT64, are used less and less as more traffic flows over IPv6. Unfortunately, using a Proxy Server to serve IPv4 clients, requires that IPv6 traffic also must use the Proxy server (if proxy protocol is to be used). Which means that as IPv6 traffic increases, the Proxy Serer remains in the data path, abrogating one of the real advantages of an IPv6-only server, a direct connection.

Wrapping up


So the good news is that it is really easy to put content on IPv6 with an IPv6-only server. The less good news is that the content can also be served to IPv4 clients, but it is overly complicated to do so and in the 21st century, it shouldn't be this hard.

Note 1: Sure you could use Google-Analytics, or something equivalently convenient, but I would rather analyze my own data. Google doesn't need to know everything about me.

Article originally appeared on www.makikiweb.com

Thursday, March 8, 2018

IPv6 Printer Support: Finally getting there

by Craig Miller


I recently replaced my printer with a Brother laser printer. I was pleasantly surprised with the level of IPv6 support. Although the printer includes WLAN support, I already had an ethernet cable in place, and it was a snap to connect to the network.

The UI designers of printers long ago realized that the small 20 character display is too limited to provide useful information, and instead there are options to print out full pages of info, including the Network Info. On this, I could not only see the usual IPv4 info, but also the SLAAC address that the printer had picked up.

Typing in the SLAAC address only once


Armed with the SLAAC address, I updated my local DNS server, since I really only wanted to type the IPv6 address once. Once that task was done, it was a snap to log into the printer's management web page over IPv6 using the DNS name.

IPv6 info


Of course, IPv6 support isn't perfect. There is no DHCPv6 support, and the IP Filtering feature is still  IPv4-only. So the next step for Brother is feature parity, but it is a good start.

Investigating IPv6 Printer Services


Even without feature parity, a quick scan by nmap reveals that the printing services are also available over IPv6.

Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-05 09:16 PST
Nmap scan report for 6kunane.hoomaha.net (2001:470:ebbd:0:3e2a:f4ff:fe37:dac4)
Host is up (0.010s latency).
Not shown: 995 closed ports
PORT     STATE SERVICE
80/tcp   open  http
443/tcp  open  https
515/tcp  open  printer
631/tcp  open  ipp
9100/tcp open  jetdirect

Printing from an IPv6-only network


And I was able to successfully print to the Brother printer which sits on my dual-stack network from my IPv6-only network. Kudos to Brother.


Wednesday, January 17, 2018

Writing IPv6 Apps: Python Webserver

by Craig Miller


Moving to IPv6 starts at home. Applications have to speak IPv6 as well as the network. The good news is that there is lots of software available which already supports IPv6. Unfortunately, there is much more that doesn't.

For example, a Python-based webserver. Certainly not ready for a production network, but handy as a learning tool about how easy it can be to support IPv6 in your application.

Why Python? Python is a wonderful programming language, and getting only better with version 3. There are libraries for most needs, including one which serves up the web. And it runs just about anywhere that Python runs (Windows, Linux, BSD, Mac, Pi, ODROID, etc)

Python module SimpleHTTPServer


The python module SimpleHTTPServer supports IPv4 out of the box with the simple command:
python -m SimpleHTTPServer
However it does not support IPv6. There is no one-line equivalent to support IPv6, so a small script is required.

Looking at the code


The ipv6-httpd script is a short script supporting both IPv4 and IPv6. Looking at the following sections:
  1. Initialization of the HTTPServer object (from SimpleHTTPServer library)
  2. Class creation (of HTTPServerV6) to support IPv6
As with all Python scripts, the details roll backwards from the bottom. Initialization occurs in main
def main():
    global server
    server = HTTPServerV6(('::', listen_port), MyHandler)
    print('Listening on port:' + str(listen_port) + '\nPress ^C to quit')
    server.serve_forever()
    os._exit(0)

The IPv6 part


In order to support IPv6, we use a bit of object oriented inheritance trickery to modify the default of an existing class HTTPServer
class HTTPServerV6(HTTPServer):
    address_family = socket.AF_INET6
This creates a new class (which is used in our server ) with the address_family set to AF_INET6 (aka IPv6). This two-line change to the script transforms an IPv4-only script into an application that also supports IPv6.

Running the code


Now that we have a server which supports both IPv4 and IPv6, all we need to do is cd to the directory we wish to share, and start the server
$ cd public/
$ ~/bin/ipv6-httpd.py 
Listening on port:8080
Press ^C to quit
2001:db8:ebbd:0:4d18:71cd:b814:9508 - - [09/Jul/2017 11:49:41] "GET / HTTP/1.1" 200 -
^CCaught SIGINT, dying

The webserver log is sent to standard out (stdout), and can be redirected to a file if desired. In the above example, an IPv6 client ..:9508 requests an index, then the server is terminated with a ^C.
Want to server from a different directory? Stop the server, cd to another directory, and restart the server, it will now serve files from the new current working directory (cwd) location.

Security (or lack there of)


This example is a personal webserver, designed to be started and stopped whenever you need it. It is NOT a production quality webserver that you should put on the internet. However it shows that supporting IPv6 doesn't have to be a hardship.

Adding IPv6 Support to your App


As you can see, it doesn't have to be difficult to add IPv6 to your apps, you just need to give it some thought when planning your App. By adding IPv6, you will future proof your App by being ready for the future of the Internet.