IPv6 is a new and complicated piece of technology; like any new technology it takes a while to get used to and discover what does and doesn’t really work.

Who am I? I’m Robert Mibus, and I helped roll out many of the IPv6-enabled services at Internode (now part of iiNet). I’m writing this based on my personal experience and opinion, so even though I reference Internode frequently, please don’t consider any of the below official communication from them, OK? :)

What this page is:

• Systems-focused
• Some network stuff you need to know
• What I wish I knew three years ago

What this page is not:

• Waxing on about infinite address space for the sake of itself
• Pretty graphs on consumer uptake
• Spiffy future IPv6-could-do-this explanations

Network Stuff

Connectivity

The first thing about IPv6 is how you’re going to get your connectivity. You have two basic options - obtain it natively (eg. via ethernet or PPP from your ISP or colo provider), or via an IPv6-specific tunnel (eg. from a tunnel provider like HurricaneElectric).

So… Don’t use tunnels; just harass your ISP. IPv6 is critical to future growth on the Internet, and is no longer a “maybe sometime in the future thing”.

Addressing

An IPv4 address is like this: 192.231.203.132.

An IPv6 address is like this: 2001:44b8:0001:0000:0000:0000:0000:0001

In order to make the addresses shorter, you can drop leading zeroes in any group, ie: 2001:44b8:1:0:0:0:0:1

You can also replace one (and *only* one) string of :0:0:0:... (for any number of :0:), with “::”…

2001:44b8:1::1

That’s Internode’s IPv6 DNS resolver address. The IPv4 address is actually longer:

192.231.203.132

It’s worth pointing out that many apps don’t accept IPv6 addresses “as-is”; primarily because adding port identifiers (like :8080) causes ambiguity. In those cases, typically the address has square brackets placed around it, eg. [2001:db8:1::1]:8080

Allocations

An ISP will get a large allocation of IPv6 addresses. Typically this will be a /32 or larger (leaving at least 96 bits for the ISP to play with).

eg.

2001:db8::/32 → ISP

The ISP is likely to hand a customer a block somewhere between a /48 and /64. Internode currently gives out /56 as standard, so a customer’s allocation from an ISP might look like this:

2001:db8:1c7:2d00::/56 → ISP Customer

Individual VLANs are typically cut up as /64s, for reasons we’ll get in to later. A /56 thus gives you 256 fully addressable VLANs. You might carve them up like this:

2001:db8:1c7:2d00::/64 → Servers

2001:db8:1c7:2d01::/64 → Ethernet PCs

2001:db8:1c7:2d02::/64 → Wireless PCs

The first 64 bits is then the “network part” and the second 64 bits is the “host part” of the address.

Allocations - Pitfalls!

Is 2001:db8:1c7:2d::/56

• 2001:db8:1c7:2d00::/56, or
• 2001:db8:1c7:002d::/56?

Leading zeroes are removed, so it’s 002d not 2d00. If you’re at all uncertain when subnetting, put the zeroes back in when you’re working it out! A tool like “sipcalc” on Linux can also help with calculations like that.

While I’m on subnetting - only subnet on 4-bit boundaries, please… (each hex character is 4 bits). Why? You know how in IPv4, it’s easy to see if a given IPv4 address falls inside or outside a /24 subnet, but harder on a /28 or /29? Subnetting inside the 4 bits is going to drive you insane with craziness in a similar way.

(ie., don’t go making a /63, /62, or /61 - round up and make it a /60!)

Topological Division

Without trying to contradict the previous section too much, all those extra bits _can_ be useful for route aggregation. If you run a large network with multiple campuses, you can use all the extra bits to create routes that can be easily aggregated together.

Imagine the topology for connecting a large set of schools. You might take a chunk of your address like this - :0000:. That’s really 0000 0000 0000 0000 in binary. You might take a bit for region (East-coast or not?), a couple of bits for sub-region (City, or other major routing point), and a couple for the specific site number.

One thing you should always try to do is keep the top-most bit reserved.

Why?

Because it means that if you’ve divided yourself too far and run out of usable networks, you can make a whole new subdivision scheme because you still have half of our addresses left!

Network Stuff - Going Smaller Than /64

It’s really not going to be worth it. “Saving address space” isn’t the argument it used to be!

Among other things, you won’t be able to use SLAAC (stateless address autoconfiguration).

Chances are, plenty of other things will be awkward - probably the one I’d be most concerned with is that you’re going to be breaking lots of people’s expectations about what a network should look like, which may confuse others looking at your network.

Technically, going smaller than /64 works in some cases - like point-to-point links - and may even have some benefits in those edge cases. Feel free to do it if there’s a really really really good reason.

But allocate the /64, at least notionally - That way you can still change your mind later!

You may end up being given something smaller than a /64 in cases where you might only normally get a single IP - eg. in a cheap VPS. Take what you can get :)

Hardware Support

For non-enterprise-grade routers, you can check places like Whirlpool. You can also take a look at Internode’s router offerings - all should now support native IPv6.

Be warned though; some hardware might “support” IPv6, but do so in a limited fashion - it might be software-routing rather than hardware, or it might not expose all of the same statistics via SNMP for IPv6, etc. As always, it’s best to test hardware before making a big purchase.

ICMP Firewalling

Don’t. It wasn’t a great idea for IPv4, and IPv6 relies even heavier on ICMP control messages.

There’s RFC4890 to cover precisely how you can filter ICMP safely - it’s 38 pages long.

But, seriously, just don’t. (Unless you need to)

Firewalling!

The suckiest part of dealing with IPv6 firewalls right now, is that they’re typically wholly separate from IPv4 firewalls.

Using iptables? You need to use ip6tables too. Configuring it on your CPE? It’s probably not smart enough to have the two really intermingled either.

What this means is that documentation is even more important than ever. Each and every set of firewall entries in both firewalls should be documented in enough detail that you can see clearly that the two sets of firewalls are functionally identical.

One nice benefit of IPv6 when firewalling, is that you can afford to have an IPv6 address per service on a given host. That makes firewalling easier, and can make it easier to migrate the service to another host later. But, if you’re lazy (like me! :), you’ll probably still be happy using just one IP per host. (Virtualisation does mean we get to have a lot of hosts, though…).

ARP! No, Neighbour Discovery…

In IPv4, “ARP” is used to discover the MAC address of a host you want to talk to. In IPv6, the comparable protocol is Neighbor Discovery (ND) - which uses ICMPv6. The response comes in the form of a Neighbor Advertisment, also ICMPv6.

With ARP, your request is broadcast across the whole VLAN. ND uses link-local multicast to target just a subset of hosts - typically a small enough subset that it is only hitting the one host! (For the technically-inclined, the multicast packet is sent to all hosts that share the last 24 bits of their IPv6 address with the address being looked for).

Like with ARP, this isn’t routed traffic, it’s specific to the layer-2 network that the host is physically connected to.

Broadcast Ping

Much like ARP’s replacement, the replacement for a broadcast ping uses local network multicast too.

All nodes on a network will join a special multicast group with an address of ff02::1. Ping that instead of an address like 192.168.1.255.

There’s a similar one that is only joined by routers (rather than hosts), ff02::2.

Duplicate Address Detection

An in-built part of IPv6 is “Duplicate Address Detection”, or DAD.

In short, when an IP address is brought up on an interface, it uses Neighbour Discovery to make sure it’s the only node with that address. Due to this, freshly-added addresses on an interface will show as “tentative”, and not actually be usable.

This includes all configured IPv6 addresses during boot, so applications need to cope with this if they try to bind to a specific address on startup. Sometimes they don’t cope, and just bomb out with an error that they couldn’t bind to the IP address as it’s not configured. (ISC BIND is an example of this).

Router Advertisements

Similar to a Neighbor Advertisement, a Router Advertisement is a network router saying “Hi, I’m here!”. This can be used by hosts to auto-configure themselves with an IPv6 address, and set a default route for IPv6. (This is called SLAAC - StateLess Address AutoConfiguration).

An advertisement like that includes a “preferred” and “valid” lifetimes - how long they should be used for new connections, and how long they should be allowed for existing connections, respectively. Routers can re-send the advertisements and refresh those times, kinda like DHCP.

It’s not, however, a full DHCP replacement - it doesn’t have acknowledgements, logging, DDNS, or any number of other fancy features.

For raw usefulness getting a set of desktops on an existing network to use IPv6… priceless.

Neighbor Discovery Security

ARP isn’t really secure by design, so it should be no surprise that Neighbor Discovery has similar sorts of issues possible. If you’re the particularly security-conscious sort, you should check out “Secure Neighbor Discovery” (SEND) or “RA-Guard”.

Unique Local Addresses (ULA)

Roughly equivalent to RFC1918 space, but (probably) still unique to a single environment.

It’s really not needed in most cases - you’re normally expected to just use addresses from your service provider.

Prefix Delegation (PD)

This is the means by which large blocks of address space (prefixes!) can be requested by clients. It’s actually a function of DHCPv6!

Prefix Delegation is how your ADSL modem can request a chunk of address-space from your ISP, and get back a /56 (or whatever is configured for your account) to further subdivide.

Hosts

Desktops

If you want to get a set of desktop PCs on a LAN using IPv6, you’re in luck.

Once your router has an IPv6 range allocated to it (by Prefix Delegation or some other means), then it can send on Router Advertisements to local LAN segments; PCs and other devices will use SLAAC to autoconfigure themselves and use the connectivity.

DHCP (via IPv4) still reigns supreme - handing out DNS and the like - but you nearly instantly have IPv6 connectivity on any modern system.

It’s worth noting that most current-generation OSes will actually end up randomly creating and cycling through IPv6 addresses on a regular basis; this is due to “Privacy Extensions”, an extension of SLAAC’s normally determininstic address selection that uses random numbers instead.

Desktops, the future!

DHCPv6 will, one day, be the norm. But not today, so I’ll leave that for another day :)

Servers

SLAAC is a really bad idea for servers, as it means that changing your host machine (or just a NIC) would change it’s IP! Statically configure the addresses instead.

You can still use Router Advertisements for setting a default gateway, if you want - as mentioned above, there are security concerns, but it can still be pretty convenient.

Services

Common Architecture: Reverse Proxy

One very easy way of making a service IPv6-capable, is by sticking some sort of reverse proxy in front of it. IPv4 traffic can flow as per usual - but when you configure an IPv6 address for the service, the address is actually on a dual-stacked box. This box can then redirect traffic back to the IPv4-only host.

For HTTP, you can set the X-Forwarded-For header with the original IPv6 address (since it’ll be lost otherwise). Make sure your backend application knows to trust the header when (and only when) it comes from the proxy, and make sure it can deal with IPv6 addresses properly!

You’ll also want to make sure that any ACLs or other IP-based restrictions (eg. Geo-IP blocking) take into consideration the proxy’s existence. It’s probably good to set a fair amount of logging on the proxy, too, so you can tie things together when troubleshooting later.

Proxies would work fairly well for:

• HTTP/S
• IMAP/POP3
• DNS, NTP

They won’t work for cases where you actually care a lot about the source - IP address reputation checks for SMTP, rate-limits on web pages, IP-blacklists on forum software, etc.

Anything using bidirectional communications is pretty much out too (eg. FTP, backups), and you also have to note that (like NAT) it risks unbalancing load-balancers if you use IP-based sticky sessions.

Common Architecture: IPv4-only Load Balancer

There’s a couple of different ways to work around an IPv4-only load balancer:

• Make an IPv6 load-balancer in software
• Go directly to one backend
• Round-robin to all backends

I actually think the “direct to backend” solutions aren’t too bad… hacky, but survivable. The vast majority of applications fall back to IPv4 sufficiently well, don’t forget. Make sure to set a low TTL on the DNS, so you can remove it in event of a failure.

Your app sucks

Let’s face it, most sysadmins have to look after any number of custom in-house applications. They’re probably going to need some love. I can’t help you with that.

There’s also amusing side-effects; several software products still truncate IPv6 addresses like “last” outputs in some *nix-based systems, or in logs in database tables. Speaking of databases, what sort of field type are your applications using to store IP addresses? Will it fit an IPv6 address in it? Is it just a 32-bit int? Many databases now support a native “IP address” type - use it if you can.

Rate-limits

While we’re on a tangent, let’s talk about those rate limits too.

• Is it useful to rate-limit a single IPv6 IP? (A /128). Probably not, it’s too easy to change.
• Is it useful to rate-limit a /64? Probably; there’s a good chance the user has multiple /64s at their disposal, but only so many…
• Is it useful to rate-limit a /56? Useful? It’s not even safe - Internode hand out /56s, other ISPs can (and do!) regularly hand out smaller.

Limit the /64 - it’s not perfect, and it’s still not even entirely “fair” in some edge cases, but it’s the best you’ve got.

DNS

DNS is a super awesome and easy service to enable for IPv6, so it’s a great first choice. The protocol has native fallback to secondary servers in case of failure, for both a client accessing a DNS resolver and for a DNS resolver talking to nameservers.

DNS: Aside

You do also end up having a rather large reverse address space to manage DNS for. A /56 has 4722366482869645213696 addresses!!

The only sane solution (other than having no reverse DNS) is procedural DNS generation. Internode do this for their customers (using the “pymds” open source project), optionally delegating it to nameservers of the customer’s choosing. After that, it’s up to you :)

SMTP MTA

Don’t forget to configure your Reverse DNS for SMTP - Yes, people really still do look at this, even when using IPv6.

Connectivity is key; if your MTA software thinks it has IPv6 connectivity then it really needs to work. Check your fallback to IPv4 when IPv6 connectivity is missing - some software doesn’t!

It’s worth noting that people don’t interactively use their mail servers very often, so there’s no implicit verification of connectivity, or firewall validity. Mistakes can go unnoticed for a very long time!

The other common issue running an IPv6-capable MTA is broken [far-end] DNS; entirely too many systems return SERVFAIL rather than an empty NOERROR response, causing extra retries and often making the sending MTA never fall back to IPv4 at all. There’s no real fix here, this is just a heads-up…

SMTP MX

Running an IPv6 MX means a moderate amount of pain right now (though it’s getting quickly better now that GMail runs an IPv6-capable MX).

Running an IPv6-only MX, with an IPv4-capable secondary MX, is not currently feasible - a small number of real-world MTAs break horribly trying to deliver email to a domain using such a setup.

It might be worthwhile using a dual-stack primary MX, and an IPv4-only secondary MX (even if it’s the same IPv4 address!), to encourage semi-broken MTAs to fall back to the secondary MX (and the IPv4 address it carries) if they don’t fall back properly from IPv6 to IPv4 on the primary MX.

Postfix will only (by default) check five addresses from DNS before stopping the delivery attempt; so if Postfix thinks trying over IPv6 is OK, and you have 5+ IPv6 addresses on your MX, *it will never fall back to IPv4 if something goes wrong*. So, don’t have more than 3-4 IPv6 addresses for your MX.

Finally, you still have to deal with broken far-end DNS, because many modern systems will check if a bounce message could be delivered in principle before accepting a message (by checking the sending domain exists and has a valid MX).

It’s worth doing if you have the patience, and things will only get better, but it’s not necessarily going to be the smoothest ride - use your own judgement.

(Internode have done it and are striving forwards with it, because that’s how we roll. I’d like to think that you’re going to do the same! :)

Monitoring

You need to dual-stack your monitoring system, naturally - all those IPv6 services aren’t really in production if they’re not monitored… but, which IP stack will actually get used to do the checks?

Do all of your hosts support all features on both protocols?

I can’t tell you the solution here, because it’s going to be very situation-specific - but there’s a good chance you’ll have to define a lot of your checks twice (once per protocol).

Don’t forget, if you can’t IPv6-enable your existing host (eg. because it’d suddenly make a lot of legacy checks try to use IPv6 and that won’t work) then you can always proxy checks via dual-stacked host.

Opinions

Lots of IPs…

Since your VLANs now have a very large address space, you can put it to some good use.

For example, if you’re mixing different services in the same /64 network, why not use the first few bits to define what sort of service they are, and create yor ACLs around that?

eg., using /80 ACLs:

An IP could be formatted like so - 2001:db8:n:n:apptype:x:x:id - where “apptype” is one of the following:

• Infrastructure = 0000
• Web Server = 1000
• Mail = 2000
• SMTP = 2010
• IMAP = 2020
• Database = 3000

So you could have a firewall rule allowing access to [2001:db8:n:n:1000::/80]:80 from the Internet, making your website public but nothing else. Conveniently, you can spin up further web servers inside that /80 without needing to alter your firewalls.

Technically, you should always pay attention to bit 7 inside the /64, it should always be zero for statically-assigned addresses as it’s the “Unique vs Local interface” bit, used to determine if a given host-part of the address (the last 64 bits of the IP address) is globally unique, or just locally unique. I don’t think anything really uses this right now, but you’ve been warned :)

Server Numbering

If you decide that the suffix of the address is the host number, that’s not a bad idea. BUT!

If:

• Fooserver1 → ::1, and
• Fooserver2 → ::2

Is:

• Fooserver10 → ::10, or
• Fooserver10 → ::A?

It’s important to remember that IPv6 addresses are hexadecimal!

You can also embed IPv4 IPs in to IPv6 addresses - a host on 192.231.203.132 could have an IPv6 address of 2001:db8:1234:1234:192:231:203:132 - but, should it?

It’s also tempting to be clever and use port numbers - ::25 for mail servers, ::53 for DNS servers, and ::80 for web sites - but again, should you? That’s hex… for 37, 83, & 128!

Conclusion

IPv6 isn’t really all that hard, just new. The best way to get familiar with it, is just to get on with using it :)