Matthew Marlowe's Blog

Problem is caused by someone or some committee deciding that "gosh darn it! we can make ipv6 adoption go much faster if we bug the hell out of the ipv4 only network admins". 


    So they did......



On latest versions of RHEL, not only do you get ipv6 enabled by default, all dns requests go to ipv6 first, even if think you have told server it should process ipv4 only  That means that if a particular domain has a large set of nameservers, it is possible in certain circumstances for your critical app which is trying to get a simple ipv4 ip will end up waiting while each nameserver is checked one by one for an ipv6 response.....which can take a really really looooooooong time.




Step 1: verifying what is going on -- use dig to query the nameservers directly for AAAA responses (if all the nameservers are responding refused), you got a candidate.

example: dig api-read.facebook.com/AAAA/IN @glb01.sf2p.tfbnw.net




Step 2: use "getent ahostsv4 hostname" and "getent ahostsv6 hostname" to confirm it is the ipv6 queries that are causing the problem.

You may also see something like the following in named (even though you have forwarding disabled):

named[2837]: FORMERR resolving 'api-read.facebook.com/AAAA/IN': 69.63.176.101#53




Step 3: redhat does provide a few solutions here (thankfully), but it is not as straightforward as it should be.

I assume you already have NETWORKING_IPV6=no in /etc/sysconfig/network.

You may even have been smart and put the following /etc/modprobe.conf:

alias net-pf-10 off
alias ipv6 off

And, of course, you've turned off all ipv6 daemons including iptables6.

No, that is not enough, it is very easy for ipv6 to be forced to load, especially in RHEL 5.5.  You probably also need to create the file /etc/sysconfig/disable-ipv6 with the contents:

options ipv6 disable=1




step 4: pretend you're running a windows box that needs its daily reboot, go ahead.

When system comes back up, do lsmod  | grep ipv6 ..and the ipv6 module should not be loaded.

Run getent ahosts hostname and see ipv4 only results come up very fast.



If you're still seeing problems, review lsmod to see what is forcing ipv6 to load and you may need to perform magic on your dns servers too -- such as adding to /etc/named.conf:

listen-on-v6 { none; };

Adding "-4" to the options field in /etc/sysconfig/named.

Restart named.

Performing steps above to kill ipv6 completely  on the local resolving dns servers.

Note: RedHat has their own technote about the problem here at:  http://kbase.redhat.com/faq/docs/DOC-17904 . 

I suspect this is more specific to RHEL 5.4 than 5.5    RHEL 5.5 does seem to do the right thing when you go through the numerous hoops to disable the ipv6 module.

The primary authoritive DNS nameserver for our client domains is:

     thepostman.deploylinux.net

This is because unlike everyone else, I thought thepostman was a good movie.

Our secondary nameservers are:

     consul.deploylinux.net

     centurion.deploylinux.net

     ns5.dnsmadeeasy.com

     ns6.dnsmadeeasy.com

     ns7.dnsmadeeasy.com
   
Thepostman, consul, and centurion are located within our San Diego datacenter and also provide responses to recursive dns queries made from virtual machines.

The three dnsmadeeasy name servers are geographically distributed and built to handle significant DoS attacks.     The combination of both local and distributed servers allows us to maximize reliability w/o depending on any single platform.

To migrate a domain to DeployLinux hosted DNS, clients should:
     a) first open a ticket within our support portal at https://support.deploylinux.net/
     b) DeployLinux techs will verify that a new zone file is created for your domain and that any prior records are transfered.  If you desire, we may also migrate your domain to our domain registry and handle all future domain renewals on your behalf.
     c) Update their domain register to point domains to the nameservers above.

Future changes to DNS should also be handled by opening up support tickets.

DeployLinux does also provide domain registration services at a flat rate of $15/yr/domain.  We realize this is a little more expensive than $7.99 at godaddy/etc, but in our experience the cheaper domain registration vendors tend to have reliability issues which result in at least a few hours/yr of domain downtime.    They may also have limits of how popular your site can get and how many dns responses they will process.   The $15/yr flat rate allows us to host the largest sites w/o any additional charges and with personal support/service.

Our hosting clients typically receive a limited number of free domain registrations and DNS hosting w/ each VM.  Note that information above is specific to .com domains.    Other domain extensions may have different handling and rates.