[11.3] Network traffic VERY VERY SLOW -- System not responding to DNS queries

openSUSE 11.3

Symptoms: all network traffic is VERY VERY slow.

Wireshark shows the following:

1. DNS client outputs a DNS A query and a AAAA query.

2. The DNS A response arrives in a few milliseconds, with the correct IPv4 IP address.

3. Nothing further happens for 5 seconds.

4. Steps 1,2 and 3 repeat.

This 5-second loop repeats 3 times, then the app uses the IP address. In other words, there is a 15-second delay following an initial DNS query before the app can use the IP address.

Yes, I have disabled IPv6 and rebooted the system.

Questions:

  1. Why is the system sending the AAAA request, when IPv6 is disabled?

  2. Does the 15-second delay occur because the system is waiting for a response to the AAAA request, eventually timing out because the AAAA response never materialises?


Ian

Problem solved, thanks to a hint from gropiuskalle in another thread about a slow network.

In /boot/grub/menu.lst, in the first section (Desktop), I added the following to the end of the list of kernel parameters:

**ipv6.disable=1**

This prevents the DNS client from issuing an AAAA query, which was the cause of the very slow network response I reported earlier.

[My understanding from another thread was that this manual fix was necessary in 11.2, but not in 11.3. However, it still seems to be necessary in 11.3, at least for me].


Ian

On 11/30/2010 10:06 AM, IanWade wrote:
>
> Problem solved, thanks to a hint from gropiuskalle in another thread
> about a slow network.
>
> In /boot/grub/menu.lst, in the first section (Desktop), I added the
> following to the end of the list of kernel parameters:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> IPV6.DISABLE=1
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> This prevents the DNS client from issuing an AAAA query, which was the
> cause of the very slow network response I reported earlier.
>
> [My understanding from another thread was that this manual fix was
> necessary in 11.2, but not in 11.3. However, it still seems to be
> necessary in 11.3, at least for me].

The boot-line parameter is needed for any kernel in which ipv6 is built into the
kernel. If it is a module, then it is not needed.

There is a command-line way to tell:


zcat /proc/config.gz | grep IPV6

If IPV6 is enabled, the first line of the output will say “CONFIG_IPV6=X”. If X
is “y”, ipv6 is built in and the YaST part is irrelevant. If it is an “m”, IPV6
is a module and the YaST part is sufficient. If IPV6 is not in your kernel, the
response will be “# CONFIG_IPV6 is not set”.

I am not running a standard kernel in 11.3; however the kernel for 11.4 M4 has
“CONFIG_IPV6=y”. It seems as if all openSUSE kernels need the boot-line parameter.

Understood, but as YaST has an option to let you switch IPv6 off, then it should really switch it off, not just pretend it has switched it off.

[The same goes for dual monitor display settings, which I mentioned in another thread today. YaST lets you extend the desktop over two monitors, but doesn’t retain that setting over a reboot. Very confusing and timewasting].


Ian

On 11/30/2010 01:36 PM, IanWade wrote:

> Understood, but as YaST has an option to let you switch IPv6 off, then
> it should really switch it off, not just pretend it has switched it
> off.

I inquired about why IPV6 was built in on the openSUSE Kernel mailing list. The
reason was boot speed; however Greg Kroah-Hartmann thought the decision had been
reversed. It may be changed by the time 11.4 comes out.

Greg KH also thought that a bug had been filed about YaST not being able to
disable IPV6 when it was built in. Please check on that, and if there is not
one, then enter such a bug report.

This explanation was posted in the Kernel mailing list by Jiri Bohac:

==========================
originally, IPv6 got moved in the kernel in our moblin kernel
branch to speed up booting. We quickly found that having it as a
module in some branches and built-in in other branches caused a
lot of pain (different behaviour when setting various ipv6
sysctl parameters or when disabling it). Thus we decided to unify
all the branches by always building it in.

Also, the bonding module now depends on the ipv6 module, so the
old method of disablling IPv6 by blacklisting the ipv6 module
breaks bonding.

> There are a lot of systems where every DNS lookup has to timeout on the IPV6
> lookup, thus IPV6 must be disabled.

… or the network infrastructure should be fixed, really.

Normally, you should not see any timeouts with IPv6 enabled.
When a host has both AAAA and A records in the DNS, an attempt to
connect to it will(*) try IPv6 first and then fall back to IPv4.
When there is no IPv6 connectivity on the network, the connect(2)
call should return immediately and cause no delay at all. One
common cause of possible timeout delays is badly configured IPv6
router that advertises an IPv6 default route on the local network
without providing the connectivity or sending proper ICMPv6
Destination Unreachable messages.

> When it is a module, the YaST setting suffices, but if built
> in, it must be disabled on the kernel boot line in GRUB.

Another method of disabling IPv6 is sysctl net.ipv6.conf.all.disable_ipv6=1
Also, YaST still has the option to disable IPv6 and does this
by setting this sysctl option in /etc/sysctl.conf

(*) this is an over-simplification the policy can be tuned with
gai.conf and also IPv6 does not always have precedence in some
cases (e.g. 2002::/16 … )

@IanWade: I checked and YaST is setting the option in /etc/sysctl.conf, at least
for 11.4 M4. Please check yours.

Is the router yours? It seems to be the real problem. On my system, it doesn’t
matter whether IPv6 is enabled or disabled. I guess mine is OK.

lwfinger, thanks for the explanation.

Yes, the router is mine. In fact, it is a new one that I only installed a week or so ago. This might explain why I didn’t seem to have this problem before, with the old router. I will investigate further.

However, I still think YaST has to be fixed. If you disable IPv6 in YaST, then IPv6 should be completely disabled – DNS should not issue an AAAA request.


Ian

On 12/01/2010 02:06 AM, IanWade wrote:
> Yes, the router is mine. In fact, it is a new one that I only installed
> a week or so ago. This might explain why I didn’t seem to have this
> problem before, with the old router. I will investigate further.
>
> However, I still think YaST has to be fixed. If you disable IPv6 in
> YaST, then IPv6 should be completely disabled – DNS should not issue an
> AAAA request.

I agree that the disabling of IPv6 in YaST is not correct, and you should file a
bug on that issue. However, your router is NOT OK. Despite it being new, it is
misprogrammed. I would also complain to the manufacturer.

IanWade wrote:

>
> lwfinger;2259913 Wrote:
>> This explanation was posted in the Kernel mailing list by Jiri Bohac:
>>
>> <snip>
>>
>> Is the router yours? It seems to be the real problem. On my system, it
>> doesn’t
>> matter whether IPv6 is enabled or disabled. I guess mine is OK.
>
> lwfinger, thanks for the explanation.
>
> Yes, the router is mine. In fact, it is a new one that I only installed
> a week or so ago. This might explain why I didn’t seem to have this
> problem before, with the old router. I will investigate further.
>
> However, I still think YaST has to be fixed. If you disable IPv6 in
> YaST, then IPv6 should be completely disabled – DNS should not issue an
> AAAA request.

Just to clarify… the RESOLVER (gethost*) should not try to do AAAA
queries. DNS is what DNS is. DNS can hold records of whatever type,
including AAAA entries… even if it doesn’t listen on any IPv6 interface.

In my own experimentation using SLES 11 SP1, I found that name lookups DID
not attempt AAAA queries when IPv6 was disabled in YaST (and rebooted). And
that it DID do AAAA queries when IPv6 was enabled through YaST (and
rebooted). With that said, I WAS strictly looking at apps that do name
lookups, I did NOT try to write my own program to ABSOLUTELY determine this.
(I could be wrong)

The “host” command by default will do both queries (there are options to
force one or the other). The “ping” command does only A lookups, again, you
have to use something like ping6 for AAAA lookups.

I did not try to look at this under openSUSE 11.3 yet… so it may VERY well
be some sort of bug (?) there… not sure. But SLES 11 SEEMS to be ok…

Done: Access Denied


Ian