NB: this is actually on Red Hat, but I suspect it is a generic Linux feature. I’ve been using S.u.S.E. since 5.1, so I’m asking here first.
We have several linux boxes with two ethernet cards. The eth0’s are all configured with unique IP addresses, and are part of a real subnet. The eth1’s are all configured with the same out-of-range IP address (1.1.2.5). Only one of these eth1’s is supposed to be connected to the network at a time.
What we have discovered is that bringing up a single eth1 while physically connected to the LAN results in IP address conflicts with the disconnected eth1’s. The connected eth0’s are responding to arps for their eth1 IP addresses.
IOW, let’s say hostone has eth0 IP address 192.168.0.11 connected to the ethernet LAN with MAC address ab.cd.ef.11. It also has eth1 IP address 1.1.2.5 and MAC address fe.dc.ba.51 which does not have a network cable attached.
hosttwo has eth0 IP address 192.168.0.12 connected (or not) to the ethernet LAN and MAC address ab.cd.ef.12. In /etc/sysconfig/network/ifcfg-eth1, eth1 is assigned IP address 1.1.2.5 and has MAC address fe.dc.ba.52 which does have a network cable attached.
When the network on hosttwo is started at boot time eth1 sends out a Who has' broadcast ARP request from MAC address fe.dc.ba.52. hostone replies directly to fe.dc.ba.52 from ab.cd.ef.11 (eth0) with a Gratuitous ARP for 1.1.2.5 (reply)’.
What is actually happening here? Is this a configurable behavior?
ARP caches do time out. So it may be the case that the entry was still used/valid less then the time out minutes ago. IIRC the time out is something in the order of 15-30 minutes. May be that is configurable, but I do not know how. I also think that lowering this value will have a negative impact on normal LAN performance. There is a possibility in arp to send a package that an entry should be dropped, but I do not know if all implementations do react on this. See: Address Resolution Protocol - Wikipedia, the free encyclopedia
talks about gratuitous ARP replies. Maybe hostone decided to tell everybody that it has the address 1.1.2.5 to help detect clashes (or potential clashes).
Is there a reason you can’t disable eth1 on all the hosts that are inactive? This seems a better solution than physically unplugging the cable.
VERSIONS
The struct arpreq changed in Linux 2.0 to include the arp_dev member and the ioctl numbers changed at the same time. Support for the old ioctls was dropped in Linux 2.2.
Support for proxy arp entries for networks (netmask not equal 0xffffffff) was dropped in Linux 2.2. It is replaced by automatic proxy arp setup by the kernel for all reachable hosts on other interfaces (when forwarding and proxy arp is enabled for the interface).
The neigh/* sysctls did not exist before Linux 2.2.
I’m not really sure what “when forwarding and proxy arp is enabled for the interface” really means. Where can that feature be feature enabled? It seems like a runtime configuration, not a compile time option.
I know that “IP forwarding” is something that can (and for security reasons should) be switched of in the kernel. I have done some searching through YaST > System > Sysconfig editor, but I can not find it.
I also tried to find somehing usefull inside /proc/, but in vain. Maybe a bit of googling will help.
122 52.994209 IntelCor_59:1c:a5 Broadcast ARP Who has 1.1.2.5? Tell 0.0.0.0
The responses are directed specifically to the MAC address IntelCor_59:1c:a5.
123 52.994295 Standard_5d:1a:5d IntelCor_59:1c:a5 ARP Gratuitous ARP for 1.1.2.5 (Reply)
124 52.994318 Standard_54:17:5d IntelCor_59:1c:a5 ARP Gratuitous ARP for 1.1.2.5 (Reply)
125 52.994319 Standard_5d:1e:ff IntelCor_59:1c:a5 ARP Gratuitous ARP for 1.1.2.5 (Reply)
I’m confused about the “Gratuitous” adjective. I get that even when the APR response comes directly from the NIC (eth1) with the requested IP address. According to the wireshark wiki, “A gratuitous ARP reply is a reply to which no request has been made.” That seems inconsistent with the situation. A request was made.
I inherited the configuration. The configuration is not (directly) under my control. Even if we do decide to go with that option, I still want to understand what’s going on.
Maybe it’s gratuitous because eth0 is on a different subnet and that subnet normally would not care about other subnets. It looks like hostone decided to tell everyone that it has that IP address, albeit on a different interface. Sorry, your guess is as good as mine why the networking stack behaves that way. Maybe there’s a kernel control to suppress this behaviour, dunno.
That’s what I suspected. So I tested and found that I get the same thing even if both eth1’s are connected to the LAN. My senior colleague suggested that it’s a flaw in WireShark. Perhaps I should ask on one of their forums.
But that is a side issue. I read the ARP RFP, and it provided no insight into the original question.
I now tested this with openSUSE 11.2 and discovered the gratuitous arp no longer occurs. It was apparently due to a spurious target IP address in the reply. That doesn’t happen with OS 11.2.
I did some more digging, and it seems that arp filter is supposed to prevent this behavior.
"arp_filter - BOOLEAN
1 - Allows you to have multiple network interfaces on the same
subnet, and have the ARPs for each interface be answered
based on whether or not the kernel would route a packet from
the ARP’d IP out that interface (therefore you must use source
based routing for this to work). In other words it allows control
of which cards (usually 1) will respond to an arp request.
0 - (default) The kernel can respond to arp requests with addresses
from other interfaces. This may seem wrong but it usually makes
sense, because it increases the chance of successful communication.
IP addresses are owned by the complete host on Linux, not by
particular interfaces. Only for more complex setups like load-
balancing, does this behaviour cause problems.
arp_filter for the interface will be enabled if at least one of
conf/{all,interface}/arp_filter is set to TRUE,
it will be disabled otherwise
It does NOT prevent eth0 from answering for the ip address assigned to eth1. Perhaps there is something I need to configure with “source routing”, but I am not sure what nor how.