Problem with wicked-0.6.39-12.1 IPv6 SIT - default route doesn't install in kernel route table

gt2847c · January 6, 2017, 10:07pm

Leap 42.1 x86_64
Lenovo T61p

I patched my IPv6 gateway device (connection to Hurricane Electric) with the current released version of Wicked (wicked-0.6.31-9.1 -> wicked-0.6.39-12.1). After the reboot, the default route for the SIT interface won’t populate on boot. I can manually add the route with the ip command, but rebooting or cycling the interface with ifdown/ifup does not work. I turned on debug logging for wicked and have done both reboots and ifdown/ifup. No errors or complaints that I was able to locate in those logs related to routes.

Configuration is as follows:

eth0 - IPv4 static (ISP) and IPv6 static (from HE assigned /48 block)
sit1 - IPv6 HE static assigned tunnel address

quagga/zebra running to advertise the routes

ifcfg-sit1:
STARTMODE=‘onboot’
BOOTPROTO=‘static’
TUNNEL=‘sit’
TUNNEL_LOCAL_IPADDR=‘x.x.x.x’
TUNNEL_REMOTE_IPADDR=‘y.y.y.y’
IPADDR=‘2001:470:x:xxx::2/64’
TUNNEL_TTL=‘64’

ifroute-sit1:
default 2001:470:x:xxx::1 - -

Wicked apparently is reading the config and not complaining about it:
Jan 6 11:01:08 sixoverfour wicked[899]: ni_sysconfig_read(/etc/sysconfig/network/ifcfg-sit1)
Jan 6 11:01:08 sixoverfour wicked[899]: ni_suse_read_routes(/etc/sysconfig/network/ifroute-sit1)
Jan 6 11:01:08 sixoverfour wicked[899]: Parsed route: ipv6 ::/0 via 2001:470:x:xxx::1 dev sit1 type unicast table main scope universe protocol boot

Further down is:
Jan 6 11:01:09 sixoverfour wickedd[887]: __ni_rtnl_send_newroute(replace ipv6 ::/0 via 2001:470:x:xxx::1 dev sit1 type unicast table main scope universe protocol boot)
Jan 6 11:01:09 sixoverfour wickedd[887]: sit1: successfully updated existing route ipv6 ::/0 dev sit1#5 type unicast table main scope universe protocol zebra priority 1024
Jan 6 11:01:09 sixoverfour wickedd[887]: sit1: applying routes for ipv6:static lease in state applying: success [after 0m0.0s]

Based on this, and no errors/complaints in the debug logs it would seem that it should have worked. When referring to the actual route table, this is not the case:

ip -6 route show

2001:470:x:xxx::/64 dev sit1 proto kernel metric 256 pref medium
2001:470:xxxx::/64 dev eth0 proto kernel metric 256 pref medium
2001:470:xxxx:1::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:2::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:3::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:4::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:5::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:6::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:7::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:8::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:9::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
2001:470:xxxx:a::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
fda0:xxxx:4465:1::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
fda0:xxxx:4465:2::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
fda0:xxxx:4465:3::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
fda0:xxxx:4465:4::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
fda0:xxxx:4465:5::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
fda0:xxxx:4465:6::/64 via 2001:470:xxxx::2 dev eth0 metric 1024 pref medium
fe80::/64 dev sit1 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium

Not sure why at this point it’s not installing the default IPv6 route in the table, the log doesn’t appear to have any complaints about adding/replacing it.

Routes for the eth0 interface (/etc/sysconfig/network/routes) install OK. Adding the default IPv6 route for the SIT interface to the routes produces the following error when ifdown/ifup on sit1:

wicked: discarding route not matching any interface: ipv6 ::/0 via 2001:470:x:xxx::1 type unicast table main scope universe protocol boot

Anyone have an idea what’s going on? Seems like a bug (I posted it over in bugzilla), but figured I post here too in case anyone else has seen this or something like it.

tsu2 · January 7, 2017, 6:17am

Setting up a tunnel broker is pretty specialized stuff…

First,
It’s hard to tell if there are any errors when you sanitize using very generic placeholders. So, for example everywhere you use xxxx, I wonder if every time it’s the same value or not.

Recommend instead that you post real addresses, but modified for privacy by replacing specific numbers only you know, that gives your addresses more realism. Even so, you may need to be careful to make it clear if you are using any link local (self generated) IPv6 addresses(those can likely be left unmodified since they shouldn’t work for anyone else).

Also, you should post whatever guide you’re following to set up your tunnel.

In theory, for whatever guide you’re following wicked should not be any different than the classical ifup/ifdown so a guide describing using ifup/ifdown should be followed almost exactly.

To address your specific question, typically any modifications you make to your routing table by command line always needs the “persistent” flag for it to survive reboots. If you add the route using YAST, your modification should be automatically persistent.

TSU

deano_ferrari · January 7, 2017, 9:24am

I’d like to refer you to this opensuse mailing list archive where similar behaviour was described for openSUSE 13.2.

Have you tried using the following route table entries:

default via fe80:: dev sit1 metric 1024

OR

default dev sit1 metric 1024 —> route according to manual config
presented on HE.net

I was able to successfully (using HE.net tunneling and a sit tunnel configured
via wicked) ping6 google.com.
I tried pinging 2001:hhh:hhhh:hhh::1 as well, but saw the same Address
Unreachable error you described. I tried
this with a sit tunnel created under wicked, and with one created via ip tunnel. Both showed the same
results.

deano_ferrari · January 7, 2017, 9:58am

Based on this archlinux wiki page: Setting_up_Hurricane_Electric_tunnel see if you can invoke successfully manually like this…

ip tunnel add he-ipv6 mode sit remote **Server_IPv4_Address** local **Client_IPv4_Address** ttl 255

ip link set he-ipv6 up mtu 1480

ip addr add **Client_IPv6_Address** dev he-ipv6

ip -6 route add ::/0 dev he-ipv6

Then check

ip -6 route show

ping6 google.com

gt2847c · January 7, 2017, 4:24pm

Sorry I wasn’t clear with the first post. The tunnel itself works properly and is not the issue I was writing about. The situation appears to be a problem with Wicked not adding the IPv6 default route (from the ifroute-sit1 file) to the kernel table when the tunnel comes up. If I manually add the default route, it works as expected:

sixoverfour:/var/log # ping6 google.comPING google.com(yk-in-x64.1e100.net) 56 data bytes
64 bytes from yk-in-x64.1e100.net: icmp_seq=1 ttl=49 time=37.2 ms
64 bytes from yk-in-x64.1e100.net: icmp_seq=2 ttl=49 time=37.3 ms
64 bytes from yk-in-x64.1e100.net: icmp_seq=3 ttl=49 time=37.3 ms
64 bytes from yk-in-x64.1e100.net: icmp_seq=4 ttl=49 time=37.3 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 37.237/37.306/37.335/0.040 ms

traceroute6 google.com   traceroute to google.com (2607:f8b0:4002:c07::64), 30 hops max, 80 byte packets
 1  gt2847c-1.tunnel.tserv13.ash1.ipv6.he.net (2001:470:x:xxx::1)  30.824 ms  42.435 ms  38.248 ms
 2  ge5-4.core1.ash1.he.net (2001:470:0:90::1)  43.580 ms  44.688 ms  57.750 ms
 3  2001:4860:1:1:0:1b1b:0:35 (2001:4860:1:1:0:1b1b:0:35)  44.433 ms  44.655 ms  44.797 ms
 4  2001:4860::1:4000:c8a3 (2001:4860::1:4000:c8a3)  45.116 ms  44.942 ms  44.798 ms
 5  2001:4860::c:4000:da1a (2001:4860::c:4000:da1a)  57.025 ms  56.022 ms 2001:4860::8:0:caf7 (2001:4860::8:0:caf7)  57.255 ms
 6  2001:4860::c:4000:d673 (2001:4860::c:4000:d673)  57.073 ms 2001:4860::c:0:d109 (2001:4860::c:0:d109)  37.465 ms 2001:4860::c:0:d108 (2001:4860::c:0:d108)  33.066 ms
 7  2001:4860::8:0:ce2d (2001:4860::8:0:ce2d)  65.750 ms  64.820 ms 2001:4860::8:0:ce2e (2001:4860::8:0:ce2e)  45.927 ms
 8  2001:4860::2:0:878b (2001:4860::2:0:878b)  47.857 ms  47.526 ms 2001:4860::2:0:878c (2001:4860::2:0:878c)  46.445 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  yk-in-x64.1e100.net (2607:f8b0:4002:c07::64)  46.776 ms  46.938 ms  46.725 ms

I think a bug/feature has cropped up either with Wicked or possibly a kernel update as both were in the pile of patches I applied on Jan 5th. The configuration for the tunnel has been otherwise unchanged since February 2016 when the system was built.

deano_ferrari
I added the route with the link local IPv6 address as documented in the other posting you referred to. That does correct the problem, so I will leave it in place (verified with a reboot). At a rough guess, it would seem there may be a race condition that has cropped up in Wicked about the IPv6 address being assigned and when the route is being added. Given the link local address is immediately assigned when the port comes up, and the global IPv6 static address goes through the duplicate address detection, the lag might be enough for the route to be discarded (albeit silently) for lack of an connected subnet with the next hop. The only times I see the discarding route message in the logs was when I added the default IPv6 route to the /etc/sysconfig/network/routes file which would happen if it attempts to apply that prior to bringing up the sit1 interface.

This system was built as a clean Leap 42.1 after a drive crash on the original 13.1 system encouraged me to upgrade last year. It would appear that the jump between 13.1 and 42.1 skipped over the problem mentioned in the post. As the configuration with the ifroute-sit1 has been working up until this point, it looks like the problem described in your referred post (and this one of mine) had gotten corrected and either a new problem was introduced or the old one wandered back in and wasn’t caught in regression testing. Being a less common scenario as TSU mentioned, they might not have a test for it…

Thanks for the find on that post. My Google-fu was insufficient. I was being a little too specific in my search terms…

deano_ferrari · January 7, 2017, 9:00pm

deano_ferrari
I added the route with the link local IPv6 address as documented in the other posting you referred to. That does correct the problem, so I will leave it in place (verified with a reboot). At a rough guess, it would seem there may be a race condition that has cropped up in Wicked about the IPv6 address being assigned and when the route is being added. Given the link local address is immediately assigned when the port comes up, and the global IPv6 static address goes through the duplicate address detection, the lag might be enough for the route to be discarded (albeit silently) for lack of an connected subnet with the next hop. The only times I see the discarding route message in the logs was when I added the default IPv6 route to the /etc/sysconfig/network/routes file which would happen if it attempts to apply that prior to bringing up the sit1 interface.

[FONT=&amp]This system was built as a clean Leap 42.1 after a drive crash on the original 13.1 system encouraged me to upgrade last year. It would appear that the jump between 13.1 and 42.1 skipped over the problem mentioned in the post. As the configuration with the ifroute-sit1 has been working up until this point, it looks like the problem described in your referred post (and this one of mine) had gotten corrected and either a new problem was introduced or the old one wandered back in and wasn’t caught in regression testing. Being a less common scenario as TSU mentioned, they might not have a test for it…
[/FONT]
[FONT=&amp]Thanks for the find on that post. My Google-fu was insufficient. I was being a little too specific in my search terms…

Yes, that sums it up well. Glad to have been of help with the researching.

Here’s a link to your bug report (for others who may come searching)…

[/FONT]1018636 – wicked-0.6.39-12.1 IPv6 SIT - default route doesn't install in kernel route table

tsu2 · January 7, 2017, 9:34pm

As I described in my prior post, you need to add the persistent flag to make your route survive a reboot.

If you don’t know what I’m describing (which should be documented everywhere including the MAN pages), post the route add command you’re invoking.

(Yes, I see you believe you think your problem is solved but unless you did what I’m describing I’m guessing you’ll be posting here again).

TSU

gt2847c · January 7, 2017, 10:24pm

TSU,

The route is persisted in the ifroute-sit1 config file which is read when Wicked brings up the interface. It survives restarts and ifup/ifdown (I tested this several times to make sure it actually fixed the issue). As I mentioned in my later post, this configuration had been working as-is since February of last year when I built the Leap 42.1 box from the wreckage of my 13.1 box - backups save the day again. The problem cropped up after the lastest run of patching. I still believe this is probably a bug introduced in the latest repo release version of Wicked, but we’ll see what they say on Bugzilla. It can certainly be worked around by using the link local address with the device, but it’s not to my way of thinking an ideal way to do it as I can’t have quagga/zebra advertise that way to the rest of the network. I have to put in a static route in the quagga config and have it redistribute that instead.

On the item of a persistent flag, the way this system is configured, there isn’t one. This system is a text only console system with no GUI, and I generally edit the conf files directly rather than use Yast (nice enough interface, but much faster to edit the files directly than fiddle with the menus). I tend to be rather old-school by habit as I did all my system admin learning back before there was any GUI or menu interfaces (well SGI’s Irix did kinda sort have one) for doing system admin work (dating myself here, but - VMS 5.x, SunOS, DEC Ultrix, etc). So the way I configure the box is in the /etc/sysconfig/network conf files and as such, there isn’t a “persistant flag” when you’re editing the files directly.

Thank you for responding to the post.

tsu2 · January 8, 2017, 5:08am

Maybe I’m misunderstanding what you are doing.

AFAIK adding a route entry by command line in a console needs that “persist” flag.

But, if you write that command into a sysconfig file, then your command will be** reloaded **(technically not persisted) because sysconfig files are read on boot.

Is that what you are doing?

TSU

gt2847c · January 8, 2017, 7:08am

If you’re adding a route via command line like:


# ip -6 route add ::/0 via 2001:470:x:xxx::1

Then you would be correct, that’s not persistent between reboots, that’s only manipulating the kernel route table in memory.

AFAIK adding a route entry by command line in a console needs that “persist” flag.

But, if you write that command into a sysconfig file, then your command will be* reloaded *(technically not persisted) because sysconfig files are read on boot.

The configuration I’m using does not specify a script or equivalent command to add the route like the one above. In fact, none of the command line methods (that I’ve ever seen or used) to change the routing table have a flag that will “persist” a route. The only way to persist a route on UNIX/Linux is to add it to a configuration file through direct edits or through one of the administrative menu driven tools (i.e. Yast) which just makes the changes to the configuration files under the covers. Even in Windows, the -p option for the route command to persist route changes is just editing the registry so that on next boot/wakeup it will get re-read by the network service.

On the sysconfig files, you are partially correct. Typically most of the sysconfig files are used during boot (when systemd or an init script starts them), but may also be used when a service is started/stopped/reloaded or some equivalent event occurs like plugging in a USB network device. The network configuration files are not in the general case scripts, but they are read by scripts (/usr/sbin/ifup, /usr/sbin/ifdown, etc) or the Wicked service daemon. If you edit one of the interface configuration files like /etc/sysconfig/network/ifroute-sit1 (route changes applicable when interface sit1 goes up or down) or the “global” routes file /etc/sysconfig/network/routes, those are applied when Wicked detects a change or an administrator (through the ifup/ifdown commands) changes the interface state.

Wicked does allows for some fairly complicated configuration changes through scripting or executables where you could put things like the ip command for routes, but the standard configuration files typically handle the common usages like changing static routes without scripted commands. As an aside, I worked on a project where we used a script to spin up a DHCP server when a specific USB Wi-Fi dongle was attached and then start SSHd on that interface. When the dongle was detached, it shut DHCP and SSHd down. (It was a digital signage solution that for security reasons we wanted to make sure you couldn’t log in remotely to that device unless our custom dongle was attached) If you’re interested, check the man page on ifcfg(5) details in the section PRE_UP_SCRIPT, POST_UP_SCRIPT, PRE_DOWN_SCRIPT and POST_DOWN_SCRIPT.

The common networking configuration files are:

/etc/sysconfig/network/config – contains global configuration items when the network is started (Wicked service)
/etc/sysconfig/network/ifsysctl – contains global configuration changes to the kernel in-memory variables when networking is started (in the format of /etc/sysctl.conf)
/etc/sysconfig/network/routes – contains static routes that will be installed when networking starts - the next hop has to exist on an attached subnet (man page routes(5) for specifics)
/etc/sysconfig/network/ifcfg-xxx# – contains interface specific configuration details for interface xxx# (eth0 would be ifcfg-eth0)
/etc/sysconfig/network/ifroute-xxx# – contains routes that should be installed when interface xxx# comes up (ifup) or removed when xxx# comes down (ifdown)
/etc/sysconfig/network/ifsysctl-xxx# – contains changes to the kernel in-memory variables that should be applied when interface xxx# comes up

deano_ferrari · January 8, 2017, 7:18am

I think this is the case. If you read from the beginning, the issue appears to lie with wicked and perhaps some sort of issue in which the tunnel is not set up when the tunnel route is attempted. The persistent tunnel route added via ifroute-sit1 is not applied as expected. The mailing list thread I linked to describes the same regression behaviour.

But, if you write that command into a sysconfig file, then your command will be** reloaded **(technically not persisted) because sysconfig files are read on boot.

That is all described well in ‘man routes’ and the OP is well aware of that (based on their dialogue in this thread), but wicked doesn’t appear to handle this properly with respect to the tunnel. It is working manually. I had to study the opening post a number of times (along with some googling) before I understood the apparent problem at hand.

gt2847c · January 8, 2017, 5:49pm

I had to study the opening post a number of times (along with some googling) before I understood the apparent problem at hand.

Sorry about that. It was one of those situations where I knew what I was trying to say, but didn’t state it clearly or with enough detail. I’d had my head in the problem for long enough that it made sense to me, but not to someone coming at it cold. I broke one of the key rules around troubleshooting of assuming that everyone knew what I was thinking without actually stating it. Mea culpa.