Page 1 of 3 123 LastLast
Results 1 to 10 of 23

Thread: et0 not (auto) enabled at boot time, no static ip addresses are assigned

  1. #1

    Default et0 not (auto) enabled at boot time, no static ip addresses are assigned

    Hi all,


    I've a very ugly issue: I'm running an OpenSuse Leap 15.0 VPS, for years now. Usually rock solid and stable. A few days ago the hosting company has had multiple issues on the host system, so they had to "hard reboot" my VPS a couple of times. They're using KVM.
    Since these multiple reboots, which I thought shouldn't really be a problem, for whatever reason my network config for the eth0 virtual nic seems to messed, somehow.

    What I've found out already:

    /etc/sysconfig/network/ifcfg-eth0 has STARTMODE='auto'

    Nevertheless journalctl -b shows eth0 as device-not-running, and systemctl status wickedd-nanny.service says

    Code:
    device eth0: call to org.opensuse.Firewall.firewallUp() failed: DBus method call timed out
    eth: failed to bring up device, still continuing
    "continuing" forever, VPS will never get eth0 enabled anymore.

    Connected via VNC to virtual console I've tried three things:

    - started yast, tried to start System/Network Settings: HANGS forever
    - tried ifup eth0: results in device-not-running
    - tried ip link set eth0 up: Works, eth0 is getting enabled. But NO static ip addresses are assigned. Not ipv4, not ipv6.

    Compared /etc/sysconfig/network/ifcfg-eth0 from my VPS and my Leap 15 box @home, except the ip addresses they're the same, so the VPS one is correct.

    Any hints how to narrow down this issue? This is really painful, as there's a postfix instance running on my VPS, and I'm loosing maybe relevant emails because of this

    Regards,
    Michael

  2. #2

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned

    One step further, but still no solution:

    As described before, no static ip addresses are assigned. And as I was getting DBUS timeouts related to firewalld, I've stopped firewalld.

    Voila, ifup eth0 works as expected, etho is now "really" up with assigned ips, both v4 and v6. Restarted firewalld immediately after enabling eth0, of course.

    Whyever, this manual enablement of eth0 does not survive the next reboot, so something seems to have got corrupted in my firewall settings. Unchanged by me for months, promised :-)

    No idea why firewalld might be able to "block" enabling eth0, mystic .....

    Any hints would be great, would this be worth to be tracked in bugzilla?

  3. #3

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned


  4. #4
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,395
    Blog Entries
    2

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned

    First,
    Although this is a networking issue, it's more specifically networking in a virtualized environment so you'd probably get more targeted help in the Virtualization forum.

    You may have to verify that your Cloud Provider has fixed all their problems, that no one else is having a similar problem.
    From what you've posted you seem to suspect the problem is not that the networking service can't connect to any other machines during bootup, it's an internal DBus issue caused by timing.

    Generally speaking your description suggests to me you have an image corruption problem, your bootlog should be more descriptive about what dependency is missing. About your firewalld, it's simply a iptables/ebtables manager, so the actual problem might be related to what firewalld does but not actually firewalld itself.

    The first thing I"d probably try is to go into YaST, and flip your Network management to NM, and then back to Wicked, to see if that resets and fixes your bootup

    To address that kind of issue would probably require a "repair" but how that might be done depends greatly on how your Provider supports images... ie
    - Is the base image provided to you by the Provider or the result of a fairly normal install? If the image is provided by the Provider, you'll have to ask them what their supported procedures are to do a repair, otherwise you can try various well known options...
    - If you have access to a virtual optical disk, point to an openSUSE LEAP 15 DVD and go through the repair process..Remember that this essentially returns your system to the day the DVD was launched, so you'll have to follow up with an update to install all patches and feature updates released since then.
    - run a "zypper dup" relying on your online repos to provide the packages required to re-install your system.

    HTH,
    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  5. #5

    Default Re: eth0 not (auto) enabled at boot time, no static ip addresses are assigned

    TSU, thank you so much for answering in that detail!


    Quote Originally Posted by tsu2 View Post
    You may have to verify that your Cloud Provider has fixed all their problems, that no one else is having a similar problem.
    They will never disclose this :-) But as said: Stopping firewalld immediately enables that eth0 could be enabled, and from then on I can also restart e.g. sshd, and connect. But only until the next reboot..
    Plus: Provider offers in parallel to the VPS itself several "rescue systems" (below more about in context), starting a separate VM with full access to your own VPS's virtual disks, not mounted at first. And all "rescue systems" I've tested have no issue with network connectivity, same ipv4/6 settings.

    So I strongly believe (now) that my issue is an my-VPS-internal-only issue.


    Quote Originally Posted by tsu2 View Post
    your bootlog should be more descriptive about what dependency is missing.
    Trying with journalctl -b and dmesg, I can't really find any error msg or dependency missing msg, except the ones to be expected, like

    Code:
    systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped
    - same msg on my local well working Leap 15 box.

    Quote Originally Posted by tsu2 View Post
    The first thing I"d probably try is to go into YaST, and flip your Network management to NM, and then back to Wicked, to see if that resets and fixes your bootup
    Same idea I had, but unfortunately I can start yast, but not the System / Network module ( /sbin/yast lan ) hangs after started, infinite. Probably same underlying issue


    Quote Originally Posted by tsu2 View Post
    To address that kind of issue would probably require a "repair" but how that might be done depends greatly on how your Provider supports images... ie
    - Is the base image provided to you by the Provider or the result of a fairly normal install? If the image is provided by the Provider, you'll have to ask them what their supported procedures are to do a repair, otherwise you can try various well known options...
    My VPS was started by me from a provider-preinstalled "basic as possible" system. Leap 42.1 that time, as far as I remember the "server only" install option was added to OpenSuse somewhat later. So some sort of custom install, but very minimum. For whatever software I've later added mysef, I've had to resolve much more dependencies than on my Leap box at home. Which is as it should be, for s server, IMHO.

    Distribution upgrades have been made by me all the way, from 42.1 to now Leap 15.0. Smaller issues, but solvable in short time. More zypper dup issues on my box at home :-)

    The provider offers several different rescue systems, probably widely known "SystemRescueCD", a "Debain 9 Live" system. Plus Clonezilla.

    I did very frequent backups by rsnapshot(underlying rsync), triggered from my machine at home, fetching all data (files, along with Percona online backups of all mysql databases). So I have all relevant files before the corruption at hand.
    I also made once, to test, a clonezilla based backup, worked fine, so I also could "take" the VPS, get it to my local Leap box, an run it in qemu.

    But currently I still have the hope that this issue might be related to corrupted config files, caused by the "hard reboots".

    I'm currently trying to compare alle (hopefully) relevant files (/etc/wicked, /etc/sysconfig/network, /etc/firewalld) by size and content between latest backup and current vps.

    Until now no differences found....

  6. #6
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,395
    Blog Entries
    2

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned

    Because you say that you can enable networking by executing an "ifup" I suspect that whatever problem that is happening during boot is resolved by the time you can log in, which suggests to me it's not a problem with a config file but a problem in the system boot parallellism which is a bit of a mystery. From what I understand and conjecture, the boot process has various "waypoints" or critical nodes in the flow where no matter what order the components are loaded up to then, have to be completed before moving beyond that point. Something has happened that networking requirements cannot complete before starting networking, so networking can't start during boot.

    You may have to have an in-depth discussion with your Provider on your Repair options, you need to know best to repair your image. This is not the same as the boot repair disks you describe, which are mainly used to recover when a system won't boot at all.

    You may want to think about rebuilding your system from scratch, and this is an example of why I am an advocate of creating a Build script that memorializes what is installed in your machine and how it's configured, this was asked about in a recent Forum post

    https://forums.opensuse.org/showthre...35#post2894435

    You can also take a look at your available backups, whether you can restore your system without also restoring your problems, in particular how usable are your backups that were made before your Provider experienced their disaster?

    You should also ask your Provider if under the circumstances they can provide you with a VPS to recover or rebuild your system without destroying your existing VPS, then when you have a replacement to your satisfaction you can simply "flip a switch" to replace old with new.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  7. #7

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned

    SOLVED!


    Documented in my bug report here: https://bugzilla.opensuse.org/show_bug.cgi?id=1126094.

    Reason was a "stale" lock file, left over by a process called ebtables, when my VPS "crashed" the last time.

    File is /var/lib/ebtables/lock.
    Deleted.
    Done.
    Sometimes the most painful errors have such small reasons, a pity that they're that hard to find


    @TSU: Thank you very, very much, again!

  8. #8
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,395
    Blog Entries
    2

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned

    Congrats on finding your problem.

    Although I don't know for sure in your case,
    lck files are typically found in virtual machines, not installations on metal and are more typical of memory backing files which have to be deleted before a virtual machine which had a dirty shutdown can be booted back up again.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  9. #9
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,395
    Blog Entries
    2

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned

    After reading your referenced RHEL Bug report and thinking about it a bit...
    I wonder if this lock method and file is because of firewalld's runtime vs permanent settings modes so is a fundamental flaw in firewalld design.

    Someone may want to evaluate whether this could be a show-stopper moving forward or something sufficiently addressed by better documentation. It does look like they tried to address the problem but still assumes that systemd would "behave properly." I'd also suspect that they probably implemented their flock solution incorrectly, it's probably invoked as a normal method instead of as a recovery, ie a default behavior when a lock is encountered... But of course that should be evaluated based on the real reason the lock is implemented (because I'm guessing on that point) so as not to possibly open a security hole.

    Speculating,
    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  10. #10
    Join Date
    Jan 2017
    Location
    Nürnberg, Germany
    Posts
    189

    Default Re: et0 not (auto) enabled at boot time, no static ip addresses are assigned

    A tad late to the action in this thread, I’d just like to mention a third option here: systemd-networkd.

    As most who install openSUSE, I started off five years ago with wicked by default, and I experienced 5 to 10 seconds of delay while booting into KDE/Plasma. Apparently, wicked and my DSL router misunderstood each other when it came to IPv6 and/or DHCP.

    I then switched to NetworkManager (NM), and boot times improved instantly: NM only needed around 100 milliseconds to bring up eth0 with a DHCP-assigned dynamic IPv4 address. I also tried reserving a static IP address in my DSL router and bypass DHCP, but this didn’t improve things noticeably. My impression was that without DHCP and manually assigned DNS servers, name resolution even seemed a bit more sluggish than with DHCP enabled. Further experiments with IPv6 lead me to the conclusion to leave IPv6 disabled system-wide. My NM config hadn’t changed for over 2 years.

    Two weeks ago then, I tested out systemd-networkd for the first time, motivated by the verbosity of NM messages (and by curiosity; ok, and by the prospect of being able to free up a bit of disk space occupied by NM).
    Here’s what I did:
    • systemctl stop NetworkManager.service # as root
    • systemctl disable NetworkManager.service
    • systemctl mask NetworkManager.service
    • mv -i /etc/sysconfig/network/ifcfg-eth0 /root/off/etc_sysconfig_network_ifcfg-eth0 # back up any previous eth0 configs
    • cp -i /etc/resolv.conf /root/off/etc_resolv.conf # back up DNS stuff for reference
    • vi /etc/systemd/network/50-static.network # create a static entry for eth0, see contents below
    • add to /etc/resolv.conf the nameservers my ISP recommends
    • make sure that any DHCP functionality is disabled in /etc/sysconfig/network/dhcp
    • add the names of my openSUSE Leap 15.0 main rig and of my DSL router to /etc/hosts, including their static ipv4 addresses, very old-school
    • sudo systemctl enable systemd-networkd
    • uninstall NetworkManager and ModemManager (agents, CLI and GUI tools as well as NM widgets)
    • create a new initrd with dracut --hostonly --force --no-compress --omit "img-lib cifs fcoe fcoe-uefi rdma multipath iscsi qemu lvm mdraid dm dmraid cdrom pollcdrom plymouth btrfs wacom convertfs wicked ipv6 mtp-probe" (YMMV, your mileage may vary), just to make sure that no artifacts of wickedd or NM mess up the start of the newly configured system
    • reboot

    Contents of my above mentioned /etc/systemd/network/50-static.network (again, YMMV):
    Code:
    [Match]
    Name=eth0
    
    [Network]
    Address=192.168.1.2/24
    Gateway=192.168.1.1
    Result: systemd now manages to bring up my eth0 in 19ms (for comparison: NM was around 100ms, wicked 5s to 10s), and it works quietly and flawlessly. Together with other optimizations (one single SSD-aligned ext4 partition for everything; MBR instead of GPT; minimal KDM instead of SDDM; exim for postfix; chronyd for ntpd; no Plymouth, no btrfs, no compressed initrd's etc.), my boot times according to systemd-analyze are around 1.5 seconds now; personal best was 1.319s five days ago.

    For completeness, add to that about 2s of BIOS initializations (testing RAM, building hardware tree, doing ACPI stuff) and about a second of KDM+KDE+Plasma startup. Not bad for a five years old Core-i5 rig.

    Conclusion: for minimal static IP configs (usually found in virtualized guest operating systems, but apparently perfectly suited for home/desktop/workstation use as well), I can recommend networkd. Give it a try. Cheers!

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •