Problems with nfs server after upgrade to Leap 15.0

gostal · August 7, 2019, 5:13pm

Hi,

Leap 42.3 used to run my nfs server without problem but ever since the upgrade this is broken. After boot I get this:

sudo systemctl status nfsserver
● nfsserver.service - Alias for NFS server
   Loaded: loaded (/usr/lib/systemd/system/nfsserver.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2019-08-06 18:29:46 CEST; 22h ago
 Main PID: 1023 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfsserver.service

Aug 06 18:29:45 k2003734 systemd[1]: Starting Alias for NFS server...
Aug 06 18:29:46 k2003734 systemd[1]: Started Alias for NFS server.
gostal@k2003734:~> showmount -e
Export list for k2003734.win.foi.se:

and restarting it makes no difference. However, if I do first:

sudo systemctl enable nfsserver

then I get:


● nfsserver.service - Alias for NFS server
   Loaded: loaded (/usr/lib/systemd/system/nfsserver.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2019-08-07 16:43:57 CEST; 10min ago
  Process: 23444 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 23444 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfsserver.service

Aug 07 16:43:57 k2003734.win.foi.se systemd[1]: Starting Alias for NFS server...
Aug 07 16:43:57 k2003734.win.foi.se systemd[1]: Started Alias for NFS server.
gostal@k2003734:~> showmount -e
Export list for k2003734.win.foi.se:
/usr/local   ki003685.win.foi.se
/disk2       ki003685.win.foi.se
/opt         ki003685.win.foi.se
/home/gostal ki003685.win.foi.se

so now my shares are exported. There are other symptoms: using the nfs-server module in yast the firewall part says:

Some firewalld services are not available: 
- nfs-kernel-server (Not available.)

So I fire up the yast firewall module which then says:

Current status: stopped

although Start During System Boot is marked.
Starting the firewall makes no change to the nfs-server module which still complains that nfs-kernel-server is not available. The package is installed, however,

sudo zypper se -i nfs-kernel-server
[sudo] password for gostal: 
Loading repository data...
Warning: Repository 'update-nonoss' appears to be outdated. Consider using a different mirror or server.
Reading installed packages...

S | Name              | Summary                           | Type   
--+-------------------+-----------------------------------+--------
i | nfs-kernel-server | Support Utilities for Kernel nfsd | package

What is missing in all this? Please help!

There is a Tumbleweed thread with like symptoms but unfortunately not concluded.

Cheers,
gostal

dcurtisfra · August 7, 2019, 7:06pm

@gostal:

Is the systemd “nfs-server.service” enabled?

The systemd “nfsserver.service” is an alias for the “nfs-server.service” …

The status of the “nfs-server.service” should be like this:


 # systemctl status nfs-server.service nfsserver.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/nfs-server.service.d
           └─nfsserver.conf, options.conf
        /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Wed 2019-08-07 09:16:10 CEST; 9h ago
  Process: 2300 ExecStart=/usr/sbin/rpc.nfsd $NFSD_OPTIONS (code=exited, status=0/SUCCESS)
  Process: 2250 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 2300 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfs-server.service

Aug 07 09:16:10 xxx systemd[1]: Starting NFS server and services...
Aug 07 09:16:10 xxx systemd[1]: Started NFS server and services.
 #

Is the systemd “firewalld.service” enabled?

The status should be as follows:


 # systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-08-07 09:15:58 CEST; 9h ago
     Docs: man:firewalld(1)
 Main PID: 1083 (firewalld)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/firewalld.service
           └─1083 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid

Aug 07 09:15:57 xxx systemd[1]: Starting firewalld - dynamic firewall daemon...
Aug 07 09:15:58 xxx systemd[1]: Started firewalld - dynamic firewall daemon.
 #

The following Firewall daemon packages should be installed:

firewalld, firewalld-lang, python3-firewall, yast2-firewall, firewall-config, firewall-macros.

The previous SuSE Firewall package should have been removed: SuSEfirewall2.

gostal · August 8, 2019, 11:32am

@dcurtisfra: Thanks for responding!

Packages: firewall-config was not installed. SuSEfirewall2 was still installed. Fixed those.

Rebooted

No, nfs-server.service is not enabled. I get this:


systemctl status nfs-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/nfs-server.service.d
           └─nfsserver.conf, options.conf
   Active: active (exited) since Thu 2019-08-08 11:07:51 CEST; 6min ago
 Main PID: 1284 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfs-server.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Yes, firewalld is enabled:


systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-08-08 11:07:51 CEST; 6min ago
     Docs: man:firewalld(1)
 Main PID: 972 (firewalld)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/firewalld.service
           └─972 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Still the yast2-nfs-server module says that nfs-kernel-server is not available.

I then enabled nfs-server:


sudo systemctl enable nfs-server
[sudo] password for gostal: 
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service → /usr/lib/systemd/system/nfs-server.service.

Perhaps now it will start enabled. Update after next reboot.

Cheers,
gostal

gostal · August 8, 2019, 11:54am

Yes, now nfs-server is enabled after boot but still my exports do not show until after restart of nfs-server:


systemctl status nfs-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/nfs-server.service.d
           └─nfsserver.conf, options.conf
   Active: active (exited) since Thu 2019-08-08 11:39:02 CEST; 2min 41s ago
 Main PID: 1339 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfs-server.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
gostal@k2003734:~> showmount -e
Export list for k2003734.win.foi.se:
gostal@k2003734:~> sudo systemctl restart nfs-server
[sudo] password for gostal: 
gostal@k2003734:~> showmount -e
Export list for k2003734.win.foi.se:
/usr/local   ki003685.win.foi.se
/disk2       ki003685.win.foi.se
/opt         ki003685.win.foi.se
/home/gostal ki003685.win.foi.se

And still:

nfs-kernel-server (Not available.)

from yast2-nfs-server module.

Cheers,
gostal

deano_ferrari · August 8, 2019, 12:07pm

I’m speculating that this might happen if the network is not active when nfs-server is started perhaps? Which network management framework are you using (NetworkManager or wicked)?

gostal · August 8, 2019, 4:15pm

Judging from systemctl status output NetworkManager is used:


~> systemctl status wickedd
● wickedd.service - wicked network management service daemon
   Loaded: loaded (/usr/lib/systemd/system/wickedd.service; indirect; vendor preset: disabled)
   Active: inactive (dead)
~> systemctl status NetworkManager
● NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/NetworkManager.service.d
           └─NetworkManager-ovs.conf
   Active: active (running) since Thu 2019-08-08 11:39:02 CEST; 4h 31min ago
     Docs: man:NetworkManager(8)
 Main PID: 1252 (NetworkManager)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/NetworkManager.service
           ├─1252 /usr/sbin/NetworkManager --no-daemon
           └─1890 /sbin/dhclient -d -q -sf /usr/lib/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid ->

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

What file is determining the order in which services are loaded?

/gostal

gostal · August 8, 2019, 5:02pm

Yes, nfs-server starting before network is up seems to be the problem. Running now systemctl status with sudo shows the journal and the PID. Although the main PID of NetworkManager is 1177 the PID of dhcp-client is 1819 which is larger than the PID of nfs-server which is 1251. Also the time stamps of the journal entries are later for NetworkManager than for nfs-server:


sudo systemctl status nfs-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/nfs-server.service.d
           └─nfsserver.conf, options.conf
   Active: active (exited) since Thu 2019-08-08 16:37:01 CEST; 8min ago
 Main PID: 1251 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfs-server.service

Aug 08 16:37:01 k2003734 systemd[1]: Starting NFS server and services...
Aug 08 16:37:01 k2003734 exportfs[1245]: exportfs: Failed to resolve ki003685
Aug 08 16:37:01 k2003734 exportfs[1245]: exportfs: Failed to resolve ki003685
Aug 08 16:37:01 k2003734 exportfs[1245]: exportfs: Failed to resolve ki003685
Aug 08 16:37:01 k2003734 exportfs[1245]: exportfs: Failed to resolve ki003685
Aug 08 16:37:01 k2003734 exportfs[1245]: exportfs: Failed to resolve ki003685
Aug 08 16:37:01 k2003734 exportfs[1245]: exportfs: Failed to resolve ki003685
Aug 08 16:37:01 k2003734 exportfs[1245]: exportfs: Failed to resolve ki003685
Aug 08 16:37:01 k2003734 systemd[1]: Started NFS server and services.
~> sudo systemctl status NetworkManager
● NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/NetworkManager.service.d
           └─NetworkManager-ovs.conf
   Active: active (running) since Thu 2019-08-08 16:37:01 CEST; 18min ago
     Docs: man:NetworkManager(8)
 Main PID: 1177 (NetworkManager)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/NetworkManager.service
           ├─1177 /usr/sbin/NetworkManager --no-daemon
           └─1819 /sbin/dhclient -d -q -sf /usr/lib/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-f11eadde-66ce-3934-bfd4-359f9e24a217-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0

Aug 08 16:37:06 k2003734 NetworkManager[1177]: <info>  [1565275026.7457] device (eth0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Aug 08 16:37:06 k2003734 NetworkManager[1177]: <info>  [1565275026.7462] device (eth0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Aug 08 16:37:06 k2003734 NetworkManager[1177]: <info>  [1565275026.7464] device (eth0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Aug 08 16:37:06 k2003734 NetworkManager[1177]: <info>  [1565275026.7465] manager: NetworkManager state is now CONNECTED_LOCAL
Aug 08 16:37:06 k2003734 NetworkManager[1177]: <info>  [1565275026.7548] manager: NetworkManager state is now CONNECTED_SITE
Aug 08 16:37:06 k2003734 NetworkManager[1177]: <info>  [1565275026.7549] policy: set 'Wired connection 1' (eth0) as default for IPv4 routing and DNS
Aug 08 16:37:06 k2003734.win.foi.se NetworkManager[1177]: <info>  [1565275026.7549] policy: set-hostname: set hostname to 'k2003734.win.foi.se' (from system configuration)
Aug 08 16:37:06 k2003734.win.foi.se NetworkManager[1177]: <info>  [1565275026.8663] device (eth0): Activation: successful, device activated.
Aug 08 16:37:06 k2003734.win.foi.se NetworkManager[1177]: <info>  [1565275026.8669] manager: startup complete

How do I set this right?

gostal · August 8, 2019, 5:37pm

The firewall is also blocking my shares from the other computer. I stopped it and my shares showed up on the other computer. Then I did:


sudo firewall-cmd --add-service=nfs --permanent
FirewallD is not running
gostal@k2003734:~> sudo systemctl start firewalld
gostal@k2003734:~> sudo firewall-cmd --add-service=nfs --permanent
Warning: ALREADY_ENABLED: nfs
success
gostal@k2003734:~> sudo firewall-cmd --reload
success

and the shares still show on the other computer but I suspect this won’t survive reboot. I suspect this has to do with the nfs-kernel-server not available in yast2-nfs-server module but what shall I do about it?

No, it doesn’t even survive stopping and restarting nfs-server and then the only way to make the shares available is to stop firewalld and then it doesn’t matter if the firewall is running or not. How do I fix these things?

dcurtisfra · August 8, 2019, 7:01pm

@gostal:

For the case of a normal private “at home” LAN or WLAN, behind a DSL-Router with a built-in Firewall, the default Firewall zone for the Ethernet and WLAN interfaces can be “trusted” – if your LAN or WLAN interfaces have a direct connection to the “real” Internet, you’ll have to come up to speed with the Firewall zone settings …


 # firewall-cmd --get-default-zone
trusted
 #

Currently, with Leap 15.1, don’t try this with a “normal” user – ‘sudo’ is currently broken but, it’ll be repaired soon …

gostal · August 9, 2019, 11:18am

So I checked and got “public” and changed to “trusted”


firewall-cmd --set-default-zone trusted

which I take it means that everything is allowed so now the shares show up on the other machine. Thanks! I am a little confused though. The yast2-firewall module says that nfs is allowed in “public” so shouldn’t it have worked anyway and is not this a consequence of my doing:


sudo firewall-cmd --add-service=nfs --permanent
sudo firewall-cmd --reload

since the default was set to public? I guess I do have some speeding up to do regarding the zone settings. BTW I had a look at the zone setting “work” as this is a work computer and “work” has the same services allowed as “public” except for nfs which I suspect I added. It’s also confusing that the “trusted” zone shows no allowed services at all or perhaps it’s serves no point to list every known service since that would imply a limitation albeit mild. Here we don’t do dhcpv6 but shouldn’t dhcp-client be allowed, if there is such a thing or is that implied by dhcpv6-client? More speed, huh? I also got the idea that it’s better to have “default” as “public” so I changed that back and explicitly set the eth0 zone to “trusted”.

Also I would really like to know how to set things so that nfs-server doesn’t start until NetworkManager is finished setting up the NIC using dhcp. See the previous post.

Cheers,
gostal

deano_ferrari · August 9, 2019, 11:57am

Some users seem to be impacted by the timing of network connectivity… For example…
https://bugzilla.redhat.com/show_bug.cgi?id=1419351

How reproducible:

every time

Steps to Reproduce:

enable nfs-server to start on boot. Add exports to /etc/exports. Using NetworkManager to start wired network via dhcp. Have NetworkManager-wait-online.service enabled.

reboot

Actual results:

exports are not exported

Expected results:

exports will be exported

Additional info:

Note error messages above: nfs server is trying to do export even before local host name is set from dhcp using systemd-hostnamed.service.

Adding

After=network-online.target

to nfs-server.service solves the problem.

A custom service unit file (/etc/systemd/system/nfs-server.service) could be created from /usr/lib/systemd/system/nfs-server.service, such that this directive is added to make sure that this target is reached before starting the nfs-server.service.

Since you are using NetworkManager, it is a good idea to enable the ‘NetworkManager-wait-online.service’ as well, so that network.target is reached after this. Others feel free to chime in here to add or correct my thinking about this.

More detail here.
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

dcurtisfra · August 9, 2019, 12:18pm

The explanation of the default Firewalld zones is here: <https://firewalld.org/documentation/man-pages/firewalld.zones.html>.

Strange – “apropos firewall” doesn’t find anything but, “man firewall*<Tab>*
” finds “man firewalld.zones” – which has the same information as in the URL above …

gostal · August 12, 2019, 5:18pm

@deano_ferrari: Worked like a charm! Thanks!

NetworkManager -> /sbin/dhcp … PID 1672
nfs-server.service Main PID 1800

firewalld config read-up will have to wait till tomorrow.

Cheers,
gostal

deano_ferrari · August 12, 2019, 8:00pm

Fantastic!

gostal · August 13, 2019, 9:56am

To be quite clear I didn’t put any custom nfs-server.service under /etc/systemd/system/ but left the symlink (/etc/systemd/system/multi-user.target.wants/nfs-server.service) untouched and modified the file it linked under /usr/lib/systemd/system/. I wasn’t quite sure how the alias nfsserver.service would react if I replaced the symlink. I realise that the modified file may be written over in future updates so I made copies of both the original and the modified version as these won’t get written over.

Perhaps my fears regarding nfsserver.sevice are ungrounded and would it in that case be better to replace the symlink with the custom file i.e. would a future update rather than replacing that file put an .rpmnew-file there instead? Or would a custom nfs-server.service file under /etc/systemd/system/ take precedence over the symlink under /etc/systemd/system/multi-user.target.wants/?

Cheers,
gostal

deano_ferrari · August 13, 2019, 10:37am

I would have done it as I already suggested. Once the /etc/systemd/system/nfs-server.service exists, then

sudo systemctl disable nfs-server

sudo systemctl enable nfs-server

is sufficient to get the customized (system administrator) service symlinked. It over-rides the original .service file in the /usr/lib/systemd/system/ directory.

gostal · August 13, 2019, 12:03pm

Yes, and much better since no update process will delete /etc/systemd/system/nfs-server.service!

I’ll move the custom nfs-server.service to /etc/systemd/system/ and redirect the symlink. The alias will be none the wiser since they both are symlinked in /etc/systemd/system/multi-user.target.wants and as you say, the file in /usr/lib/systemd/system/ invisible to system boot. I suppose I’ll have to watch out for any .rpmnew -files in the context, though.

Cheers,
gostal

gostal · August 13, 2019, 4:17pm

This firewall is really hard to cope with. I read about the zones in man firewalld.zones and about configuration runtime and permanent in man firewall-cmd but I can’t get to grips with it. The man pages conform pretty much to my gut feeling but still the behaviour is really strange. Now I have:

showmount -e
Export list for k2003734.win.foi.se:
/usr/local   ki003685.win.foi.se
/disk2       ki003685.win.foi.se
/opt         ki003685.win.foi.se
/home/gostal ki003685.win.foi.se

so this is working as expected. Now I also have:

sudo cat /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='dhcp'
DHCLIENT_SET_DEFAULT_ROUTE='yes'
STARTMODE='auto'
ZONE=work

and

sudo firewall-cmd --permanent --zone=work --list-services  
ssh dhcpv6-client nfs

and I also have

sudo firewall-cmd --permanent --zone=work --list-interfaces  
eth0

but still after reboot I get this:

sudo firewall-cmd --get-active-zones
public
  interfaces: eth0

To me this is contradictory and since I removed nfs from public:

sudo firewall-cmd --permanent --zone=public --list-services  
ssh dhcpv6-client

my exports don’t show on the other computer.

And then I did:

sudo firewall-cmd --permanent --zone=public --remove-interface=eth0  
Warning: NOT_ENABLED: eth0
success

NOT_ENABLED??? How come? Just now firewall-cmd reported that eth0 is in public!

So now I do:

sudo firewall-cmd --reload  
success
sudo firewall-cmd --get-active-zones
work
  interfaces: eth0

but still my exports don’t show on the other computer. And I really shouldn’t have to do that after a reboot.
I just don’t get this. What am I doing wrong?

Then I do:

sudo systemctl restart firewalld.service
[sudo] password for gostal: 
sudo firewall-cmd --get-active-zones
public
  interfaces: eth0

Back to public again??

I seems that systemctl ignores whatever configuration changes I have made to the firewall. That explains why things get screwed up after reboot. How do I propagate the firewall config changes to systemctl?

Cheers,
gostal

deano_ferrari · August 13, 2019, 9:55pm

After applying any run-time changes to firewalld you need to make them permanent, otherwise a restart of the firewall will reload the original configuration.

https://firewalld.org/documentation/man-pages/firewall-cmd

gostal · August 14, 2019, 10:13am

I know and according to man firewall-cmd the option --permanent sets, gets etc. what should happen when firewalld is started/restarted i.e. it should conform to the run-time situation after start/restart before any run-time changes have been made. This does not happen and there is something fishy going on or I still haven’t got it!! Consider the following consequtive sequence of commands (I put in the comments afterwards):

sudo firewall-cmd --get-active-zones
public
  interfaces: eth0
#
# This says that  run-time zone of eth0 is public ...
#
gostal@k2003734:/etc> sudo firewall-cmd --get-zone-of-interface=eth0
public
#
# ... and this says the same thing. But the following says ...
#
gostal@k2003734:/etc> sudo firewall-cmd --permanent --get-zone-of-interface=eth0
work
#
# --- that permanent zone should be work and is what I should get after restarting firewalld.
#
gostal@k2003734:/etc> sudo systemctl restart firewalld.service
#
# firewalld restarted
#
gostal@k2003734:/etc> sudo firewall-cmd --get-zone-of-interface=eth0
public
#
# Restarting firewalld does not put eth0 in work!! ...
#
gostal@k2003734:/etc> sudo firewall-cmd --permanent --get-zone-of-interface=eth0
work
#
# ... as the above says it should!

Where is the logic in the above? You tell me! I don’t get it!

I have also tried the option --runtime-to-permanent which should make the runtime configuration permanent by overwriting the currently saved permanent configuration. This after having thoroughly checked that the runtime configuration is to my liking. Same thing happens after restarting firewalld i.e. eth0 in public. It is as if the systemctl start/restart command looks for firewall configuration in a place where firewall-cmd has not been able to put it. At least this is what it looks like to me. Incidentally doing firewall-cmd --reload sets the runtime configuration according to permant as is should. So it seems that firewall-cmd behaves consistently but systemctl start/restart firewalld.service does it’s own thing. Have I missed something here, please, do tell!

Since I don’t seem to be able to get eth0 anywhere but in public I have enabled nfs there again to get a working runtime but clearly I do no not have this under control.

Cheers,
gostal