NFSv4 oddity

English Network/Internet
#1

I’m running OpenSuSE 11.3 ( 64 bit ) with the latest kernel update as of yesterday.

I’ve recently moved from NFSv3 to NFSv4 on 4 servers. Between the 4 servers there are 9 exported directories. All of them work EXCEPT 1 on 1 of the 4 servers.

/etc/exports

NFSv4 fsid=0 for the top level directory

/export *(rw,fsid=0,async,no_subtree_check,insecure)

End

NFSv3

/export/apps/oracle *(rw,async,fsid=7,no_subtree_check,insecure)
/export/ftpserver *(rw,async,fsid=6,no_subtree_check,insecure)

End

I had to keep the NFSv3 entry as I need that as the mount doesn’t work with v4.

/export directory listing
/export:
total 5
drwxr-xr-x 4 root root 104 2002-08-08 16:47 .
drwxr-xr-x 33 root root 944 2011-01-25 17:28 …
drwxr-xr-x 10 root root 232 2010-01-26 10:01 apps
drwxr-xr-x 8 root root 4096 2010-12-13 12:31 ftpserver

/export/apps:
total 0
drwxr-xr-x 10 root root 232 2010-01-26 10:01 .
drwxr-xr-x 4 root root 104 2002-08-08 16:47 …
drwx------ 3 root root 72 2008-08-06 16:40 etc
drwxr-xr-x 2 root root 240 2011-01-24 03:00 htdig
drwx------ 3 root root 72 2008-08-06 16:40 lib
drwx------ 2 root root 48 2002-07-02 13:18 lost+found
drwxr-xr-x 4 oracle oinstall 104 2009-01-07 09:59 oracle
drwx------ 5 root root 120 2008-08-06 16:40 usr

Now on any of the 4 servers if I do:

mount -t nfs4 machinename:/apps/oracle /mnt

mount.nfs4: mounting lister:/apps/oracle failed, reason given by server:
No such file or directory

But the other sub-dir of /export is fine:

mount -t nfs4 machinename:/apps/ftpserver /mnt

If I mount the pseudo root:

mount -t nfs4 machinename:/ /mnt

ls -l /mnt

total 4
drwxr-xr-x 2 root root 48 2002-08-08 15:20 apps
drwxr-xr-x 8 root root 4096 2010-12-13 12:31 ftpserver

But the apps sub-dir is empty

ls -l /mnt/apps

total 0

On the server:

ls -l /export/apps

total 0
drwx------ 3 root root 72 2008-08-06 16:40 etc
drwxr-xr-x 2 root root 240 2011-01-24 03:00 htdig
drwx------ 3 root root 72 2008-08-06 16:40 lib
drwx------ 2 root root 48 2002-07-02 13:18 lost+found
drwxr-xr-x 4 oracle oinstall 104 2009-01-07 09:59 oracle
drwx------ 5 root root 120 2008-08-06 16:40 usr

Well that explains why the NFSv4 mount of /apps/oracle fails because for some reason it can’t “see” any files below that, but as I say the ftpserver sub-dir & mount are fine.

#2

Sorry that was

mount -f nfs4 machinename:/ftpserver /mnt

#3

Sorry, I can’t help with your specific problem, but I can tell you I had nfs4 issues under 11.2 and 11.3, the last took forever to identify, because it affected gnome in strange ways because of some problem talking to the nsf4 mount (note, this did initially work). I liked the single export of nfs4, but I went back to nfs3 and everything worked – and still does. I hope you have better luck that I.

#4

Thanks C0ff33. The only reason I went with NFSv4 is that NFSv3 was playing up. It all worked one week then without reason some of the exports stopped working. No reboot or change to the config files:

mount -t nfs machinename:/export/blah

Think it said either Access Denied or Permission Denied. Checked the logs and the fact that the IP address could be reverse looked up to the hostname.

Moved to NFSv4 everything worked except this 1 problem which I am using NFSv3 to solve. Kinda like your scenario in reverse.

#5

Do you have the same Domain defined in server and client in /etc/idmapd.conf ?
Do you have an oracle user and oinstall group on the client ?
What happens if you change ownership of directory ‘oracle’ (for testing purpose) ?

#6

All /etc/idmapd.conf files are the same acros the 4 servers:

[General]

Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = localdomain

[Mapping]

grep nobody /etc/passwd

nobody:x:65534:65533:nobody:/var/lib/nobody:/bin/bash

grep nobody /etc/group

nobody::65533:nobody,root
nogroup::65534:nobody,root

It’s not just the oracle sub-dir that’s the problem if I try:

mount -t nfs4 hostname:/apps/htdig /mnt

mount.nfs4: mounting hostname:/apps/htdig failed, reason given by server:
No such file or directory

The same applies to any subdir in /export/apps on the server so i can’t see it being an oracle user issue. htdig is 755 owner by root as is /export/ftpserver which works. The hub of the problem would seeem to be when I do:

mount -t nfs4 hostname:/apps /mnt

The mount works but an ls of /mnt reveals there is nothing there.

#7

Compare the nfs4 exported in /etc/exports with the output of

mount | grep bind
  • /proc doesn’t count
#8

unfortunately I dont have my old files, but a few comments (from memory) on nfs4.

nfsv4 is very different than v3. you can do v3 style shares/mounts, but the nice feature of v4 is exporting a pseudo file system. Basically, you create an empty folder somewhere (/nfs4exp for example) then you “bind mount” all the real directories to that location by adding entries to the server’s fstab. Only the single nfs4exp (which has fsid=0) needs to be mounted by clients.

I would be interested to know if others here are using v4 without problems. My setup suddenly died in a non-obvious way, took way to long to figure out the problem was v4, went back to v3 without problems. I’d hate for you to get it working --briefly. And given you current v4 setup, I’m not sure what benefit it provides.

#9

Unfortunately mount binding (or bind mounting, whatever is the most approriate) in /etc/fstab only worked for me under Ubuntu. I have no idea why. It conflicted with something else (I have to use v3 too for BSD clients). Under openSUSE, Fedora and others, I used that command from script (/etc/rc.d/after.local in the case of openSUSE):

# mounting nfs4 exported
eval $(exportfs -v | awk '/nfs4/ { fs=$1 ; sub(/\/export\/nfs4/, "", fs) ; if (fs == "") fs="/" ; printf "mount --bind %s %s ; ", fs, $1 }')
#10

Thanks all. No fancy mount binds:

mount | grep bind

/proc on /var/lib/ntp/proc type none (ro,nosuid,nodev,bind)
/export/apps/oracle on /net/oracle type none (rw,bind)

I use autofs in conjunction with NFSv3 for /net/oracle as NFSv4 doesn’t work for /apps/oracle.

The config is most basic. /export is part of the root filesystem & the 2 subdirs below that are actually partitions, both of which are mounted:

Part of /etc/fstab

/dev/disk/by-id/scsi-SFUJITSU_MAP3367NP_UPS0P3100CSM-part1 /export/apps
reiserfs acl,user_xattr 1 2
/dev/disk/by-id/scsi-35000000000000000-part1 /export/ftpserver
ext3 acl,user_xattr,usrquota 1 2

Ok, 1 is reiserfs & the other ( which works ) ext3, buit my other 3 servere export reiserfs filesystems fine.

#11

Hmmm … However this looks like NFSv4. Only v4 uses such bindings. So if that filesystem is exported with NFSv3, you should not have that.

Yeah, I don’t think that matters.