Increase RAM usage for nfs server on opensuse 12.2

English Network/Internet
#1

I’m running nfs v3 on 64bit opensuse 12.2 as an NFS server to a number of clients but I’d like (if possible) to increase RAM usage to offset slow disks on the server.

The machine isn’t tasked with anything else so the increased ram usage can only improve things I feel by getting more out of the hardware I have and it doesn’t appear to be maxing out the RAM or CPU when under load.

I realise the network is the pinch point but its on a 1Gbit connection with jumbo frames (MTU) enabled along the entire length of the connection but it seems wasteful to have RAM available and not to make use of it, I also realise it won’t make up for v slow disks but any improvement would be welcome

What I’ve done so far (not to increase ram but improve performance)

  • I’ve increased the thread count from the default
  • I’ve changed the mounting option to async

I’m not sure whats the best way to configure NFS to consume more RAM as a way to increase performance so any advice is welcome

Attached is my **/etc/syconfig/nfs

**

#
## Path:                Network/File systems/NFS server
## Description:         number of threads for kernel nfs server
## Type:                integer
## Default:             4
## ServiceRestart:      nfsserver
#
# the kernel nfs-server supports multiple server threads
#
USE_KERNEL_NFSD_NUMBER="16"
## Path:                Network/File systems/NFS server
## Description:         use fixed port number for mountd
## Type:                integer
## Default:             ""
## ServiceRestart:      nfsserver
#
#  Only set this if you want to start mountd on a fixed
#  port instead of the port assigned by rpc. Only for use
#  to export nfs-filesystems through firewalls.
#
MOUNTD_PORT=""
## Path:                Network/File systems/NFS server
## Description:         GSS security for NFS
## Type:                yesno
## Default:             yes
## ServiceRestart:      nfs nfsserver
#
# Enable RPCSEC_GSS security for NFS (yes/no)
#
NFS_SECURITY_GSS="no"
## Path:                Network/File systems/NFS server
## Description:         NFSv3 server support
## Type:                yesno
## Default:             yes
## ServiceRestart:      nfsserver
#
# Enable NFSv3 server support (yes/no)
# This causes the NFS server to respond to
# NFSv2 and NFSv3 requests.  Only disable this
# if you want to ensure only NFSv4 is used.
#
NFS3_SERVER_SUPPORT="yes"
## Path:                Network/File systems/NFS server
## Description:         NFSv4 protocol support
## Type:                yesno
## Default:             yes
## ServiceRestart:      nfs nfsserver
#
# Enable NFSv4 support (yes/no)
#
NFS4_SUPPORT="no"
## Path:                Network/File systems/NFS server
## Description:         Network Status Monitor options
## Type:                string
## Default:             ""
#
#
SM_NOTIFY_OPTIONS=""
#2

I’m not sure where you’re going with this. By default the Linux kernel
will cache files that are used, so if you have NFS files being read over
and over they are probably already cached and benefiting NFS, and even
better they should be available from that cache for all applications, not
just NFS clients across the wire.

Do you have a reason to believe that low RAM usage is the slow point here?
Pretend that all remaining RAM was used by NFS (assuming it’s not
effectively that way already). Which files would be put into this cache,
and how would they be chosen? What happens to these cached files when
other files are requested instead and no RAM remains? What tests have you
done so far to find the performance bottleneck?

What is the output of the ‘free’ command on your server?

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below…

#3

The files hosted on the NFS are virtual machine disks so are large (256GB) files that are accessed constantly by the 1 vm they are attached to, as that vm reads and writes to its disk. Also while the storage is presented to more than 1 client, the majority of traffic would be from 1 client at a time (the vm host where the owner of the virtual disks currently resides and gets its vCPU, vRAM from) so in my innocence I’m assuming that caching isn’t benefitting client access speeds because there is no common file in use between clients to cache, that the files themselves (256GB) are too large to cache in 1 go and the requests would be randomly spread across the file anyway as it represents a virtual disk

My hope was to find similar options and recommendations for tweaking RAM usage, just like increasing the thread count for CPU usage in NFSConfig or increasing the MTU on the nic to increase the size of the packets

I’m not interested in writing or implementing additional caching procedures for NFS, just modifying the existing options that pertain to RAM usage or that would benefit cases where disk I/O could be better. I don’t expect it to turn a pig into a racehorse

#4

On 2013-11-20 14:16, mobcdi wrote:
>
> The files hosted on the NFS are virtual machine disks so are large
> (256GB) files that are accessed constantly by the 1 vm they are attached
> to, as that vm reads and writes to its disk.

So, you should ask a moderator to move this thread to the virtualization
forum (use the triangle report button), because being a VM has a lot to
do with your problem.

Me, I would simply use a real machine for an NFS server.

Consider that Linux applies all available ram to disk cache, so
increasing memory for NFS serves no purpose - and you have not posted
the output of “free” as requested.

–
Cheers / Saludos,

Carlos E. R.
(from 12.3 x86_64 “Dartmouth” at Telcontar)

#5

On 11/20/2013 06:16 AM, mobcdi wrote:
>
> The files hosted on the NFS are virtual machine disks so are large
> (256GB) files that are accessed constantly by the 1 vm they are attached
> to, as that vm reads and writes to its disk. Also while the storage is
> presented to more than 1 client, the majority of traffic would be from 1
> client at a time (the vm host where the owner of the virtual disks
> currently resides and gets its vCPU, vRAM from) so in my innocence I’m
> assuming that caching isn’t benefitting client access speeds because
> there is no common file in use between clients to cache, that the files
> themselves (256GB) are too large to cache in 1 go and the requests would
> be randomly spread across the file anyway as it represents a virtual
> disk

Just to be clear, I’m not an NFS expert, but I do a bit with Linux and
computers so I think my original question is still pertinent:

How will using more RAM help when the file access isn’t able to be stored
in RAM? If your system has 32 GB of RAM all dedicated to NFS, and you are
trying to transfer just 256 GB of data regularly, you’ll only ever be able
to cache 1/8 of the data, so at best you’ll get better performance 1/8 of
the time. If you have multiple 256-GB files, that performance improvement
decreases, and this all assumes that the system reliably caches that bit
of data regularly instead of freeing up the RAM to try to cache something
else that is requested besides the 1/8 data.

Adding more RAM to the system could improve your chances here, but I am
guessing that the potential increase in performance will not be justified
by investing in a computer that can support hundreds of gigabytes (or
more) of RAM, much less the RAM cost in addition to the system that can
support it (by definition a server-class system since it’s > 32 GB).

To focus on the root of the problem (performance), what kind of
performance are you getting right now via NFS? What kind of performance
can you get copying files from your source to, for example, /dev/shm (a
built-in virtual memory device) when copying a gigabyte or two of data
after dropping caches (google for ‘drop caches’ and use the first link)?

You mentioned “jumbo frames” are enabled, but what is the actual MTU and
do you see the same number on both sides to rule out something in between
incorrectly splitting up packets? I believe you that jumbo frames are
enabled throughout, but every time I assume I have things configured one
way I find out that some other guy has tweaked things, so I do a lot of
verification.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below…

#6

Carlos E. R. wrote:
> On 2013-11-20 14:16, mobcdi wrote:
>> The files hosted on the NFS are virtual machine disks so are large
>> (256GB) files that are accessed constantly by the 1 vm they are attached
>> to, as that vm reads and writes to its disk.
>
> So, you should ask a moderator to move this thread to the virtualization
> forum (use the triangle report button), because being a VM has a lot to
> do with your problem.

Well, it’s one aspect of it, but the network is more fundamental, so I
think it belongs here.

> Me, I would simply use a real machine for an NFS server.

The OP is doing. The files it serves out are the VM images.

#7

mobcdi wrote:
> so in my innocence I’m
> assuming that caching isn’t benefitting client access speeds because
> there is no common file in use between clients to cache, that the files
> themselves (256GB) are too large to cache in 1 go and the requests would
> be randomly spread across the file anyway as it represents a virtual
> disk

Cacheing is of individual blocks, not entire files.

As others have asked, please post more information about your system’s
config and performance. Here are some more questions: what filesystems,
what mount options?

#8

Correct the NFS Server isn’t virtual it just provides the storage for hosting some virtual machine disk files. The mtu is set correctly for jumbo frames because there was a problem previously where 1 of the switches along the route stopped supporting jumbo frames due to a config change that didn’t survive a switch reboot so the nfs share would disconnect and reconnect from the client side and wouldn’t allow data transfer which was “interesting” to troubleshoot. (used ping destination -d -s 8972 to see if packets of the right MTU size less overheads were getting through)

I’m wondering if there is a way for me to tell NFS its ok to store more (capacity permitting) blocks in ram before writing them to disk, just like I can tell it to use more threads from the default 4

I’m not trying to get massive improvements, I’m just asking is there anything obvious I’m missing having already changed the mounting option to async (and accepting the consequences of that), enabled jumbo frames along the route, increased the thread count. The next option would seem to make more use of faster or more efficient use of ram I/o speeds by matching it to the block size on (server) disk, increasing the default caching or something else entirely just to avoid having to hit the NFS servers own disks if possible given I can’t change the them to SSD’s or other changes that required the nfs share to be down for longer than a normal server reboot. I’m not trying or evening thinking its possible to do away with reading/writing to the server disks but if I can make them more efficient or less frequent surely that would make a difference for the better

Are there options in NFSv3 or is it just threadcount that I can change that has an effect on the hardware usage?

Pings from NFS server to clients are around 1.2ms

Mounts are rw,no_root,squash,async

The vm who’s disks reside on the NFS share is reporting
Read: 96.8MiB/s
Write: 110.3MiB/s
Seeks: 1238.3seeks/s

Output of NFSStat on the server

Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
667355748   0          0          0          0Server nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 74335     0% 4665      0% 2106826   0% 198899    0% 0         0%
read         write        create       mkdir        symlink      mknod
555076594 83% 109764864 16% 13380     0% 15        0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
11556     0% 7         0% 1719      0% 0         0% 0         0% 20793     0%
fsstat       fsinfo       pathconf     commit
81897     0% 3         0% 0         0% 0         0%
Client rpc stats:
calls      retrans    authrefrsh
0          0          0

fdisk -lDisk /dev/sda: MYSIZE GB, MYSIZE bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Using LVM on /dev/sda

#9

free -m
total used free shared buffers cached
Mem: 3949 3804 145 0 9 3230
-/+ buffers/cache: 564 3385
Swap: 2047 103 1944

#10

mobcdi wrote:
> I’m wondering if there is a way for me to tell NFS its ok to store more
> (capacity permitting) blocks in ram before writing them to disk, just
> like I can tell it to use more threads from the default 4

It’s not NFS that does that, it’s not even the native filesystem, it’s
the kernel.

> The vm who’s disks reside on the NFS share is reporting
> Read: 96.8MiB/s
> Write: 110.3MiB/s
> Seeks: 1238.3seeks/s

Correct me if my brain is fried, but doesn’t that say the transfers are
saturating the network? 100 MiB/s ~ 1 Gb/s

#11

On 11/20/2013 08:37 AM, Dave Howorth wrote:
> mobcdi wrote:
>> I’m wondering if there is a way for me to tell NFS its ok to store more
>> (capacity permitting) blocks in ram before writing them to disk, just
>> like I can tell it to use more threads from the default 4
>
> It’s not NFS that does that, it’s not even the native filesystem, it’s
> the kernel.

Exactly.

>> The vm who’s disks reside on the NFS share is reporting
>> Read: 96.8MiB/s
>> Write: 110.3MiB/s
>> Seeks: 1238.3seeks/s
>
> Correct me if my brain is fried, but doesn’t that say the transfers are
> saturating the network? 100 MiB/s ~ 1 Gb/s

No, that’s correct. 1,000 mbit = 120 MByte, which is a theoretical max
that I’ve never seen hit in any environment. Getting 110 MiB/s,
especially on a write (vs. read), is pretty awesome. If you want to go
faster, upgrade your network to 10 Gbit.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below…

#12

Seeing that I’m a bit dubious about the numbers produced. Are there any native opensuse commands I can use to get eth0 live usage?

#13

Code:

ip -s link

That should show you the statistics for all devices, physical or virtual.
Run it now, wait ten seconds while copying data via NFS, run it again,
find the difference and divide by ten for bytes per second.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below…