Results 1 to 10 of 10

Thread: Hard Disk Activity Memory Hole

  1. #1

    Default Hard Disk Activity Memory Hole

    I do not know where else to post this, so here goes. I have a clean basic XFCE installation of OpenSUSE LEAP 15.2. This behavior happens on both the Asus R541U laptop I used to have (8GB RAM, 512MB Swap) and my new Dell Inspiron 7591 (16GB RAM, 1GB swap). I would boot into OpenSUSE to do some "hard drive wrangling", i.e. making disk images of hard drives via USB adapters (dd if=/some/device | gzip -c > imagefile) or zeroizing old disks (dd if=/dev/zero of=/some/device).

    As soon as I begin the dd process, my RAM and swap climb through the roof. Almost no applications are open when this occurs. For example:

    1) When I was using my Asus to read the 512GB SSD via an adapter to another USB external hard drive (BACKUP) (i.e. dd if=/dev/nvme0n1 | gzip -c > /run/media/robert/BACKUP/Windows/dell7591.img.gz)

    2) When I was backing up my files on the Asus (rsync -Hav /home/robert/ /run/media/robert/BACKUP/dell/robert/)

    3) When I was copying the dd image to the new 1TB SSD upgrade for my dell (gunzip -c /run/media/robert/BACKUP/Windows/dell7591.img.gz | dd of=/dev/nvme0n1)

    4) When I was just now zeroizing the old 512GB SSD via the same USB adapter (dd if=/dev/zero of=/dev/sdb)

    5) When I was synchronizing my incremental monthly backups (both 2TB external USB drives running LUKS) (rsync -Hav --delete --progress /run/media/robert/BACKUP/ /run/media/robert/BACKUP2/ )

    It always seems connected to rsync/gzip/dd, i.e. heavy use of filesystems. If I boot OpenSUSE and I am just sitting in OpenSUSE using applications, usually it does not cause me to dig into swap.

    At the height of the zeroizing action, for example, swap use (16GB RAM, 1GB swap, new Dell Inspiron 7591) climbed to 108MB. It dropped to 11MB.

    Given that I have 16GB of RAM, such behavior is absolutely unacceptable. All the I/O should be happening on disks. I have not been able to triangulate, using top, what process is eating RAM so much.

    I am using EXT4 exclusively, no BTRFS anywhere.

    I have remounted all tmpfs entries to only give them 1GB of RAM to work with, as in the past this has prevented such excessive swappiness (believe it or not; it's difficult to prove; older versions of OpenSUSE, etc).

    I am willing to run experiments to see what's going on.

    I noticed that there were some btrfs components of systemd that were installed. I uninstalled them, but the problem remains.

    I don't understand how even running something complex as rsync + gzip + dd should need to dig into that much system RAM. I mean, I have 16GB!

    Have any memory leaks been reported on OpenSUSE?

  2. #2
    Join Date
    Jul 2018
    Location
    Loma Linda, Mo
    Posts
    321

    Default Re: Hard Disk Activity Memory Hole

    You can always do a swapoff to stop the swapping.

    All linux kernels map memory and when memory gets used fast will swap memory that has not been used for a long time to swap space. It does not matter how much memory you have.

    In your dd's, I/O is pending and buffers are filling up faster than emptying so the memory manager is going to swap out little used memory to swap.
    Opensuse 15.2 with VirtualBox VM's (XP, 10 & OpenSUSE 15.0)
    Pi4 with Ubuntu MATE 20.04
    Unix since 1974 (pdp-11 in "B" , Interdata 7/32 in "C") (AT&T, Tandy, Convergent, IBM, NCR, HP flavors)
    Linux since 1995 (mandrake, redhat, fedora, centos, now OpenSUSE)

  3. #3
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,903
    Blog Entries
    1

    Default Re: Hard Disk Activity Memory Hole

    Quote Originally Posted by radelahunt View Post
    I do not know where else to post this, so here goes. I have a clean basic XFCE installation of OpenSUSE LEAP 15.2. This behavior happens on both the Asus R541U laptop I used to have (8GB RAM, 512MB Swap) and my new Dell Inspiron 7591 (16GB RAM, 1GB swap). I would boot into OpenSUSE to do some "hard drive wrangling", i.e. making disk images of hard drives via USB adapters (dd if=/some/device | gzip -c > imagefile) or zeroizing old disks (dd if=/dev/zero of=/some/device).

    As soon as I begin the dd process, my RAM and swap climb through the roof. Almost no applications are open when this occurs. For example:

    1) When I was using my Asus to read the 512GB SSD via an adapter to another USB external hard drive (BACKUP) (i.e. dd if=/dev/nvme0n1 | gzip -c > /run/media/robert/BACKUP/Windows/dell7591.img.gz)

    2) When I was backing up my files on the Asus (rsync -Hav /home/robert/ /run/media/robert/BACKUP/dell/robert/)

    3) When I was copying the dd image to the new 1TB SSD upgrade for my dell (gunzip -c /run/media/robert/BACKUP/Windows/dell7591.img.gz | dd of=/dev/nvme0n1)

    4) When I was just now zeroizing the old 512GB SSD via the same USB adapter (dd if=/dev/zero of=/dev/sdb)

    5) When I was synchronizing my incremental monthly backups (both 2TB external USB drives running LUKS) (rsync -Hav --delete --progress /run/media/robert/BACKUP/ /run/media/robert/BACKUP2/ )

    It always seems connected to rsync/gzip/dd, i.e. heavy use of filesystems. If I boot OpenSUSE and I am just sitting in OpenSUSE using applications, usually it does not cause me to dig into swap.

    At the height of the zeroizing action, for example, swap use (16GB RAM, 1GB swap, new Dell Inspiron 7591) climbed to 108MB. It dropped to 11MB.

    Given that I have 16GB of RAM, such behavior is absolutely unacceptable. All the I/O should be happening on disks. I have not been able to triangulate, using top, what process is eating RAM so much.

    I am using EXT4 exclusively, no BTRFS anywhere.

    I have remounted all tmpfs entries to only give them 1GB of RAM to work with, as in the past this has prevented such excessive swappiness (believe it or not; it's difficult to prove; older versions of OpenSUSE, etc).

    I am willing to run experiments to see what's going on.

    I noticed that there were some btrfs components of systemd that were installed. I uninstalled them, but the problem remains.

    I don't understand how even running something complex as rsync + gzip + dd should need to dig into that much system RAM. I mean, I have 16GB!

    Have any memory leaks been reported on OpenSUSE?
    You are not alone: https://forums.opensuse.org/showthre...ng-large-files

    See also: https://bugzilla.opensuse.org/show_bug.cgi?id=1159882

    Currently I use btrfs and experience no problems related to swapping:

    Code:
    erlangen:~ # free -h
                  total        used        free      shared  buff/cache   available
    Mem:           31Gi       2.2Gi        25Gi       315Mi       2.9Gi        28Gi
    Swap:          15Gi          0B        15Gi
    erlangen:~ #
    Turning swap off indeed helps most of the time.
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), AMD Ryzen 5 3400G (2020), openSUSE Tumbleweed, KDE Plasma 5

  4. #4

    Default Re: Hard Disk Activity Memory Hole

    Quote Originally Posted by karlmistelberger View Post
    You are not alone: https://forums.opensuse.org/showthre...ng-large-files

    See also: https://bugzilla.opensuse.org/show_bug.cgi?id=1159882

    Currently I use btrfs and experience no problems related to swapping:

    Turning swap off indeed helps most of the time.
    But this seems illogical if I have just booted into Linux and just zeroizing one drive leads to incredibly high swap use.

    I'd rather not engage in any trickery where I have to make swap TMPFS or disable swap entirely.

    Maybe it's that the kernel is far too eager to hand over tons of memory to I/O operations where it would be waiting on a drive anyways?

    Is there a way to tune this aside from making swappiness 1?

    Especially when using zeroize operations (dd if=/dev/zero) you'd think there should be no memory I/O or almost none because your input device is virtual. Why should buffers fill up with zeros?

    Am I conceptualizing this correctly? Or am I confused?

    I can set swappiness to 1 for now, but this just feels like a really weird problem to have, especially on a laptop. (Can we please please please get a kernel that's intended for laptops?)

    Is there any other information that I can provide or harvest from my machine to see if maybe there's some problem within something like systemd or other base level component?

    I really appreciate everyone taking the time to reply!

  5. #5
    Join Date
    Jul 2018
    Location
    Loma Linda, Mo
    Posts
    321

    Default Re: Hard Disk Activity Memory Hole

    Even if you write /dev/zero - it goes into a buffer to be written to the disk - a buffer is still a 512 bytes of buffer even when it is filled with zeros.

    dd will fill buffers as fast as the kernel will give it buffers. Once the kernel reaches some level of buffers allocated it will try to free memory to swap if available.

    That is the readers digest of what is happening. The main reason today for swap is to provide an area to hibernate memory into. When you can put lots of memory on a computer the 1980's reason for swap is no longer needed. Veritos Disk Manager (Logical Volume Manager in Linux) was created to allow multiple 20MB disk to look like 1 large disk. Mirroring was added for reliability - now we use it to make a large disk look like many smaller disks.

    As for what is running even after a reboot ; do a ps -ef

    It will show all programs that are in memory that might need to be swapped.

    Use iotop to see what is doing I/O

    Use top or htop to see how much memory programs are using.
    Opensuse 15.2 with VirtualBox VM's (XP, 10 & OpenSUSE 15.0)
    Pi4 with Ubuntu MATE 20.04
    Unix since 1974 (pdp-11 in "B" , Interdata 7/32 in "C") (AT&T, Tandy, Convergent, IBM, NCR, HP flavors)
    Linux since 1995 (mandrake, redhat, fedora, centos, now OpenSUSE)

  6. #6
    Join Date
    Dec 2008
    Location
    FL, USA
    Posts
    2,377

    Default Re: Hard Disk Activity Memory Hole

    There aren't a whole lot of people (%-wise) with 16GB or more RAM and don't hibernate who actually need swap. I ran this PC over 5 years with 16G of RAM and swap off. Last month I added another 16G RAM, still have swap off, but haven't yet tried any organized evaluation of whatever differences were made.

    If you really need swap, but do a lot of the copying you describe, I suggest to put the process in a script that disables swap at the start and re-enables it when done.
    Reg. Linux User #211409 *** multibooting since 1992
    Primary: 15.1,TW,15.2 & 13.1 on Haswell w/ RAID
    Secondary: eComStation (OS/2)&15.1 on 965P/Radeon
    Tertiary: TW,15.2,15.1,Fedora,Debian,more on Kaby Lake,Q45,Q43,G41,G3X,965G,Cedar,Caicos,Oland,GT218&&&

  7. #7
    Join Date
    Sep 2012
    Posts
    5,905

    Default Re: Hard Disk Activity Memory Hole

    Quote Originally Posted by radelahunt View Post
    As soon as I begin the dd process, my RAM and swap climb through the roof.
    In case of using dd to write on device oflag=direct should help - it will bypass kernel filesystem buffering and will write directly to device. You may want to increase obs, otherwise it will be rather slow.

  8. #8
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,903
    Blog Entries
    1

    Default Re: Hard Disk Activity Memory Hole

    Quote Originally Posted by radelahunt View Post
    But this seems illogical if I have just booted into Linux and just zeroizing one drive leads to incredibly high swap use.

    I'd rather not engage in any trickery where I have to make swap TMPFS or disable swap entirely.

    Maybe it's that the kernel is far too eager to hand over tons of memory to I/O operations where it would be waiting on a drive anyways?

    Is there a way to tune this aside from making swappiness 1?

    Especially when using zeroize operations (dd if=/dev/zero) you'd think there should be no memory I/O or almost none because your input device is virtual. Why should buffers fill up with zeros?

    Am I conceptualizing this correctly? Or am I confused?

    I can set swappiness to 1 for now, but this just feels like a really weird problem to have, especially on a laptop. (Can we please please please get a kernel that's intended for laptops?)

    Is there any other information that I can provide or harvest from my machine to see if maybe there's some problem within something like systemd or other base level component?

    I really appreciate everyone taking the time to reply!
    I think the observed behavior is a bug. I observed it for some time on several desktop machines running Tumbleweed. It disappeared on those machines some time ago. You may try to reopen Bug 1159882.
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), AMD Ryzen 5 3400G (2020), openSUSE Tumbleweed, KDE Plasma 5

  9. #9

    Default Re: Hard Disk Activity Memory Hole

    Quote Originally Posted by karlmistelberger View Post
    I think the observed behavior is a bug. I observed it for some time on several desktop machines running Tumbleweed. It disappeared on those machines some time ago. You may try to reopen Bug 1159882.
    Thank you, I think I successfully reopened the bug and moved it to LEAP 15.2. I replied with text of this post, and the link, and told them I'm very eager to help provide information to substantiate or troubleshoot.

  10. #10

    Default Re: Hard Disk Activity Memory Hole

    Bug https://bugzilla.opensuse.org/show_bug.cgi?id=1159882

    Information added to help everyone troubleshoot this.

    Thank you very much for all your help. Again, please notify me ASAP if there's anything else I can help provide.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •