Memory management

Hi all, this is my first post to this group.
I spent some time before deciding on which forum this request could be sent. I finally decided for “Install/Boot/Login” because the answer to my problem could be related to installation / tuning / kernel rebuild, so this might be the right place.
Please excuse me if I am wrong!

I am trying to understand how exactly memory management in linux (OpenSUSE 11.2) works.
I have compiled the following program (memorytest.c):

#include <stdlib.h>
#include <stdio.h>
int main(int argc, char **argv)
{
  long int n=0;
  int *x;

  if (argc != 2) exit (0);
  else {
    int mem = atoi(argv[1]);
    while(1)
    {
      x = (int*) malloc (mem);  // no point in testing if not zero:
                                // I'll know that from the segmfault :-)
      printf("
 n=%ld x=%p", n, x);
      *(x+20) = 123456; // just to use the allocated memory
                        // (or to trigger the segmfault)
      n++;
      sleep(1); // wait a little and let me see what's happening
    } // while
  } // else
  return 0;
}

on a linux workstation running 64bit OpenSuse, with 24 GB RAM.
I run the program with
memorytest 1073751800
thus allocating memory 1 GB per cycle.

The program output is

n=0 x=0x7f751b1cd010
n=1 x=0x7f74db1ca010
n=2 x=0x7f749b1c7010
n=3 x=0x7f745b1c4010
<…omissis…>
n=21 x=0x7f6fdb18e010
n=22 x=0x7f6f9b18b010
n=23 x=0x7f6f5b188010
n=24 x=0x7f6f1b185010
Segmentation fault

The program stops after allocating 24 GB, that is the physical memory
limit: why does it not go as far as 64 bit allocation allows? I’d expect
it to continue allocating memory (and paging) till all the paging memory
is exhausted, or virtual space addressing is saturated, whichever
the first.

Also consider that the same code works with no problem on a Mac Pro, continuing to allocate memory on and on (till you are bored and stop the program).

What’s going on? Can someone enlighten me please?
I need all that memory (far more than the available physical RAM) so the problem is not just curiosity…
It is evident that linux is not allowing memory paging: how can I verify it and convince linux to work as I expect?
Thanks
Giorgio

PS This request was already posted to other forums, so please excuse me if you have already seen this help cry somewhere else…

I tested your program on my machine with 4 GB physical RAM and it can
allocate 9 GB before it aborts. So it really uses the swap.
What does


free

show on your machine, does it report swap space?


PC: oS 11.3 64 bit | Intel Core2 Quad Q8300@2.50GHz | KDE 4.6.1 | GeForce
9600 GT | 4GB Ram
Eee PC 1201n: oS 11.4 64 bit | Intel Atom 330@1.60GHz | KDE 4.6.0 | nVidia
ION | 3GB Ram

On 04/14/2011 10:06 PM, giorgiodn wrote:
>
> The program stops after allocating 24 GB, that is the physical memory
> limit: why does it not go as far as 64 bit allocation allows?

how large is your /swap partition?

please show us the terminal output from


sudo /sbin/fdisk -l
cat /etc/fstab

copy/paste the output back to this thread using the instructions here:
http://goo.gl/i3wnr


CAVEAT: http://is.gd/bpoMD
[NNTP via openSUSE 11.3 + KDE4.5.5 + Thunderbird3.1.8]
Q: Why do you upgrade?
A: Because the Gecko is always greener on the other side!
So said k428 in http://is.gd/Pwc3xq

On 04/14/2011 03:06 PM, giorgiodn wrote:
>
> Hi all, this is my first post to this group.
> I spent some time before deciding on which forum this request could be
> sent. I finally decided for “Install/Boot/Login” because the answer to
> my problem could be related to installation / tuning / kernel rebuild,
> so this might be the right place.
> Please excuse me if I am wrong!
>
> I am trying to understand how exactly memory management in linux
> (OpenSUSE 11.2) works.
> I have compiled the following program (memorytest.c):
>
>
> Code:
> --------------------
> #include<stdlib.h>
> #include<stdio.h>
> int main(int argc, char **argv)
> {
> long int n=0;
> int x;
>
> if (argc != 2) exit (0);
> else {
> int mem = atoi(argv[1]);
> while(1)
> {
> x = (int
) malloc (mem); // no point in testing if not zero:
> // I’ll know that from the segmfault :slight_smile:
> printf("
n=%ld x=%p", n, x);
> *(x+20) = 123456; // just to use the allocated memory
> // (or to trigger the segmfault)
> n++;
> sleep(1); // wait a little and let me see what’s happening
> } // while
> } // else
> return 0;
> }
>
> --------------------
>
>
> on a linux workstation running 64bit OpenSuse, with 24 GB RAM.
> I run the program with
> memorytest 1073751800
> thus allocating memory 1 GB per cycle.
>
> The program output is
>
> n=0 x=0x7f751b1cd010
> n=1 x=0x7f74db1ca010
> n=2 x=0x7f749b1c7010
> n=3 x=0x7f745b1c4010
> <…omissis…>
> n=21 x=0x7f6fdb18e010
> n=22 x=0x7f6f9b18b010
> n=23 x=0x7f6f5b188010
> n=24 x=0x7f6f1b185010
> Segmentation fault
>
> The program stops after allocating 24 GB, that is the physical memory
> limit: why does it not go as far as 64 bit allocation allows? I’d
> expect
> it to continue allocating memory (and paging) till all the paging
> memory
> is exhausted, or virtual space addressing is saturated, whichever
> the first.

How much swap space do you have?

> Also consider that the same code works with no problem on a Mac Pro,
> continuing to allocate memory on and on (till you are bored and stop the
> program).

Does the Mac have OSX? Does it use a dynamic page/swap file?

> What’s going on? Can someone enlighten me please?
> I need all that memory (far more than the available physical RAM) so
> the problem is not just curiosity…
> It is evident that linux is not allowing memory paging: how can I
> verify it and convince linux to work as I expect?

I suspect that you don’t have very much swap allocated and that swap + swappable
RAM + free RAM is that 24 GB limit.

Hi again, and thanks for your kind replies!
Here are some answers to your questions:

free, fdisk, and cat fstab give:

giorgio.denunzio@pc-gdenunzio:~> free
             total       used       free     shared    buffers     cached
Mem:      24735456    9390952   15344504          0     424584    8253004
-/+ buffers/cache:     713364   24022092
Swap:      8385888     541412    7844476

giorgio.denunzio@pc-gdenunzio:~> sudo /sbin/fdisk -l
root's password:

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000080

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       60801   488384001    f  W95 Ext'd (LBA)
/dev/sda5               1        1306    10490382   83  Linux
/dev/sda6            1307       20887   157284351   83  Linux
/dev/sda7           20888       21540     5245191   83  Linux
/dev/sda8           21541       21553      104391   83  Linux
/dev/sda9           21554       22597     8385898+  82  Linux swap / Solaris
/dev/sda10          22598       23250     5245191   83  Linux
/dev/sda11          23251       60801   301628376   83  Linux

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000081

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       26109   209720511   83  Linux

giorgio.denunzio@pc-gdenunzio:~> cat /etc/fstab
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164864-part9 swap                 swap       defaults              0 0
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164864-part5 /                    ext4       acl,user_xattr        1 1
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164864-part8 /boot                ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164864-part11 /home                ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164864-part10 /tmp                 ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164864-part6 /usr                 ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164864-part7 /var                 ext4       acl,user_xattr        1 2
proc                 /proc                proc       defaults              0 0
sysfs                /sys                 sysfs      noauto                0 0
debugfs              /sys/kernel/debug    debugfs    noauto                0 0
usbfs                /proc/bus/usb        usbfs      noauto                0 0
devpts               /dev/pts             devpts     mode=0620,gid=5       0 0
/dev/disk/by-id/ata-WDC_WD5000AAKS-75V0A0_WD-WMAWF1164476-part1 /data                ext4       acl,user_xattr        1 2
giorgio.denunzio@pc-gdenunzio:~> 

Moreover, issuing three times the free command **during ** the run of the malloc program in another shell (issuing “free” just before running, during the run after the program allocated 10 GB, during the run after the program allocated 20 GB) gives:

giorgio.denunzio@pc-gdenunzio:~> free
             total       used       free     shared    buffers     cached
Mem:      24735456    9393768   15341688          0     424680    8253784
-/+ buffers/cache:     715304   24020152
Swap:      8385888     541412    7844476
giorgio.denunzio@pc-gdenunzio:~> 
giorgio.denunzio@pc-gdenunzio:~> free
             total       used       free     shared    buffers     cached
Mem:      24735456    9362404   15373052          0     424692    8253772
-/+ buffers/cache:     683940   24051516
Swap:      8385888     541412    7844476
giorgio.denunzio@pc-gdenunzio:~> free
             total       used       free     shared    buffers     cached
Mem:      24735456    9367480   15367976          0     424692    8253776
-/+ buffers/cache:     689012   24046444
Swap:      8385888     541412    7844476
giorgio.denunzio@pc-gdenunzio:~> 


Somebody proposed to me to check the output of

swapon -s ; free -t

so I put it here too:

giorgio.denunzio@pc-gdenunzio:~> /sbin/swapon -s ; free -t
Filename                                Type            Size    Used    Priority
/dev/sda9                               partition       8385888 541412  -1
             total       used       free     shared    buffers     cached
Mem:      24735456    9361496   15373960          0     424732    8253776
-/+ buffers/cache:     682988   24052468
Swap:      8385888     541412    7844476
Total:    33121344    9902908   23218436

I also checked command ulimit, and the reply was “unlimited”, while /proc/sys/vm/swappiness is 60.

Finally, I know nothing about the Mac Pro on which the program runs well: it belongs to one of my fellows and I just asked him to compile and try because I knew the machine was 64-bits, but I know no more details (I’ll ask).

24 GB is the physical memory; it was 12 GB but I upgraded one year ago.

Thanks again for all your kind replies!!
Regards
Giorgio

To add a bit more info:

If your free command (and the questions DenverD asked you) lead to the
conclusion that you have no or a very small swap space and you do not want
to change your partitition then you should consider to try a swap file under
linux.
A howto is here:
http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

You can also have several swap files at the same time (just repeat the
steps).


PC: oS 11.3 64 bit | Intel Core2 Quad Q8300@2.50GHz | KDE 4.6.1 | GeForce
9600 GT | 4GB Ram
Eee PC 1201n: oS 11.4 64 bit | Intel Atom 330@1.60GHz | KDE 4.6.0 | nVidia
ION | 3GB Ram

ulimit without parameter does not tell you much.
Run

ulimit -a

the important parameter is -v which tells you how much virtual memory is allowed.
You can set it to a higher value with

ulimit -v <kbytes>

If you want try the suggestion with the additional swap file, I just did that myself with an additional swap file to check and now I can run your example on a 4GB machine up to 20GB allocation.
I think its worth a try.


Hi Martin, command:
ulimit -v
did work! ulimit -v alone gave about 26,000,000; I set ulimit -v 40000000 and could run my program till 37-GB allocation! That’s great but… strange! At present, if I understand what “free” gives, I have 8 GB swap. 24+8=32, how can 40 GB ulimit -v work?
Another curious thing: after ulimiting to (say) -v 30000000, I could no more issue the same command with a larger -v, but only with a lower one, otherwise the system would respond: “-bash: ulimit: virtual memory: cannot modify limit: Operation not permitted”.
I’ll now add a swap file as you suggest (and later an auxiliary swap partition), and try to ulimit as much as needed.
The former doubt anyway remains: if “free” gave me the total amount of swap memory (8GB) how could I make the program run till 37 GB before segmfault?
Thanks again!
Giorgio

giorgiodn wrote:

>
> Code:
> --------------------
>
> --------------------
> Hi Martin, command:
> ulimit -v
> did work! ulimit -v alone gave about 26,000,000; I set ulimit -v
> 40000000 and could run my program till 37-GB allocation! That’s great
> but… strange! At present, if I understand what “free” gives, I have 8
> GB swap. 24+8=32, how can 40 GB ulimit -v work?
> Another curious thing: after ulimiting to (say) -v 30000000, I could no
> more issue the same command with a larger -v, but only with a lower one,
> otherwise the system would respond: “-bash: ulimit: virtual memory:
> cannot modify limit: Operation not permitted”.
> I’ll now add a swap file as you suggest (and later an auxiliary swap
> partition), and try to ulimit as much as needed.
> The former doubt anyway remains: if “free” gave me the total amount of
> swap memory (8GB) how could I make the program run till 37 GB before
> segmfault?
> Thanks again!
> Giorgio
>
Please be very carefull with ulimit -v you can fake an amount of memory the
system does not have and your prorams will at some point fail randomly
because the memory space does not really exist. Set it to something which is
really below the amount of physical ram + swap space!
Sorry, I should have added this warning before.


PC: oS 11.3 64 bit | Intel Core2 Quad Q8300@2.50GHz | KDE 4.6.1 | GeForce
9600 GT | 4GB Ram
Eee PC 1201n: oS 11.4 64 bit | Intel Atom 330@1.60GHz | KDE 4.6.0 | nVidia
ION | 3GB Ram

martin_helm wrote:

> Please be very carefull with ulimit -v you can fake an amount of memory
> the system does not have and your prorams will at some point fail randomly
> because the memory space does not really exist. Set it to something which
> is really below the amount of physical ram + swap space!
> Sorry, I should have added this warning before.
>
The conservative way (just checked the Novell documentation for SLES so I
think it is the same for openSUSE) is to set ulimit by default to 80% of the
total swap + total RAM. This would explain the 26 GB limit you see by
default with a total of 32 GB = 24 GB RAM + 8 GB swap you see.
You can see this in the file
/etc/sysconfig/ulimit
which shows
SOFTVIRTUALLIMIT=“80”
So if you add swap space (and add that to /etc/fstab so the setting is
persistent) this should (after the next reboot) automaticaly increase the
ulimit default size for the virtual memory you can use.
But I did not check that myself right now.


PC: oS 11.3 64 bit | Intel Core2 Quad Q8300@2.50GHz | KDE 4.6.1 | GeForce
9600 GT | 4GB Ram
Eee PC 1201n: oS 11.4 64 bit | Intel Atom 330@1.60GHz | KDE 4.6.0 | nVidia
ION | 3GB Ram

Thanks again, Martin! I’ve done what you suggested: I have now a 80 GB file called swap_1 that I am using as an additional swap area (by swapon), I have defined ulimit -v at about 80% of the total swap space (I’ll later check if this is done automatically when fstab is modified).
My test program ran ok, and now I have launched the real program from which my problem started (a program working on a binary mask of a lung, whose aim is finding pleural nodules: I work in the field of medical physics and imaging). Everything is going ok, and I can see free virtual memory going down as the program runs. If all is ok, I’ll play with /etc/fstab and make my changes permanent.
Thanks again to Martin and to the whole forum for helping!
Giorgio

giorgiodn wrote:

> I work in the field of medical
> physics and imaging).

Wow, a colleague (my main focus is meanwhile medical databases and project
management and leading a team and all that boring stuff, but I was working
myself as a developer in medical imaging for several years, way back in the
stoneage when I was young, I am a mathematician).
I hope everything will go well, if there are further problems you know where
to ask.


PC: oS 11.3 64 bit | Intel Core2 Quad Q8300@2.50GHz | KDE 4.6.1 | GeForce
9600 GT | 4GB Ram
Eee PC 1201n: oS 11.4 64 bit | Intel Atom 330@1.60GHz | KDE 4.6.0 | nVidia
ION | 3GB Ram

On 2011-04-15 22:36, giorgiodn wrote:
>
> Thanks again, Martin! I’ve done what you suggested: I have now a 80 GB
> file called swap_1 that I am using as an additional swap area (by

Interesting. If you need that much swap, you could set a separate hard disk
as swap, it is much faster. Another trick, if you have several disks, is to
distribute several swap spaces on each, at the same priority. The priority
is important: if the values are different, the spaces will be used in
succession. If the same, they will be used simultaneously: two disks will
be twice as fast.

Ideal would be a solid state disk: not flash, they are very slow on writes.

> My test program ran ok, and now I have launched the real program from
> which my problem started (a program working on a binary mask of a lung,
> whose aim is finding pleural nodules: I work in the field of medical
> physics and imaging). Everything is going ok, and I can see free virtual
> memory going down as the program runs. If all is ok, I’ll play with
> /etc/fstab and make my changes permanent.

That’s a very interesting job :slight_smile:


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

Carlos E. R. wrote:
>
> Ideal would be a solid state disk: not flash, they are very slow on
> writes.
>
I’ll second that definitely solid state will obviously greatly improve
performance here, at Parallax Inc. they use such an environment for running
the layout/simulation software for their microcontrollers (I saw that on
their forum where the CTO reported about that).

The suggestion with the swap file was a quick solution to prove that the
software works and to have it up and running without much problems and
buying new hardware or rearanging the partitions.


PC: oS 11.3 64 bit | Intel Core2 Quad Q8300@2.50GHz | KDE 4.6.1 | GeForce
9600 GT | 4GB Ram
Eee PC 1201n: oS 11.4 64 bit | Intel Atom 330@1.60GHz | KDE 4.6.0 | nVidia
ION | 3GB Ram

On 2011-04-15 23:38, martin_helm wrote:
> The suggestion with the swap file was a quick solution to prove that the
> software works and to have it up and running without much problems and
> buying new hardware or rearanging the partitions.

Oh, yes, I understood that. I would have done the same. :slight_smile:


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

Hi Martin and Carlos, thanks for your additional information :slight_smile:

Martin (Medical Imaging): happy of meeting a colleague in this forum! I have a small team working on various subjects (in particular, Computer Assisted Detection systems for brain glioma, lung tumor, and Alzheimer’s Disease, in DTI, CT, and MRI respectively). Besides leading the team, I still do a lot of programming myself because (1) it amuses me a lot :slight_smile: [why should I let the others have fun, and just deal with politics, money and similar boring stuff??] and (2) in particular situations, or when starting a new work, I prefer to check myself how things can go. By the way, I am a physicist and work as a university researcher.

Carlos and Martin: in point of fact I already knew that I could (should) use the swap file just as a quick hack, and then change my disk partitions and have ad additional partition (or disk) devoted to paging. What was completely unknown to me was the “ulimit” stuff, and for this reason I’ll never stop saying thanks :slight_smile: Moreover, I had not thought of the possibility of using solid-state disks, of which I was almost unaware, and I did not know about priority, too.

Performance is an issue: my program has run for 24 hours, too slow! I’ll explore the solid-state disk option.

Thanks again!!
Sincerely
Giorgio