Sure, but swap will allow you to cram more running processes into that 4GB until the performance suffers due to the working set exceeding RAM space. That has always been swap’s role, to free up valuable RAM space by paging out currently unused pages. Nobody claims that swap extends the memory addressing space.
On 2010-11-25 02:36, ken yap wrote:
>
> robin_listas;2257460 Wrote:
>> That’s the important poing: non-pae means no mapping. Forget swap, if
>> you have 8 GiB ram only 4 are visible. Swap will not add to it.
>
> Sure, but swap will allow you to cram more running processes into that
> 4GB until the performance suffers due to the working set exceeding RAM
> space. That has always been swap’s role, to free up valuable RAM space
> by paging out currently unused pages. Nobody claims that swap extends
> the memory addressing space.
Yes, see the original post.
>> Have had various FF windows open for a long time even with no network
>> connection which resulted in memory leaks (understandable). Eventually
>> the machine slowed to a crawl with numerous FF processes running (each
>> about 8% of CPU) and 4GB memory map (4GB physical RAM) exhausted but
>> surprisingly the 2GB swap was totally untouched.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
Again, if I understand what ken_yap posted what you state is not true…
Given…
Yes, a non-PAE system has a maximum 4GB memory map. Yes, this means that memory addresses <directly accessible> by the system (both User and System) totals 4GB.
Does this mean that a hard limit of 4GB exists, that all processes must fit within this space and no more space?
Apparently not, and it’s because the way swap file contents is mapped… <not> using a method extending the method that’s used to address main memory but using a <different> method. By <not> using a method with limits (the 32-bit memory map), it’s possible to map additional resources, although obviously at cost.
If this is how things work, then I think the following are differences between PAE and swap mapping…
- PAE is fixed. Specific directly addressable addresses within the 32-bit memory map (4GB total addresses) are mapped to specific blocks of addresses beyond the 4GB barrier. Configured prior to the main OS loading, the OS “knows” exactly where everything is located in the extended memory map. The extended memory map is not automatically 64GB(a virtual memory map), it’s limited to the amount of physical (real) memory in the system.
- Swap mapping is dynamic. specific addresses within the 32-bit memory map are mapped <not> to specific blocks of addresses but to some other “block” measurement, possibly actual physical disk blocks, maybe virtual blocks of some sort. This means that it’s likely when the OS looks for data in swap, it is an “on demand” process which has to execute a swap file lookup because the data isn’t assigned to a specific fixed address.
- PAE is mapped based on the amount of physical memory in the system, so I would expect all available memory addresses are mapped prior to loading the OS and the usage of directly addressable addresses to map to extended addresses depends on the number of extended addresses.
- Swap is dyanmic, so I’m guessing that unlike PAE it’s likely that there isn’t a fixed ratio of pointers in addressable memory to blocks in swap, it’s more likely based on tasks or processes (CPU unit) and created (and possibly destroyed) as needed.
Still learning and someone could still post I’m all wrong,
Tony
32 bit is 32 bit no process can have to itself more then 4 gig addresses. But the OS in PAE can map multiple processes to a larger address space up to 64 gig. In non-PAE there is a further limitation in that 1gig is reserved for the OS and total user space is 3gig. PAE is a kludge and can introduces a 5-10% performance hit. Better to just go to 64bit.
In my investigations on this subject, it’s noted that the 1GB reserved for “System” (OS most protected kernel processes) is only default and can be modified. Since unlike Windows practically all applications run only in Linux User space this can more efficiently apportion the use of physical RAM(Windows runs authentication/authorization and several Server apps in kernel mode instead). Yes, PAE is a kludge and depending on available resources may be a noticeable drag but like many things YMMV.
I’d have to doublecheck to be certain but IIRC you’re right that for any single process, today’s OS will present a virtual memory map so that the process is fooled into believing it has the whole available address space to itself… The OS will do the housekeeping though to actually manage the co-existence of and how simultaneous running processes actually operate and that applies regardless whether you’re talking about PAE or not. But, unless you’re running a complex database or other type of application I’m not sure people will be concerned with this restriction, I haven’t yet even on most Server apps I’ve supported because as a matter of practice I try to make all Server calls and processing as short as possible.
Tony
On 2010-11-26 18:36, gogalthorp wrote:
>
> 32 bit is 32 bit no process can have to itself more then 4 gig
> addresses. But the OS in PAE can map multiple processes to a larger
> address space up to 64 gig. In non-PAE there is a further limitation
> in that 1gig is reserved for the OS and total user space is 3gig. PAE is
> a kludge and can introduces a 5-10% performance hit. Better to just go
> to 64bit.
Both PAE and non PAE kernels in 32 bit give 3 GiB to processes and keep 1
to the kernel. The difference is that with PAE there are several 3 GiB blocks.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
On 2010-11-26 17:36, tsu2 wrote:
>
> Again, if I understand what ken_yap posted what you state is not
> true…
No, you are mistaken.
> Given…
> Yes, a non-PAE system has a maximum 4GB memory map. Yes, this means
> that memory addresses <directly accessible> by the system (both User and
> System) totals 4GB.
>
> Does this mean that a hard limit of 4GB exists, that all processes must
> fit within this space and no more space?
Yes. Absolutely.
> Apparently not, and it’s because the way swap file contents is
> mapped… <not> using a method extending the method that’s used to
> address main memory but using a <different> method. By <not> using a
> method with limits (the 32-bit memory map), it’s possible to map
> additional resources, although obviously at cost.
In non PAE mode, this is irrelevant. Yes, swap is mapped differently, but
nevertheless, you can not address more than 4 GiB because registers are 32
bits and there is no mapping (non pae mode). Total usable memory is 4 GiB
adding both RAM and SWAP. This is a hard limit.
You have the proof is the first post of this thread, system full and
absolutely no swap was used.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
You were doing fine until you wrote this. You don’t add RAM and SWAP. They are different classes of storage, not comparable. Total usable RAM is 4GiB. SWAP does not add to RAM. It merely assists in cramming a larger set of processes into 4GiB.
As for why swap was not used in his case, who knows why the system did not move some pages out of RAM to swap. One should not draw general conclusions from single observations.
To expand on how swap makes it possible to put more processes in, here are the situations with and without swap.
Without swap, all processes must fit within 4GiB RAM. Note that execution addresses in different processes can and will overlap. As a matter of fact, Linux processes are linked to start at the second page of memory space. (The first page is mapped as invalid for catching accesses via the null pointer.) However that’s not a problem, that’s what memory address mapping hardware is meant to do. The processes reside in various physical addresses in RAM. In fact they will be scattered all over the physical address space.
With swap, it’s the same, except that the OS can overcommit RAM. Processes can have pages moved out of RAM into swap and the physical memory page that was occupied given to another process. So the sum of sizes of all the processes can exceed 4GiB. However you can’t do too much of this. When processes continually use a working set of pages and this set will not fit into memory, then the system spends a lot of time bringing in pages from swap when a process runs, only to get them thrown out when a competing process runs. This is known as thrashing and the system slows down dramatically.
So in summary, swap is not addressed via the virtual memory mechanism, and swap does not extend the memory space. It only allows the OS to overcommit the existing memory space.
On 2010-11-27 00:06, ken yap wrote:
>
> robin_listas;2258209 Wrote:
>> Total usable memory is 4 GiB
>> adding both RAM and SWAP. This is a hard limit.
>
> You were doing fine until you wrote this. You don’t add RAM and SWAP.
> They are different classes of storage, not comparable. Total usable RAM
> is 4GiB. SWAP does not add to RAM. It merely assists in cramming a
> larger set of processes into 4GiB.
No, you are misinterpreting what swap is.
Non-pae. 2 GiB ram, 2 GiB swap: total usable memory (aka virtual memory) is
way more than the 2 GiB ram, and less than the 4 GiB limit. How much
exactly I do not know, but it adds.
> As for why swap was not used in his case, who knows why the system did
> not move some pages out of RAM to swap. One should not draw general
> conclusions from single observations.
I don’t.
I draw the conclusion from computer science - and the conclusion is
supported by tests. Hypothesis -> experiments.
Ok, you don’t believe me. Get a good kernel dev and ask him.
Or, write a Bugzilla complaining that with a non pae kernel, 4 GiB ram and
4 GiB swap, swap is not used; then wait for the closing as INVALID.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
On 2010-11-27 01:36, ken yap wrote:
> So in summary, swap is not addressed via the virtual memory mechanism,
Correct.
> and swap does not extend the memory space.
Incorrect.
> It only allows the OS to
> overcommit the existing memory space.
Partly correct.
If you have (non-pae) 1 GiB ram and 4 GiB swap, you can have a single
process with 3 GiB used memory: the process believes it has 3 GiB of ram,
even if its not true, most of it is disk.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
When I write extend the memory space, it is commonly understood that this means increase the range of addressable RAM. Swap plainly does not do that. It allows some of the process pages to live on disk some of the time.
As a counter example, consider a machine that has 1GiB RAM and a process that has a footprint of say 10MiB. It can however mmap(2) in a memory page above 2GiB (process address space, not hardware address space). Obviously the mapped hardware addresses of the page will be within the existing 1GiB RAM. However the address as seen by the process is truly > 2GiB, or 0x80000000. This can be done regardless of whether or not swap is active.
I think you may be failing to take into account the memory mapping hardware in your mental model.
If that’s not your understanding, I shall leave you to live in unenlightenment then. 