Requested - Authoritative Source on OpenSuSE kernels

English Install/Boot/Login
#1

OpenSuSE 11.3
2.6.34.7-0.5-desktop i686

Installation used default options.

Discovered that my “standard” Desktop kernel isn’t likely PAE enabled…

Have had various FF windows open for a long time even with no network connection which resulted in memory leaks (understandable). Eventually the machine slowed to a crawl with numerous FF processes running (each about 8% of CPU) and 4GB memory map (4GB physical RAM) exhausted but surprisingly the 2GB swap was totally untouched.

Searching the Forums there are numerous anecdotal opinions that PAE should be or was enabled automatically at least for Desktop kernels. Viewing the OpenSuSE repository, there are kernel-pae packages for this kernel version which aren’t installed.

So before I start installing packages willy-nilly, is there an authoritative published source that documents what is is in an OpenSuSE kernel package, and what combination of packages plus if necessary additional manual configurations to achieve desired goals?

BTW - I’m somewhat surprised that today PAE is not automatically included in today’s kernels considering how cheap hardware is… I don’t know if PAE typically should be a noticeable performance hit on low resource machines(like netbooks) and would be necessary for anything configured with more than 4GB total (physical plus swap) memory.

IMO and TIA,
Tony

#2

kernel-desktop is PAE enabled and is designed for desktop use, hence PAE is not selected by the installer

#3

Hmmm…

Then I’m still looking for enough documentation to explain the scenario I described. I’d like to definitively first eliminate the PAE enable issue through some official documentation before I start looking at other possible explanations (heap? restrictions on what can be swapped?).

Tony

#4

If you look in Yast > Software Management
What do the details show on kernel-desktop ( I can’t check ATM, I’m on 11.4 with kernel-default)

#5

Here is the detail from Yast

kernel-desktop - Kernel optimized for the desktop
This kernel is optimized for the desktop. It is configured for lower latency and has many of the features that aren’t usually used on desktop machines disabled. This kernel supports up to 64GB of main memory. It requires Physical Addressing Extensions (PAE), which were introduced with the Pentium Pro processor. PAE is not only more physical address space but also important for the “no execute” feature which disables execution of code that is marked as non-executable. Therefore, the PAE kernel should be used on any systems that support it, regardless of the amount of main memory.

#6

tsu2, what makes you think PAE is not working? PAE is not referring to any performance issues in general, but only the ability to address up to 64GiB of RAM (as opposed to actual 'default-Kernel, which addresses only up to 3.something GiB of memory).

#7

The 32-bit memory map totals 4GB. Typically an OS then partitions total addresses into those to be used in “User Mode” and “kernel mode,” the former typically for Applications and the latter typically for lower level, more basic/protected OS processing. When referring to “3GB of addressable memory” that usually means the way the 32-bit memory map is partitioned for User mode, which leaves 1GB for kernel mode addresses.

PAE extends the amount of usable memory addresses by taking a small number of addresses from the 32-bit memory map, remapping to the extra addresses beyond the now slightly less than 4GB memory address barrier.

So, without PAE, the limits of the 32-bit memory map should be considered a hard limit, total memory addresses should total 4GB and no more. On my system, I installed 32-bit OpenSuSE on a machine with 4GB physical RAM with a 2GB swap partition, totaling 6GB.

because total memory exceeds 4GB (total 6GB) , I would have expected that when top starts reporting that all 4GB of physical RAM is being used that applications should continue to run with only a slight degradation of performance due to having to access addresses using the swap file.

That didn’t happen, I experienced a hard limit at 4GB and the machine ground to an unusable state while reporting that the swap file was totally unused.

Tony

#8

OK,
I think I finally was able to Google an answer, and with strong kudos to the author of this reerence (perhaps the only one currently on the Internet in the English language)

LinuxRAMlimits

According to this post there is one very relevant comment

PAE extends physical addresses only (to 36 bits), and does nothing
for virtual space
.

That means that all “total” memory addresses may not be considered equal… PAE may not be able to access virtual addresses in the SWAP file, can only map real, physical memory beyond the 4GB barrier.

Something good to know, if installing 32-bit OS on a machine with 4GB or physical RAM, there is absolutely no point in configuring a swap partition/file.

Skimming the rest of the posts on this page, the only thing I found likely innacurate is a possible mis-spoke saying 1GB of addresses are allocated for I/O, somewhere a long time ago I remember it was more like about 16mb or so and in a very specific address range due to legacy BIOS architecture.

Other than that, everything re-affirms what I stated and much more.

Tony

#9

I don’t see how you can jump to that conclusion about swap. Swap isn’t addressed using memory mapping hardware.

#10

Also any single process can not address more then 4gig with or without PAE on 32bit. Without PAE 32 bit user address space is limited to 3gig with 1 gig reserved for the OS )an Intel thing). PAE introduces overhead into the process to “bank switch” address space.

#11

On 2010-11-20 22:36, tsu2 wrote:

> That didn’t happen, I experienced a hard limit at 4GB and the machine
> ground to an unusable state while reporting that the swap file was
> totally unused.

Have a look at “/proc/meminfo”. There is a value for “SwapTotal”. I think
“Vmalloc*” is mem plus swap available.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

#12

Is there a source description then that describes how swap (virtual) memory addresses are defined? If what you say were to be true, those addresses can’t be part of the 32-bit memory map since all such addresses have been exhausted applied to physical (real) memory. For what you say to be true, the virtual memory map must be larger than the default 32-bit memory map or a separate memory map (I should think).

And, then I would be back to wondering why I ran into a situation where the kernel absolutely refused to swap to the swap file when the situation very obviously needed to do so.

Tony

#13

cat /proc/meminfo

Interesting memory usage reporting.

This is interesting enough that I think I will intentionally create memory leaks again to exhaust physical memory, then take readings again.

Will be interesting to see first if the scenario can be replicated (likely) and if more detailed info improves on just using top and htop, but I’m going to first see if there is a way to report heaps usage.

Tony

#14

Just study some Operating System textbooks. There’s no need to map addresses in swap files 1:1 with virtual memory. When the memory mapping hardware detects an access to a non-existent page which causes an instruction fault, the process is suspended and the swapper reads the requested block into memory, then the memory maps are adjusted to point to this block, and the process is resumed. As far as the process is concerned, the memory access succeeded. Like Rip Van Winkle, it doesn’t know that a lot of stuff happened while it was suspended.

So the swap file bookkeeping consists of tracking the owning process and the block numbers held in swap.

All this is standard OS practice and even predates Unix. No need to go reimagine the wheel.

#15

On 2010-11-23 01:36, tsu2 wrote:

> cat /proc/meminfo
>
> Interesting memory usage reporting.
>
> This is interesting enough that I think I will intentionally create
> memory leaks again to exhaust physical memory, then take readings again.

You can do that easily with dd:

dd if=/dev/zero of=/dev/null bs=4G count=1

Will try to reserve 4GiB of ram (and fail on a 32 bit computer).

> Will be interesting to see first if the scenario can be replicated
> (likely) and if more detailed info improves on just using top and htop,
> but I’m going to first see if there is a way to report heaps usage.

I really think that swaped memory is listed in the memory map in the
“physical” address range, thus without PAE you are limited to 4 GiB of
swap, too. Total size, I mean.

Why? Suppose you have 2 gigs, and 2 swap, so 4 in total. If you address in
the program the address 3e9. The kernel sees it is not in ram, so that it
brings it from disk swap and “maps” that address to another in the 2 gig
range. But the software, which does not know where it is in fact, uses
address in the 3 gig range… so without pae, total (ram + swap) can not
exceed 4gig.

I don’t think I have been clear enough, perhaps somebody else understands
what I meant and explain it better?

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

#16

Something like that but not quite. On a non-PAE machine you can configure swap to be larger than 4GB, say 2x3GB.

Obviously the amount of swap that is useful to a single process is limited by the amount of virtual memory that it can address (ignoring bank switching schemes which have gone out of fashion). However swap is not addressed directly using memory mapping hardware. So there is no direct relationship between the max address space of a process and the configured size of the swap partition.

So it’s not true that in a PAE situation, swap is not useful. Remember that the swap partition is not dedicated to a single process but is a common OS resource. It extends the RAM space. Thus with a PAE machine you would still use the same rule of thumb connecting RAM space and swap space.

#17

ken_yap, thx very much for going the extra distance, so it does appear that swapping should “just work” and the swapping method simply maps to disk blocks and isn’t to addresses.

It looks then that the “hard limit” I ran into isn’t strictly a 32-bit memory map issue despite what top reported but something else(maybe memory exhaustion was just a consequence of something else). Better armed with some application memory usage tools, I’m going to try to replicate the problem over the 4 day holiday and see if I can determine what is happening.

Tony

#18

Carlos,
If I understand what ken_yap posted, you are suffering from the same mis-conception I had that the inside of the swap file is address-mapped either as an extension of or similar to the regular addressable memory map (4GB in a 32-bit system, orders of magnitude larger in a PAE or 64-bit system), so that when an address is accessed in the regular memory map, it might point to a virtual address within the swap file.

I believe that what ken_yap posted however states that there isn’t a memory map using similar addresses within the swap file, the swap file is using some other method to locate the data written to swap, designated as “blocks” which I assume may be physical disk blocks or a virtual version thereof, maybe using methods similar to a physical disk table (a pointer to the beginning of a series of blocks plus the number of contiguous blocks).

If that is any way close to what ken_yap posted, then the capability of the swap file is completely disconnected from, not related to and not restricted by the system’s memory map.

Tony

#19

On 2010-11-25 01:06, tsu2 wrote:
>
> robin_listas;2256611 Wrote:

> Carlos,
> If I understand what ken_yap posted, you are suffering from the same
> mis-conception I had that the inside of the swap file is address-mapped
> either as an extension of or similar to the regular addressable memory
> map (4GB in a 32-bit system, orders of magnitude larger in a PAE or
> 64-bit system), so that when an address is accessed in the regular
> memory map, it might point to a virtual address within the swap file.

No, you haven’t understood what I meant.

We are talking of a non-pae system. The kernel can not address more than 4
GiB, and that memory can be all RAM, or part ram, part swap. The address
space is 4 GiB, ie, 2³². A process, including the kernel, can not address
anything beyond 2³². It simply can not store any more in a register - and
there is no mapping, it is a non-pae kernel.

So, suppose there is 2 GiB in ram, and 4 GiB in swap. In this situation
(non-pae, remember) the maximum total addressable memory is 4 GiB - not 6.
If you have 4 GiB ram, swap will not be used at all, except for hibernation.

And this concurs with what you originally reported. I could try to
replicate this, but it is a long procedure, several hours, so I will not :slight_smile:

Even if swap is addressed differently, it does not matter: processes do not
know they are using swap. An address has to fit in the address register, 32
bits.

When a pae-kernel is used, then the addressing method changes, there is
mapping, and the kernel can address those 6 GiB (4 GiB per process), ram +
swap.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

#20

On 2010-11-23 05:06, ken yap wrote:
>
> robin_listas;2256611 Wrote:
>> I really think that swaped memory is listed in the memory map in the
>> “physical” address range, thus without PAE you are limited to 4 GiB of
>> swap, too. Total size, I mean.
>
> Something like that but not quite. On a non-PAE machine you can
> configure swap to be larger than 4GB, say 2x3GB.
>
> Obviously the amount of swap that is useful to a single process is
> limited by the amount of virtual memory that it can address (ignoring
> bank switching schemes which have gone out of fashion). However swap is
> not addressed directly using memory mapping hardware. So there is no
> direct relationship between the max address space of a process and the
> configured size of the swap partition.

That is true - but in non-pae the total virtual memory space is 4 GiB,
because there is no mapping. You may have 2 GiB ram and 8 GiB swap, but the
virtual space available to the kernel is 4 GiB.

The same combination with PAE would give 10 GiB virtual memory.

That’s the important poing: non-pae means no mapping. Forget swap, if you
have 8 GiB ram only 4 are visible. Swap will not add to it.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)