KVM shows wrong number of CPU's

senblue · June 27, 2012, 2:04pm

On my Fujitsu Primergy RX300S6 i got 2x Intel Xeon CPU X5650
and the Out-Of-The-Box-Xen-Kernel shows me 24 CPU’s.
Also the virt-manager offers me 24 CPU’s to use for my virtual machines.

When i start the machine with the Standard-Dektop-Kernel the kernel
only sees 12 CPU’s.

Below is the output of lscpu for both kernels.

According to /boot/config-3.1.9-1.4-desktop and /boot/config-3.1.9-1.4-xen
the number of CPU’s is set to 512 (CONFIG_NR_CPUS=512).

How can i teach my kernel to see the rest of the CPU’s ?

Any hint will be greatly appreciated.

Thanks

Xen – openSUSE 12.1 - 3.1.9-1.4

server01:~ # lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 24
Vendor ID: GenuineIntel
CPU family: 6
Model: 44
Stepping: 2
CPU MHz: 2666.832
BogoMIPS: 5401.35
Hypervisor vendor: Xen
Virtualization type: none
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K

Desktop – openSUSE 12.1 - 3.1.9-1.4

server01:~ # lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 1
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 44
Stepping: 2
CPU MHz: 2666.988
BogoMIPS: 5332.55
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0-11

lwfinger · June 27, 2012, 5:37pm

On 06/27/2012 07:06 AM, senblue wrote:
>
> On my Fujitsu Primergy RX300S6 i got 2x Intel Xeon CPU X5650
> and the Out-Of-The-Box-Xen-Kernel shows me 24 CPU’s.
> Also the virt-manager offers me 24 CPU’s to use for my virtual
> machines.
>
> When i start the machine with the Standard-Dektop-Kernel the kernel
> only sees 12 CPU’s.
>
> Below is the output of lscpu for both kernels.
>
> According to /boot/config-3.1.9-1.4-desktop and
> /boot/config-3.1.9-1.4-xen
> the number of CPU’s is set to 512 (CONFIG_NR_CPUS=512).
>
> How can i teach my kernel to see the rest of the CPU’s ?
>
> Any hint will be greatly appreciated.

You should compare the two versions of dmesg for a clue regarding why the
desktop only sees half of the CPUs. It may be a kernel bug.

senblue · June 28, 2012, 6:34pm

I dont want to spam this board with excessive listings
therefore below only some of the output of dmesg concerning cpus.
And only from the desktop-kernel.
The kernel sees only one cpu (CPU0).
There is no obvious error message, while booting
the desktop-kernel.
Or am i missing something ?

As of my initial posting lscpu shows
two sockets, six cores per socket an one thread
per core , so 2x6x1 => 12 logical cpus

I made an online update to kernel 3.1.10-1.9
and tried the default-kernel and the vanilla-kernel.
Always the same result.

Booting an ubuntu-12.04-live-system with kernel 3.2.0-23-generic
gives me the 24 cpus like the xen-kernel of opensuse 12.1.

One can say, why not use the opensuse-xen-kernel ?
I changed to kvm because of totally unpredictable and
unexplainable system freezes with the xen-kernel.

Possibly these freezes where caused by bugs in the acpi-code,
because a second RX300 running stable for two month with acpi=off.

But i invested lot of time in buildding the virtual machines under
KVM. So i would prefer to stay with it.

0.000000] Initializing cgroup subsys cpuset
0.000000] Initializing cgroup subsys cpu
0.000000] Linux version 3.1.9-1.4-desktop (geeko@buildhost) (gcc version 4.6.2 (SUSE Linux) ) #1 SMP PREEMPT Fri Jan 27 08:55:10 UTC 2012 (efb5ff4)
0.000000] Command line: root=/dev/system/root resume=/dev/system/swap splash=silent quiet showopts acpi=off vga=0x31a
0.000000] BIOS-provided physical RAM map:
0.000000]  BIOS-e820: 0000000000000000 - 000000000009c400 (usable)
0.000000]  BIOS-e820: 000000000009c400 - 00000000000a0000 (reserved)
0.000000]  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
0.000000]  BIOS-e820: 0000000000100000 - 00000000bf7a0000 (usable)
0.000000]  BIOS-e820: 00000000bf7a0000 - 00000000bf7b9000 (ACPI data)
0.000000]  BIOS-e820: 00000000bf7b9000 - 00000000bf7ba000 (ACPI NVS)
0.000000]  BIOS-e820: 00000000bf7ba000 - 00000000c0000000 (reserved)
0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
0.000000]  BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
0.000000]  BIOS-e820: 00000000ffa00000 - 0000000100000000 (reserved)
0.000000]  BIOS-e820: 0000000100000000 - 0000002440000000 (usable)
0.000000] NX (Execute Disable) protection: active
0.000000] DMI present.
0.000000] DMI: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.08.2619.N1           10/06/2010
0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==&gt; (reserved)
0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
0.000000] No AGP bridge found
0.000000] last_pfn = 0x2440000 max_arch_pfn = 0x400000000
0.000000] MTRR default type: uncachable
0.000000] MTRR fixed ranges enabled:
0.000000]   00000-9FFFF write-back
0.000000]   A0000-BFFFF uncachable
0.000000]   C0000-FFFFF write-protect
0.000000] MTRR variable ranges enabled:
0.000000]   0 base 00C0000000 mask FFC0000000 uncachable
0.000000]   1 base 0000000000 mask E000000000 write-back
0.000000]   2 base 2000000000 mask FC00000000 write-back
0.000000]   3 base 2400000000 mask FFC0000000 write-back
0.000000]   4 disabled
0.000000]   5 disabled
0.000000]   6 disabled
0.000000]   7 disabled
0.000000]   8 disabled
0.000000]   9 disabled
0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
0.000000] e820 update range: 00000000c0000000 - 0000000100000000 (usable) ==&gt; (reserved)
0.000000] last_pfn = 0xbf7a0 max_arch_pfn = 0x400000000
0.000000] found SMP MP-table at [ffff8800000f7ef0] f7ef0
0.000000] initial memory mapped : 0 - 20000000
0.000000] Base memory trampoline at [ffff880000097000] 97000 size 20480
0.000000] Using GB pages for direct mapping
0.000000] init_memory_mapping: 0000000000000000-00000000bf7a0000
0.000000]  0000000000 - 0080000000 page 1G
0.000000]  0080000000 - 00bf600000 page 2M
0.000000]  00bf600000 - 00bf7a0000 page 4k
0.000000] kernel direct mapping tables up to bf7a0000 @ 1fffd000-20000000
0.000000] init_memory_mapping: 0000000100000000-0000002440000000
0.000000]  0100000000 - 2440000000 page 1G
0.000000] kernel direct mapping tables up to 2440000000 @ bf79f000-bf7a0000
0.000000] RAMDISK: 37b34000 - 37ff0000
0.000000] No NUMA configuration found
0.000000] Faking a node at 0000000000000000-0000002440000000
0.000000] Initmem setup node 0 0000000000000000-0000002440000000
0.000000]   NODE_DATA [000000243ffec000 - 000000243fffffff]
0.000000]  [ffffea0000000000-ffffea007edfffff] PMD -&gt; [ffff8823af600000-ffff88242d5fffff] on node 0
0.000000] Zone PFN ranges:
0.000000]   DMA      0x00000010 -&gt; 0x00001000
0.000000]   DMA32    0x00001000 -&gt; 0x00100000
0.000000]   Normal   0x00100000 -&gt; 0x02440000
0.000000] Movable zone start PFN for each node
0.000000] early_node_map[3] active PFN ranges
0.000000]     0: 0x00000010 -&gt; 0x0000009c
0.000000]     0: 0x00000100 -&gt; 0x000bf7a0
0.000000]     0: 0x00100000 -&gt; 0x02440000
0.000000] On node 0 totalpages: 37746476
0.000000]   DMA zone: 56 pages used for memmap
0.000000]   DMA zone: 5 pages reserved
0.000000]   DMA zone: 3919 pages, LIFO batch:0
0.000000]   DMA32 zone: 14280 pages used for memmap
0.000000]   DMA32 zone: 765912 pages, LIFO batch:31
0.000000]   Normal zone: 505344 pages used for memmap
0.000000]   Normal zone: 36456960 pages, LIFO batch:31
0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
0.000000] Intel MultiProcessor Specification v1.4
0.000000] MPTABLE: OEM ID: FSCD2619
0.000000] MPTABLE: Product ID: PRIMERGY    
0.000000] MPTABLE: APIC at: 0xFEE00000
0.000000] Processor #0 (Bootup-CPU)
0.000000] Processor #2
0.000000] Processor #4
0.000000] Processor #16
0.000000] Processor #18
0.000000] Processor #20
0.000000] Processor #32
0.000000] Processor #34
0.000000] Processor #36
0.000000] Processor #48
0.000000] Processor #50
0.000000] Processor #52
0.000000] IOAPIC[0]: apic_id 6, version 32, address 0xfec00000, GSI 0-23
0.000000] IOAPIC[1]: apic_id 7, version 32, address 0xfec80000, GSI 24-47
0.000000] Processors: 12
0.000000] SMP: Allowing 12 CPUs, 0 hotplug CPUs
0.000000] nr_irqs_gsi: 64
0.000000] PM: Registered nosave memory: 000000000009c000 - 000000000009d000
0.000000] PM: Registered nosave memory: 000000000009d000 - 00000000000a0000
0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e4000
0.000000] PM: Registered nosave memory: 00000000000e4000 - 0000000000100000
0.000000] PM: Registered nosave memory: 00000000bf7a0000 - 00000000bf7b9000
0.000000] PM: Registered nosave memory: 00000000bf7b9000 - 00000000bf7ba000
0.000000] PM: Registered nosave memory: 00000000bf7ba000 - 00000000c0000000
0.000000] PM: Registered nosave memory: 00000000c0000000 - 00000000e0000000
0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000
0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fed00000
0.000000] PM: Registered nosave memory: 00000000fed00000 - 00000000fee00000
0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ffa00000
0.000000] PM: Registered nosave memory: 00000000ffa00000 - 0000000100000000
0.000000] Allocating PCI resources starting at c0000000 (gap: c0000000:20000000)
0.000000] Booting paravirtualized kernel on bare hardware
0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:12 nr_node_ids:1
0.000000] PERCPU: Embedded 27 pages/cpu @ffff88243fc00000 s78208 r8192 d24192 u131072
0.000000] pcpu-alloc: s78208 r8192 d24192 u131072 alloc=1*2097152
0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 -- -- -- -- 
0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 37226791
0.000000] Policy zone: Normal
0.000000] Kernel command line: root=/dev/system/root resume=/dev/system/swap splash=silent quiet showopts acpi=off vga=0x31a
0.000000] bootsplash: silent mode.
0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
0.000000] Checking aperture...
0.000000] No AGP bridge found
0.000000] Memory: 148832796k/152043520k available (5797k kernel code, 1057616k absent, 2153108k reserved, 7771k data, 940k init)
0.000000] Preemptible hierarchical RCU implementation.
0.000000] NR_IRQS:33024 nr_irqs:1184 16
0.000000] Console: colour dummy device 80x25
0.000000] console [tty0] enabled
0.000000] allocated 1207959552 bytes of page_cgroup
0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
0.001000] Fast TSC calibration using PIT
0.002000] Detected 2666.988 MHz processor.
0.000004] Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.97 BogoMIPS (lpj=2666988)
0.000007] pid_max: default: 32768 minimum: 301
0.000063] Security Framework initialized
0.000076] AppArmor: AppArmor initialized
0.018399] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes)
0.090049] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes)
0.120631] Mount-cache hash table entries: 256
0.120761] Initializing cgroup subsys cpuacct
0.120765] Initializing cgroup subsys memory
0.120785] Initializing cgroup subsys devices
0.120788] Initializing cgroup subsys freezer
0.120790] Initializing cgroup subsys net_cls
0.120791] Initializing cgroup subsys blkio
0.120797] Initializing cgroup subsys perf_event
0.120825] CPU: Physical Processor ID: 0
0.120826] CPU: Processor Core ID: 0
0.120831] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
0.120831] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
0.120834] mce: CPU supports 9 MCE banks
0.120843] CPU0: Thermal monitoring enabled (TM1)
0.120849] using mwait in idle threads.
0.121600] Switched APIC routing to physical flat.
0.121710] ExtINT not setup in hardware but reported by MP table
0.122053] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
0.132030] CPU0: Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz stepping 02
0.335108] Performance Events: PEBS fmt1+, Westmere events, Intel PMU driver.
0.335114] ... version:                3
0.335115] ... bit width:              48
0.335116] ... generic registers:      4
0.335117] ... value mask:             0000ffffffffffff
0.335118] ... max period:             000000007fffffff
0.335119] ... fixed-purpose events:   3
0.335120] ... event mask:             000000070000000f
0.343573] NMI watchdog enabled, takes one hw-pmu counter.
0.351083] Booting Node   0, Processors  #1
0.351087] smpboot cpu 1: start_ip = 97000
0.441928] NMI watchdog enabled, takes one hw-pmu counter.
0.447825]  #2
0.447827] smpboot cpu 2: start_ip = 97000
0.538666] NMI watchdog enabled, takes one hw-pmu counter.
0.544556]  #3
0.544558] smpboot cpu 3: start_ip = 97000
0.635396] NMI watchdog enabled, takes one hw-pmu counter.
0.641291]  #4
0.641293] smpboot cpu 4: start_ip = 97000
0.732134] NMI watchdog enabled, takes one hw-pmu counter.
0.738020]  #5
0.738022] smpboot cpu 5: start_ip = 97000
0.828860] NMI watchdog enabled, takes one hw-pmu counter.
0.834751]  #6
0.834753] smpboot cpu 6: start_ip = 97000
0.925576] NMI watchdog enabled, takes one hw-pmu counter.
0.931478]  #7
0.931480] smpboot cpu 7: start_ip = 97000
1.022304] NMI watchdog enabled, takes one hw-pmu counter.
1.028210]  #8
1.028212] smpboot cpu 8: start_ip = 97000
1.119036] NMI watchdog enabled, takes one hw-pmu counter.
1.124942]  #9
1.124944] smpboot cpu 9: start_ip = 97000
1.215767] NMI watchdog enabled, takes one hw-pmu counter.
1.221673]  #10
1.221676] smpboot cpu 10: start_ip = 97000
1.312501] NMI watchdog enabled, takes one hw-pmu counter.
1.318409]  #11 Ok.
1.318411] smpboot cpu 11: start_ip = 97000
1.409275] NMI watchdog enabled, takes one hw-pmu counter.
1.411132] Brought up 12 CPUs
1.411135] Total of 12 processors activated (63991.89 BogoMIPS).
1.419679] devtmpfs: initialized
1.449907] PM: Registering ACPI NVS region at bf7b9000 (4096 bytes)
1.450050] print_constraints: dummy: 
1.450079] RTC time:  9:49:07, date: 06/27/12
1.450140] NET: Registered protocol family 16
1.450359] PCI: Using configuration type 1 for base access
1.462855] bio: create slab &lt;bio-0&gt; at 0
1.462968] ACPI: Interpreter disabled.

lwfinger · June 28, 2012, 7:09pm

On 06/28/2012 11:36 AM, senblue wrote:
>
> lwfinger;2471612 Wrote:

>> You should compare the two versions of dmesg for a clue regarding why
>> the
>> desktop only sees half of the CPUs. It may be a kernel bug.
>
>
> I dont want to spam this board with excessive listings
> therefore below only some of the output of dmesg concerning cpus.

If you read carefully, I never asked for you to post the dmesg output, but
recommended that you look at it carefully.

> Booting an ubuntu-12.04-live-system with kernel 3.2.0-23-generic
> gives me the 24 cpus like the xen-kernel of opensuse 12.1.

Obviously, that kernel bug was fixed between 3.1.0 and 3.2.0. You need to
implement a later version, either by installing one from the openSuSE kernel
repos, or by downloading the kernel source for 3.4.4 and building your own kernel.

senblue · June 29, 2012, 5:34pm

lwfinger:

On 06/28/2012 11:36 AM, senblue wrote:
>
> lwfinger;2471612 Wrote:

>> You should compare the two versions of dmesg for a clue regarding why
>> the
>> desktop only sees half of the CPUs. It may be a kernel bug.
>
>
> I dont want to spam this board with excessive listings
> therefore below only some of the output of dmesg concerning cpus.

If you read carefully, I never asked for you to post the dmesg output, but
recommended that you look at it carefully.

I thought the with the (minimal) dmesg output you or somebody
else would get a clue what is wrong.

> Booting an ubuntu-12.04-live-system with kernel 3.2.0-23-generic
> gives me the 24 cpus like the xen-kernel of opensuse 12.1.

Obviously, that kernel bug was fixed between 3.1.0 and 3.2.0. You need to
implement a later version, either by installing one from the openSuSE kernel
repos, or by downloading the kernel source for 3.4.4 and building your own kernel.

Unfortunately it seemed not to b a kernel bug. I installed kernel-default-3.4.4-1.1.x86_64.rpm
and the problem remains.
I read the dmesg output, found the line:
0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:12 nr_node_ids:1

The NR_CPUS expands to CONFIG_NR_CPUS in the kernel-config. But i did not find something
like CONFIG_NR_CPU_IDS. After some googling i assumed that its somehow related to or controlled
by acpi. Since my machine is booted with “acpi=off” i changed that and voila got the
full numeber of 24 cpus.
But acpi was off for a reason (unpredictable server freezes). So i will give acpi=ht a shot.
It should disable ACPI system almost completely, but allows hyperthreading.

I wonder why the xen-kernel gives me my full 24 cpus in spite of acpi=off.
It must be some kernel-configuration i am not aware of. And i think there is lot of it .

lwfinger · June 29, 2012, 7:15pm

On 06/29/2012 10:36 AM, senblue wrote:
>
> lwfinger;2471785 Wrote:
>> On 06/28/2012 11:36 AM, senblue wrote:
>>>
>>> lwfinger;2471612 Wrote:
>>
>>>> You should compare the two versions of dmesg for a clue regarding
>> why
>>>> the
>>>> desktop only sees half of the CPUs. It may be a kernel bug.
>>>
>>>
>>> I dont want to spam this board with excessive listings
>>> therefore below only some of the output of dmesg concerning cpus.
>>
>> If you read carefully, I never asked for you to post the dmesg output,
>> but
>> recommended that you look at it carefully.
>>
>> I thought the with the (minimal) dmesg output you or somebody
>> else would get a clue what is wrong.
>>
>>> Booting an ubuntu-12.04-live-system with kernel 3.2.0-23-generic
>>> gives me the 24 cpus like the xen-kernel of opensuse 12.1.
>>
>> Obviously, that kernel bug was fixed between 3.1.0 and 3.2.0. You need
>> to
>> implement a later version, either by installing one from the openSuSE
>> kernel
>> repos, or by downloading the kernel source for 3.4.4 and building your
>> own kernel.
>
>
> Unfortunately it seemed not to b a kernel bug. I installed
> kernel-default-3.4.4-1.1.x86_64.rpm
> and the problem remains.
> I read the dmesg output, found the line:
> 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:12
> nr_node_ids:1
> CONFIG_NR_CPUS
> The NR_CPUS expands to CONFIG_NR_CPUS in the kernel-config. But i did
> not find something
> like CONFIG_NR_CPU_IDS. After some googling i assumed that its somehow
> related to or controlled
> by acpi. Since my machine is booted with “acpi=off” i changed that and
> voila got the
> full numeber of 24 cpus.
> But acpi was off for a reason (unpredictable server freezes). So i will
> give acpi=ht a shot.
> It should disable ACPI system almost completely, but allows
> hyperthreading.

The CONFIG_NR_CPUS parameter controls the maximum number that are allowed. The
discovery process while booting tells you how many there are. The standard
setting for the kernel parameter CONFIG_SCHED_SMT is “y”, thus they should be
enabled.

The thread at http://www.gossamer-threads.com/lists/linux/kernel/1182330 has
some discussion on the value of enabling the HT pseudo-CPUs.