Kernel Paging And Eeek Problems

Hello,

I have a machine running openSuSE 10.3 (64-Bit) Linux OS acting as an E-Mail spam filter gateway; small mail flow, about 500-1,000 messages a day.

Hardware: Intel Core2Duo, 2GB Ram.
Software: ASSP, MailArchiva, Postfix, SpamAssassin, Spamass-Milter, Annoyance-Filter, and CACTI.
Linux mail 2.6.22.13-0.3-default #1 SMP 2007/11/19 15:02:58 UTC x86_64 x86_64 x86_64 GNU/Linux

About once a month, the machine would hang and have to be rebooted; however, this has been occurring more frequently. I know enough to setup and run a Linux machine, but not enough to do serious debug work.

I've disabled applications individually in hopes of discovering a offending program and still have yet to experience increased stability. Prior, I had SLES SP1 (64-Bit) installed on the same hardware with the same applications installed and still had Kernel problems; somewhat different, of the panicking variety. I can't determine if this is a hardware, software, or 64-Bit issue.

Below are two snippets from the messages log; the errors occur frequently and don't seem to be a direct cause of a single program. I'll be happy to provide further any information about the system and software or log dumps on request.

Jan 22 18:29:34 mail kernel: Unable to handle kernel paging request at 0000000001000000 RIP:
Jan 22 18:29:34 mail kernel:**[<ffffffff80290910>] do_lookup+0x3e/0x1ae
Jan 22 18:29:34 mail kernel: PGD 31ecc067 PUD 12321067 PMD 0
Jan 22 18:29:34 mail kernel: Oops: 0000 [1] SMP
Jan 22 18:29:34 mail kernel: last sysfs file: /devices/pci0000:00/0000:00:1f.3/i2c-adapter/i2c-0/name
Jan 22 18:29:34 mail kernel: CPU 0
Jan 22 18:29:34 mail kernel: Modules linked in: st ide_disk ide_cd ide_core af_packet nls_utf8 joydev ip6t_LOG nf_conntrack_ipv6 xt_pkttype ipt_LOG xt_limit snd_pcm_oss snd_mixer_oss snd_seq ip6t_REJECT xt_tcpudp ipt_REJECT xt_state iptable_mangle iptable_nat nf_nat iptable_filter snd_seq_device ip6table_mangle nf_conntrack_ipv4 nf_conntrack nfnetlink cpufreq_conservative cpufreq_userspace ip_tables cpufreq_powersave ip6table_filter acpi_cpufreq ip6_tables x_tables ipv6 microcode firmware_class apparmor loop dm_mod parport_pc parport floppy i2c_i801 i2c_core rtc_cmos r8169 rtc_core iTCO_wdt sr_mod cdrom rtc_lib iTCO_vendor_support snd_hda_intel snd_pcm snd_timer snd soundcore snd_page_alloc button intel_agp sg sd_mod ehci_hcd uhci_hcd usbcore edd ext3 mbcache jbd fan pata_jmicron ata_piix ahci libata scsi_mod thermal processor
Jan 22 18:29:34 mail kernel: Pid: 14425, comm: pyzor Tainted: G******N 2.6.22.13-0.3-default #1
Jan 22 18:29:34 mail kernel: RIP: 0010:[<ffffffff80290910>]**[<ffffffff80290910>] do_lookup+0x3e/0x1ae
Jan 22 18:29:34 mail kernel: RSP: 0018:ffff81003d0d7c08**EFLAGS: 00010206
Jan 22 18:29:34 mail kernel: RAX: 0000000001000000 RBX: ffff81001abe58a0 RCX: 0000000000000000
Jan 22 18:29:34 mail kernel: RDX: 0000000000000000 RSI: ffff81004f218028 RDI: ffff81001abe5958
Jan 22 18:29:34 mail kernel: RBP: ffff81003d0d7ea8 R08: 0000000000000001 R09: 0000000000000000
Jan 22 18:29:34 mail kernel: R10: ffff81007d209450 R11: ffffffff882c0c0a R12: ffff810015e8ad18
Jan 22 18:29:34 mail kernel: R13: 0000000000000000 R14: ffff81003d0d7ea8 R15: ffff81003d0d7c88
Jan 22 18:29:34 mail kernel: FS:**00002b08fa195130(0000) GS:ffffffff80500000(0000) knlGS:0000000000000000
Jan 22 18:29:34 mail kernel: CS:**0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 22 18:29:34 mail kernel: CR2: 0000000001000000 CR3: 0000000016ecd000 CR4: 00000000000006e0
Jan 22 18:29:34 mail kernel: Process pyzor (pid: 14425, threadinfo ffff81003d0d6000, task ffff8100097f77d0)
Jan 22 18:29:34 mail kernel: Stack:**ffff810015e841ed ffff81003d0d7c98 ffff81007d62b0c0 ffff81003d0d41ed
Jan 22 18:29:34 mail kernel:**ffff81003d0d7ea8 ffff810015e8ad18 0000000000000000 ffff81004f218000
Jan 22 18:29:34 mail kernel:**ffff81004f218000 ffffffff80292d2b ffff81003d0d7eb0 ffff81004f218028
Jan 22 18:29:34 mail kernel: Call Trace:
Jan 22 18:29:35 mail kernel:**[<ffffffff80292d2b>] __link_path_walk+0x8d9/0xd8b
Jan 22 18:29:35 mail kernel:**[<ffffffff802931b9>] __link_path_walk+0xd67/0xd8b
Jan 22 18:29:35 mail kernel:**[<ffffffff80293235>] link_path_walk+0x58/0xe0
Jan 22 18:29:35 mail kernel:**[<ffffffff802887d9>] do_filp_open+0x1c/0x3d
Jan 22 18:29:35 mail kernel:**[<ffffffff802935ad>] do_path_lookup+0x1ab/0x227
Jan 22 18:29:35 mail kernel:**[<ffffffff80293fdd>] __path_lookup_intent_open+0x56/0x97
Jan 22 18:29:35 mail kernel:**[<ffffffff8029416c>] open_namei+0x7a/0x674
Jan 22 18:29:35 mail kernel:**[<ffffffff802887d9>] do_filp_open+0x1c/0x3d
Jan 22 18:29:35 mail kernel:**[<ffffffff8028883e>] do_sys_open+0x44/0xc1
Jan 22 18:29:35 mail kernel:**[<ffffffff80209c2e>] system_call+0x7e/0x83
Jan 22 18:29:35 mail kernel:
Jan 22 18:29:35 mail kernel:
Jan 22 18:29:35 mail kernel: Code: 48 8b 00 48 85 c0 0f 85 05 01 00 00 48 8b 54 24 08 48 8b 44
Jan 22 18:29:35 mail kernel: RIP**[<ffffffff80290910>] do_lookup+0x3e/0x1ae
Jan 22 18:29:35 mail kernel:**RSP <ffff81003d0d7c08>
Jan 22 18:29:35 mail kernel: CR2: 0000000001000000

Jan 24 14:22:58 mail kernel: Eeek! page_mapcount(page) went negative! (-1)
Jan 24 14:22:58 mail kernel:** page pfn = 358f3
Jan 24 14:22:58 mail kernel:** page->flags = 100000000000060
Jan 24 14:22:58 mail kernel:** page->count = 2
Jan 24 14:22:58 mail kernel:** page->mapping = ffff81006ed86b11
Jan 24 14:22:58 mail kernel:** vma->vm_ops = _stext+0x7fdf9000/0x1a 
Jan 24 14:22:58 mail kernel: ------------[ cut here ]------------
Jan 24 14:22:58 mail kernel: kernel BUG at mm/rmap.c:628!
Jan 24 14:22:58 mail kernel: invalid opcode: 0000 [2] SMP 
Jan 24 14:22:58 mail kernel: last sysfs file: /devices/virtual/net/lo/type
Jan 24 14:22:58 mail kernel: CPU 1 
Jan 24 14:22:58 mail kernel: Modules linked in: joydev ip6t_LOG nf_conntrack_ipv6 xt_pkttype ipt_LOG xt_limit microcode firmware_class ip6t_REJECT xt_tcpudp snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device ipt_REJECT cpufreq_conservative xt_state cpufreq_userspace iptable_mangle cpufreq_powersave acpi_cpufreq iptable_nat nf_nat iptable_filter ip6table_mangle nf_conntrack_ipv4 nf_conntrack nfnetlink ip_tables ip6table_filter ip6_tables x_tables ipv6 apparmor loop dm_mod i2c_i801 i2c_core floppy r8169 iTCO_wdt parport_pc parport iTCO_vendor_support button rtc_cmos rtc_core rtc_lib sr_mod cdrom snd_hda_intel snd_pcm snd_timer snd soundcore snd_page_alloc intel_agp sg ehci_hcd sd_mod uhci_hcd usbcore edd ext3 mbcache jbd fan pata_jmicron ata_piix ahci libata scsi_mod thermal processor
Jan 24 14:22:58 mail kernel: Pid: 668, comm: su Tainted: G******N 2.6.22.13-0.3-default #1
Jan 24 14:22:58 mail kernel: RIP: 0010:[<ffffffff802781c9>]**[<ffffffff802781c9>] page_remove_rmap+0xed/0x108
Jan 24 14:22:58 mail kernel: RSP: 0000:ffff810068dadd38**EFLAGS: 00010246
Jan 24 14:22:58 mail kernel: RAX: 0000000000000000 RBX: ffff81007ee99528 RCX: 00000000001f9453
Jan 24 14:22:58 mail kernel: RDX: 00000000ffffffff RSI: 0000000000000046 RDI: ffffffff804bf01c
Jan 24 14:22:58 mail kernel: RBP: ffff810061792e48 R08: ffff810001160000 R09: 0000000000000020
Jan 24 14:22:58 mail kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff81007ed4fa38
Jan 24 14:22:58 mail kernel: R13: ffff81006111e298 R14: ffff810061792e48 R15: ffff81007c8053c0
Jan 24 14:22:58 mail kernel: FS:**00002b1f001dbb90(0000) GS:ffff810001114ec0(0000) knlGS:0000000000000000
Jan 24 14:22:58 mail kernel: CS:**0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 14:22:58 mail kernel: CR2: 00007fffab653268 CR3: 0000000054aa7000 CR4: 00000000000006e0
Jan 24 14:22:58 mail kernel: Process su (pid: 668, threadinfo ffff810068dac000, task ffff81003145e080)
Jan 24 14:22:58 mail kernel: Stack:**ffff81006111e298 0000000000000002 ffff81007ee99528 ffffffff8027014f
Jan 24 14:22:58 mail kernel:**ffff81007f81dea0 ffff81002fe9bad8 00007fffab653268 ffff81007f81dea0
Jan 24 14:22:58 mail kernel:**ffff810000000000 ffff81006111e298 ffff810061792e48 80000000358f3065
Jan 24 14:22:58 mail kernel: Call Trace:
Jan 24 14:22:58 mail kernel:**[<ffffffff8027014f>] do_wp_page+0x39b/0x482
Jan 24 14:22:58 mail kernel:**[<ffffffff80271b1c>] __handle_mm_fault+0x9e0/0xa3e
Jan 24 14:22:58 mail kernel:**[<ffffffff8029a139>] dput+0x26/0x115
Jan 24 14:22:58 mail kernel:**[<ffffffff802c2ae7>] proc_flush_task+0xb5/0x1ee
Jan 24 14:22:58 mail kernel:**[<ffffffff803fdb8f>] do_page_fault+0x400/0x769
Jan 24 14:22:58 mail kernel:**[<ffffffff803947a6>] sock_init_data+0xa3/0x1c2
Jan 24 14:22:58 mail kernel:**[<ffffffff802affb8>] inotify_d_instantiate+0x3e/0x68
Jan 24 14:22:58 mail kernel:**[<ffffffff80299e85>] d_instantiate+0x53/0x8b
Jan 24 14:22:58 mail kernel:**[<ffffffff803fc0fd>] error_exit+0x0/0x84
Jan 24 14:22:58 mail kernel: 
Jan 24 14:22:58 mail kernel: 
Jan 24 14:22:58 mail kernel: Code: 0f 0b eb fe 8b 77 18 5a 5b 5d 83 e6 01 f7 de 83 c6 04 e9 8b
Jan 24 14:22:58 mail kernel: RIP**[<ffffffff802781c9>] page_remove_rmap+0xed/0x108
Jan 24 14:22:58 mail kernel:**RSP <ffff810068dadd38>

Jan 24 14:44:54 mail kernel: Bad page state in process 'kswapd0'
Jan 24 14:44:54 mail kernel: page:ffff81007eeda128 flags:0x0100000000000008 mapping:0000000000000000 mapcount:-1 count:0
Jan 24 14:44:54 mail kernel: Trying to fix it up, but a reboot is needed
Jan 24 14:44:54 mail kernel: Backtrace:
Jan 24 14:44:54 mail kernel:
Jan 24 14:44:54 mail kernel: Call Trace:
Jan 24 14:44:54 mail kernel:**[<ffffffff802680a4>] bad_page+0x63/0x8d
Jan 24 14:44:54 mail kernel:**[<ffffffff80268695>] free_hot_cold_page+0x8e/0x13c
Jan 24 14:44:54 mail kernel:**[<ffffffff8026876c>] __pagevec_free+0x29/0x3b
Jan 24 14:44:54 mail kernel:**[<ffffffff8026b93a>] __pagevec_release_nonlru+0x52/0x5f
Jan 24 14:44:54 mail kernel:**[<ffffffff8026ce3b>] shrink_inactive_list+0x568/0x838
Jan 24 14:44:54 mail kernel:**[<ffffffff8026c6ee>] shrink_active_list+0x517/0x529
Jan 24 14:44:54 mail kernel:**[<ffffffff8026d200>] shrink_zone+0xf5/0x11e
Jan 24 14:44:54 mail kernel:**[<ffffffff8026dcb2>] kswapd+0x341/0x4f3
Jan 24 14:44:54 mail kernel:**[<ffffffff8024433b>] autoremove_wake_function+0x0/0x2e
Jan 24 14:44:54 mail kernel:**[<ffffffff8026d971>] kswapd+0x0/0x4f3
Jan 24 14:44:54 mail kernel:**[<ffffffff80244214>] kthread+0x47/0x73
Jan 24 14:44:54 mail kernel:**[<ffffffff8020aa48>] child_rip+0xa/0x12
Jan 24 14:44:54 mail kernel:**[<ffffffff802441cd>] kthread+0x0/0x73
Jan 24 14:44:54 mail kernel:**[<ffffffff8020aa3e>] child_rip+0x0/0x12

Apps seldom cause that sort of problem, especially if the crash is at different places each time. 64-bit is usually not a problem, compatibility issues are normally quite plain and cause failure in predictable places.

So that leaves the kernel or the hardware. If the crash is usually at the same place or a small set of places, it might be the kernel. You could try an upgraded kernel from the kernel of the day builds.

My guess is hardware. You could try to see if you can drive the hardware a little less agressively in the BIOS settings, if that's possible. If you have an identical machine you could try a HD transplant to see if that fares better there. Oh and check that basic things like CPU cooling are adequate.

These sorts of problems are quite hard to diagnose. Good luck.

Do a memory test with OpenSuse DVD and let me know how you get on..
My guess is you will have few incorrect entries.

Regards