openSUSE 12.3 64 bit English, Version 5.3.17

andreacon · March 4, 2014, 12:12pm

Hi,
we have a critical problem with OpenSUSE (see subject).
We in random way have this crash and we have to restart the machine (you can to suggest a solution or workaround) we have a virtual environment with esxi 4.1
Thanks in advanced
Andrea

2014-03-04T04:01:03.742946+01:00 RMHBDB03 systemd-logind[484]: New session 1529 of user script.
2014-03-04T04:01:04.355181+01:00 RMHBDB03 sshd[2332]: Received disconnect from 10.28.22.149: 11: disconnected by user
2014-03-04T04:01:04.359582+01:00 RMHBDB03 sshd[2330]: pam_unix(sshd:session): session closed for user script
2014-03-04T04:01:04.367397+01:00 RMHBDB03 systemd-logind[484]: Removed session 1529.
2014-03-04T04:21:06.415990+01:00 RMHBDB03 sshd[25077]: Accepted publickey for script from 10.28.22.151 port 38740 ssh2
2014-03-04T04:21:06.420380+01:00 RMHBDB03 sshd[25077]: pam_unix(sshd:session): session opened for user script by (uid=0)
2014-03-04T04:21:06.422592+01:00 RMHBDB03 systemd-logind[484]: New session 1601 of user script.
2014-03-04T04:21:06.783491+01:00 RMHBDB03 sshd[25079]: Received disconnect from 10.28.22.151: 11: disconnected by user
2014-03-04T04:21:06.787038+01:00 RMHBDB03 sshd[25077]: pam_unix(sshd:session): session closed for user script
2014-03-04T04:21:06.802833+01:00 RMHBDB03 systemd-logind[484]: Removed session 1601.
2014-03-04T04:21:06.935398+01:00 RMHBDB03 sshd[25114]: Accepted publickey for script from 10.28.22.151 port 38741 ssh2
2014-03-04T04:21:06.936367+01:00 RMHBDB03 sshd[25114]: pam_unix(sshd:session): session opened for user script by (uid=0)
2014-03-04T04:21:06.938188+01:00 RMHBDB03 systemd-logind[484]: New session 1602 of user script.
2014-03-04T04:21:07.289466+01:00 RMHBDB03 sshd[25130]: Received disconnect from 10.28.22.151: 11: disconnected by user
2014-03-04T04:21:07.292695+01:00 RMHBDB03 sshd[25114]: pam_unix(sshd:session): session closed for user script
2014-03-04T04:21:07.305888+01:00 RMHBDB03 systemd-logind[484]: Removed session 1602.
2014-03-04T04:53:25.153009+01:00 RMHBDB03 rsyslogd: – MARK –
2014-03-04T05:53:25.253197+01:00 RMHBDB03 rsyslogd: – MARK –
2014-03-04T06:53:25.353286+01:00 RMHBDB03 rsyslogd: – MARK –
2014-03-04T07:53:25.453398+01:00 RMHBDB03 rsyslogd: – MARK –
2014-03-04T08:34:39.760809+01:00 RMHBDB03 kernel: [42047.048147] BUG: unable to handle kernel NULL pointer dereference at (null)
2014-03-04T08:34:39.760833+01:00 RMHBDB03 kernel: [42047.048156] IP: < (null)>] (null)
2014-03-04T08:34:39.760835+01:00 RMHBDB03 kernel: [42047.048191] PGD 0
2014-03-04T08:34:39.760837+01:00 RMHBDB03 kernel: [42047.048194] Oops: 0000 #1] PREEMPT SMP
2014-03-04T08:34:39.760838+01:00 RMHBDB03 kernel: [42047.048198] Modules linked in: dm_mod xt_tcpudp xt_pkttype xt_LOG xt_limit ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT iptable_raw xt_CT iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ppdev vmw_balloon mperf coretemp microcode pcspkr i2c_piix4 shpchp pci_hotplug parport_pc parport floppy mptctl sg e1000 sr_mod button container ac cdrom autofs4 vmwgfx ttm drm processor thermal_sys scsi_dh_alua scsi_dh_hp_sw scsi_dh_emc scsi_dh_rdac scsi_dh ata_generic mptspi mptscsih mptbase scsi_transport_spi ata_piix
2014-03-04T08:34:39.760841+01:00 RMHBDB03 kernel: [42047.048239] CPU 0
2014-03-04T08:34:39.760847+01:00 RMHBDB03 kernel: [42047.048251] Pid: 27436, comm: telnet Tainted: G W 3.7.10-1.1-desktop #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
2014-03-04T08:34:39.760850+01:00 RMHBDB03 kernel: [42047.048254] RIP: 0010:<0000000000000000>] < (null)>] (null)
2014-03-04T08:34:39.760852+01:00 RMHBDB03 kernel: [42047.048257] RSP: 0018:ffff8807a61ebe30 EFLAGS: 00010083
2014-03-04T08:34:39.760858+01:00 RMHBDB03 kernel: [42047.048259] RAX: ffff8807a951efe8 RBX: 0000000000000000 RCX: 0000000000000000
2014-03-04T08:34:39.760860+01:00 RMHBDB03 kernel: [42047.048261] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8807a951efe8
2014-03-04T08:34:39.760861+01:00 RMHBDB03 kernel: [42047.048263] RBP: ffff8807a61ebe78 R08: 0000000000000000 R09: 0000000000000000
2014-03-04T08:34:39.760862+01:00 RMHBDB03 kernel: [42047.048265] R10: 00007fd585b553e0 R11: 0000000000000000 R12: ffff8807a56c5698
2014-03-04T08:34:39.760867+01:00 RMHBDB03 kernel: [42047.048267] R13: ffffffffffffffe8 R14: 0000000000000000 R15: 0000000000000003
2014-03-04T08:34:39.760868+01:00 RMHBDB03 kernel: [42047.048284] FS: 00007fd58462c700(0000) GS:ffff8807d1c00000(0000) knlGS:0000000000000000
2014-03-04T08:34:39.760870+01:00 RMHBDB03 kernel: [42047.048286] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
2014-03-04T08:34:39.760871+01:00 RMHBDB03 kernel: [4202014-03-04T07:48:21.695576+01:00 RMHBDB03 rsyslogd: [origin software=“rsyslogd” swVersion=“7.2.5” x-pid=“504” x-info=“http://www.rsyslog.com”] start
2014-03-04T07:48:21.697317+01:00 RMHBDB03 kernel: 0.000000] Initializing cgroup subsys cpuset
2014-03-04T07:48:21.697328+01:00 RMHBDB03 kernel: 0.000000] Initializing cgroup subsys cpu
2014-03-04T07:48:21.697330+01:00 RMHBDB03 kernel: 0.000000] Linux version 3.7.10-1.1-desktop (geeko@buildhost) (gcc version 4.7.2 20130108 [gcc-4_7-branch revision 195012] (SUSE Linux) ) #1 SMP PREEMPT Thu Feb 28 15:06:29 UTC 2013 (82d3f21)
2014-03-04T07:48:21.697353+01:00 RMHBDB03 kernel: 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.7.10-1.1-desktop root=UUID=7868e841-377b-4598-ba6d-c8ebff4d23a8 resume=/dev/sda1 splash=silent quiet showopts
2014-03-04T07:48:21.697355+01:00 RMHBDB03 kernel: 0.000000] Disabled fast string operations
2014-03-04T07:48:21.697357+01:00 RMHBDB03 kernel: 0.000000] e820: BIOS-provided physical RAM map:
2014-03-04T07:48:21.697358+01:00 RMHBDB03 kernel: 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable
2014-03-04T07:48:21.697359+01:00 RMHBDB03 kernel: 0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved
2014-03-04T07:48:21.697360+01:00 RMHBDB03 kernel: 0.000000] BIOS-e820: [mem 0x00000000000ca000-0x00000000000cbfff] reserved
2014-03-04T07:48:21.697368+01:00 RMHBDB03 kernel: 0.000000] BIOS-e820: [mem 0x00000000000dc000-0x00000000000e3fff] reserved
2014-03-04T07:48:21.697370+01:00 RMHBDB03 kernel: 0.000000] BIOS-e820: [mem 0x00000000000e8000-0x00000000000fffff] reserved
2014-03-04T07:48:21.697371+01:00 RMHBDB03 kernel: 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bfeeffff] usable
2014-03-04T07:48:21.697372+01:00 RMHBDB03 kernel: 0.000000] BIOS-e820: [mem 0x00000000bfef0000-0x00000000bfefefff] ACPI data

tsu2 · March 4, 2014, 4:26pm

Looks to me like the client (un named) is trying to negotiate a SSH session and something is going wrong.

Recommend you start there.
What is the client app running on what OS?
Have you tried connecting from a different client and OS?
It looks like you’re using a login script, instead of using the script you should try executing each command in the script from a terminal to determine exactly which command fails.

The above are troubleshooting steps I’d recommend.
As for the detailed part of your post, it looks like the following is the critical issue reported by your log

2014-03-04T08:34:39.760809+01:00 RMHBDB03 kernel: [42047.048147] BUG: unable to handle kernel NULL pointer dereference at           (null)

Which leads me to suspect that something in your login script is likely missing a value(null generally means missing).

Why a missing value in a login script would cause a system crash, I wouldn’t know off-hand but it sure seems dastardly, maybe even a bug (else would be an interesting potential DoS attack)

HTH,
TSU

andreacon · March 4, 2014, 6:27pm

Thanks TSU,
we are verifying your feedback
andrea

andreacon · March 5, 2014, 9:48am

Hi TSU,
we have checked your answer and all the script in the server.

Below some explanation:
The server use protocol ssh/scp to dialogue with other server in local network (without using DNS - all resoluton are made by file /etc/hosts)
The server is inside a sicure network (so the eventually of DOS attack is very unprobably)
The server is crashing 1/2 time a day and all the script run every 15 minutes by 24 hor (we have tryied to run script handly without reproduce the crash).
There are other log that we can activete to collect more information or log?

BR
Sergio

tsu2 · March 5, 2014, 7:50pm

IMO the fault likely is in your ssh login script.
I’m not sure why the SSH session is being closed client-side.
Are you able to sanitize your login script and post it?
(If it’s short, is ok to post here in code blocks. Else, post on SUSE PASTE or somewhere else similar).

Since this a problem most likely unrelated to virtualization, I highly recommend you also search for posts in the Applications forum., eg “ssh client disconnect”
There may also be stray threads posted in the Networking forum.

TSU