I’m testing upgrading from 11.1 to 11.3 and running into a major roadblock. When I try to ssh to the server after upgrading, I am unable to use password or pubkey auth to connect via ssh with my ldap user due to sshd segfault. I can however connect via pubkey to a local account on the system. Both auth methods work (for local users) when I disable UsePAM in sshd_config, but auth via ldap is required.
My configuration works fine on fully patched installs of both 11.1 and 11.3, but not a fully patched 11.1 upgraded to 11.3. I’ve been at this for a while now trying various things but don’t seem to be making much progress…any help would be appreciated.
/var/log/messages:
Oct 6 20:33:15 susetest kernel: 1829.251921] sshd[3602]: segfault at 7f4bb0521240 ip 00007f4bb0509354 sp 00007fffdf212850 error 7 in libcrypto.so.1.0.0[7f4bb0449000+188000]
/usr/sbin/sshd -ddd:
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 431
debug2: parse_server_config: config /etc/ssh/sshd_config len 431
debug3: /etc/ssh/sshd_config:21 setting Protocol 2
debug3: /etc/ssh/sshd_config:41 setting PermitRootLogin no
debug3: /etc/ssh/sshd_config:61 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:93 setting UsePAM yes
debug3: /etc/ssh/sshd_config:98 setting X11Forwarding no
debug3: /etc/ssh/sshd_config:120 setting Subsystem sftp /usr/lib64/ssh/sftp-server
debug3: /etc/ssh/sshd_config:123 setting AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
debug3: /etc/ssh/sshd_config:124 setting AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
debug3: /etc/ssh/sshd_config:125 setting AcceptEnv LC_IDENTIFICATION LC_ALL
debug1: sshd version OpenSSH_5.4p1
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug3: oom_adjust_setup
Set /proc/self/oom_adj from 0 to -17
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 431
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.1.100 port 50586
debug1: Client protocol version 2.0; client software version OpenSSH_5.6
debug1: match: OpenSSH_5.6 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.4
debug2: fd 3 setting O_NONBLOCK
debug2: Network child is on pid 3603
debug3: preauth child monitor started
debug3: mm_request_receive entering
debug3: privsep user:group 71:65
debug1: permanently_set_uid: 71/65
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug3: mm_request_send entering: type 0
debug3: monitor_read: checking request 0
debug3: mm_answer_moduli: got parameters: 1024 1024 8192
debug3: mm_request_send entering: type 1
debug2: monitor_read: 0 used once, disabling now
debug3: mm_request_receive entering
debug3: mm_choose_dh: waiting for MONITOR_ANS_MODULI
debug3: mm_request_receive_expect entering: type 1
debug3: mm_request_receive entering
debug3: mm_choose_dh: remaining 0
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug2: dh_gen_key: priv key bits set: 122/256
debug2: bits set: 487/1024
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug2: bits set: 504/1024
debug3: mm_key_sign entering
debug3: mm_request_send entering: type 4
debug3: monitor_read: checking request 4
debug3: mm_answer_sign
debug3: mm_answer_sign: signature 0x7f4bb12e8bd0(143)
debug3: mm_request_send entering: type 5
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN
debug3: mm_request_receive_expect entering: type 5
debug3: mm_request_receive entering
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user user service ssh-connection method none
debug1: attempt 0 failures 0
debug3: mm_getpwnamallow entering
debug3: mm_request_send entering: type 6
debug3: monitor_read: checking request 6
debug3: mm_answer_pwnamallow
debug3: Trying to reverse map address 192.168.1.100.
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM
debug3: mm_request_receive_expect entering: type 7
debug3: mm_request_receive entering
debug2: parse_server_config: config reprocess config len 431
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: mm_request_receive entering
debug2: input_userauth_request: setting up authctxt for user
debug3: mm_start_pam entering
debug3: mm_request_send entering: type 45
debug3: mm_inform_authserv entering
debug3: mm_request_send entering: type 3
debug2: input_userauth_request: try method none
debug3: mm_auth_password entering
debug3: mm_request_send entering: type 10
debug3: mm_auth_password: waiting for MONITOR_ANS_AUTHPASSWORD
debug3: mm_request_receive_expect entering: type 11
debug3: mm_request_receive entering
debug3: monitor_read: checking request 45
debug1: PAM: initializing for "user"
debug1: PAM: setting PAM_RHOST to "192.168.1.100"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 45 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 3
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 3 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 10
debug3: mm_answer_authpassword: sending result 0
debug3: mm_request_send entering: type 11
Failed none for user from 192.168.1.100 port 50586 ssh2
debug3: mm_request_receive entering
debug3: mm_auth_password: user not authenticated
debug1: userauth-request for user user service ssh-connection method publickey
debug1: attempt 1 failures 0
debug2: input_userauth_request: try method publickey
debug1: test whether pkalg/pkblob are acceptable
debug3: mm_key_allowed entering
debug3: mm_request_send entering: type 20
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED
debug3: mm_request_receive_expect entering: type 21
debug3: mm_request_receive entering
debug3: monitor_read: checking request 20
debug3: mm_answer_keyallowed entering
debug3: mm_answer_keyallowed: key_from_blob: 0x7f4bb12f0750
debug1: temporarily_use_uid: 10001/10 (e=0/0)
debug1: trying public key file /export/home/wheel/user/.ssh/authorized_keys
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 10001/10 (e=0/0)
debug1: trying public key file /export/home/wheel/user/.ssh/authorized_keys2
debug1: restore_uid: 0/0
Failed publickey for user from 192.168.1.100 port 50586 ssh2
debug3: mm_answer_keyallowed: key 0x7f4bb12f0750 is not allowed
debug3: mm_request_send entering: type 21
debug2: userauth_pubkey: authenticated 0 pkalg ssh-rsa
debug3: mm_request_receive entering
debug1: userauth-request for user user service ssh-connection method publickey
debug1: attempt 2 failures 1
debug2: input_userauth_request: try method publickey
debug3: mm_key_allowed entering
debug3: mm_request_send entering: type 20
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED
debug3: mm_request_receive_expect entering: type 21
debug3: mm_request_receive entering
debug3: monitor_read: checking request 20
debug3: mm_answer_keyallowed entering
debug3: mm_answer_keyallowed: key_from_blob: 0x7f4bb12f07f0
debug1: temporarily_use_uid: 10001/10 (e=0/0)
debug1: trying public key file /export/home/wheel/user/.ssh/authorized_keys
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 10001/10 (e=0/0)
debug1: trying public key file /export/home/wheel/user/.ssh/authorized_keys2
debug1: restore_uid: 0/0
Failed publickey for user from 192.168.1.100 port 50586 ssh2
debug3: mm_answer_keyallowed: key 0x7f4bb12f07f0 is not allowed
debug3: mm_request_send entering: type 21
debug2: userauth_pubkey: authenticated 0 pkalg ssh-dss
debug3: mm_request_receive entering
debug1: userauth-request for user user service ssh-connection method keyboard-interactive
debug1: attempt 3 failures 2
debug2: input_userauth_request: try method keyboard-interactive
debug1: keyboard-interactive devs
debug1: auth2_challenge: user=user devs=
debug1: kbdint_alloc: devices 'pam'
debug2: auth2_challenge_start: devices pam
debug2: kbdint_next_device: devices <empty>
debug1: auth2_challenge_start: trying authentication method 'pam'
debug3: mm_sshpam_init_ctx
debug3: mm_request_send entering: type 48
debug3: mm_sshpam_init_ctx: waiting for MONITOR_ANS_PAM_INIT_CTX
debug3: mm_request_receive_expect entering: type 49
debug3: mm_request_receive entering
debug3: monitor_read: checking request 48
debug3: mm_answer_pam_init_ctx
debug3: PAM: sshpam_init_ctx entering
debug3: mm_request_send entering: type 49
debug3: mm_sshpam_query
debug3: mm_request_send entering: type 50
debug3: mm_sshpam_query: waiting for MONITOR_ANS_PAM_QUERY
debug3: mm_request_receive_expect entering: type 51
debug3: mm_request_receive entering
debug3: mm_request_receive entering
debug3: monitor_read: checking request 50
debug3: mm_answer_pam_query
debug3: PAM: sshpam_query entering
debug3: ssh_msg_recv entering
debug3: mm_request_send entering: type 51
debug3: mm_request_receive entering
debug3: mm_sshpam_query: pam_query returned -1
debug3: mm_sshpam_free_ctx
debug3: mm_request_send entering: type 54
debug3: mm_sshpam_free_ctx: waiting for MONITOR_ANS_PAM_FREE_CTX
debug3: mm_request_receive_expect entering: type 55
debug3: mm_request_receive entering
debug3: monitor_read: checking request 54
debug3: mm_answer_pam_free_ctx
debug3: PAM: sshpam_free_ctx entering
debug3: PAM: sshpam_thread_cleanup entering
debug3: mm_request_send entering: type 55
debug2: monitor_read: 54 used once, disabling now
Failed keyboard-interactive/pam for user from 192.168.1.100 port 50586 ssh2
debug1: Unable to open the btmp file /var/log/btmp: No such file or directory
debug3: mm_request_receive entering
debug1: userauth-request for user user service ssh-connection method password
debug1: attempt 4 failures 3
debug2: input_userauth_request: try method password
debug3: mm_auth_password entering
debug3: mm_request_send entering: type 10
debug3: mm_auth_password: waiting for MONITOR_ANS_AUTHPASSWORD
debug3: mm_request_receive_expect entering: type 11
debug3: mm_request_receive entering
debug3: monitor_read: checking request 10
debug1: do_cleanup
debug3: PAM: sshpam_thread_cleanup entering
Segmentation fault
