No access after upgrade to Leap 15!

I recently upgraded to Leap 15.0 from 42.3 using the zypper-method. The upgrade finished without problem but after reboot I could not log in. This is the error message that is shown if I try a TTY login (graphical login gives no hint):

User unknown to the underlying authentication module

This happens for all users and root! I have tried rebooting several times with no success. There are two users defined: one locally authenticated and one network user authenticated via LDAP.

It happened now and then in Leap 42.3 the same thing with root so I could not use graphical Yast. The reason for this was that the order of the lines in one or more of the following files got screwed up:


/etc/pam.d/common-account -> common-account-pc
/etc/pam.d/common-auth -> common-auth-pc
/etc/pam.d/common-password -> common-password-pc
/etc/pam.d/common-session -> common-session-pc
/etc/pam.d/gdm-password -> gdm

Why this happened was never found out but more machines on the local network had the same issue. But as sudo never was compromised I could use that to fire up text-mode Yast and go into Network Services->LDAP and Kerberos Client, hit Change Settings and then OK on the following screen. The files were then rewritten with the lines in the correct order and all was fine. I thought that I had made backups of the correct files but no such luck. Below I show what they look like now (except comment lines) and I would like to know if there is anything wrong with the order of the lines and in that case what the correct order should be. I imagine that somehow LDAP requirement obscures local authentication but unfortunately that doesn’t help me.


cat /etc/pam.d/common-account-pc
...
account    requisite    pam_unix.so    try_first_pass 
account    required    pam_krb5.so    use_first_pass 
account    sufficient    pam_localuser.so 
account    required    pam_ldap.so    use_first_pass


cat /etc/pam.d/common-auth-pc
...
auth    required    pam_env.so    
auth    optional    pam_gnome_keyring.so
auth    sufficient    pam_unix.so    try_first_pass 
auth    sufficient    pam_krb5.so    use_first_pass 
auth    required    pam_ldap.so    use_first_pass


cat /etc/pam.d/common-password-pc
...
password    requisite    pam_cracklib.so    
password    optional    pam_gnome_keyring.so    use_authtok
password    [default=ignore success=1]    pam_succeed_if.so    uid > 999 quiet 
password    sufficient    pam_unix.so    use_authtok nullok shadow try_first_pass 
password    sufficient    pam_krb5.so 
password    required    pam_ldap.so    try_first_pass use_authtok


cat /etc/pam.d/common-session-pc
...
session  optional    pam_mkhomedir.so    
session    optional    pam_systemd.so
session    required    pam_limits.so    
session    required    pam_unix.so    try_first_pass 
session    optional    pam_krb5.so    
session    optional    pam_ldap.so    
session    optional    pam_umask.so    
session    optional    pam_gnome_keyring.so    auto_start only_if=gdm,gdm-password,lxdm,lightdm 
session    optional    pam_env.so


cat /etc/pam.d/gdm
...
auth     requisite      pam_nologin.so
auth     include        common-auth
account  include        common-account
password include        common-password
session  required       pam_loginuid.so
session  optional       pam_keyinit.so force revoke
session  include        common-session

If the line order has indeed been screwed up there is hope if someone can tell me the right order. Thanks beforehand for any help that you may provide.

Cheers,
gostal

So the first step is to compare PAM configuration for sudo and login.


cat /etc/pam.d/common-account-pc
...
account    required    pam_krb5.so    use_first_pass 

[/QUOTE]
Do you have kerberos infrastructure? Is kerberos client configured? Does keyinit for user in question work?

Thanks for responding!

Sudo and login:


cat /etc/pam.d/sudo
#%PAM-1.0
auth     include        common-auth
account  include        common-account
password include        common-password
session  optional       pam_keyinit.so revoke
session  include        common-session
# session  optional       pam_xauth.so


cat /etc/pam.d/login
#%PAM-1.0
auth     requisite    pam_nologin.so
auth     include    common-auth
account  include     common-account
password include    common-password
session  required    pam_loginuid.so
session     include    common-session
#session  optional       pam_lastlog.so nowtmp showfailed
session  optional       pam_mail.so standard

I guess these files are OK as they have not been changed since 2018.

arvidjaar: Do you have kerberos infrastructure? Is kerberos client configured? Does keyinit for user in question work?

Cannot say for sure. The machine was setup by a presently retired man at our support group. The new people favour Ubuntu. I had a look at the corresponding files in Ubuntu but they were so different I could not use them for comparison. I imagine that kerberos was/is used because the only info I have is that the line ordering was screwed up. Keyinit? How can I tell? I cannot login to Leap 15. I booted the machine from a USB-stick (Linux Mint 19.1) and can access the hard disks from there. There is no keyinit command on the USB Mint but even if there was it wouldn’t be relevant.

/gostal

There is clearly something wrong with these files:


/etc/pam.d/common-account -> common-account-pc
/etc/pam.d/common-auth -> common-auth-pc
/etc/pam.d/common-password -> common-password-pc
/etc/pam.d/common-session -> common-session-pc

There were both pam-config backups and .rpmnew backups of these files under /etc/pam.d created at different times:


ll /etc/pam.d/common*
lrwxrwxrwx 1 root root  17 Oct  9  2017 /etc/pam.d/common-account -> common-account-pc
-rw-r--r-- 1 root root 380 Jul  7  2017 /etc/pam.d/common-account.pam-config-backup
-rw-r--r-- 1 root root 380 Aug  5 13:58 /etc/pam.d/common-account-pc
-rw-r--r-- 1 root root 578 Aug  5 13:56 /etc/pam.d/common-account-pc-190805
-rw-r--r-- 1 root root 380 Dec  3  2018 /etc/pam.d/common-account.rpmnew
lrwxrwxrwx 1 root root  14 Oct  9  2017 /etc/pam.d/common-auth -> common-auth-pc
-rw-r--r-- 1 root root 462 Jul  7  2017 /etc/pam.d/common-auth.pam-config-backup
-rw-r--r-- 1 root root 462 Aug  5 13:58 /etc/pam.d/common-auth-pc
-rw-r--r-- 1 root root 658 Aug  5 13:56 /etc/pam.d/common-auth-pc-190805
-rw-r--r-- 1 root root 462 Dec  3  2018 /etc/pam.d/common-auth.rpmnew
lrwxrwxrwx 1 root root  18 Oct  9  2017 /etc/pam.d/common-password -> common-password-pc
-rw-r--r-- 1 root root 510 Jul  7  2017 /etc/pam.d/common-password.pam-config-backup
-rw-r--r-- 1 root root 510 Aug  5 13:58 /etc/pam.d/common-password-pc
-rw-r--r-- 1 root root 644 Aug  5 13:57 /etc/pam.d/common-password-pc-190805
-rw-r--r-- 1 root root 510 Dec  3  2018 /etc/pam.d/common-password.rpmnew
lrwxrwxrwx 1 root root  17 Oct  9  2017 /etc/pam.d/common-session -> common-session-pc
-rw-r--r-- 1 root root 482 Jul  7  2017 /etc/pam.d/common-session.pam-config-backup
-rw-r--r-- 1 root root 482 Aug  5 13:59 /etc/pam.d/common-session-pc
-rw-r--r-- 1 root root 731 Aug  5 13:57 /etc/pam.d/common-session-pc-190805
-rw-r--r-- 1 root root 482 Dec  3  2018 /etc/pam.d/common-session.rpmnew

Using USB-Mint I backed up the current ones and copied the earlier backups which had nothing in them but local authentication stuff:


cat /etc/pam.d/common*rpmnew
#
# /etc/pam.d/common-account - account settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the account modules that define
# the central access policy for use on the system.  The default is to
# only deny service to users whose accounts are expired.
#
account required        pam_unix.so     try_first_pass
#
# /etc/pam.d/common-auth - authentication settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the authentication modules that define
# the central authentication scheme for use on the system
# (e.g., /etc/shadow, LDAP, Kerberos, etc.).  The default is to use the
# traditional Unix authentication mechanisms.
#
auth    required        pam_env.so
auth    required        pam_unix.so     try_first_pass
#
# /etc/pam.d/common-password - password-related modules common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of modules that define  the services to be
# used to change user passwords.
#
# The "nullok" option allows users to change an empty password, else
# empty passwords are treated as locked accounts.
#
password        requisite       pam_cracklib.so
password        required        pam_unix.so     use_authtok nullok try_first_pass
#
# /etc/pam.d/common-session - session-related modules common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of modules that define tasks to be performed
# at the start and end of sessions of *any* kind (both interactive and
# non-interactive).
#
session required        pam_limits.so
session required        pam_unix.so     try_first_pass
session optional        pam_umask.so
session optional        pam_env.so
session optional        pam_systemd.so

Now I can login to Leap 15 local user and root works. So clearly those files had got it wrong so that local authentication got obscured. Apparently there is also something wrong with the LDAP authentication setup as I could not login the network user either. Perhaps you can tell me whats wrong here.

/gostal

Figured out the correct order for the lines in /etc/pam.d/common-account-pc in order that local authentication does not break. For some reason, as yet
unknown, lines 2 and 3 got switched. This is the correct order or at least an order that doesn’t break local authentication:


account    requisite    pam_unix.so    try_first_pass 
account    sufficient    pam_localuser.so 
account    required    pam_krb5.so    use_first_pass
account    required    pam_ldap.so    use_first_pass

But still LDAP authentication for the network user does not work so there is still something wrong with the /etc/pam.d-files. I have looked at this page and tried various
things without success. Ideas, anyone?

Cheers,
gostal

I had a quick look at the man-pages for pam.d and pam-config. For pam-config there are some interesting option that I decided to try out:


pam-config --initialize
pam-config --update
pam-config --verify

So I did after having backed up the /etc/pam.d/common-account-pc that doesn’t break local authentication. To be on the safe side I also had a root terminal session open. I ran the commands in that order using sudo. The first one complained it could not read the old config. The next one went through silent. The third one gave me this:


sudo pam-config --verify
ERROR: module /lib/security/pam_ldap.so is not installed.

To get rid of this error I symlinked /lib64/security/pam_ldap.so

Then I looked at /etc/pam.d/common-account-pc and it turns out that again the order of the lines were such that local authentication was broken. Good thing I had the backup file and the root window. So now we know three things:

  • pam-config is the culprit behind the screwed up order of the lines
  • the yast-module makes no use of pam-config otherwise it wouldn’t be possible to correct the order
  • some updates for Leap 42.3 made use of pam-config and hence the screwed up order of the lines in the files and the Leap 15.0 version is no better.

It seems that it’s time to make another bug report.

Cheers,
gostal

So finally I managed to log in my network user. :slight_smile: I edited the /etc/pam.d-files like so (comment lines omitted):


# /etc/pam.d/common-account-pc
...
account    requisite    pam_unix.so    try_first_pass 
account    sufficient    pam_localuser.so 
account required        pam_krb5.so     use_first_pass
account    required    pam_ldap.so    use_first_pass


# /etc/pam.d/common-auth-pc
...
auth    required    pam_env.so    
auth    optional    pam_gnome_keyring.so
auth    sufficient    pam_unix.so    try_first_pass 
auth    sufficient    pam_krb5.so    use_first_pass 
auth    required    pam_ldap.so    use_first_pass


# /etc/pam.d/common-password-pc
...
password    requisite    pam_cracklib.so    
password    optional    pam_gnome_keyring.so    use_authtok
password    sufficient      pam_unix.so     use_authtok nullok shadow try_first_pass
password    [default=ignore success=1]    pam_succeed_if.so    uid > 999 quiet 
password    sufficient    pam_krb5.so 
password    required    pam_ldap.so    try_first_pass use_authtok 


# /etc/pam.d/common-session-pc
...
session optional        pam_mkhomedir.so
session    required    pam_limits.so    
session    required    pam_unix.so    try_first_pass 
session optional        pam_krb5.so
session optional        pam_ldap.so
session    optional    pam_umask.so    
session optional        pam_systemd.so
session    optional    pam_gnome_keyring.so    auto_start only_if=gdm,gdm-password,lxdm,lightdm 
session    optional    pam_env.so    

Of these it would appear that the critical one for local authentication to work is the first, common-account-pc. Remember that lines 2 and 3 got switched. If I understand the docs correctly a line with required in it must succeed or there will be an error stopping further execution. If kerberos fails and sits before critical lines then local authentication breaks. However, the modification of these files alone was not sufficient. As I got no further I looked at /etc/ldap.conf which looked like this:


uri ldap://ldap-kst.foi.se
base dc=foi,dc=se
nss_connect_policy persist
ssl no
bind_policy soft
bind_timelimit 30
timelimit 30

Inspired by the same file on another machine running Ubuntu 18.04 I did the following changes:


### uri ldap://ldap-kst.foi.se
uri ldap://ldap.foi.se
base dc=foi,dc=se
ldap_version=2
nss_connect_policy persist
### ssl no
bind_policy soft
bind_timelimit 30
timelimit 30

I suspect that the critical change was the addition of the line *ldap_version=2. *Both ldap servers are reachable but have different IP numbers. In the Ubuntu version of the file it says that default version now is 3 so it’s plausible that the reason why the LDAP-configuration broke already before the upgrade is that the client software assumed version 3 whereas the servers still are running version 2. I may test it if I find the time but there are other more urgent issues.

I would, however, greatly appreciate suggestions of improvement or any other comment.

Also, I’ll be sure to make backup copies of the files in point, now that they are finally working!

Cheers,
gostal

Today I reported the pam-config bug 1146178.

Cheers,
gostal