My YubiKey broke Snapper! How do I unbreak it?

English Install/Boot/Login
1

Hi everybody

I’m new to OpenSUSE (after playing with it a tiny bit in the late 90s). Have spent some time on Debian lately. But I wanted to try out btrfs and snapshots and booting into them and all that. So I started setting it up on Debian, but it’s a pain! Then I stumbled upon OpenSUSE Tumbleweed and voilà, it’s just so simple!

One thing, however, that is not working for me, is using my YubiKey when booting into a snapshot. After a bit of log digging, I think the reason may be that /dev is part of the snapshot and therefore read-only. But my YubiKey is expected be mounted in /dev/hidraw1 - which I’m guessing is not possible due to the read-only-ness of the snapshot. The real problem is that this means I cannot run sudo snapper rollback because I set sudo up to require my YubiKey. I cannot restore snapshots! I broke snapper! :face_with_peeking_eye:

How do I fix this? The only solution I can think of is to exclude /dev from the screenshots by turning it into a btrfs subvolume. But is that advisable or rather a stupid thing to do? (Also, I don’t now yet how that would be done.)

Or does the problem lie elsewhere entirely?

I hope somebody here can help me so I can stop tinkering (which is of course way too much fun…) and start actually working my work using OpenSUSE. :slight_smile:

2

/dev is in-memory pseudo filesystem. If not, something is extremely wrong and I do not see how it is related to snapshots.

This should be created by loading driver hidraw (assuming drivers for physical devices are loaded).

3

As @arvidjaar said, /dev, /proc, /sys are all API filesystems that expose the linux kernel functionalities as a filesystem for users/admins to interact with the kernel without syscalls/coding.

Btw, I just tested my Yubikey when booted into a read-only snapshot and it’s detected just fine. Tried unplugging and re-plugging it even… :white_check_mark:

You may want to re-check your Yubikey setup:

4

Thank you for responding so quickly!

Yes, I assume it’s in the in-memory pseudo filesystem. But isn’t that filesystem still read-only? I’m sorry if this is too basic a misunderstanding on my part. I’m new to OpenSUSE, btrfs, snapshots, and snapper. Need to work my way through all of these topics.

The question is why would that driver be loaded correctly in the actual system, but not when booting into a snapshot?

5

No. Only one btrfs subvolume is read-only. Everything else is not affected by snapshots.

We have no information whether this was or was not loaded. You did not show any logs, any command output or something else. You can start with lsmod. And you can redirect output to a file on any other btrfs subvolume to save it permanently.

6

Thanks to you too for responding so quickly!

Aaah, so I probably showed my incompetence in my previous reply, didn’t I? I thought /dev would be in-memory because it’s from a snapshot, but it just always is in-memory? Have I mentioned I’m no developer but just a simple nerd. :smile:

I stumbled upon /dev in the YubiKey debug logs. Line 4 is the point where things behave differently if booted from snapshot:

debug(pam_u2f): util.c:797 (get_authenticators): Working with 1 authenticator(s)
debug(pam_u2f): util.c:800 (get_authenticators): Checking whether key exists in authenticator 0
debug(pam_u2f): util.c:808 (get_authenticators): Authenticator path: /dev/hidraw1
debug(pam_u2f): util.c:835 (get_authenticators): Key not found in authenticator 0
debug(pam_u2f): util.c:845 (get_authenticators): Key not found
debug(pam_u2f): util.c:1263 (do_authentication): Device for this keyhandle is not present

But maybe the problem lies somewhere else entirely then?

Hehe that’s good and bad news. Good because it must work somehow, bad because the error probably lies with me. Thanks for checking!

Oh man these instructions would have saved me a lot of headache yesterday. Have you considered sending them to Yubico? I sent them some of my info. The info on their website is not very helpful if you’re on OpenSUSE.

But to the topic at hand:

  • pam_u2f is installed and all looks good with ykman info and ykman fido info.
  • I used pamu2fcfg > /etc/Yubico/u2f_keys instead of echo `pamu2fcfg -N` | sudo tee -a /etc/Yubico/u2f_keys. Does that make a difference?
  • This is what I added to my pam config file: auth required pam_u2f.so authfile=/etc/Yubico/u2f_keys. Should be OK?

What else can I check?

7

But /dev is not a btrfs subvolume, it’s just a subfolder of /, isn’t it? But I probably just don’t understand the details here, which is fine, since this doesn’t seem to be the culprit anyway?

My first time using lsmod. Shall I paste it here? It’s quite long. Or do I look for something specific first?

8

We all have gaps in our knowledge, being here helps others to fill in those gaps. :brain::handshake:
More about API filesystems.

The -N flag does PIN verification in addition to presence (by touch).
It’s necessary when you want the Yubikey to be the sole authenticator (PIN+presence MFA) instead of being just another 2FA.

Depends on which PAM config you added it to, if you have a secondary machine try to set it up from scratch according to the Gist. Comparing the two should help you fix the current machine.

9

It is mount point where /dev filesystem is mounted.

You can upload it to https://paste.opensuse.org/. Once when booting normally and once when booting from snapshot.

10

Thank you for your patience. I’m starting to understand.

Here they come!
normal boot: openSUSE Paste
snapshot boot: openSUSE Paste

11

I like the brain-handshake! I think I need to put that on a poster in some form for my wall! I’m also starting to understand, thank you.

Ah yes, I forgot about that flag. Will keep it in mind.

I added it to /usr/lib/pam.d/sudo.
I just noticed something in your instructions. It says there “Default pam config files can be found in /usr/lib/pam.d directory.” Which is where I found mine and those are the ones I edit. But then in your code block you write: “nano /etc/pam.d/sudo”. Is one location of the two a mistake or are both correct in some way?

12

You should never edit files in /usr - this directory is reserved for installed packages and package update will silently replace modified files. Files under /etc/pam.d override files with the same name under /usr/lib/pam.d.

13

I was mistaken, openSUSE kernel does not build hidraw as separate module.

Are you sure /dev/hidraw* are missing? Show

ls -l /dev/hidraw*

both in normal and snapshot boot.

14

Thank you, I just corrected that mistake. I was hoping this would fix the problem, but unfortunately it didn’t.

The results are weird. At first, the result was the same in both cases:

crw-rw----+ 1 root root 245, 0 Apr 18 14:51 /dev/hidraw0
crw-rw----+ 1 root root 245, 1 Apr 18 14:51 /dev/hidraw1
crw-------  1 root root 245, 2 Apr 18 14:51 /dev/hidraw2
crw-------  1 root root 245, 3 Apr 18 14:51 /dev/hidraw3
crw-------  1 root root 245, 4 Apr 18 14:51 /dev/hidraw4

But after I authenticated once, the normal boot result changed to this:

crw-------  1 root root 245, 1 Apr 18 15:00 /dev/hidraw1
crw-------  1 root root 245, 2 Apr 18 14:56 /dev/hidraw2
crw-rw----+ 1 root root 245, 3 Apr 18 15:01 /dev/hidraw3
crw-------  1 root root 245, 4 Apr 18 14:56 /dev/hidraw4
crw-rw----+ 1 root root 245, 5 Apr 18 15:01 /dev/hidraw5

When I look at the pam_u2f debug log, I can see that hidraw5 is indeed where the YubiKey is queried when in normal boot, while hidraw1 is queried when booted into a snapshot.

The only other difference I can find in those logs is the first line. Could that be relevant?
pam_u2f debug log - normal boot: openSUSE Paste
pam_u2f debug log - snapshot boot: openSUSE Paste

15

Well, devices are present in both cases. I am not familiar with YubiKeys, hopefully someone can chime in.

16

No hidraw5 on my machine:

pavin@suse-pc:~> ll /dev/hidraw*
crw-------  1 root root 245, 0 Apr 16 03:40 /dev/hidraw0
crw-------  1 root root 245, 1 Apr 16 03:40 /dev/hidraw1
crw-rw----+ 1 root root 245, 2 Apr 18 16:21 /dev/hidraw2
crw-------  1 root root 245, 3 Apr 16 03:40 /dev/hidraw3
crw-------  1 root root 245, 4 Apr 16 03:40 /dev/hidraw4

hidraw2 is the Yubikey as per the following script that matches hidraw to device name:

#!/bin/bash

FILES=/dev/hidraw*
for f in $FILES
do
  FILE=${f##*/}
  DEVICE="$(cat /sys/class/hidraw/${FILE}/device/uevent | grep HID_NAME | cut -d '=' -f2)"
  printf "%s \t %s\n" $FILE "$DEVICE"
done

Source

17

Thank you @arvidjaar and @pavinjoseph for helping me out here. I decided to set up a fresh system last night and now my YubiKeys work as expected. I don’t know what caused the issue - maybe I messed up some dependencies on my quest to get thing going.

But thanks to the :brain: :handshake: I learned a few new valuable things about OpenSUSE and Linux more generally (API filesystems yay, and don’t touch /usr!) and my setup is better now than it would have otherwise been. Also, getting it confirmed here that YubiKeys should work in snapshots was a strong factor in not abandoning the test case entirely.

Thanks again, it was a great first contact with the OpenSUSE community! A few more tests and I’ll give it a go on my first production machine. :partying_face:

18

Ah, now I just had another question that maybe could be answered here. I read here about adding a YubiKey to /etc/pam.d/common-auth, which would enable it for all authentication including TTY, SSH, sudo etc.

Is this a good or a bad idea?

19

It is up to you to decide, but if you are going to add it, it makes sense to use pam-config which supports pam_u2f.

1 Like
20

Something to consider with regard to SSH specifically - if you’re not taking your Yubikey with you, or if it isn’t supported on devices you authenticate from, then you might not want to do that. For example, if you only have a USB-A key without NFC capability and use an Android SSH client, you wouldn’t be able to authenticate from your phone.

Or if you use one of the mini-USBC Yubikeys (like I have) and leave it plugged in all the time. Or if you’re afraid of leaving the key somewhere if you don’t leave it plugged in.

All things to consider.