Cryptsetup on a separate /home partition goes into emergency mode

This issue is the same issue created by me here: Setting up Cryptsetup on a separate /home partition goes into emergency mode

The solution that I found out was sort of an “Ostrich Algorithm” solution, but now I need to encrypt. Thus, I am opening this issue again in a new post since the old issue was closed. Pasting the same issue description down here, along with the output of “journalctl -b” and “/etc/crypttab” as recommended by @arvidjaar in the previous post (UUID for cr_home might have changed since previous post, but the mount points and setup are same):

/etc/crypttab

cr_home  UUID=9d4dcb77-c318-4ef1-bb04-8cbf0fb6e4f5

jounralctl -b
openSUSE Paste

Previous post

Hi,

I recently made a fresh installation of Tumbleweed (snapshot 20241109). While setting up the system, I created 4 partitions:

  • /boot/eft, FAT, 0.50 GiB
  • swap, swap, 32 GiB
  • /, btrfs, snapshots enabled, 150GiB
  • /home, ext4, rest of the space.

Now I face a bizarre issue. When I turn on my system, it sometimes boots up just fine, and sometimes it fails to boot, and goes inside emergency mode (which probably meant something failed in fstab). Both issues happen after asking the cryptsetup password. Going inside the emergency mode by entering the root password, I ran the following command to see what failed:

systemctl --failed -l

  UNIT                               LOAD   ACTIVE SUB    DESCRIPTION
● systemd-cryptsetup@cr_home.service loaded failed failed Cryptography Setup for cr_home

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

1 loaded units listed.

I have setup other openSUSE instances (both Leap and Tumbleweed) with a similar setup before (infact there is Leap 15.6 in my laptop which is setup exactly this way), and this has never occurred.

Here is an output of the cryptsetup logs:

journalctl -xu systemd-cryptsetup@cr_home.service

Nov 11 14:40:13 BarunesPC systemd[1]: Starting Cryptography Setup for cr_home...
░░ Subject: A start job for unit systemd-cryptsetup@cr_home.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit systemd-cryptsetup@cr_home.service has begun execution.
░░ 
░░ The job identifier is 179.
Nov 11 14:40:17 BarunesPC systemd-cryptsetup[878]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/16a2cccc-5e55-4319-9a6d-439f50db7dea.
Nov 11 14:40:19 BarunesPC systemd-cryptsetup[878]: Cannot use device /dev/disk/by-uuid/16a2cccc-5e55-4319-9a6d-439f50db7dea which is in use (already mapped or mounted).
Nov 11 14:40:19 BarunesPC systemd-cryptsetup[878]: Failed to activate with specified passphrase: Device or resource busy
Nov 11 14:40:19 BarunesPC systemd[1]: systemd-cryptsetup@cr_home.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit systemd-cryptsetup@cr_home.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Nov 11 14:40:19 BarunesPC systemd[1]: systemd-cryptsetup@cr_home.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit systemd-cryptsetup@cr_home.service has entered the 'failed' state with result 'exit-code'.
Nov 11 14:40:19 BarunesPC systemd[1]: Failed to start Cryptography Setup for cr_home.
░░ Subject: A start job for unit systemd-cryptsetup@cr_home.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit systemd-cryptsetup@cr_home.service has finished with a failure.
░░ 
░░ The job identifier is 179 and the job result is failed.
Nov 11 14:40:19 BarunesPC systemd[1]: systemd-cryptsetup@cr_home.service: Consumed 2.123s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit systemd-cryptsetup@cr_home.service completed and consumed the indicated resources.

And here is my fstab file if it helps:

cat /etc/fstab

UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /                       btrfs  defaults                      0  0
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /var                    btrfs  subvol=/@/var                 0  0
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /usr/local              btrfs  subvol=/@/usr/local           0  0
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /srv                    btrfs  subvol=/@/srv                 0  0
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /root                   btrfs  subvol=/@/root                0  0
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /opt                    btrfs  subvol=/@/opt                 0  0
/dev/mapper/cr_home                        /home                   ext4   data=ordered                  0  2
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=10a56f4c-2628-4b9e-b935-ebbcacbaa7ef  swap                    swap   defaults                      0  0
UUID=4c36332c-30a9-454e-ba57-9ca1e8b2417c  /.snapshots             btrfs  subvol=/@/.snapshots          0  0
UUID=02CB-89B4                             /boot/efi               vfat   utf8                          0  2

Thanks in advance!

Dec 25 13:59:00 BarunesPC systemd[1]: Starting Cryptography Setup for cr_home...
Dec 25 13:59:00 BarunesPC systemd[1]: Starting Cryptography Setup for home...

Show the full /etc/crypttab, not some arbitrarily chosen line.

/etc/crypttab seems to have only one line, which has

cr_home  UUID=9d4dcb77-c318-4ef1-bb04-8cbf0fb6e4f5

Show

systemctl --all list-units | grep home
systemctl list-unit-files | grep home

The issue happens randomly, and it boots properly sometimes and goes into emergency mode at other times. At this moment it booted fine, and the the outputs are given below. If you need the same output when it’s in emergency mode, please let me know:

barunespadhy@BarunesPC:~> systemctl --all list-units | grep home

  dev-disk-by\x2did-dm\x2dname\x2dcr_home.device                                                                                                         loaded    active   plugged   /dev/disk/by-id/dm-name-cr_home
  dev-disk-by\x2did-dm\x2dname\x2dhome.device                                                                                                            loaded    active   plugged   /dev/disk/by-id/dm-name-home
  dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS2\x2d9d4dcb77c3184ef1bb048cbf0fb6e4f5\x2dcr_home.device                                                   loaded    active   plugged   /dev/disk/by-id/dm-uuid-CRYPT-LUKS2-9d4dcb77c3184ef1bb048cbf0fb6e4f5-cr_home
  dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS2\x2d9d4dcb77c3184ef1bb048cbf0fb6e4f5\x2dhome.device                                                      loaded    active   plugged   /dev/disk/by-id/dm-uuid-CRYPT-LUKS2-9d4dcb77c3184ef1bb048cbf0fb6e4f5-home
  dev-mapper-cr_home.device                                                                                                                              loaded    active   plugged   /dev/mapper/cr_home
  dev-mapper-home.device                                                                                                                                 loaded    active   plugged   /dev/mapper/home
  home.mount                                                                                                                                             loaded    active   mounted   /home
  run-credentials-systemd\x2dcryptsetup\x40cr_home.service.mount                                                                                         loaded    active   mounted   run-credentials-systemd\x2dcryptsetup\x40cr_home.service.mount
  run-credentials-systemd\x2dcryptsetup\x40home.service.mount                                                                                            loaded    active   mounted   run-credentials-systemd\x2dcryptsetup\x40home.service.mount
  systemd-cryptsetup@cr_home.service                                                                                                                     loaded    active   exited    Cryptography Setup for cr_home
  systemd-cryptsetup@home.service                                                                                                                        loaded    active   exited    Cryptography Setup for home
  systemd-fsck@dev-mapper-cr_home.service                                                                                                                loaded    active   exited    File System Check on /dev/mapper/cr_home
  blockdev@dev-mapper-cr_home.target                                                                                                                     loaded    active   active    Block Device Preparation for /dev/mapper/cr_home
  blockdev@dev-mapper-home.target

systemctl list-unit-files | grep home

home.mount                                                                generated       -
systemd-cryptsetup@cr_home.service                                        generated       -
systemd-cryptsetup@home.service                                           generated       -

Show

systemd-cat systemd-cryptsetup@home.service

For some reason the output of systemd-cat systemd-cryptsetup@home.service shows Failed to execute process: No such file or directory

I tried re-boting a few more times to get the output of the previous commands (systemctl --all list-units | grep home and systemctl list-unit-files | grep home) in emergency mode. Here are the outputs incase it helps:

BarunesPC:~ # cat systemctl-all.log

  dev-disk-by\x2did-dm\x2dname\x2dhome.device                                                                                                            loaded    active   plugged   /dev/disk/by-id/dm-name-home
  dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS2\x2d9d4dcb77c3184ef1bb048cbf0fb6e4f5\x2dhome.device                                                      loaded    active   plugged   /dev/disk/by-id/dm-uuid-CRYPT-LUKS2-9d4dcb77c3184ef1bb048cbf0fb6e4f5-home
  dev-mapper-cr_home.device                                                                                                                              loaded    inactive dead      /dev/mapper/cr_home
  dev-mapper-home.device                                                                                                                                 loaded    active   plugged   /dev/mapper/home
  home.mount                                                                                                                                             loaded    inactive dead      /home
  run-credentials-systemd\x2dcryptsetup\x40cr_home.service.mount                                                                                         loaded    inactive dead      run-credentials-systemd\x2dcryptsetup\x40cr_home.service.mount
  run-credentials-systemd\x2dcryptsetup\x40home.service.mount                                                                                            loaded    active   mounted   /run/credentials/systemd-cryptsetup@home.service
● systemd-cryptsetup@cr_home.service                                                                                                                     loaded    failed   failed    Cryptography Setup for cr_home
  systemd-cryptsetup@home.service                                                                                                                        loaded    active   exited    Cryptography Setup for home
  systemd-fsck@dev-mapper-cr_home.service                                                                                                                loaded    inactive dead      File System Check on /dev/mapper/cr_home
  blockdev@dev-mapper-cr_home.target                                                                                                                     loaded    inactive dead      Block Device Preparation for /dev/mapper/cr_home
  blockdev@dev-mapper-home.target

BarunesPC:~ # cat systemctl-list-unit-files.log

home.mount                                                                generated       -
systemd-cryptsetup@cr_home.service                                        generated       -
systemd-cryptsetup@home.service                                           generated       -

I’d say it is a bug in systemd. Show

find /run/systemd -name systemd-cryptsetup@home.service

BarunesPC:~ # find /run/systemd -name systemd-cryptsetup@home.service

/run/systemd/generator.late/systemd-cryptsetup@home.service
/run/systemd/generator.late/dev-disk-by\x2ddiskseq-2\x2dpart4.device.wants/systemd-cryptsetup@home.service
/run/systemd/generator.late/cryptsetup.target.requires/systemd-cryptsetup@home.service
/run/systemd/generator.late/dev-mapper-home.device.requires/systemd-cryptsetup@home.service

And

cat /run/systemd/generator.late/systemd-cryptsetup@home.service

BarunesPC:~ # cat /run/systemd/generator.late/systemd-cryptsetup@home.service

# Automatically generated by systemd-gpt-auto-generator

[Unit]
Description=Cryptography Setup for %I
Documentation=man:crypttab(5) man:systemd-cryptsetup-generator(8) man:systemd-cryptsetup@.service(8)

DefaultDependencies=no
After=cryptsetup-pre.target systemd-udevd-kernel.socket systemd-tpm2-setup-early.service
Before=blockdev@dev-mapper-%i.target
Wants=blockdev@dev-mapper-%i.target
IgnoreOnIsolate=true
Before=umount.target cryptsetup.target
Conflicts=umount.target
BindsTo=dev-disk-by\x2ddiskseq-2\x2dpart4.device
After=dev-disk-by\x2ddiskseq-2\x2dpart4.device

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=infinity
KeyringMode=shared
OOMScoreAdjust=500
ImportCredential=cryptsetup.*
ExecStart=/usr/bin/systemd-cryptsetup attach 'home' '/dev/disk/by-diskseq/2-part4' '' ''
ExecStop=/usr/bin/systemd-cryptsetup detach 'home'

@barunespadhy ,
Please make it easier to everybody (including probably you), by copying/pasting including the line with the prompt and command up to and including the line with the next prompt.
Then you do not need to add the command separate (where others then have to gather those two together again) or other comments (like there is no output) because we all see what you saw in one glance.

1 Like

Understood, subsequent replies will be created in one formatted text bloc instead of multiple.

1 Like

I am afraid, anything I would say about it will be censored by this forum software.

The immediate workaround is to add systemd.gpt_auto=0 to the kernel command line.

Showing

gdisk -l /dev/sda

and

gdisk /dev/sda
i

for the home partition would be useful. Replace /dev/sda with your disk name.

P.S. of course, you could also comment out cr_home in /etc/crypttab, adjust /etc/fstab and rely on gpt-auto to do the right thing.

Understood. Before I add it to the kernel boot parameter, could please let me know what is wrong? Is this a systemd bug or did I set something up wrong?

Also, here is an output of the queries requested:

BarunesPC:~ # gdisk -l /dev/nvme0n1
GPT fdisk (gdisk) version 1.0.10

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/nvme0n1: 3907029168 sectors, 1.8 TiB
Model: WD_BLACK SN770 2TB                      
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 694698DF-D280-43F2-BC59-CC09D9351AD1
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1333247   650.0 MiB   EF00  
   2         1333248        68442111   32.0 GiB    8200  
   3        68442112       320100351   120.0 GiB   8304  
   4       320100352      3907029134   1.7 TiB     8302  





BarunesPC:~ # gdisk /dev/nvme0n1
GPT fdisk (gdisk) version 1.0.10

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): i
Partition number (1-4): 1
Partition GUID code: C12A7328-F81F-11D2-BA4B-00A0C93EC93B (EFI system partition)
Partition unique GUID: C1620209-759B-4E0B-B2C3-DAD71899B0E5
First sector: 2048 (at 1024.0 KiB)
Last sector: 1333247 (at 651.0 MiB)
Partition size: 1331200 sectors (650.0 MiB)
Attribute flags: 0000000000000000
Partition name: ''

Command (? for help): i
Partition number (1-4): 2
Partition GUID code: 0657FD6D-A4AB-43C4-84E5-0933C84B4F4F (Linux swap)
Partition unique GUID: 5612DC7E-F78F-4BA8-880A-765C96EC8889
First sector: 1333248 (at 651.0 MiB)
Last sector: 68442111 (at 32.6 GiB)
Partition size: 67108864 sectors (32.0 GiB)
Attribute flags: 0000000000000000
Partition name: ''

Command (? for help): i
Partition number (1-4): 3
Partition GUID code: 4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709 (Linux x86-64 root (/))
Partition unique GUID: D5A23C43-4718-45A5-AC61-3181439F5523
First sector: 68442112 (at 32.6 GiB)
Last sector: 320100351 (at 152.6 GiB)
Partition size: 251658240 sectors (120.0 GiB)
Attribute flags: 0000000000000000
Partition name: ''

Command (? for help): i
Partition number (1-4): 4
Partition GUID code: 933AC7E1-2EB4-4F13-B844-0E14E2AEF915 (Linux /home)
Partition unique GUID: 1C0C88DA-5097-4BFC-8318-446601AB3E58
First sector: 320100352 (at 152.6 GiB)
Last sector: 3907029134 (at 1.8 TiB)
Partition size: 3586928783 sectors (1.7 TiB)
Attribute flags: 0000000000000000
Partition name: ''

OK, that just confirms it.

You did not do anything wrong. systemd folks always believed they know better what users need, so they come up with the clever idea to automatically setup some filesystems. Having this specific partition GUID is not really common, I wonder what software did it.

If anything, it is a systemd bug. It should have checked that there is already another LUKS device configured for the same physical partition and skip its “helpful” “users are too dumb, let me do it” assistance. But it is not trivial. At the time generators run systemd did not even enumerate devices yet …

hmmm, so is there really nothing I can do except for adding the boot parameter? Should me (or you) be raising some kind of a bug report on systemd’s github?

Re-read my post again. You can also change partition GUID to something more generic, like Linux filesystem data (0FC63DAF-8483-4772-8E79-3D69D8477DE4 or 0x8300 as abbreviated by gdisk).

Sorry, I won’t do it for several reasons. One of them - I am pretty sure the answer will be “if you do not want autoconfiguration, do not use this partition type”. The first implementation (which already included /home) appeared over 10 years ago. It is probably too late to complaint about it. See man systemd-gpt-auto-generator for details. BTW it is also possible to set partition flag to turn off auto-mount.

The question is how this partition got its type. What program did you use to create it?

When installing tumbleweed, I chose to create my own partitions. While creating my own partitions, I did the following:

I created 4 partitions:

  • /boot/eft, FAT, 0.50 GiB

  • swap, swap, 32 GiB

  • /, btrfs, snapshots enabled, 150GiB

  • /home, ext4, rest of the space. When creating this partition, I checked the “Encrypt the partition”, and then chose LUKS2 to encrypt it.

That’s pretty much it. I wonder something in the installer messes it up?