openSUSE boots into emergency console when other partitions / drives are broken or disappear

English Install/Boot/Login
1

No, my system is not broken now. But an experiment I did in VirtualBox reminded me of this, and I remember it happening years ago with my real openSUSE installation. I haven’t looked a lot into the problem, but this is what I know:

If openSUSE is configured to mount multiple drives / partitions at startup (excluding external drives) it will fail to boot if one of those partitions goes missing. For instance (from what I know), if openSUSE mounts a secondary hard drive to something like /windows/D/, removing that drive would cause openSUSE to no longer boot, although it’s an optional disk. Instead of seeing the logon screen, I am taken to the “emergency console” and asked to login as root, without any information about what to do. I really don’t want to find myself in this situation with my actual system if a secondary drive goes away or some partition other than openSUSE’s gets broken.

In my VM experiment today, I have a virtual drive of 8GB. I used VirtualBox to expand the image to 32GB, then started the VM and SUSE booted just fine. I then went to the Partitioner tool and created an extended partition in the empty space that was generated, filling it up with a logical partition. I formatted that partition as FAT then using gparted as NTFS. I think a permanent mount point was assigned, not sure. When I restarted the VM, I got the emergency console I mentioned above, without even a message telling me why. I wrote the exit command and only then it printed some error about “FAT bogus sectors” or something like that. As far as I could tell the system was broken beyond repair… for messing with a partition that was NOT relevant for openSUSE starting up.

I’m disappointed that even today with 12.2, SUSE would choose to not boot instead of printing a warning and simply not mounting the partition it can’t access. But I’m still new to Linux so I don’t have enough experience to judge… maybe there’s a reason for this although I don’t see it. My curiosity is what command can be used in that console, and basically what can be done to ignore / forget any partition openSUSE expects and no longer finds (or finds and cannot mount / understand). I don’t plan to play with partitions or drives, but in case something happens I don’t want to be left with a broken system that needs to be completely reinstalled.

2

You may think that a file system is optional, but how can the system know? When you have it in* /etc/fstab *without further parameteers. it is a must. You could e.g. add he option *noauto, *but then it will not mount at boot and when you need it, you have to mount it seperately.

You could also leave it out of the fstab completley and then let the end-user ask for a mount (it will btw then mounted on a directory inside /media and on behalf of that user and not on the mount point you choose to use in fstab and for usage as defined in the fstab).

3

Ran the experiment again on Virtualbox… the result was as expected. Initially there were two partitions (excluding SWAP), the ext4 one with openSUSE and a NTFS one after it (inside an extended partition). This setup worked fine and openSUSE booted while mounting the NTFS in /windows/D/. I then booted the Live-CD and went to the Partitioner, deleting the extended partition and the NTFS partition in it. When booting the normal installation it showed the splash screen for a bit, then I was to a console in which the only things printed were:

doing fast boot
Creating device nodes with udev
Welcome to emergency mode. Use "systemctl default" or ^D to enter default mode.
Give root password for login:

I type the root password then I can write. I tried ‘systemctl default’ then it freezes for several seconds, taking me back to the very same console, asking root password and starting over. I’m keeping a copy of this vbox drive to see if I can learn to debug it after someone clears me up on this. But really, this feels wrong… why would openSUSE do this and create so much trouble instead of simply not caring and booting as usual :frowning:

4

@ hcvv: Sorry, didn’t see your reply. Like I said, I still don’t understand this. Why can’t openSUSE try to ignore the missing drive and go on as usual, and only fail if it can’t find the necessary items to boot? If it needs to know whether the drive is crucial for booting, why not a fstab option to specify it as a non-boot drive? Removing automount would be bad because I’d have to mount it manually each time.

Anyway, is there any way to remove it from fstab from this console? If I’m ever in this situation, I’d really like to know the command to use for editing this drive out so the system will boot as normal again. Still have my vbox image to try it out.

5

Sorry for the triple post, no spam intended. I asked about this on IRC and found two solutions for my problem. Tested on the broken virtual machine used to experiment, and it worked like a charm.

The first solution is to add the parameter “nofail” to the fstab options of that drive, which does exactly what I wanted (boot and not care). Verified and functional. Already added it to all other drives on my real system to automatically prevent such situations (and allow unplugging my other hard disk if I ever need to). I think openSUSE should automatically do this for any partition that isn’t / or swap since newbies that don’t have any idea about this (like I was until a few minutes ago) don’t accidentally unplug a drive, then face a system mysteriously getting broken. Most importantly, I think it should print a message to the recovery console explaining why you get there and what’s broken!

Second option is booting with a live-CD and editing /etc/fstab manually. Either remove the abandoned partition / drive (a line of text for each drive) or manually add nofail there. With these two solutions, I consider my question solved.

6

On 2012-09-27 22:06, MirceaKitsune wrote:
>
> No, my system is not broken now. But an experiment I did in VirtualBox
> reminded me of this, and I remember it happening years ago with my real
> openSUSE installation. I haven’t looked a lot into the problem, but this
> is what I know:
>
> If openSUSE is configured to mount multiple drives / partitions at
> startup (excluding external drives) it will fail to boot if one of
> those partitions disappears.

Yes.

> For instance (from what I know), if
> openSUSE mounts a secondary hard drive to something like /windows/D/,
> removing that drive would cause openSUSE to no longer boot, although
> it’s an optional disk.

But the system doesn’t know it is optional, you do. Thus you, the administrator, must tell the
system that a partition is optional by adding “nofail” to the options in fstab for that partition.

:slight_smile:

> In my VM experiment today, I have a virtual drive of 8GB. I used
> VirtualBox to expand the image to 32GB, then started the VM and SUSE
> booted just fine. I then went to the Partitioner tool, and created an
> extended partition in the empty space that was generated, filling it up
> with a logical partition. I formatted that partition as FAT then using
> gparted as NTFS. I think a permanent mount point was assigned, not
> sure. When I restarted the VM, I got the emergency console I mentioned
> above, without even a message telling me why.

If you can reproduce this situation in a VM and certify there is no message in the screen, you
will do us a lot of good by reporting the issue in Bugzilla. I have my suspicions about this
and systemd, but no proof, so I can’t report.

In classical system V the message is there, although not easy to interpret.

As to Linux refusing to boot in this situation, it is intentional, so don’t report as bug.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

7

On 2012-09-27 23:36, MirceaKitsune wrote:
>
> Sorry for the triple post, no spam intended. I asked about this on IRC
> and found two solutions for my problem. Tested on the broken virtual
> machine used to experiment, and it worked like a charm.
>
> The first solution is to add the parameter “nofail” to the fstab
> options of that drive, which does exactly what I wanted (boot and not
> care).

Ah, you found out. :slight_smile:

> Verified and functional. Already added it to all other drives on
> my real system to automatically prevent such situations (and allow
> unplugging my other hard disk if I ever need to). I think openSUSE
> should automatically do this for any partition that isn’t / or swap
> since newbies that don’t have any idea about this (like I was until a
> few minutes ago) don’t accidentally unplug a drive, then face a system
> mysteriously getting broken.

No, it can not be the default option. First, only the admin knows which partitions are
optional. Second, if a partition fails to mount for some reason, it will do so silently,
without any error message. If someone, or a script, tries to use that partition, the writes
will go to the parent partition where it should be mounted.

For example, suppose you mount “/backup” with nofail, and it is a 2 TiB drive. It fails to
mount, then the automated backup script triggers and does a backup to “/backup”, which writes
into “/” instead which is a partition of 20 GiB, fills it fast, and crashes the system (which
will not boot if you try).

> Most importantly, I think it should print a
> message to the recovery console explaining why you get there and what’s
> broken!

Yes, and it did, with system V. System D is new, I don’t like it, and here you have one more
thing to dislike of it. So go and report in bugzilla :slight_smile:

> Second option is booting with a live-CD and editing /etc/fstab
> manually. Either remove the abandoned partition / drive (a line of text
> for each drive) or manually add nofail there. With these two solutions,
> I consider my question solved.

Yes, that was the original fault. You can not modify the setup from a live and not tell the
system about it… And it could be worse, if you for example change /home from ext4 to xfs that
way… It is not enough to modify fstab, you may also have to add kernel modules at boot to the
initrd, and to the initial fsck.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

8

Yeah, figured out how to tell the OS it’s an optional drive (had assumed it would automatically know). As for the FAT error, I sadly deleted that VM. Maybe tomorrow or in the next days I’ll make a copy and try to reproduce it again.

Like I said, in Partitioner I created an extended partition in the entire empty area of the drive, filled it with a logical partition, and formatted that as FAT (also set it to auto-mount IIRC). Then I went to gparted and formatted that FAT partition to NTFS (cuz Partitioner won’t allow formatting to NTFS). Then restarting it caused the recovery console, and would print something about “FAT bogus sectors”. If someone else can try it on a virtual machine with 12.2 it would make sense for the result to be the same.

9

On 2012-09-28 01:46, MirceaKitsune wrote:

> Like I said, in Partitioner I created an extended partition in the
> entire empty area of the drive, filled it with a logical partition, and
> formatted that as FAT (also set it to auto-mount IIRC). Then I went to
> gparted and formatted that FAT partition to NTFS (cuz Partitioner won’t
> allow formatting to NTFS). Then restarting it caused the recovery
> console, and would print something about “FAT bogus sectors”. If someone
> else can try it on a virtual machine with 12.2 it would make sense for
> the result to be the same.

If I have time I can try to reproduce the issue and report. But I need a big enough stretch of
time. Many things on my plate…

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

10

This is a nice story, but as you do not tell each and every step you took, we can not realy find out if you did something wrong here.

I could imagine that you first made a partition ad gave it the type FAT (4 or 6) (automaticaly done by the Partitioner because you also asked it to create a FAT fs on it). Then you went to gparted to create a NTFS on the partition, but I am not sure if that changed also the partition type to HPFS/NTFS/exFAT (7) because it is done on an existing partition. I realy do not know if all tthose diferent tools used each for different tasks (partitioning and fs creation) check everything for consistancy. Maybe gparted fans know more about this.

BTW, I see that you do understand a lot about partitions, but maybe this is interesting for you: SDB:Basics of partitions, filesystems, mount points - openSUSE

11

On 2012-09-28 10:16, hcvv wrote:
>
> This is a nice story, but as you do not tell each and every step you
> took, we can not realy find out if you did something wrong here.
>
> I could imagine that you first made a partition ad gave it the type FAT
> (4 or 6) (automaticaly done by the Partitioner because you also asked it
> to create a FAT fs on it). Then you went to gparted to create a NTFS on
> the partition, but I am not sure if that changed also the partition type
> to HPFS/NTFS/exFAT (7) because it is done on an existing partition. I
> realy do not know if all tthose diferent tools used each for different
> tasks (partitioning and fs creation) check everything for consistancy.
> Maybe gparted fans know more about this.

The exact details do not matter much :slight_smile:

What happened was that the partition was reformatted with another type (ntfs instead of fat)
and fstab was not corrected to reflect the change. That alone can cause the problem described
on boot. I know because I have caused similar problems before :wink:

I guess that gpart altered the partition type in the table, I do not know for sure; but that
would not have altered the results.

What I want to see is what, if anything, says systemd when this happens. I know how systemv
does it, but not systemd, and I guess there is a problem there.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

12

Ahhh, now it makes sense (if it’s what I think it is). When I used Yast - Partitioner and formatted it there as FAT, it was added to fstab as a FAT partition. Then in gparted I made it NTFS. But gparted didn’t change it in fstab as well, so when openSUSE booted it was still trying to mount a NTFS partition as FAT. Not sure what can be done here, except maybe the gparted devs addressing the issue if possible.

13

On 2012-09-28 15:16, MirceaKitsune wrote:
>
> robin_listas;2491417 Wrote:

> Ahhh, now it makes sense (if it’s what I think it is). When I used Yast
> - Partitioner and formatted it there as FAT, it was added to fstab as a
> FAT partition. Then in gparted I made it NTFS. But gparted didn’t change
> it in fstab as well, so when openSUSE booted it was still trying to
> mount a NTFS partition as FAT. Not sure what can be done here, except
> maybe the gparted devs addressing the issue if possible.

No, it is how the tool works.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

14

That is nonsense. Gparted is only for partitioning and file system management (creation and shrinking/extending when possible). Not or managing. It only works on the disk. Not on the system. YaST is openSUSE’s system management tool. And it does a good job there because it does all sorts of logical related tasks together. Presenting the system manager with a GUI interface to those combined tasks.

Same as YaST will e.g. when you start a service, also open the Firewall and similar. YaST is very good in letting a system manager not forget parts of the action(s)he is executing. A great help.

But you can not accuse the different tools for not doing the tasks of other tools. fdisk partitions. mkfs creates file systems. vi edits /etc/fstab. YaST combines these things (ecept when you do excotic combinations where you reverse to the CLI tools).

15

I didn’t mean to accuse gparted, just wasn’t sure if it’s a bug until getting more insight on this. If it’s only intended to edit partitions (not change fstab options) then it’s not anyone’s fault… just a case of bad usage where the user might not know. I only used gparted because YAST Partitioner wouldn’t allow formatting to NTFS… IMO this should be changed if possible.

Anyway, I submitted a feature request about automatically setting secondary drives as ‘nofail’ if openSUSE / Partitioner can tell which is root and which is something else. If anyone’s interested, https://features.opensuse.org/314483

16

That’s what “auto” fstype is for, although I am not sure whether it covers ntfs as well. In the past it tried pretty much hardcoded list of file system types. Possible implementation could be to use (lib)blkid to find out filesystem type.

Of course, blindly doing it may open up the whole new can of worms.

17

On 2012-09-28 16:06, MirceaKitsune wrote:
>
> I didn’t mean to accuse gparted, just wasn’t sure if it’s a bug until
> getting more insight on this. If it’s only intended to edit partitions
> (not change fstab options) then it’s not anyone’s fault… just a case
> of bad usage where the user might not know. I only used gparted because
> YAST Partitioner wouldn’t allow formatting to NTFS… IMO this should be
> changed if possible.

Not prioritary, few people want to format as ntfs. I would not do it, I prefer to use Windows
for that.

> Anyway, I submitted a feature request about automatically setting
> secondary drives as ‘nofail’ if openSUSE / Partitioner can tell which is
> root and which is something else. If anyone’s interested,
> https://features.opensuse.org/314483

No, as I explained already this can not be done.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

18

Of course it can. Just make all of them automounted :slight_smile:

19

Maybe YaST will get that feature when Windows users wil be able to create ext4 and Btrfs file systems. rotfl!

20

On 2012-09-28 16:26, arvidjaar wrote:
>
> robin_listas;2491417 Wrote:

> That’s what “auto” fstype is for, although I am not sure whether it
> covers ntfs as well.

Yes, it does. Think that when you plug in a usb stick formatted as ntfs it is automatically
mounted.

> In the past it tried pretty much hardcoded list of
> file system types. Possible implementation could be to use (lib)blkid to
> find out filesystem type.
>
> Of course, blindly doing it may open up the whole new can of worms.

Mmmm… :-?

It takes some time to do the auto thing.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)