A completely messed up system (fstab)

English Install/Boot/Login
#1

This is an install of openSUSE-11.2 32bit. Obvoiusly, during the fresh install something went wrong. The HDD has 4 primary partitions, meant for /boot / swap /home (in that order). Instead of /home partition /dev/sda4 is mounted on /usr.

Currently it looks like this:

# fdisk -l
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000cbdf6
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1         257     2064321   83  Linux
/dev/sda2             258       16641   131604480   83  Linux
/dev/sda3           16642       17154     4120672+  82  Linux swap / Solaris
/dev/sda4           17155       60801   350594527+  83  Linux

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2            129537660  53126008  69831428  44% /
udev                   1009032       312   1008720   1% /dev
/dev/sda1              2031888     86140   1842532   5% /boot
/dev/sda4            345093228  62382236 265181268  20% /usr

# cat /etc/fstab
/dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part3 swap                 swap       defaults              0 0
/dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part2 /                    ext4       acl,user_xattr        1 1
/dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part1 /boot                ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part4 /usr                 ext4       acl,user_xattr        1 2
proc                 /proc                proc       defaults              0 0
sysfs                /sys                 sysfs      noauto                0 0
debugfs              /sys/kernel/debug    debugfs    noauto                0 0
usbfs                /proc/bus/usb        usbfs      noauto                0 0
devpts               /dev/pts             devpts     mode=0620,gid=5       0 0

To fix this I have the following plan:

  1. Copy (rsync) /usr to a new directory /usrbak. Obviously /usrbak will be created under / on /dev/sda2.

  2. Comment out the fstab line for /dev/sda4

  3. Make a copy of the mv command in /root/bin

  4. umount /usr (here it may bite me :open_mouth: ) ???

  5. Rename /usrbak to /usr

  6. Reboot

  7. Mount /dev/sda4 on a new mountpoint /homebak

  8. Wipe out everything on /homebak, copy /home to /homebak

  9. umount /home, fix the fstab entry for /dev/sda4 to mount on /home

  10. Reboot

Any comments? This is a remote server, only accessible with ssh.

#2

On 2010-08-11 14:36, vodoo wrote:
>
> This is an install of openSUSE-11.2 32bit. Obvoiusly, during the fresh
> install something went wrong. The HDD has 4 primary partitions, meant
> for /boot / swap /home (in that order). Instead of /home partition
> /dev/sda4 is mounted on /usr.
>
> Currently it looks like this:
>
>
> Code:
> --------------------
> # fdisk -l
> Disk /dev/sda: 500.1 GB, 500107862016 bytes
> 255 heads, 63 sectors/track, 60801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x000cbdf6
> Device Boot Start End Blocks Id System
> /dev/sda1 * 1 257 2064321 83 Linux
> /dev/sda2 258 16641 131604480 83 Linux
> /dev/sda3 16642 17154 4120672+ 82 Linux swap / Solaris
> /dev/sda4 17155 60801 350594527+ 83 Linux
> --------------------
>
>
>
> Code:
> --------------------
> # df
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sda2 129537660 53126008 69831428 44% /
> udev 1009032 312 1008720 1% /dev
> /dev/sda1 2031888 86140 1842532 5% /boot
> /dev/sda4 345093228 62382236 265181268 20% /usr
> --------------------
>
>
>
> Code:
> --------------------
> # cat /etc/fstab
> /dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part3 swap swap defaults 0 0
> /dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part2 / ext4 acl,user_xattr 1 1
> /dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part1 /boot ext4 acl,user_xattr 1 2
> /dev/disk/by-id/ata-WDC_WD5000AAKS-00UU3A0_WD-WCAYU3940633-part4 /usr ext4 acl,user_xattr 1 2
> proc /proc proc defaults 0 0
> sysfs /sys sysfs noauto 0 0
> debugfs /sys/kernel/debug debugfs noauto 0 0
> usbfs /proc/bus/usb usbfs noauto 0 0
> devpts /dev/pts devpts mode=0620,gid=5 0 0
> --------------------
>
>
> To fix this I have the following plan:

Hold on. Look first at the contents of the current /usr directory. Does it contain “/usr” files?

I will assume it does.

I understand that it had the contents of /home previously, but as a new install formats /usr, they
will no longer be there.

  1. Consider switching to runlevel 3.
    Alternatively, make sure no one is logged in, and no X is running (it uses lots of files in
    /usr).

> 1. Copy (rsync) /usr to a new directory /usrbak. Obviously /usrbak will
> be created under / on /dev/sda2.

Yes.
But on the chance that you still have /home files on /usr, do not copy them.

You can also use “mc” for the copy.

> 2. Comment out the fstab line for /dev/sda4

No, first you have to umount it, which might not be possible. If there are in use files, the system
will refuse.

>
> 3. Make a copy of the mv command in /root/bin

What is that, copy the binary? No need, it is not there.

>
> 4. umount /usr (here it may bite me :open_mouth: ) ???

You have to do it before “2”.

Instead, edit here the fstab line for sd4, to mount it somewhere else, create the mount point, and
then mount it.

>
> 5. Rename /usrbak to /usr
>
> 6. Reboot

Maybe you can skip rebooting. If you were able to umount sda4, then just try to use something in
/usr, like “mc”.

>
> 7. Mount /dev/sda4 on a new mountpoint /homebak
>
> 8. Wipe out everything on /homebak, copy /home to /homebak
>
> 9. umount /home, fix the fstab entry for /dev/sda4 to mount on /home
>
> 10. Reboot

No need to reboot.

After all is done, then reboot to test it, not because it is necessary.

> Any comments? This is a remote server, only accessible with ssh.

That’s the tricky part. On a local system I would use a live, or runlevel 1.

You don’t have another runable system on the machine?

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” GM (Elessar))

#3

Hi Carlos, and thanks a lot for your suggestions so far.

Hold on. Look first at the contents of the current /usr directory. Does it contain “/usr” files?

It was a fresh install, and /usr mounted on /dev/sda4 contains exactly all /usr files; nothing else.

  1. Consider switching to runlevel 3.
    Alternatively, make sure no one is logged in, and no X is running (it uses lots of files in
    /usr).

Makes sense.

> 2. Comment out the fstab line for /dev/sda4
No, first you have to umount it, which might not be possible. If there are in use files, the system will refuse.

You are right. Most probably some files are in use. But I thought that fstab is only read when something is being mounted, so I could edit it any time?

> 3. Make a copy of the mv command in /root/bin
What is that, copy the binary? No need, it is not there.

What I mean is: copy the mv command to /root/bin, but this is not necessary, because mv is in /bin/mv and should be available even when /usr is umounted.

> 4. umount /usr (here it may bite me) ???

You have to do it before “2”.

Instead, edit here the fstab line for sd4, to mount it somewhere else, create the mount point, and then mount it.

Here is the tricky part. I’m logged in from remote and need a running system. I can’t have two directories named /usr at the same time and I want to switch over. I can’t see any advantage to mount /dev/sda4 on some other mountpoint, because I no longer need it for anything. What are your intentions here? Just have the commands available?

Maybe it could work with mv:

mv /usr /usr-old && mv /usrbak /usr

I guess that after this /dev/sda4 is still mounted on /usr-old but I might be able to umount it now.

On a local system I would use a live, or runlevel 1.

Me too. And there is no other bootable system on this machine. In the worst case this operation has to wait until I get physical access to that machine.

#4

On 2010-08-11 17:36, vodoo wrote:
>
> Hi Carlos, and thanks a lot for your suggestions so far.

Welcome.

>>> 2. Comment out the fstab line for /dev/sda4
>> No, first you have to umount it, which might not be possible. If there
>> are in use files, the system will refuse.
>
> You are right. Most probably some files are in use. But I thought that
> fstab is only read when something is being mounted, so I could edit it
> any time?

Mmmm… good question.

I think I have had weird results when done that way when trying to umount, sort of umounting the
wrong thing. I’m unsure of what it was, but I got something “funny”.

Better be safe and umount first. WHich perhaps is weird as it should use “mtab” for umounting.

By the way, without usr you have to use “vi” as editor.

>>> 3. Make a copy of the mv command in /root/bin
>> What is that, copy the binary? No need, it is not there.
>
> What I mean is: copy the mv command to /root/bin, but this is not
> necessary, because mv is in /bin/mv and should be available even when
> /usr is umounted.

Right.

>
>>> 4. umount /usr (here it may bite me) ???
>>
>> You have to do it before “2”.
>>
>> Instead, edit here the fstab line for sd4, to mount it somewhere else,
>> create the mount point, and then mount it.
>
> Here is the tricky part. I’m logged in from remote and need a running
> system. I can’t have two directories named /usr at the same time and I
> want to switch over. I can’t see any advantage to mount /dev/sda4 on
> some other mountpoint, because I no longer need it for anything. What
> are your intentions here? Just have the commands available?

Just in case. Dunno :slight_smile:

What about sshd? Is it in /usr?

> Maybe it could work with mv:
>
>
> Code:
> --------------------
> mv /usr /usr-old && mv /usrbak /usr
> --------------------
>
>
> I guess that after this /dev/sda4 is still mounted on /usr-old but I
> might be able to umount it now.

I think it will fail. You can test it locally. You could even create a minimal system and test the
procedure instead of trying first the remote one.

>> On a local system I would use a live, or runlevel 1.
>
> Me too. And there is no other bootable system on this machine. In the
> worst case this operation has to wait until I get physical access to
> that machine.

Perhaps it is safer. No, sure it is safer, that is not the word I want… advisable?

Sorry. Administering remote systems not designed to be administered remotely, with safeguards, is a
nightmare.

I’m right now thinking what the chaps that have to do some software change on a satellite or a space
explorer thing suffer. I read of one Russian thing that was lost because of this. It only had one
processor. After the change the antenna did not reorient back to Earth and all control was lost,
forever.

Shudder!

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” GM (Elessar))

#5

Perhaps it is safer. No, sure it is safer, that is not the word I want… advisable?

I agree. That computer is running quite well for now and the “lost” HD space will not be required for anything in the next few weeks or months. The staff on location will not be able to fix it when things go wrong. So I will do it on the next occasion booting from a live DVD. Better safe than sorry.

#6

On 2010-08-12 10:36, vodoo wrote:
>
>> Perhaps it is safer. No, sure it is safer, that is not the word I
>> want… advisable?
>
> I agree. That computer is running quite well for now and the “lost” HD
> space will not be required for anything in the next few weeks or months.
> The staff on location will not be able to fix it when things go wrong.
> So I will do it on the next occasion booting from a live DVD. Better
> safe than sorry.

Right.

You can always make a symlink to /usr if you need more space.

However, remember that during boot, if there is an fsck run that fails and it drops you in rescue
mode, that many tools are in /usr and that you get a minimal set of repair tools. For instance,
there is no “joe” editor, only vi. Even vi complains. You would have no sshd, either. It will break
for you in that situation, I think.

It is a question of going there when you choose or in a hurry >;-)

(unless you have two computers there, internconnected via serial port, so that one machine can
rescue the other)

There is a thread currently in the opensuse-factory mail list discussing whether a separate /usr
partition should be still supported or not. It is not clear if bugs regarding this will be solved.

Just for other readers and as a reminder: According to the FHS, /usr is (or can be) read-only, and
can be shared between several machines with the same arch. Thus, you can have a shop of stations
with very small disks sharing a common /usr via NFS. Or, you can have a host server running dozens
or hundreds of virtualized linuxes, all sharing the same /usr in the host, saving several gigabytes
of space for each client. It does a lot of sense having a separate /usr partition.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” GM (Elessar))