dm devices not detached when shutting down

Hi,
i’m using a freshly installed openSuSe 12.2 64 bit, with this partition layout:

sda1 + sdb2 -> md0 (raid1) -> /boot (ext4)
sda2 + sdb2 -> md1 (raid1) -> swap
sda3 + sdb3 -> md2 (raid1) -> LVM volume group “SystemVG”->
Logical volume dm0 -> /home (ext4)
Logical volume dm1 -> /root (ext4)
Logical volume dm2 -> /tmp (ext4)
Logical volume dm3 -> /var (ext4)
sda5 -> /data1 (ext4)
sdb5 -> /data2 (ext4)

Boot sequence seems ok (I have some problem but they aren’t “storage-related”), but when halting or rebooting the system the shutdown sequence has some errors on closing raid and lvm volume. Those are messages that look more significative:


boot.lvm can’t deactivate volume group “sysmetVG” with 3 open logical volume

boot.md Not shutting down MD RAID - reboot/halt scripts do this…missing

could not unmount /var: device or resource busy

could not delete dm /dev/dm-3: device o resource busy

Not all DM devices detached, 2 left

Cannot finalize remaining file system and devices, giving up

I’m sorry, I cannot cut and paste, those are shutdown messages only available on the display
I really don’t know where to start searching…any ideas ?

Thank you all!

jjletho wrote:
> Hi,
> i’m using a freshly installed openSuSe 12.2 64 bit, with this partition
> layout:
>
> sda1 + sdb2 -> md0 (raid1) -> /boot (ext4)
> sda2 + sdb2 -> md1 (raid1) -> swap
> sda3 + sdb3 -> md2 (raid1) -> LVM volume group “SystemVG”->
> Logical
> volume dm0 -> /home (ext4)
> Logical
> volume dm1 -> /root (ext4)
> Logical
> volume dm2 -> /tmp (ext4)
> Logical
> volume dm3 -> /var (ext4)
> sda5 -> /data1 (ext4)
> sdb5 -> /data2 (ext4)

The list is incomplete. Specifically, where is your root partition mounted?

Have you tried any other systems with this h/w? Either older or newer
kernels in particular; I’ve seen some correspondence about shutdown
problems with md in certain circumstances.

Did this cause any problems?

In particular, was there an “fsck” on the next boot, or a message on the next boot about disks not being cleanly unmounted? (I think you can check in “/var/log/boot.log” to look for such messages).

If this does not cause problems, then best to not worry about it. If it does cause problems, then consider reporting a bug.

I’m sorry, i did a mistake describing partition layout: dm1 is the logical volume where “/” is mounted!

Volume dm1 -> / (ext4)

Marco

jjletho wrote:
> djh-novell;2525528 Wrote:
>> jjletho wrote:
>>
>> The list is incomplete. Specifically, where is your root partition
>> mounted?
>>
>> Have you tried any other systems with this h/w? Either older or newer
>> kernels in particular; I’ve seen some correspondence about shutdown
>> problems with md in certain circumstances.
>
> I’m sorry, i did a mistake describing partition layout: dm1 is the
> logical volume where “/” is mounted!
>
> Volume dm1 -> / (ext4)

Right, so LVM on RAID is definitely not the easiest configuration to use
for the root partition. I would definitely try some other kernels, both
older and newer, to see what symptoms you see.

But as nrickert asked, do the messages indicate a real problem, or are
they just noise?

That’s pretty much the explanation.

The system needs to access the root file system during shutdown, so attempts to disconnect the crypto for that file system should fail.

When I last checked this, the root file system was being remounted as read-only. And, as long as that is properly done, there will be a clean filesystem shutdown in spite of those error messages.

As long as the file systems come up clean on the next boot, everything is fine.

Just noting that it looks like you specified your own layout instead of allowing systemd to suggest mount points according to newer principles. Most importantly, by default systemd in 12.2 would by default mount /var/run in tmpfs instead of a disk volume which would make it faster and non-persistent. If no fsck on next boot, would be circumstantial evidence this could be related to your problem.

I’d suggest taking a look at a default 12.2 layout and consider creating mount points missing that exist in a default install.

IMO,
TSU

Hi,
I’m not sure this boot sequence is completely ok.


mdadm: /dev/md1 not identified in config file.
  4 logical volume(s) in volume group "systemVG" now active
  4 logical volume(s) in volume group "systemVG" now active
Trying manual resume from /dev/md1
resume device /dev/md1 not found (ignoring)
Trying manual resume from /dev/md1
resume device /dev/md1 not found (ignoring)
Waiting for device /dev/mapper/systemVG-rootLV to appear:  ok
fsck from util-linux 2.21.2
[/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a /dev/mapper/systemVG-rootLV 
rootFS: clean, 208673/1966080 files, 1803322/7864320 blocks
fsck succeeded. Mounting root device read-write.
Mounting root /dev/mapper/systemVG-rootLV
mount -o rw,acl,user_xattr -t ext4 /dev/mapper/systemVG-rootLV /root

Welcome to [0;32mopenSUSE 12.2 (Mantis) (x86_64)[0m!

....

Starting File System Check on /dev/md0...                                                                                                                                                                        
Starting File System Check on /dev/systemVG/homeLV...                                                                                                                                                            
Starting File System Check on /dev/systemVG/tmpLV...                                                                                                                                                             
Starting File System Check on /dev/systemVG/varLV...                                                                                                                                                             
Starting File System Check on /dev/disk/by-id/ata-WDC_WD2500AAJS-07B4A0_WD-WCAT12873197-part5...                                                                                                                 
Starting File System Check on /dev/disk/by-id/ata-WDC_WD2500AAJS-07B4A0_WD-WCAT13047480-part5...                                                                                                                 
systemd-fsck[698]: bootFS: clean, 362/131072 files, 40407/523760 blocks
systemd-fsck[702]: vmdisk1FS: clean, 113/8060928 files, 3620785/32214830 blocks
systemd-fsck[703]: vmdisk2FS: clean, 11/8060928 files, 553925/32214830 blocks
systemd-fsck[699]: homeFS: clean, 1165/262144 files, 298843/1048576 blocks
systemd-fsck[700]: tmpFS: clean, 74/524288 files, 70814/2097152 blocks
systemd-fsck[701]: varFS: clean, 1650/524288 files, 209616/2097152 blocks
Started File System Check on /dev/disk/by-id/ata-WDC_WD2500AAJS-07B4A0_WD-WCAT12873197-part5                                                                                                             [1;32m  OK  [0m]
Starting /vmdisk1...                                                                                                                                                                                             
Started File System Check on /dev/disk/by-id/ata-WDC_WD2500AAJS-07B4A0_WD-WCAT13047480-part5                                                                                                             [1;32m  OK  [0m]
Starting /vmdisk2...                                                                                                                                                                                             
Started /vmdisk1                                                                                                                                                                                         [1;32m  OK  [0m]
Started File System Check on /dev/md0                                                                                                                                                                    [1;32m  OK  [0m]
Starting /boot...                                                                                                                                                                                                
Started /boot                                                                                                                                                                                            [1;32m  OK  [0m]
Started File System Check on /dev/systemVG/tmpLV                                                                                                                                                         [1;32m  OK  [0m]
Starting /tmp...                                                                                                                                                                                                 
Started /vmdisk2                                                                                                                                                                                         [1;32m  OK  [0m]
Started File System Check on /dev/systemVG/homeLV                                                                                                                                                        [1;32m  OK  [0m]
Started File System Check on /dev/systemVG/varLV                                                                                                                                                         [1;32m  OK  [0m]
Starting /var...                                                                                                                                                                                                 
Starting /home...                                                                                                                                                                                                
Started /var                                                                                                                                                                                             [1;32m  OK  [0m]
Starting Load Random Seed...                                                                                                                                                                                     
Starting Runtime Directory...                                                                                                                                                                                    
Starting Lock Directory...                                                                                                                                                                                       
Started /tmp                                                                                                                                                                                             [1;32m  OK  [0m]
Started /home                                                                                                                                                                                            1;32m  OK  0m                                         

I omitted unessential and uncorrelated parts of boot.log.
Do you think everything is fine ?

That looks okay to me. There would be other lines of output from “fsck” if there were problems. After a forced power off, I usually see a message about recovering from journal, and I am not seeing that in your logs.

Today i have had the opportunity to test on a different machine with a different disks layout:

single disk , no md raid

sda1 -> /boot (ext4)
sda2 -> swap
sda3 -> LVM Volume Group
dm0 -> /home (ext4)
dm1 -> / (ext4)
dm2 -> /tmp (ext4)
dm3 -> /var (ext4)

I observed exactly the same error messages while halting the system. the file boot.log looks clear, very similar to the other one i have posted before.

nrikert i think (and hope) you are right. Now i’m going to have a look into the halt script to understand something more.
For sure we can exclude md raid.The (probably cosmetic only) problem could reside in LVM or in the particular partition layout i use.

Recommend you compare your custom mount points with a default system. IMO is likely that you can add additional mount points but default systemd mount points should be unaltered

Here is mine (a default system not using RAID but is unlikely important) for comparison


# df
Filesystem     1K-blocks      Used Available Use% Mounted on
rootfs         121009300  26136332  93766524  22% /
devtmpfs         2019248        16   2019232   1% /dev
tmpfs            2028444      2252   2026192   1% /dev/shm
tmpfs            2028444     12328   2016116   1% /run
/dev/sda2      121009300  26136332  93766524  22% /
tmpfs            2028444         0   2028444   0% /sys/fs/cgroup
tmpfs            2028444     12328   2016116   1% /var/lock
tmpfs            2028444         0   2028444   0% /media
tmpfs            2028444     12328   2016116   1% /var/run
tmpfs            2028444       428   2028016   1% /tmp
/dev/sdb1      940790496 370275932 569558776  40% /home

So, on my system the only custom mount point I specified is that /tmp be tmpfs.
Am installed across 2 disks, as you can see my root and swap are installed on sda and my data (/home) is installed on sdb.
Of particular interest to you is that however you might specify /var, systemd should be mounting all directories including /var/run which are used for very quick temporary functions in tmpfs. Of course, these tmpfs mounts should be unmounted quickly, easily and cleanly on shutdown.

If you think about this default layout, you may also possibly want to re-think your rationale for some of your mountpoints like /var… It may not make as much sense today as it did in the past.

HTH,
TSU

You might be misreading what jjletho posted.

Here’s an excerpt from “df” output on my system:


devtmpfs                   2005272        4   2005268   1% /dev
/dev/mapper/homesys-root  41284928  6669840  32517936  18% /
/dev/mapper/homesys-home  61927420   995568  59862308   2% /home
/dev/sda1                   193367    59785    123598  33% /boot
/dev/mapper/cr_shared    501586304 70895424 405211764  15% /shared

That’s basically the “df” output with the tmpfs mounts removed from listing.

I could, alternately, describe the same thing as:

sda1 –> /boot
sda2 –> LVM volume group
dm-1 –> /home
dm-2 –> /
dm-3 –> swap
dm-4 –> /shared

That’s the way crypto is setup. For example, “/dev/mapper/homesys-home” is a symlink to “/dev/dm-1”.

Apart from having a separate “/var”, jjletho is using a reasonably standard crypto setup.

Thank you all for your suggestions!

This is the output of df:


rootfs                       30963708  6093548  23297296  21% /
devtmpfs                      1873536       36   1873500   1% /dev
tmpfs                         1881692      160   1881532   1% /dev/shm
tmpfs                         1881692      808   1880884   1% /run
/dev/mapper/systemVG-rootLV  30963708  6093548  23297296  21% /
tmpfs                         1881692        0   1881692   0% /sys/fs/cgroup
tmpfs                         1881692        0   1881692   0% /media
/dev/sdb5                   126835676   192056 124066436   1% /vmdisk2
/dev/sda5                   126835676 17349936 106908556  14% /vmdisk1
/dev/mapper/systemVG-tmpLV    8256952   150296   7687228   2% /tmp
/dev/mapper/systemVG-homeLV   4128448  1749488   2169248  45% /home
/dev/mapper/systemVG-varLV    8256952   823608   7013916  11% /var
/dev/md0                      2062096   128684   1828660   7% /boot
tmpfs                         1881692      808   1880884   1% /var/run
tmpfs                         1881692      808   1880884   1% /var/lock

So apparently yast has added to my personalized partitioning schema some specific optimization like /var/run /var/lock etc. mounted as tmpfs. The only line which looks obscure to me is the first…Is it a sort of alias of my root filesystem (/dev/mapper/systemVG-rootLV) ? I came from red hat based distro, this is my first experience with opensuse…

Why do you suggest to do not keep a separated /var ? I have always thought that keeping log files separated from the root filesystem was a good idea.

by the way, i’m NOT using any encrypted filesystem just ext4 on top of lvm on top of mdraid

On 2013-02-13 09:56, jjletho wrote:
> Why do you suggest to do not keep a separated /var ? I have always
> thought that keeping log files separated from the root filesystem was a
> good idea.

Yes… if you have a significant volume of data there, perhaps.


Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

First,
It looks to me that your system is setup properly by systemd with the additional mount points (nrickert - It was precisely the info you excised from your data that was important).

As for deploying /var on its own disk (not just mount point), that can be a good practice on a machine with heavy and constant load… IF writing to the logfiles creates an undue additional load on the disk subsystem. But, if your machine is not under constant load (eg not a heavily accessed Server) I doubt that it should make that much diff. Today, disks are typically running at 7800rpm, not 5400 rpm (or slower). As you can see, systemd on openSUSE has created special mount points <in RAM> for various “run” and “temp” (although not /tmp by default) which are mainly used by apps for very temporary write/reads significantly removing a normally heavy load (in earlier systems).

Consider though that by deploying on multiple disks that the disks themselves may be introducing some timing issues not typical of a “normal” shutdown, at the least there will be latencies. I cannot even begin to speculate if something more might be happening but IMO the syslog is the first place to analyze.

So, I doubt that mounting /var on its own disk is important today for most people, and almost certainly not necessary for a Desktop or light or intermittently used small server. But, you never know… maybe someone will do some testing and prove me wrong… :slight_smile:

HTH,
TSU

On 2013-02-14 20:16, tsu2 wrote:
> As for deploying /var on its own disk (not just mount point), that can
> be a good practice on a machine with heavy and constant load… IF
> writing to the logfiles creates an undue additional load on the disk
> subsystem.

It is not only the log files, there are other interesting places there.
There can be databases (mysql, postgress), mail (smtp, imap), faxes
(hylafax), rpms downloaded by yast/zypper… There are a lot of things
stored under /var in different subdirectories, so there can be reasons
to increase the space and add dedicated partitions or even disks, as you
say. But doubtful for desktops or small servers, the reason for using a
separate /var has to be explained better.

For example, I use a separate /var/spool/news because I use reiserfs for
it, thousands of small files and inode waste.


Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

I think there are a lot of good reasons to have a separated /var. First of all is to avoid root filesystem becoming full due to some log files getting too big for some reasons. The other main reason because i prefer a separated /var is that i can resize it independently from the rest of the system (using lvm).

So my main doubt now is to know if this can lead to some strange shutdown problem if used in combination with md raid and lvm.

There is someone of you with a similar configuration ( separated /var, lvm on top of mdraird) able to confirm the presence of the messages i signaled on my first post while halting the system ? I would like to be sure they are really normal.