VG activation + LVM Snaphots = long time = systemctl timeout

ceinma · October 3, 2014, 10:00pm

Hi ,

Opensuse 13.1 .

This is a fresh install + online updates, where I’ve write some scripts to create a snapshot of MySQL file system each 4 hours.
All work’s fine for few days and fews snapshots already taken.

Now I’ve restarted this linux and the systemctl always timeout when it mounting this FS and this **** (sorry , I hate systemctl) “enter” in emergency mode but don’t give me the prompt to take any action. (always , for some reason it enter in emergency mode, I need to boot with DVD or rescue disc and use they emergency mode)
So , after reboot with my rescue DVD , I’ve comment at /etc/fstab the mount of this FS to able boot this linux.

After boot successfully , without mount this FS and I execute a “vgchange -ay” , take close of 10 minutes to LVM “check” the snapshots…

This is it checking the snapshots…


root@jdivm04:~
# lvs
  LV         VG      Attr      LSize   Pool Origin  Data%  Move Log Copy%  Convert
  root       system  rwi-aor--  40.00g
  swap       system  -wi-ao---   2.00g
  lvdados    vgdados rwi-aor-- 300.00g
  lvSnap_004 vgmysql swi-i-s--  10.00g      lvmysql 100.00
  lvSnap_005 vgmysql swi-i-s--  10.00g      lvmysql 100.00
  lvSnap_006 vgmysql swi-i-s--  10.00g      lvmysql 100.00
  lvSnap_007 vgmysql swi-d-s--  10.00g      lvmysql 100.00
  lvSnap_008 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_009 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_010 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_011 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_012 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_013 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_014 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_015 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_016 vgmysql swi---s--  10.00g      lvmysql
  lvSnap_017 vgmysql swi---s--  10.00g      lvmysql
  lvmysql    vgmysql owi-i-r--  32.00g

So , I consider two problems here…

This slowness behave of LVM + snapshots at “vgchange -ay” are expected ?
Maybe this is some misconfiguration ?
The systemctl don’t know how dealing with this situation… (and the emergency mode don’t work) .

Suggestions ?
I need this FS mounting automatically at boot , because it is the database of my zabbix which monitor my environments…

gogalthorp · October 4, 2014, 12:28am

Maybe you are out of space??

ceinma · October 4, 2014, 12:41am

Hi,
No… I have lot of free space (inside of VG , and FS) and any snapshot have filled yet… the oldest is in 45% used…

arvidjaar · October 4, 2014, 8:56am

You could try to add x-systemd.device-timeout=10m to mount options in fstab (make it 15m to be sure). This should cause mount to wait enough time for device to appear. Or add nofail - in this case it will skip if device still is not available after timeout and continue boot.

At the end there should be some timeout - you do not want your system to wait indefinitely if device is not present.

ceinma · October 4, 2014, 3:44pm

Thank’s arvdjaar!!!

With your tip I found more information at man systemd.mount .
I consider this systemd very confusing… I like the things at KISS style (Keep It Simple and Stupid) … SystemV just work’s for me…

Well… I update my fstab and will test on next days!
Thank you again.

arvidjaar · October 4, 2014, 8:13pm

If mandatory filesystem cannot be mounted on boot it stops booting and enters emergency mode. Or do you say that it will continue boot in this case?

ceinma · October 4, 2014, 8:50pm

I say that, because as far I remember, with SystemV the things works and at the end the system always boot…
This FS for example (database FS), as far I remember, with SystemV behave it will take the time is need to mount and if fail they just continue , since it isn’t the root.
At least you are able to connect the host remotely and check what happen.

With Systemd , for my point of view and felling as old linux user (desktop and server), appear to be fragile, if anything go weird they stop at emergency mode and stop the boot.
If this is a sever/machine which you expected to run 24x7 and able to suffer a hard reset sometime (any reason) , with systemd you can (and at my opinion, you will) be forced to open the machine console frequently to discovery a fsck take longer and it just stop into emergency mode.

At my environment (two opensuse VMs) there is one aggravation , which is the emergency mode do not work.
It give me the message is entering in emergency mode and “freeze” , don’t give the prompt to type any command and the message entering into emergency mode start to repeat indefinitely .
This behave (don’t give me the prompt) do not occur at first time I was “obligated” to use systemd (opensuse distro upgrades) , then after some update or configuration of my environment , this start occur…

arvidjaar · October 5, 2014, 7:00am

I think the primary difference is not so much sysvinit or systemd, but the fact that in the past LVM was started synchronously during boot (i.e. boot process waited until “vgchange -ay” completes) while now activation happens concurrently in response to new devices. I think it’s worth to raise this issues on both LVM and systemd lists to at least make them aware of such conditions.

which is the emergency mode do not work.

Yes, this bug is almost one year old but for some reason openSUSE maintainer is unwilling to release update for 13.1.

ceinma · October 6, 2014, 5:06pm

Nice to know.
Do you know for some alternative repository where I can add to my zypper and get a more recent version of systemd ?
(which I can trust it’s is a stable version…)

Coincidently, today I just received at my linkedin news, a post of Suse user group about SysV and Systemd …
This post just help me a little to understand why of this and the advantages of systemd.

http://blog.jorgenschaefer.de/2014/07/why-systemd.html