System recovery with chroot and systemd

English Install/Boot/Login
1

Managed to break booting on my 12.3 system by attempting to disable consolekit, after thinking it was no longer needed (oops). Fortunately, my system has several other Linux distros I can boot from, so I can easily get into my broken 12.3 system with a chroot. Tried to enable the consolekit units again, but much to my dismay, systemctl actively refuses to run when it detects it is running in a chroot environment!

“Running in chroot, ignoring request.”

Huh? I really haven’t been happy with the whole switch to systemd in the first place (on any distro), but intentionally breaking a useful method- chrooting in to fix things -smacks of poor and user-hostile design to me. Mutter, mutter, get off my lawn, etc.

Anyway, rant over. How can I get systemctl to work while chrooted? Am I not bind mounting enough, or will the systemd tools never work while chrooted no matter what I do? Only other option is to manually emulate “systemctl enable” by looking at the unit files and linking them back, I suppose.

2

On 2013-03-28 20:16, hallbw wrote:
>
> but much to my dismay, systemctl actively
> refuses to run when it detects it is running in a chroot environment!
>
> “Running in chroot, ignoring request.”

Wow.

> Huh? I really haven’t been happy with the whole switch to systemd in
> the first place (on any distro), but intentionally breaking a useful
> method- chrooting in to fix things -smacks of poor and user-hostile
> design to me. Mutter, mutter, get off my lawn, etc.

Maybe ask in the factory mail list, see if the devs there can explain it.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

3

When you say you broke booting, what does that mean? Does Grub 2 load? Can you select the kernel and then it blows up? For me, I would have already reloaded after making such a mistake I think. I found this link on the subject:

https://bugzilla.novell.com/show_bug.cgi?id=790147

Thank You,

4

If you have a question about program behavior, you should always copy and paste exact program invocation and its output. Not explain what you think may have happened.

The message above simply means that systemctl cannot request systemd to re-read services configuration after it was changed. That’s all. If you tried “systemctl enable”, systemctl did enable service (if not, it is a bug).

If you tried something else, then please show exactly what you did.

5

systemd-nspawn can be used to boot another Linux OS from within a particular environment

The following guide might be helpful

Wunschkonzert, Ponyhof und Abenteuerspielplatz

6

Actually after having actually played around with systemd-nspawn for awhile now, I fully understand that particular blog and in it he describes exactly why you can’t run systemctl from within the chroot. Also, systemd-nspawn is useful for launching and managing the chroot from the Host, it I understand what the OP is describing that won’t be helpful here.

So,
It’s important to know what the Host is, whether an advanced version of systemd is running in the Host. Today, there are two other main distros that fit this requirement, Arch and Fedora 17. YMMV with other distros and versions. You’re going to be SOL running Debian, RHEL or Ubuntu today.

If the Host is one of these three (and others <might> work) you can actually run systemctl from outside the chroot on the Host. See the referenced blog for the naming scheme so you can know how to manage units running in the chroot.

Yes, this a big change even in how chroot runs on a systemd system, no longer are many tasks and apps running as a hierarchical cascading sequence of scripts, now everything runs as a flattened configured Unit. And after understanding how things work in systemd you might find it an improvement over systemVinit.

HTH,
TSU

7

After thinking about this, am wondering how you disabled consolekit. If you did this by modifying the Unit configuration, then you also have another option.

So first, yes you can use “systemctl enable serivicename” to re-enable autostart, Of course if you went further so the Unit isn’t even configured to load, you’d have to address that, too.

But, you also have an option to edit the Unit config file with a text editor instead of using systemctl, although it’s highly inadvisable because you’d be exposed to making serious mistakes.

If you need to customize Unit files, the location of the default (original) files is at

 /usr/lib/systemd/system

Although you edit the file directly, it’s inadvisable. Best Practice is to copy whatever Unit file you want to modify the following location and then edit, your file will take precedence over the default

/etc/systemd/system

But, that begs the question in the first place how you disabled consolekit, I can’t find a Unit by that name on my system.

HTH,
TSU

8

OK, mea culpa. Login problem was not due to removing consolekit, it was due to an invalid /dev/shm tmpfs line in /etc/fstab. Fixed that, and now I get a login prompt. To find that, I had to use “journalctl -b” in an emergency console (which systemd dumped me to after it sat there for 5 minutes or so). Yes, I realize I could have done that from the boot prompt, but it timed out while I was looking at the systemd pages on another machine. Not fond of systemd having a binary log either- just use the system logger like everyone else, please!

I was also able to restore the console kit unit in a chroot, but the syntax is critical- leaving off the .service extension when enabling won’t work, but it will when disabling, so beware.

systemctl enable console-kit-daemon.service

ln -s ‘/usr/lib/systemd/system/console-kit-daemon.service’ ‘/etc/systemd/system/graphical.target.wants/console-kit-daemon.service’

I originally tried to disable consolekit with:

systemctl disable console-kit-daemon
systemctl disable console-kit-log-system-start

Apparently consolekit is still needed for xfce4, since when I removed the ck-launch-session prefixing startxfce4, I no longer had authorization to suspend or mount, and the XDG_SESSION_ID was inactive. Now, it all works but I have to start X from a login manager (ack)- currently using xdm for that until I can get it working from a console login, which should be possible based on tips I’ve found elsewhere.

I still maintain systemd doesn’t need to disallow everything except enable/disable units in a chroot- that it does is a fact, see Pottering’s systemd for sysadmins series- I just disagree. BTW, for anyone else who goes down this rabbit hole, bind mounting /run in the chroot does NOT get around the problem. Might have to hack the systemctl source for that.

My main complaint about systemd is that is breaks backwards compatibility where it isn’t necessary. I agree it is better than SysV init, but really, is it too much to ask for systemd to have a backwards compatibility layer, particularly for when things break? Text logs, full chroot functionality, and wrapper scripts would not prevent a better boot experience but would make the transition much less painful.

9

On 2013-03-29 21:16, hallbw wrote:
> My main complaint about systemd is that is breaks backwards
> compatibility where it isn’t necessary. I agree it is better than SysV
> init, but really, is it too much to ask for systemd to have a backwards
> compatibility layer, particularly for when things break? Text logs, full
> chroot functionality, and wrapper scripts would not prevent a better
> boot experience but would make the transition much less painful.

Indeed :frowning:

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

10

syslog did not disappear, you still have all your text logs you had before. What compatibility is broken here? And systemd is free to use whatever it wants for new functionality. You may argue and discuss this new functionality but it has nothing to do with compatibility.

full chroot functionality

What exactly does not work in chroot that did work before?

wrapper scripts

Example please. Wrapper scripts for what?

11

Although there is plenty of legacy support for SysVinit, chroot isn’t one of them and the blog post describes why…

Whereas in SysVinit the processes are <all> children of Parent(s), that’s not the case in systemd… In systemd, there is one enormous pool of “equal opportunity, full citizen” processes which run as configurable Units, again every Unit is “equal.”

The result is that whereas in traditional chroot you can hope that child processes might be “managed” through the parent/child relationships, in systemd processes are “just there” and invoked or torn down as needed without the hierarchical structure found in SysVinit.

For that reason although I’m still a bit unclear what happens when invoking traditional chroot and would still want to test, definitely when you invoke using systemd-nspawn the chrooted child processes should be isolated, running in a parent cgroup. After all, you don’t <really> want your chroot Users granted access to Host and other chroots, do you?

On a slight tangent, IIRC there is a way to enable Host credentials in the chroot and login as such, but I’d have to look that up… If you want to break security. But of course that won’t and shouldn’t be default.

As for the journal and tradiional system logging, at least for all current distro versions the journal messages are being piped to the legacy syslogs, so although there would not be any change how difficult it is to find events in the traditional syslog you should see everything that is also in the journal. So, this is still usable in openSUSE 12.3 just like previous versions

tail -n 300 /var/log/messages

Regarding whether the Unit suffix is required (eg .service), this is supposed to be a recent “improvement” and may not be fully debugged. Recommend submitting a bug if you believe the syntax is not consistent as you expected. But for me, I’m just used to typing the entire name including suffix, like in coding it’s better to be safe than to be sorry.

As I noted before, if systemctl is not available in your environment, there is no need to hack the C code and risk your system. I’d recommend you just open up a text editor and modify the Unit config. To be safe, leave the original Unit untouched, copy the Unit to /etc/systemd/ before making your changes.

IMO and HTH,
TSU