Custom systemd service not executed at startup

English Install/Boot/Login
#1

This is the systemd service template /usr/local/lib/systemd/system/journal-split.service:

[Unit]
Description=Journal Splitter Service / System Init Messages
Documentation=man:journalctl(1) man:systemd-journal-remote(8)
#RefuseManualStart=true

[Service]
Type=oneshot
RemainAfterExit=yes
User=root
Group=systemd-journal
ExecStart=/usr/local/lib/systemd/journal-split.sh
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target default.target

Service installation:

$ systemd-analyze verify journal-split.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable journal-split.service
Created symlink /etc/systemd/system/multi-user.target.wants/journal-split.service → /usr/local/lib/systemd/system/journal-split.service.
Created symlink /etc/systemd/system/default.target.wants/journal-split.service → /usr/local/lib/systemd/system/journal-split.service.
$ sudo reboot

After reboot, the service gets loaded but not executed:

$ systemctl status journal-split
○ journal-split.service - Journal Splitter Service / System Init Messages
     Loaded: loaded (/usr/local/lib/systemd/system/journal-split.service; enabled; vendor preset: disabled)
     Active: inactive (dead)

What may prevent execution?

I can execute the service manually using sudo systemctl start journal-split and it runs successfully.

#2

No, it is not - it is service definition. “Template” in systemd means something different.

Did you look in logs?

#3

No entries in journal. Do you have other logs in mind?

#4

Reboot and upload full output of journalctl -b --no-pager --full to https://susepaste.org/

#5

https://paste.opensuse.org/pastes/3109667845b7

#6 
Apr 30 13:17:50 xs systemd[1]: Mounted /usr/local.

Unit definition is not present when initial transaction is computed.

#7

Hmmm:

Apr 30 13:17:50 xs systemd[1]: Mounted /usr/local.
Apr 30 13:17:50 xs systemd[1]: Mounted /var.
Apr 30 13:17:50 xs systemd[1]: Condition check resulted in Lock Directory being skipped.
Apr 30 13:17:50 xs systemd[1]: Condition check resulted in Runtime Directory being skipped.
Apr 30 13:17:50 xs systemd[1]: Reached target Local File Systems.

So /usr/local is mounted before reaching local-fs.target, right?

And local-fs.target is reached long before target multi-user.target is reached, correct?

I can not follow your argumentation, sorry.

#8

systemd computes units to start on boot once when it [systemd] is started (or after switch to the real root from initrd). If unit definition is not available at this point, it is not included in boot transaction.

#9

OK, understood. Instead of a symlink, I will copy the service definition itself to /etc/systemd/system/multi-user.target.wants and see if it works then. Give me a minute.

#10

From man systemd.unit:

System Unit Search Path
/etc/systemd/system.control/*
/run/systemd/system.control/*
/run/systemd/transient/*
/run/systemd/generator.early/*
/etc/systemd/system/*
/etc/systemd/system.attached/*
/run/systemd/system/*
/run/systemd/system.attached/*
/run/systemd/generator/*
…
/usr/lib/systemd/system/*
/run/systemd/generator.late/*

/usr/local/lib/systemd/system/ is not in the above list.

#11

If you scroll down that man page until section titled Unit File Load Path, there’s a table which contains this row:

/usr/local/lib/systemd/system System units installed by the administrator

And if you think about it, the command sudo systemctl enable journal-split from the first post wouldn’t have worked if it was like you said. It would have thrown an error message that it can’t find journal-split.service.

I’d consider this a bug: If this path is explicitly foreseen to hold system units installed by the administrator, then it must be available at the time when systemd looks for unit definitions.

#12

I did this:

$ sudo systemctl disable journal-split
Removed /etc/systemd/system/multi-user.target.wants/journal-split.service.
Removed /etc/systemd/system/default.target.wants/journal-split.service.
$ sudo cp /usr/local/lib/systemd/system/journal-split.service /etc/systemd/system/multi-user.target.wants
$ sudo cp /usr/local/lib/systemd/system/journal-split.service /etc/systemd/system/default.target.wants
$ sudo reboot

No change. Service is not being started. No entries in journal.

#14

But this is my fault, it seems:

$ systemd-analyze verify journal-split.service
multi-user.target: Wants dependency dropin /etc/systemd/system/multi-user.target.wants/journal-split.service is not a symlink, ignoring.

Will try to copy the service definition to /etc/systemd/system/ and create symlinks from there.

#15

This worked:

$ sudo mv /etc/systemd/system/multi-user.target.wants/journal-split.service /etc/systemd/system/
$ sudo rm /etc/systemd/system/default.target.wants/journal-split.service
remove default.target:
$ sudo vi /etc/systemd/system/journal-split.service
$ systemd-analyze verify journal-split.service
$ sudo systemctl enable journal-split
Created symlink /etc/systemd/system/multi-user.target.wants/journal-split.service → /etc/systemd/system/journal-split.service.
$ sudo reboot

$  systemctl status journal-split
● journal-split.service - Journal Splitter Service / System Init Messages
     Loaded: loaded (/etc/systemd/system/journal-split.service; enabled; vendor preset: disabled)
     Active: active (exited) since Sun 2023-04-30 14:28:50 CEST; 43s ago
       Docs: man:journalctl(1)
             man:systemd-journal-remote(8)
    Process: 643 ExecStart=/etc/systemd/journal-split.sh (code=exited, status=0/SUCCESS)
   Main PID: 643 (code=exited, status=0/SUCCESS)

Apr 30 14:28:49 xs systemd[1]: Starting Journal Splitter Service / System Init Messages...
Apr 30 14:28:49 xs journal-split.sh[643]: System Init messages saved to /var/log/sysinit.log
Apr 30 14:28:49 xs sudo[677]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/journalctl --list-boots
Apr 30 14:28:49 xs journal-split.sh[643]: last boot: 2023-04-30 14:04:48, saving entries to file 04301404.journal
Apr 30 14:28:50 xs journal-split.sh[782]: Finishing after writing 575 entries
Apr 30 14:28:50 xs systemd[1]: Finished Journal Splitter Service / System Init Messages.

Thank you, this can be closed. The documentation or implementation bug is noted.

#16

Yep. On the other hand this option is excluded from snapshotting and comes with some fine print: directory structure - Where do I put my systemd unit file? - Unix & Linux Stack Exchange

  • I prefer to put all local stuff into /etc/systemd/system.
  • I never specified default.target with WantedBy=.
  • I was never puzzled by a service not executing with WantedBy=multi-user.target specified in the [Install section].
#17

This is what I take as the lessons learned from this, at least for system stuff. For userland stuff (except services and other systemd stuff), I think that /usr/local is the right place.

This is only needed for services running under the current user, i.e. when you use the --user option with systemd.

Good for you :slight_smile: This is because you never tried putting your service definition in /usr/local/lib/systemd/system instead of /etc/systemd/system.

#18

Note this paragraph in the linked page:

Gray Area

On the one hand, the File Hierarchy Standard (also man file-hierarchy) specifies that /etc is for local configurations that do not execute binaries. On the other hand it specifies that /usr/local/ “is for use by the system administrator when installing software locally”. You could also argue (if not just for the purpose of organization) that all system unit files should go under /usr/local/lib/systemd/system, but this is intended for unit files that are part of “software” not from a package manager. The corresponding systemd user units that are system-wide could go under /usr/local/lib/systemd/user.

If one does follow this logic, she will end up in the same mess as I did.

#19

I have:

karl@erlangen:~> systemctl --user cat save-jalbum-settings.service 
# /home/karl/.config/systemd/user/save-jalbum-settings.service

[Unit]
Description=Save jAlbum Project Files

[Service]
ExecStart=/bin/bash /home/karl/bin/save-jalbum-settings.sh

[Install]
WantedBy=default.target
karl@erlangen:~>

Yep. See above example.

When I started working with the HP 1000 - 21-MX E-Series in 1977 I needed to produce results and thus I learned at an early stage to avoid less common options. The same strategy works in 2023. I need no longer produce results. However I don’t want to waste my time either.