Apache2 fails to start on boot but start OK when system is up.

English Applications
1

I am runnning TW/KDE on a new desktop and have this strange issue where Apache2 fails to start at boot time but starts OK as soon as system is up. The system is booting from an SSD and the Apache2 document root is on an HDD mounted via fstab. Apache2 fails with


DocumentRoot '/home/stuart/Work/Web-SSI/' is not a directory, or is not readable

Actually there are two mounts needed one for the Work and one for Web-SSI. I’ guessing that Apache2 starts before the fstab mounts are completed as this never used to happen on the old system which only had HDDs.

Is there any easy way to get round this and still have Apache2 start as part of the boot process?

Stuart

2

Sure. Have apache wait for local file systems: After=local-fs.target

3

Perfect… thanks very much all working fine now.

Stuart

4

Spoke too soon, the first reboot and Apache2 started OK, seond time I rebooted it failed with the same error. Checked the /etc/systemd/system/apache.conf still has local-fs.target in the After statement.


After=network.target nss-lookup.target time-sync.target remote-fs.target local-fs.target
Before=getty@tty1.service plymouth-quit.service xdm.service

Stuart

5

What are your /etc/fstab entries apache is waiting for? They should be dependencies of local-fs.target.


erlangen:~ # systemctl list-dependencies local-fs.target
local-fs.target
● ├─-.mount
● ├─\x2esnapshots.mount
● ├─boot-efi.mount
● ├─boot-grub2-i386\x2dpc.mount
● ├─boot-grub2-x86_64\x2defi.mount
● ├─hd-idle.service
● ├─home.mount
● ├─opt.mount
● ├─root.mount
● ├─srv.mount
● ├─systemd-remount-fs.service
● ├─tmp.mount
● ├─usr-local.mount
● ├─var-lock.mount
● ├─var-run.mount
● └─var.mount
erlangen:~ #
6

Here they are:-


systemctl list-dependencies local-fs.target
local-fs.target
● ├─-.mount
● ├─boot-efi.mount
● ├─home-stuart-Work-Graphics.mount
● ├─home-stuart-Work-Graphics1.mount
● ├─home-stuart-Work-kdenlive.mount
● ├─home-stuart-Work-Shared\x2dXP1.mount
● ├─home-stuart-Work-Video\x2dWorking.mount
● ├─home-stuart-Work-VirtualBox.mount
● ├─home-stuart-Work-Web\x2dSSI.mount
● ├─home-stuart-Work.mount
● ├─home.mount
● ├─systemd-fsck-root.service
● ├─systemd-remount-fs.service
● ├─tmp.mount
● ├─var-lock.mount
● ├─var-run.mount
● └─var.mount

Looking at fstab the mount for the document root partition is last in fstab and is attached to an SATA controller in a PCI slot not a motherboard connector. Could that be a tad slow and would it make a difference if I moved the fstab entry to higher up (assuming they are mounted in order?


UUID=190442a2-32b3-47e7-bac7-a39e6841236f  /          ext4  defaults      0  1
UUID=1957a46b-74d5-40d6-baa5-e5ae1b6ecbcd  swap       swap  defaults      0  0
UUID=40e0966e-8ece-4851-9c18-a4b6b4bfc8dc  /var       ext4  data=ordered  0  2
UUID=f48f0621-5155-43bd-a051-d82568315b07  /home      ext4  data=ordered  0  2
UUID=98F0-942E                             /boot/efi  vfat  utf8          0  2
/dev/disk/by-uuid/2b996ff0-3fac-46d0-85d5-dc6e8f4a60e9 /home/stuart/Work auto nosuid,nodev,nofail,x-gvfs-show 0 0
/dev/disk/by-uuid/adea671a-5027-4843-9b61-4a96ce6b33d2 /home/stuart/Work/Graphics1 auto nosuid,nodev,nofail,x-gvfs-show 0 0
/dev/disk/by-uuid/5d31f76b-bfd9-4f11-a88d-bbb723df8f1a /home/stuart/Work/Video-Working auto nosuid,nodev,nofail,x-gvfs-show 0 0
/dev/disk/by-uuid/6c21a856-6303-46e2-ae8b-770c97ad0e24 /home/stuart/Work/VirtualBox auto nosuid,nodev,nofail,x-gvfs-show 0 0
/dev/disk/by-uuid/6d2611f7-c1bb-49ba-8a1d-084f26b2b1cb /home/stuart/Work/kdenlive auto nosuid,nodev,nofail,x-gvfs-show 0 0
/dev/disk/by-uuid/e3954a91-421b-436a-ace5-f7e1c374625f /home/stuart/Work/Shared-XP1 auto nosuid,nodev,nofail,x-gvfs-show 0 0
/dev/disk/by-uuid/cc20ba60-b8a7-4f25-bdf3-bd2d546f83ea /home/stuart/Work/Web-SSI auto nosuid,nodev,nofail,x-gvfs-show 0 0                       << this is the document root
/dev/disk/by-uuid/a179c87b-6443-4ece-935f-274391201d65 /home/stuart/Work/Graphics auto nosuid,nodev,nofail,x-gvfs-show 0 0

Stuart

7

This shuld not matter. There are several units:

erlangen:~ # systemctl list-unit-files apache*
UNIT FILE        STATE    VENDOR PRESET
apache.service   alias    -            
**apache2.service  enabled  disabled     
**apache2@.service disabled disabled     
apache2.target   static   -            

4 unit files listed.
erlangen:~ # systemctl list-units apache*
  UNIT            LOAD   ACTIVE SUB     DESCRIPTION         
  apache2.service loaded active running The Apache Webserver

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
erlangen:~ #

Verify the changes made by running: ‘systemctl status apache2.service’

8

Well here is the output from those commands


systemctl list-unit-files apache*
UNIT FILE        STATE    VENDOR PRESET
apache.service   alias    -            
apache2.service  enabled  disabled     
apache2@.service disabled disabled     
apache2.target   static   -            

4 unit files listed.
localhost:/home/stuart # ystemctl list-units apache*

systemctl list-units apache*
  UNIT            LOAD   ACTIVE SUB     DESCRIPTION         
  apache2.service loaded active running The Apache Webserver

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

and


systemctl status apache2.service
● apache2.service - The Apache Webserver
     Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2020-10-08 10:57:58 BST; 2h 7min ago
   Main PID: 3040 (httpd-prefork)
     Status: "Processing requests..."
      Tasks: 11
     Memory: 24.7M
     CGroup: /system.slice/apache2.service
             ├─3040 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3048 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3050 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3051 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3052 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3210 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3212 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3214 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3215 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             ├─3216 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>
             └─3217 /usr/sbin/httpd-prefork -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc>

Oct 08 10:57:58 localhost.localdomain systemd[1]: Starting The Apache Webserver...
Oct 08 10:57:58 localhost.localdomain systemd[1]: Started The Apache Webserver.

bear in mind that I started the apache service manually after boot up.

Stuart

9

I can’t see your changes. Show the output:

erlangen:~ # systemctl cat apache2.service 
# /usr/lib/systemd/system/apache2.service
[Unit]
Description=The Apache Webserver
After=network.target nss-lookup.target time-sync.target remote-fs.target
Before=getty@tty1.service plymouth-quit.service xdm.service
PartOf=apache2.target

[Service]
Type=notify
PrivateTmp=true
ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start
ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful
ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop
KillMode=mixed
TasksMax=infinity
NotifyAccess=all

[Install]
WantedBy=multi-user.target
Alias=httpd.service apache.service

erlangen:~ # 


erlangen:~ # systemd-analyze critical-chain apache2.service
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

apache2.service +102ms
└─time-sync.target @571ms
  └─chronyd.service @509ms +61ms
    └─network.target @505ms
      └─NetworkManager.service @415ms +89ms
        └─dbus.service @413ms
          └─basic.target @411ms
            └─sockets.target @411ms
              └─pcscd.socket @411ms
                └─sysinit.target @409ms
                  └─systemd-udevd.service @295ms +113ms
                    └─systemd-tmpfiles-setup-dev.service @273ms +11ms
                      └─kmod-static-nodes.service @254ms +7ms
                        └─systemd-journald.socket
                          └─-.mount
                            └─system.slice
                              └─-.slice
erlangen:~ #
10 

systemctl cat apache2.service 
# /usr/lib/systemd/system/apache2.service
[Unit]
Description=The Apache Webserver
After=network.target nss-lookup.target time-sync.target remote-fs.target local-fs.target
Before=getty@tty1.service plymouth-quit.service xdm.service
PartOf=apache2.target

[Service]
Type=notify
PrivateTmp=true
ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start
ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful
ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop
KillMode=mixed
TasksMax=infinity
NotifyAccess=all

[Install]
WantedBy=multi-user.target
Alias=httpd.service apache.service




systemd-analyze critical-chain apache2.service
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

apache2.service +59ms
└─time-sync.target @1.103s
  └─chronyd.service @1.027s +75ms
    └─network.target @1.026s
      └─NetworkManager.service @977ms +48ms
        └─dbus.service @975ms
          └─basic.target @972ms
            └─sockets.target @972ms
              └─pcscd.socket @972ms
                └─sysinit.target @970ms
                  └─systemd-update-utmp.service @961ms +8ms
                    └─auditd.service @949ms +11ms
                      └─systemd-tmpfiles-setup.service @937ms +11ms
                        └─local-fs.target @776ms
                          └─home.mount @528ms +247ms
                            └─systemd-fsck@dev-disk-by\x2duuid-f48f0621\x2d5155\x2d43bd\x2da051\x2dd82568315b07.service @324ms +203ms
                              └─local-fs-pre.target @321ms
                                └─systemd-tmpfiles-setup-dev.service @310ms +11ms
                                  └─kmod-static-nodes.service @294ms +7ms
                                    └─systemd-journald.socket
                                      └─-.mount
                                        └─system.slice
                                          └─-.slice

Again does it matter that I have manually started apache?

Stuart

11

I believe I may have sorted this, at least two boots in a row has apache starting each time on boot up.

I notice that there was no local-fs.target.wants whether or not this is an issue I dont know, however I searched around on the internet and found a similar issue. This was fixed but running systemctl listunits --type=mount which shows all the mounts. I selected the mount for the partition containing the document root and added home-stuart-Work.mount to the After= for apache instead of local-fs.target. Now it seems to be working.
So now I believe this is a timing issue using an SSD for the system and having HDDs for all user data. Now it could be that there is an issue with local-fs.target but checking that with systemctl shows it started OK anyway for now it seems I have apache starting on boot.

Time will tell…

Stuart

12

This is already default for all normal services.

systemd does not wait for filesystems marked as “nofail”. Add explicit

RequiresMountsFor=/home/stuart/Work/Web-SSI

to your apache service

13

Not all versions of systemd are created equal.

systemd does not wait for filesystems marked as “nofail”.
Missed that option.

14

Great. Thanks for the feedback.

15

Don’t modify unit files owned by a package. Always use ‘systemctl edit -full apache2.service’.

16

Yes,
But you should explain to the User why and what happens.

If a User edits the original Unit file directly, you alter it and might have problems undoing your changes if they don’t work out.

The above command creates a copy of the Unit file in /etc/systemd/system/
When a duplicate Unit file exists in this location and especially when you make changes to it, it will take precedence over the original.
If you decide to undo your changes, you simply remove this duplicate and the original file will be the active file.
Also, if you edits could potentially crash the process, systemd will use the original Unit file configuration instead of your custom file so your system isn’t borked.

TSU

17

Thanks for commenting. See also: How To Use Systemctl to Manage Systemd Services and Units | DigitalOcean