I’ve put together some IPtables rules which modify the INPUT chain by inserting new rules at the top of the chain. I’ve also created a service-unit which works fine when I enable and start it manually (using either the YaST Service Manager or systemctl in the root account) but it won’t start at boot-time.
However I’ve found it’s necessary to stop, disable, enable, and start the new service manually after booting, and I’d like to know whether the symlinks created by enabling a service should be part of that service’s unit definition?
Always show the journal and systemd’s point of view, e.g.:
**erlangen:~ #** journalctl -b -u postfix.service
-- Logs begin at Sat 2021-05-29 13:22:57 CEST, end at Tue 2021-06-08 06:35:50 CEST. --
Jun 08 06:35:16 erlangen systemd[1]: Starting Postfix Mail Transport Agent...
Jun 08 06:35:16 erlangen echo[792]: Starting mail service (Postfix)
Jun 08 06:35:16 erlangen postfix/postfix-script[933]: starting the Postfix mail system
Jun 08 06:35:16 erlangen postfix/master[936]: daemon started -- version 3.5.10, configuration /etc/postfix
Jun 08 06:35:16 erlangen systemd[1]: Started Postfix Mail Transport Agent.
**erlangen:~ #**
**erlangen:~ #** systemctl cat postfix.service
**# /usr/lib/systemd/system/postfix.service**
# This file is part of package postfix.
#
# Copyright (c) 2011 SuSE LINUX Products GmbH, Germany.
# Author: Werner Fink
# Please send feedback to http://www.suse.de/feedback
#
# Description:
#
# Used to start the postfix Mail Transport Agent service
# which handles all mails stored at /var/spool/postfix/ and
# all connections on port 25 aka smtp at localhost as well
# as on all other network interfaces.
#
[Unit]
Description=Postfix Mail Transport Agent
Requires=var-run.mount
After=var-run.mount nss-lookup.target network.target time-sync.target
After=amavis.service mysql.service cyrus.service ldap.service openslp.service ypbind.service
Conflicts=sendmail.service exim.service
[Service]
Type=forking
PIDFile=/var/spool/postfix/pid/master.pid
ExecStartPre=-/bin/echo 'Starting mail service (Postfix)'
EnvironmentFile=-/etc/sysconfig/postfix
ExecStartPre=/usr/lib/postfix/systemd/config_postfix
ExecStartPre=/usr/lib/postfix/systemd/update_chroot
ExecStartPre=/usr/lib/postfix/systemd/update_postmaps
ExecStart=/usr/sbin/postfix start
ExecStartPost=/usr/lib/postfix/systemd/wait_qmgr 60
ExecStartPost=/usr/lib/postfix/systemd/cond_slp register
ExecReload=/usr/sbin/postfix reload
ExecReload=/usr/sbin/postfix flush
ExecStop=/usr/sbin/postfix stop
ExecStopPost=/usr/lib/postfix/systemd/cond_slp deregister
[Install]
WantedBy=multi-user.target
**erlangen:~ #**
[FONT=monospace]**erlangen:~ #** systemctl list-unit-files postfix.service
UNIT FILE STATE VENDOR PRESET
postfix.service **enabled enabled **
1 unit files listed.
**erlangen:~ #**[/FONT]
[FONT=monospace][FONT=monospace]**erlangen:~ #** systemctl list-units postfix.service
UNIT LOAD ACTIVE SUB DESCRIPTION
postfix.service loaded active running Postfix Mail Transport Agent
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
**1 loaded units listed.** Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
**erlangen:~ #**[/FONT][/FONT]
[FONT=monospace][FONT=monospace][FONT=monospace]**erlangen:~ #** systemctl status postfix.service
**●** postfix.service - Postfix Mail Transport Agent
Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: enabled)
Active: **active (running)** since Tue 2021-06-08 06:35:16 CEST; 7min ago
Process: 792 ExecStartPre=/bin/echo Starting mail service (Postfix) (code=exited, status=0/SUCCESS)
Process: 793 ExecStartPre=/usr/lib/postfix/systemd/config_postfix (code=exited, status=0/SUCCESS)
Process: 796 ExecStartPre=/usr/lib/postfix/systemd/update_chroot (code=exited, status=0/SUCCESS)
Process: 803 ExecStartPre=/usr/lib/postfix/systemd/update_postmaps (code=exited, status=0/SUCCESS)
Process: 827 ExecStart=/usr/sbin/postfix start (code=exited, status=0/SUCCESS)
Process: 941 ExecStartPost=/usr/lib/postfix/systemd/wait_qmgr 60 (code=exited, status=0/SUCCESS)
Process: 944 ExecStartPost=/usr/lib/postfix/systemd/cond_slp register (code=exited, status=0/SUCCESS)
Main PID: 936 (master)
Tasks: 3 (limit: 4915)
CPU: 334ms
CGroup: /system.slice/postfix.service
├─936 /usr/lib/postfix/bin//master -w
├─939 pickup -l -t unix -u
└─940 qmgr -l -t unix -u
Jun 08 06:35:16 erlangen systemd[1]: Starting Postfix Mail Transport Agent...
Jun 08 06:35:16 erlangen echo[792]: Starting mail service (Postfix)
Jun 08 06:35:16 erlangen postfix/postfix-script[933]: starting the Postfix mail system
Jun 08 06:35:16 erlangen postfix/master[936]: daemon started -- version 3.5.10, configuration /etc/postfix
Jun 08 06:35:16 erlangen systemd[1]: Started Postfix Mail Transport Agent.
**erlangen:~ #**[/FONT][/FONT][/FONT]
Your subject implies service was not started on boot. If you need to “stop” it it means service was started on boot.
disable, enable
That most certainly is not needed every time.
and start the new service manually after booting
I’ll be glad to receive any feedback.
You never explained what your actual problem was. You made some conclusions and describe your conclusions instead of describing actual facts. Start with describing your problem - why do you need to stop and then start your service after boot. What is wrong until you have done it. Show actual commands and their output you used to decide that something was wrong and explain what is wrong in this output from your point of view and what output you expected instead.
Thanks Flux Capacitor and Karl, I take your points.
I’m reluctant to list O/S evidence of the problem because the additional IPtables rules are inserted correctly into the start of the INPUT chain when my service is started interactively, but not at all when the system is booted. (And the one rule I’ve tested appears to work correctly too.) However listing the INPUT chain in each case would reveal my site-dependent rules for all to see when their integrity isn’t demonstrated yet.
Here is the output of the Service Manager Details and Log:
(a) After starting the service interactively:
# systemctl status iptables-restore-ADL
● iptables-restore-ADL.service - Restore SysAdmin iptables rules at boot-time.
Loaded: loaded (/usr/lib/systemd/system/iptables-restore-ADL.service; enabled; vendor preset: >
Active: active (exited) since Tue 2021-06-08 14:13:57 AEST; 3h 13min ago
Process: 3784 ExecStart=/usr/sbin/iptables-restore -T filter -w 10 /etc/iptables/20210531-iptab>
Main PID: 3784 (code=exited, status=0/SUCCESS)
Tasks: 0
CGroup: /system.slice/iptables-restore-ADL.service
Jun 08 14:13:57 anon systemd[1]: Starting Restore SysAdmin iptables rules at boot-time....
Jun 08 14:13:57 anon systemd[1]: Started Restore SysAdmin iptables rules at boot-time..
[1]+ Stopped systemctl status iptables-restore-ADL
#
# journalctl --lines=3 --unit=iptables-restore-ADL.service
-- Logs begin at Tue 2021-06-08 13:52:10 AEST, end at Tue 2021-06-08 18:00:31 AEST. --
Jun 08 14:06:39 anon systemd[1]: Stopped Restore SysAdmin iptables rules at boot-time..
Jun 08 14:13:57 anon systemd[1]: Starting Restore SysAdmin iptables rules at boot-time....
Jun 08 14:13:57 anon systemd[1]: Started Restore SysAdmin iptables rules at boot-time..
#
(b) After attempting (presumably to start the service at boot-time:
# systemctl status iptables-restore-ADL
● iptables-restore-ADL.service - Restore SysAdmin iptables rules at boot-time.
Loaded: loaded (/usr/lib/systemd/system/iptables-restore-ADL.service; enabled; vendor preset: >
Active: active (exited) since Tue 2021-06-08 18:07:18 AEST; 2min 25s ago
Process: 923 ExecStart=/usr/sbin/iptables-restore -T filter -w 10 /etc/iptables/20210531-iptabl>
Main PID: 923 (code=exited, status=0/SUCCESS)
Tasks: 0
CGroup: /system.slice/iptables-restore-ADL.service
Jun 08 18:07:17 anon systemd[1]: Starting Restore SysAdmin iptables rules at boot-time....
Jun 08 18:07:18 anon systemd[1]: Started Restore SysAdmin iptables rules at boot-time..
[1]+ Stopped systemctl status iptables-restore-ADL
#
# journalctl --lines=6 --unit=iptables-restore-ADL.service
-- Logs begin at Tue 2021-06-08 18:06:56 AEST, end at Tue 2021-06-08 18:12:46 AEST. --
Jun 08 18:07:17 anon systemd[1]: Starting Restore SysAdmin iptables rules at boot-time....
Jun 08 18:07:18 anon systemd[1]: Started Restore SysAdmin iptables rules at boot-time..
#
And to prove I have nothing up my sleeve, here is a listing of the INPUT chain which contains the O/S default rules only:
# iptables -L INPUT
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
INPUT_direct all -- anywhere anywhere
INPUT_ZONES_SOURCE all -- anywhere anywhere
INPUT_ZONES all -- anywhere anywhere
DROP all -- anywhere anywhere ctstate INVALID
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
#
I have to say this exercise has been a learning experience…
It appears to start from boot as expected, but likely firewalld overwrites the iptables rules during this process. Is there any reason that you are trying to apply custom iptables rules outside of the firewalld framework? The use of direct rules might be another option for you?
So your service was correctly started at boot time.
And to prove I have nothing up my sleeve, here is a listing of the INPUT chain which contains the O/S default rules only:
# iptables -L INPUT
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
INPUT_direct all -- anywhere anywhere
INPUT_ZONES_SOURCE all -- anywhere anywhere
INPUT_ZONES all -- anywhere anywhere
DROP all -- anywhere anywhere ctstate INVALID
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
#
It means your service runs before firewalld applies its configuration. It is quite possible - firewalld appears to perform initialization asynchronously; so from systemd point of view firewalld is started and systemd proceeds with dependent services, later firewalld comes to the point of actually applying its configuration and the first thing it does is to flush everything.
This is something to discuss with firewalld developers. Pragmatic answer - user firewalld direct interface to add your own rules.
In any case, this is outside of scope of this sub-forum as this is application specific issue.
And it may even be repaired because 15.1 is already some time out of support.
In any case, reporting a bug on an old version will probably not get a warm welcome on any product’s bugzilla.
I think that’s probably correct, but the intention of the unit definition was to run the local rules after the firewall demon finished (and if the firewall exited for any reason then restarted), but before the network-online target was flagged.
The problem may have something to do with the symlinks created when the service is enabled, leading to exactly the scenario you described. Surely this isn’t correct:
My working hypothesis is that the reason it works when started manually is that synchronisation of the addon service no longer matters, and the reason it doesn’t start at boot time is the absence of symlinks.
As to why I’m not using the firewall GUI, I find it too far removed from firewall operation for my comfort, and the GUI probably requires just as much expertise anyway if it’s used properly.
Henk, I don’t think for a moment it’s a bug, it’s my lack of systemd understanding! And as soon as I get this problem fixed I’ll be migrating the configuration to 15.2 anyway.
# systemctl enable test
Created symlink /etc/systemd/system/multi-user.target.wants/test.service → /usr/lib/systemd/system/test.service.
Created symlink /etc/systemd/system/network-online.target.wants/test.service → /usr/lib/systemd/system/test.service.
So you should check your unit file with respect to that.
# ls -aliR /usr/lib/systemd/system/multi-user*
1191303 -rw-r--r-- 1 root root 492 Jan 28 00:43 /usr/lib/systemd/system/multi-user.target
The relevant path is /etc/systemd/system/multi-user.target.wants, and /etc/systemd/system/network-online.target.wants/
For example
dir /etc/systemd/system/network-online.target.wants/
total 0
lrwxrwxrwx 1 root root 58 Oct 7 2020 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service
lrwxrwxrwx 1 root root 36 Jun 9 10:17 test.service -> /usr/lib/systemd/system/test.service
As to why I’m not using the firewall GUI, I find it too far removed from firewall operation for my comfort, and the GUI probably requires just as much expertise anyway if it’s used properly.
You don’t need to configure via firewall-config GUI though. The firewall-cmd CLI can be used instead.
**3400G:~ #** journalctl -b -u firewalld.service -u postfix.service
-- Logs begin at Fri 2021-06-04 21:19:38 CEST, end at Wed 2021-06-09 08:08:24 CEST. --
Jun 09 08:05:07 3400G systemd[1]: Starting Postfix Mail Transport Agent...
Jun 09 08:05:07 3400G echo[772]: Starting mail service (Postfix)
Jun 09 08:05:07 3400G postfix/postfix-script[938]: starting the Postfix mail system
Jun 09 08:05:07 3400G postfix/master[940]: daemon started -- version 3.5.10, configuration /etc/postfix
Jun 09 08:05:07 3400G systemd[1]: Started Postfix Mail Transport Agent.
Jun 09 08:08:24 3400G systemd[1]: Starting firewalld - dynamic firewall daemon...
Jun 09 08:08:24 3400G systemd[1]: Started firewalld - dynamic firewall daemon.
**3400G:~ #**
Show the output of ‘journalctl -b -u firewalld.service -u iptables-restore-ADL.service’.
Sorry if I misled you there Deano, I removed “wanted by network-online.target” from the unit definition shown in a previous post in an effort to simplify & isolate the problem. However I wasn’t aware that any unit definitions are located in /etc/systemd/system/.
After enabling and checking the unit definition, I reviewed the evidence after a clean restart with the following result.
As usual, systemctl status
showed a SUCCESS exit code for unit startup and iptables -L INPUT showed the iptables changes had not been inserted into the INPUT chain. 1. ls -ali /usr/lib/systemd/system/multi-user.target.wants/
showed a total of eight directories but none for my unit, and there was no /usr/lib/systemd/system/network-online.target.wants/ directory at all. 1. However symlinks for my unit had been created in the /etc/systemd/system/ tree for both multi-user.target.wants and network-online.target.wants.
systemctl stop
and systemctl start (with no prior disabling or enabling) then resulted in a successful startup with the required changes inserted into the INPUT chain. 1. If a relevant 'wants" directory (e.g. multi-user.target.wants) exists in both places but the required symlink is only in one, is there a possible problem? I suppose the demon has both in its PATH environment variable.
The evidence is listed below.
Re (1):
systemctl status iptables-restore-ADL
● iptables-restore-ADL.service - Restore SysAdmin iptables rules at boot-time.
Loaded: loaded (/usr/lib/systemd/system/iptables-restore-ADL.service; enabled; vendor preset: disabled)
Active: active (exited) since Wed 2021-06-09 15:02:38 AEST; 2min 1s ago
Process: 913 ExecStart=/usr/sbin/iptables-restore -T filter -w 10 /etc/iptables/20210531-iptables-save-ADL.v4 (code=ex>
Main PID: 913 (code=exited, status=0/SUCCESS)
Tasks: 0
CGroup: /system.slice/iptables-restore-ADL.service
Jun 09 15:02:37 anon systemd[1]: Starting Restore SysAdmin iptables rules at boot-time…
Jun 09 15:02:38 anon systemd[1]: Started Restore SysAdmin iptables rules at boot-time…
[1]+ Stopped systemctl status iptables-restore-ADL
iptables -vL INPUT
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
7 640 ACCEPT all – any any anywhere anywhere ctstate RELATED,ESTABLISHED
7 636 ACCEPT all – lo any anywhere anywhere
0 0 INPUT_direct all – any any anywhere anywhere
0 0 INPUT_ZONES_SOURCE all – any any anywhere anywhere
0 0 INPUT_ZONES all – any any anywhere anywhere
0 0 DROP all – any any anywhere anywhere ctstate INVALID
0 0 REJECT all – any any anywhere anywhere reject-with icmp-host-prohibited
Re (2):
ls -ali /usr/lib/systemd/system/multi-user.target.wants/
Re (4) & (5):
I’m afraid I forgot to copy & paste the evidence from the root account, but it’s working fine after about 10 hours, and one of my REJECTs has something to show for it.
I’m not sure this gets me much further ahead, though, and I can’t even think of a good diagnostic. But I’ll have a closer look at the journal tomorrow (it’s getting late here).
Well then you should understand that your custom service will run according to the ordering you configured.
After enabling and checking the unit definition, I reviewed the evidence after a clean restart with the following result.
As usual, systemctl status
showed a SUCCESS exit code for unit startup and iptables -L INPUT showed the iptables changes had not been inserted into the INPUT chain.>
ls -ali /usr/lib/systemd/system/multi-user.target.wants/ showed a total of eight directories but none for my unit, and there was no /usr/lib/systemd/system/network-online.target.wants/ directory at all.
The /etc/systemd/system/*.wants directories are pertinent for unit ordering.
The logic in removing “WantedBy=…]network-online.target” from the test unit was that network security isn’t an issue in a debug environment and I wanted to simplify things. However I’m aware all these startup tasks execute in parallel subject to unit constraints, so that might be dangerous after cutover.
For the sake of brevity and clarity in the following, I’ll refer to /etc/systemd/system as the ‘A’ directory and /usr/lib/systemd/system as the ‘B’ directory.
Now I know there are two directories, A & B, I see the A version of multi-user.target wants both firewalld.service and the test.service (i.e. iptables-restore-ADL.service) however the B version doesn’t want either. I’m a little surprised there are incompatible versions of multi-user.target.wants in both places. Further, the A version of network-online.target exists, but there’s no B version.
Might this be a potential problem? Is there any way of forcing the Demon to ignore the B directory when starting the test unit? I’ll try moving the test unit-definition to A.
Yes, I should have been talking about dependencies…
Along with a unit file foo.service, the directory foo.service.wants/ may exist. All unit files symlinked from such a directory are implicitly added as dependencies of type Wants= to the unit. Similar functionality exists for Requires= type dependencies as well, the directory suffix is .requires/ in this case. This functionality is useful to hook units into the start-up of other units, without having to modify their unit files. For details about the semantics of Wants=, see below. The preferred way to create symlinks in the .wants/ or .requires/ directory of a unit file is by embedding the dependency in [Install] section of the target unit, and creating the symlink in the file system with the enable or preset commands of systemctl(1).
…or perhaps it’s about both ordering and dependencies. I think a service-unit is guaranteed to start executing after all units listed in its After: declaration and before those listed in its Before: declaration, and it will finish before all units listed in it’s WantedBy: declaration begin. But beyond that it’s a race so it’s important to know what other constraints may be relevant.
While investigating my problem, it seemed clear that my firewall changes using IPtables were sometimes overwritten (probably by the firewall demon), sometimes my unit was loaded but didn’t start at all, and sometimes it failed because another process held the xtables lock. In the end, I think it’s better to locate a unit as late as possible in the system-startup topology rather than as soon as known dependencies are satisfied.
This should guarantee it runs after the firewall’s default configuration has been set up (i.e. before network-pre.target is declared) and startup has reached a multi-user CLI state (R/L 2), but before the graphical target (R/L 5) is reached. It seems network-online.target is never reached.
And I note the firewalld.service definition “conflicts” with iptables.service but I can’t find any iptables.service unit definition. Does anyone know where that is? Is it implied when iptables is run?
to start the service at boot-time: