Service Scripts Start in Wrong Order

Hi All,

I’ve set up wireless access point via hostapd and it is working smoothly. I’ve created a simple init script (see below) in /etc/init.d/ and used chkconfig, insserv and/or yast (trying many different combinations of commands) to activate the script for certain runlevels. When ‘service hostapd start’ is called by root, the service activates as normal.

However, the service will not automatically start at boot.

According to boot.msg, the service is being started before the $remote_fs psuedo-service is brought up. Even if the ‘required-start’ field is set to $ALL, the script still starts before the network is brought up via ifconfig. I’ve tried playing with all of the other fields in the ‘init info’ header, to no avail

I’ve checked that the service is activated via chkconfig runlevels 3 and 5 (set to ‘on’) and that the appropriate symlinks are created (S05hostapd-S15hostapd, depending on what I put for ‘required-start’) in /etc/init.d/rc3.d and rc5.d. Going through Yast expert mode, I can see again that the service is activated for the runlevels. Script perms are 755, like all of the other scripts.

As an aside, I was able to add ‘service hostapd start’ as an ifup script in /etc/sysconfig/network/scripts, and it is now started at boot, after the network interface is up. I still need this problem solved for another custom service script, which is also having the same issue.

Any ideas?

/etc/init.d/hostapd

#!/bin/sh
#
#     Template SUSE system startup script for example service/daemon FOO
#     Copyright (C) 1995--2005  Kurt Garloff, SUSE / Novell Inc.
#          
#     This library is free software; you can redistribute it and/or modify it
#     under the terms of the GNU Lesser General Public License as published by
#     the Free Software Foundation; either version 2.1 of the License, or (at
#     your option) any later version.
#			      
#     This library is distributed in the hope that it will be useful, but
#     WITHOUT ANY WARRANTY; without even the implied warranty of
#     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
#     Lesser General Public License for more details.
#      
#     You should have received a copy of the GNU Lesser General Public
#     License along with this library; if not, write to the Free Software
#     Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307,
#     USA.
#
# /etc/init.d/FOO
#   and its symbolic link
# /(usr/)sbin/rcFOO
#
# Template system startup script for some example service/daemon FOO
#
# LSB compatible service control script; see http://www.linuxbase.org/spec/
# 
# Note: This template uses functions rc_XXX defined in /etc/rc.status on
# UnitedLinux/SUSE/Novell based Linux distributions. If you want to base your
# script on this template and ensure that it works on non UL based LSB 
# compliant Linux distributions, you either have to provide the rc.status
# functions from UL or change the script to work without them.
# See skeleton.compat for a template that works with other distros as well.
#
### BEGIN INIT INFO
# Provides:          hostapd
# Required-Start:    $remote_fs
# Should-Start:      
# Required-Stop:     $remote_fs
# Should-Stop:       
# Default-Start:     3 5
# Default-Stop:      0 1 2 6
# Short-Description: Starts hostapd access point at bootup
# Description:       This script will start a secured
#	access point on the ath0 interface, bridging
#	the wireless network with the ethernet connection.
### END INIT INFO
# 
# Any extensions to the keywords given above should be preceeded by 
# X-VendorTag- (X-UnitedLinux- X-SuSE- for us) according to LSB.
# 
# Notes on Required-Start/Should-Start:
# * There are two different issues that are solved by Required-Start
#    and Should-Start
# (a) Hard dependencies: This is used by the runlevel editor to determine
#     which services absolutely need to be started to make the start of
#     this service make sense. Example: nfsserver should have
#     Required-Start: $portmap
#     Also, required services are started before the dependent ones.
#     The runlevel editor will warn about such missing hard dependencies
#     and suggest enabling. During system startup, you may expect an error,
#     if the dependency is not fulfilled.
# (b) Specifying the init script ordering, not real (hard) dependencies.
#     This is needed by insserv to determine which service should be
#     started first (and at a later stage what services can be started
#     in parallel). The tag Should-Start: is used for this.
#     It tells, that if a service is available, it should be started
#     before. If not, never mind.
# * When specifying hard dependencies or ordering requirements, you can 
#   use names of services (contents of their Provides: section)
#   or pseudo names starting with a $. The following ones are available
#   according to LSB (1.1):
#	$local_fs		all local file systems are mounted
#				(most services should need this!)
#	$remote_fs		all remote file systems are mounted
#				(note that /usr may be remote, so
#				 many services should Require this!)
#	$syslog			system logging facility up
#	$network		low level networking (eth card, ...)
#	$named			hostname resolution available
#	$netdaemons		all network daemons are running
#   The $netdaemons pseudo service has been removed in LSB 1.2.
#   For now, we still offer it for backward compatibility.
#   These are new (LSB 1.2):
#	$time			the system time has been set correctly	
#	$portmap		SunRPC portmapping service available
#   UnitedLinux extensions:
#	$ALL			indicates that a script should be inserted
#				at the end
# * The services specified in the stop tags 
#   (Required-Stop/Should-Stop)
#   specify which services need to be still running when this service
#   is shut down. Often the entries there are just copies or a subset 
#   from the respective start tag.
# * Should-Start/Stop are now part of LSB as of 2.0,
#   formerly SUSE/Unitedlinux used X-UnitedLinux-Should-Start/-Stop.
#   insserv does support both variants.
# * X-UnitedLinux-Default-Enabled: yes/no is used at installation time
#   (%fillup_and_insserv macro in %post of many RPMs) to specify whether
#   a startup script should default to be enabled after installation.
#   It's not used by insserv.
#
# Note on runlevels:
# 0 - halt/poweroff 			6 - reboot
# 1 - single user			2 - multiuser without network exported
# 3 - multiuser w/ network (text mode)  5 - multiuser w/ network and X11 (xdm)
# 
# Note on script names:
# http://www.linuxbase.org/spec/refspecs/LSB_1.3.0/gLSB/gLSB/scrptnames.html
# A registry has been set up to manage the init script namespace.
# http://www.lanana.org/
# Please use the names already registered or register one or use a
# vendor prefix.


# Check for missing binaries (stale symlinks should not happen)
# Note: Special treatment of stop for LSB conformance
FOO_BIN="/usr/sbin/hostapd"
test -x $FOO_BIN || { echo "$FOO_BIN not installed"; 
	if  "$1" = "stop" ]; then exit 0;
	else exit 5; fi; }

# Check for existence of needed config file and read it
FOO_CONFIG=/etc/hostapd.conf
test -r $FOO_CONFIG || { echo "$FOO_CONFIG not existing";
	if  "$1" = "stop" ]; then exit 0;
	else exit 6; fi; }

# Read config	
#. $FOO_CONFIG

# Source LSB init functions
# providing start_daemon, killproc, pidofproc, 
# log_success_msg, log_failure_msg and log_warning_msg.
# This is currently not used by UnitedLinux based distributions and
# not needed for init scripts for UnitedLinux only. If it is used,
# the functions from rc.status should not be sourced or used.
#. /lib/lsb/init-functions

# Shell functions sourced from /etc/rc.status:
#      rc_check         check and set local and overall rc status
#      rc_status        check and set local and overall rc status
#      rc_status -v     be verbose in local rc status and clear it afterwards
#      rc_status -v -r  ditto and clear both the local and overall rc status
#      rc_status -s     display "skipped" and exit with status 3
#      rc_status -u     display "unused" and exit with status 3
#      rc_failed        set local and overall rc status to failed
#      rc_failed <num>  set local and overall rc status to <num>
#      rc_reset         clear both the local and overall rc status
#      rc_exit          exit appropriate to overall rc status
#      rc_active        checks whether a service is activated by symlinks
. /etc/rc.status

# Reset status of this service
rc_reset

# Return values acc. to LSB for all commands but status:
# 0	  - success
# 1       - generic or unspecified error
# 2       - invalid or excess argument(s)
# 3       - unimplemented feature (e.g. "reload")
# 4       - user had insufficient privileges
# 5       - program is not installed
# 6       - program is not configured
# 7       - program is not running
# 8--199  - reserved (8--99 LSB, 100--149 distrib, 150--199 appl)
# 
# Note that starting an already running service, stopping
# or restarting a not-running service as well as the restart
# with force-reload (in case signaling is not supported) are
# considered a success.

case "$1" in
    start)
	echo -n "Starting hostapd "
	## Start daemon with startproc(8). If this fails
	## the return value is set appropriately by startproc.
	/sbin/startproc $FOO_BIN -B -P /var/run/hostapd.pid $FOO_CONFIG
	
	# Remember status and be verbose
	rc_status -v
	;;
    stop)
	echo -n "Shutting down hostapd "
	## Stop daemon with killproc(8) and if this fails
	## killproc sets the return value according to LSB.

	/sbin/killproc -TERM $FOO_BIN

	# Remember status and be verbose
	rc_status -v
	;;
    try-restart|condrestart)
	## Do a restart only if the service was active before.
	## Note: try-restart is now part of LSB (as of 1.9).
	## RH has a similar command named condrestart.
	if test "$1" = "condrestart"; then
		echo "${attn} Use try-restart ${done}(LSB)${attn} rather than condrestart ${warn}(RH)${norm}"
	fi
	$0 status
	if test $? = 0; then
		$0 restart
	else
		rc_reset	# Not running is not a failure.
	fi
	# Remember status and be quiet
	rc_status
	;;
    restart)
	## Stop the service and regardless of whether it was
	## running or not, start it again.
	$0 stop
	$0 start

	# Remember status and be quiet
	rc_status
	;;
    force-reload)
	## Signal the daemon to reload its config. Most daemons
	## do this on signal 1 (SIGHUP).
	## If it does not support it, restart the service if it
	## is running.

	echo -n "Reload service hostapd "
	## if it supports it:
	/sbin/killproc -HUP $FOO_BIN
	#touch /var/run/FOO.pid
	rc_status -v

	## Otherwise:
	#$0 try-restart
	#rc_status
	;;
    reload)
	## Like force-reload, but if daemon does not support
	## signaling, do nothing (!)

	# If it supports signaling:
	echo -n "Reload service hostapd "
	/sbin/killproc -HUP $FOO_BIN
	#touch /var/run/FOO.pid
	rc_status -v
	
	## Otherwise if it does not support reload:
	#rc_failed 3
	#rc_status -v
	;;
    status)
	echo -n "Checking for service hostapd "
	## Check status with checkproc(8), if process is running
	## checkproc will return with exit status 0.

	# Return value is slightly different for the status command:
	# 0 - service up and running
	# 1 - service dead, but /var/run/  pid  file exists
	# 2 - service dead, but /var/lock/ lock file exists
	# 3 - service not running (unused)
	# 4 - service status unknown :-(
	# 5--199 reserved (5--99 LSB, 100--149 distro, 150--199 appl.)
	
	# NOTE: checkproc returns LSB compliant status values.
	/sbin/checkproc $FOO_BIN
	# NOTE: rc_status knows that we called this init script with
	# "status" option and adapts its messages accordingly.
	rc_status -v
	;;
    probe)
	## Optional: Probe for the necessity of a reload, print out the
	## argument to this init script which is required for a reload.
	## Note: probe is not (yet) part of LSB (as of 1.9)

	test $FOO_CONFIG -nt /var/run/hostapd.pid && echo reload
	;;
    *)
	echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}"
	exit 1
	;;
esac
rc_exit

Are you using ifup method or NetworkManager to start the network? If NM, then it starts up later, when you run NM, not by init script. In which case you have to use an ifup hook, not an init script for your hostap script.

Using traditional method with ifup to bring up network:
ath0 unconfigured
eth0 unconfigured
br0 bridged eth0 ath0 with static ip on LAN

There’s a seeming contradiction in what you are saying. You say the order is wrong, and yet you say that when you look at the symlink numbers they are ok. So is it starting in the correct numbered order or not? Is it really a problem of the script not doing what it’s supposed to at boot time?

IIRC there’s also another factor, bridges take some time to settle, I think about 30 seconds due to the spanning tree algorithm.

Yes, there is a contradiction here, the symlinks are in the correct order, and yet the service is still started before the network is brought up… even when the symlink is S15… very unexpected.

I’m completely stumped, everything looks like it should be working as normal; here are the logs so you can see what I’m talking about, with the init script required-start set to $ALL…

snippet of ‘insserv -v hostapd’ output:

insserv: Loading multipathd
insserv: enable service ../hostapd -> /etc/init.d/rc3.d/S15hostapd
insserv: enable service ../hostapd -> /etc/init.d/rc3.d/K01hostapd
insserv: enable service ../hostapd -> /etc/init.d/rc5.d/S15hostapd
insserv: enable service ../hostapd -> /etc/init.d/rc5.d/K01hostapd
insserv: creating .depend.boot

snippet of ‘chkconfig -l’ output:

gpm                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
haldaemon                 0:off  1:off  2:on   3:on   4:off  5:on   6:off
hostapd                   0:off  1:off  2:off  3:on   4:off  5:on   6:off
inn                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
ivman                     0:off  1:off  2:off  3:off  4:off  5:off  6:off

(YAST expert mode also lists identical to above)

double-check symlinks, output of 'ls -la /etc/init.d/rc3.d:

lrwxrwxrwx  1 root root    9 2009-08-01 16:14 S13xinetd -> ../xinetd
lrwxrwxrwx  1 root root   10 2009-08-15 20:07 S15hostapd -> ../hostapd
lrwxrwxrwx  1 root root   15 2009-08-01 16:14 S15stopblktrace -> ../stopblktrace

Wait while machine test-reboots…

hmm? Hostapd doesn’t appear to be running?

service hostapd status … unused

time to go to the log, drumroll please…

/var/log/boot.msg:
Starting acpid done
<notice>startproc: execve (/sbin/syslog-ng)  /sbin/syslog-ng -a /var/lib/dhcp/d
ev/log ],  CONSOLE=/dev/console ROOTFS_FSTYPE=ext3 TERM=linux SHELL=/bin/sh ROO
TFS_FSCK=0 LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 COLUMNS=15
6 PATH=/bin:/sbin:/usr/bin:/usr/sbin DO_CONFIRM= vga=0x31a RUNLEVEL=3 SPLASHCFG=
/etc/bootsplash/themes/openSUSE/config/bootsplash-1280x1024.cfg PWD=/ PREVLEVEL=
N LINES=60 SHLVL=2 HOME=/ SPLASH=yes splash=silent ROOTFS_BLKDEV=/dev/disk/by-id
/ata-WDC_WD7500AACS-00D6B1_WD-WCAU47212359-part5 _=/sbin/startproc DAEMON=/sbin/
syslog-ng ]
Starting hostapd done
Configuration file: /etc/hostapd.conf
<notice>startproc: execve (/sbin/klogd)  /sbin/klogd -c 1 -x ],  CONSOLE=/dev/
console ROOTFS_FSTYPE=ext3 TERM=linux SHELL=/bin/sh ROOTFS_FSCK=0 LC_ALL=POSIX I
NIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 COLUMNS=156 PATH=/bin:/sbin:/usr/bi
n:/usr/sbin DO_CONFIRM= vga=0x31a RUNLEVEL=3 SPLASHCFG=/etc/bootsplash/themes/op
enSUSE/config/bootsplash-1280x1024.cfg PWD=/ PREVLEVEL=N LINES=60 SHLVL=2 HOME=/
 SPLASH=yes splash=silent ROOTFS_BLKDEV=/dev/disk/by-id/ata-WDC_WD7500AACS-00D6B
1_WD-WCAU47212359-part5 _=/sbin/startproc DAEMON=/sbin/klogd ]
acpid: 2 rules loaded


Starting D-Bus daemondone
Starting syslog servicesdone

Hostapd silently fails (confirmed with ‘ps aux’, hostapd not listed), and further down we see why… $remote_fs is brought up later…

Starting service MySQL done
Setting up (remotefs) network interfaces:
    ath0      device: Atheros Communications Inc. Atheros AR5001X+ Wireless Netw
ork Adapter (rev 01)
              No configuration found for ath0
    ath0     
unused
    br0       
    br0       Ports: [ath0] [eth0] 
    br0       forwarddelay (see man ifcfg-bridge) ... ready
    br0       IP address: 192.168.0.12/24   
    br0      
doneSetting up service (remotefs) network  .  .  .  .  .  .  .  .  .  .done
Starting httpd2 (prefork) <notice>startproc: execve (/usr/sbin/httpd2-prefork) 
 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf ],  CONSOLE=/dev/console R
OOTFS_FSTYPE=ext3 SHELL=/bin/sh TERM=linux ROOTFS_FSCK=0 get_module_list_done=tr
ue LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 COLUMNS=156 get_in
cludes_done=true PATH=/bin:/sbin:/usr/bin:/usr/sbin vga=0x31a DO_CONFIRM= RUNLEV
EL=3 PWD=/ SPLASHCFG=/etc/bootsplash/themes/openSUSE/config/bootsplash-1280x1024
.cfg PREVLEVEL=N LINES=60 HOME=/var/lib/apache2 SHLVL=2 splash=silent SPLASH=yes
 ROOTFS_BLKDEV=/dev/disk/by-id/ata-WDC_WD7500AACS-00D6B1_WD-WCAU47212359-part5 _
=/sbin/startproc DAEMON=/usr/sbin/httpd2-prefork ]
done

When I check the service in YAST simple mode, it’s listed as ‘Enabled YES’ with an asterisk next to it… and then in expert mode it’s listed as ‘Enabled NO’.

Something is wonky.

The symlink number is not all that controls the startup order. When the service is installed by insserv or chkconfig they also have to update dependency files in /etc/init.d. You can see them as .depend.something. There are 4 of them on mine. I would look at .depend.boot and .depend.start to see if they list the correct dependencies.

.depend.boot:

loadmodules boot.device-mapper boot.md boot.localfs boot.swap boot.fuse boot.cle
anup boot.scpm boot.klog boot.cycle boot.proc boot.udev_retry boot.localnet boot
.sysctl boot.ldconfig boot.ipconfig
INTERACTIVE = boot.rootfsck boot.clock boot.localfs
boot.udev: boot.blktrace
boot.rootfsck: boot.udev
boot.clock: boot.rootfsck
boot.loadmodules: boot.udev
boot.device-mapper: boot.udev boot.rootfsck
boot.md: boot.udev boot.rootfsck
boot.localfs: boot.clock boot.udev boot.rootfsck boot.md boot.loadmodules
boot.swap: boot.rootfsck boot.localfs boot.md
boot.fuse: boot.udev boot.localfs
boot.cleanup: boot.rootfsck boot.localfs
boot.scpm: boot.rootfsck boot.localfs
boot.klog: boot.rootfsck boot.localfs
boot.cycle: boot.localfs boot.loadmodules
boot.proc: boot.localfs boot.rootfsck
boot.udev_retry: boot.rootfsck boot.localfs
boot.localnet: boot.cleanup
boot.sysctl: boot.localfs
boot.ldconfig: boot.swap boot.clock
boot.ipconfig: boot.sysctl

.depend.start, notice no dependencies are listed for hostapd:

TARGETS = halt fbset earlysyslog random dbus acpid reboot network haldaemon sys
log boot.clock splash_early rpcbind auditd nfs kbd mysql network-remotefs sshd 
pure-ftpd ntp splash nscd xend earlyxdm single cups xendomains libvirtd postfix
 nfsserver apache2 smartd cron xinetd stopblktrace hostapd
INTERACTIVE = kbd ntp single apache2
network: dbus
haldaemon: dbus acpid
syslog: network earlysyslog
splash_early: network syslog
rpcbind: network syslog
auditd: syslog
nfs: network rpcbind
kbd: nfs fbset
mysql: network nfs
network-remotefs: network nfs haldaemon
sshd: network nfs
pure-ftpd: nfs syslog
ntp: nfs syslog network
splash: nfs fbset
nscd: nfs syslog network
xend: syslog nfs
earlyxdm: earlysyslog haldaemon
single: boot.clock kbd splash fbset
cups: nfs syslog dbus network rpcbind
xendomains: syslog nfs xend
libvirtd: network nfs xend
postfix: network syslog nfs
nfsserver: network rpcbind
apache2: nfs network mysql
smartd: syslog nfs
cron: nfs syslog network postfix
xinetd: network nfs rpcbind
stopblktrace: apache2 kbd ntp nfs splash fbset earlysyslog random smartd syslog
 network dbus mysql splash_early cups rpcbind network-remotefs haldaemon cron p
ostfix acpid sshd nscd xend xinetd xendomains pure-ftpd nfsserver libvirtd audi
td earlyxdm

So, I manually added a line ‘hostapd: nfs network syslog’ and rebooted, and hostapd started a little later. So, I went back to /etc/rc.d/.depend.start and made the line ‘hostapd: network-remotefs’ and restarted… Win! Hostapd now started at boot as it should.

Thank you very much for your help.

Now… any clues as to why chkconfig is not creating the necessary dependency entry for my init scripts? ‘chkconfig -d hostapd’ will delete my custom entry from the .depend.start file, but ‘chkconfig -a hostapd’ will not recreate any entry.

Strange isn’t it? Your guess is as good as mine.

chkconfig -a is supposed to be a front-end to insserv. Try running insserv with -v to show what it’s doing. Maybe that will reveal something.

output of insserv -rv hostapd:

insserv: Loading K02fbset
insserv: Loading S01fbset
insserv: Loading S15hostapd
insserv: Loading S11nfsserver
insserv: Loading K02postfix
insserv: Loading S06kbd
...
insserv: Loading boot.open-iscsi
insserv: Loading cups
insserv: Loading svnserve
insserv: Loading multipathd
insserv: remove service /etc/init.d/rc3.d/S15hostapd
insserv: remove service /etc/init.d/rc3.d/K01hostapd
insserv: remove service /etc/init.d/rc5.d/S15hostapd
insserv: remove service /etc/init.d/rc5.d/K01hostapd
insserv: creating .depend.boot
insserv: creating .depend.start
insserv: creating .depend.halt
insserv: creating .depend.stop

then, output of ‘insserv -v hostapd’; note that there is no reference to ‘Loading S15hostapd’, not sure if there should be:

insserv: Loading K04boot.localfs
insserv: Loading S08boot.md
insserv: Loading K01boot.cleanup
insserv: Loading S04boot.clock
insserv: Loading hostapd
insserv: Loading acpid
insserv: Loading sshd
insserv: Loading network-remotefs
insserv: Loading nscd
...
insserv: Loading cups
insserv: Loading svnserve
insserv: Loading multipathd
insserv: enable service ../hostapd -> /etc/init.d/rc3.d/S15hostapd
insserv: enable service ../hostapd -> /etc/init.d/rc3.d/K01hostapd
insserv: enable service ../hostapd -> /etc/init.d/rc5.d/S15hostapd
insserv: enable service ../hostapd -> /etc/init.d/rc5.d/K01hostapd
insserv: creating .depend.boot
insserv: creating .depend.start
insserv: creating .depend.halt
insserv: creating .depend.stop

Finally I view .depend.start, still no hostapd line.

I even reinstalled insserv just to be sure, still does the above.

Ok, so now I’ve removed the $ALL from required-start in the script, and replaced it with $remote-fs, and now the hostapd entry is created in /etc/rc.d/.depend.start as ‘hostapd: nfs’ and symlink S07. The script is still starting before the network is brought up (back where I was before).

Now I’ve changed the required-start/stop to $named, and .depend.start lists ‘hostapd: network’ and makes it S09 symlink. Still started before the network is brought up.

This is illogical.

To summarize:
$ALL makes no .depend.start entry at all
$remote-fs starts script before network is brought up
$named starts script before network is brought up
The only way to get this working is to manually add hostapd: network-remote_fs to /etc/rc.d/.depend.start

Can anyone else duplicate this? Any more suggestions/ideas on how to fix this?

File a bug report and let the developers figure out what’s wrong maybe?

Seems bug-worthy, entered as bug #531612, see https://bugzilla.novell.com/show_bug.cgi?id=531612

Just adding a “me to” to receive any updates as they arrive.
Thank you.

~Martin