MySQL fails to start at boot ~4 times out of 5.

Hi,

Since upgrading from 12.3 through 13.1 to 13.2 MySQL starts automatically at boot only perhasp once every five boots.

The strange thing is that the mysqld.log indicates that mysql starts but there is no mysql process.

The following messages in the logs seem to be related to mysql:

/var/log/mysql/mysqld.log


150321 14:15:31 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2015-03-21 14:15:33 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-03-21 14:15:33 2046 [Note] Plugin 'FEDERATED' is disabled.
2015-03-21 14:15:33 2046 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-03-21 14:15:33 2046 [Note] InnoDB: The InnoDB memory heap is disabled
2015-03-21 14:15:33 2046 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-03-21 14:15:33 2046 [Note] InnoDB: Compressed tables use zlib 1.2.8
2015-03-21 14:15:33 2046 [Note] InnoDB: Using Linux native AIO
2015-03-21 14:15:33 2046 [Note] InnoDB: Using CPU crc32 instructions
2015-03-21 14:15:33 2046 [Note] InnoDB: Initializing buffer pool, size = 512.0M
2015-03-21 14:15:33 2046 [Note] InnoDB: Completed initialization of buffer pool
2015-03-21 14:15:33 2046 [Note] InnoDB: Highest supported file format is Barracuda.
2015-03-21 14:15:36 2046 [Note] InnoDB: 128 rollback segment(s) are active.
2015-03-21 14:15:37 2046 [Note] InnoDB: Waiting for purge to start
2015-03-21 14:15:37 2046 [Note] InnoDB: 5.6.17 started; log sequence number 2075391196

/var/log/messages:


2015-03-21T14:16:00.486013+01:00 gunnar1 rcmysql[1217]: Starting service MySQL warning: /var/run/mysql/mysql.sock didn't appear within 30 seconds
2015-03-21T14:16:00.487713+01:00 gunnar1 rcmysql[1217]: chmod: cannot access ‘/var/run/mysql/mysqld.pid’: No such file or directory
2015-03-21T14:16:00.487963+01:00 gunnar1 rcmysql[1217]: Something went wrong
2015-03-21T14:16:00.489013+01:00 gunnar1 systemd[1]: Failed to start MySQL server.
2015-03-21T14:16:00.489230+01:00 gunnar1 after.local[2153]: Job for mysql.service failed. See "systemctl status mysql.service" and "journalctl -xn" for details.

Output from systemctl status mysql.service:


mysql.service - MySQL server
   Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled)
   Active: failed (Result: exit-code) since l�r 2015-03-21 14:16:00 CET; 1h 27min ago
  Process: 1217 ExecStart=/usr/lib/mysql/rcmysql start (code=exited, status=1/FAILURE)

mar 21 14:16:00 gunnar1.home rcmysql[1217]: Starting service MySQL warning: /var/run/mysql/mysql.sock didn't appear within 30 seconds
mar 21 14:16:00 gunnar1.home rcmysql[1217]: chmod: cannot access ‘/var/run/mysql/mysqld.pid’: No such file or directory
mar 21 14:16:00 gunnar1.home rcmysql[1217]: Something went wrong
mar 21 14:16:00 gunnar1.home systemd[1]: Failed to start MySQL server.

The output from journalctl -xn speaks only of .Xauthority files in my home directory.

I have also added “service mysql start” to /etc/init.d/after.local but it doesn’t seem to help.

Anyone knows what is wrong or have any ideas?

TIA,

Well, I’ve seen that a few times myself, although it has become better recently. I.e. it hasn’t happened since quite a while now, maybe since I updated to a newer 3.16.7 kernel [noparse](from Kernel:openSUSE_13.2)[/noparse] because of another problem with fuse filesystems. (a kernel update is already running for that…)

I think the reason is (was) that mysql takes too long to start and then gets killed by systemd.

Running “systemctl start mysql” manually after the system has booted should work, right? (at least that worked for me every time)

Maybe setting a higher timeout would help?
From “man systemd.service”:


       TimeoutStartSec=

           Configures the time to wait for start-up. If a daemon service does
           not signal start-up completion within the configured time, the
           service will be considered failed and will be shut down again.
           Takes a unit-less value in seconds, or a time span value such as
           "5min 20s". Pass "0" to disable the timeout logic. Defaults to
           TimeoutStartSec= from the manager configuration file, except when
           Type=oneshot is used, in which case the timeout is disabled by
           default.

To do that, copy /usr/lib/systemd/system/mysql.service to /etc/systemd/system/, open it in a text editor, and add something like the following to the “[Service]” section:

TimeoutStartSec=0

I never tried that myself though. In my case it didn’t fail ~4 out of 5 times, but only once every 20 boots or so, and as mentioned I haven’t seen it for a while at all now.

I have also added “service mysql start” to /etc/init.d/after.local but it doesn’t seem to help.

This would be an ugly workaround…
And I suppose at the time /etc/init.d/after.local is called, mysql.service is still trying to start anyway so that probably doesn’t have any effect anyway.

Thanks, I’ll try that and report back in a week or two when I know if it has reliably solved the problem.