Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: systemd - no effect change service start timeout configuration

  1. #1

    Default systemd - no effect change service start timeout configuration

    Hi ,

    openSUSE 13.2 (Harlequin) (x86_64)
    # systemctl --version
    systemd 210
    +PAM -LIBWRAP +AUDIT +SELINUX -IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ +SECCOMP +APPARMOR


    I've a situation where I start a MySQL database and it's always need execute the recovery process , so take few seconds/minutes to finish.
    (this is expected , since I'm starting a backup from snapshot).

    This always works before, but few days ago , become failing constantly ( I execute this 1 time per day)
    After investigating , notice the recovery of mysql is taking +1 minute , but systemctl start is keeping they timeout fixed in 30 seconds, then if the service doesn't start in this time it's kill the process and put in failed .

    I'm pretty sure about that because the mysql log shows it's stop during recovery process and there's no PID of mysqld anymore.
    if start manually the MySQL manually , works fine...

    I've try few parameters at mysql.unit where systemd was recognized , but there no effect when I start the service...
    What I missing here ?

    Code:
    # systemctl cat mysql
    # /usr/lib/systemd/system/mysql.service
    [Unit]
    Description=MySQL server
    Wants=basic.target
    After=basic.target network.target
    # by Cesar
    JobTimeoutSec=120s
    
    
    [Service]
    Type=forking
    ExecStart=/usr/lib/mysql/rcmysql start
    ExecStop=/usr/lib/mysql/rcmysql stop
    # by Cesar
    TimeoutStartSec=95s
    
    
    [Install]
    WantedBy=multi-user.target
    
    
    
    
    # systemctl show mysql | grep -i time
    TimeoutStartUSec=1min 35s
    TimeoutStopUSec=1min 30s
    WatchdogTimestampMonotonic=0
    ExecMainStartTimestamp=Mon 2015-07-13 10:20:21 BRT
    ExecMainStartTimestampMonotonic=844288186352
    ExecMainExitTimestamp=Mon 2015-07-13 10:20:51 BRT
    ExecMainExitTimestampMonotonic=844318601781
    ExecStart={ path=/usr/lib/mysql/rcmysql ; argv[]=/usr/lib/mysql/rcmysql start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
    ExecStop={ path=/usr/lib/mysql/rcmysql ; argv[]=/usr/lib/mysql/rcmysql stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
    LimitRTTIME=18446744073709551615
    TimerSlackNSec=50000
    InactiveExitTimestamp=Mon 2015-07-13 10:20:19 BRT
    InactiveExitTimestampMonotonic=844286075868
    ActiveEnterTimestamp=Mon 2015-07-13 10:20:21 BRT
    ActiveEnterTimestampMonotonic=844288186394
    ActiveExitTimestamp=Mon 2015-07-13 10:20:51 BRT
    ActiveExitTimestampMonotonic=844317785447
    InactiveEnterTimestamp=Mon 2015-07-13 10:20:51 BRT
    InactiveEnterTimestampMonotonic=844318642487
    JobTimeoutUSec=2min
    ConditionTimestamp=Mon 2015-07-13 10:20:19 BRT
    ConditionTimestampMonotonic=844286006583
    
    
    # time systemctl start mysql
    Job for mysql.service failed. See "systemctl status mysql.service" and "journalctl -xn" for details.
    
    real    0m32.736s
    user    0m0.002s
    sys     0m0.003s
    Last edited by ceinma; 13-Jul-2015 at 07:46. Reason: forgot the question...

  2. #2
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,296
    Blog Entries
    2

    Post Re: systemd - no effect change service start timeout configuration

    Maybe you should set a longer value, eg 3min ? According to the MAN pages, the service fails if your timeout isn't configured long enough and 95sec might be enough only for a very small database. If you database grows, you may want to consider a much larger number.

    TSU

    http://www.dsm.fordham.edu/cgi-bin/m...ystemd.service
    From the MAN pages
    Code:
     TimeoutStartSec=           
               Configures the time to wait for start-up. If a daemon service does
               not signal start-up completion within the configured time, the
               service will be considered failed and will be shut down again.
               Takes a unit-less value in seconds, or a time span value such as
               "5min 20s". Pass "0" to disable the timeout logic. Defaults to
               DefaultTimeoutStartSec= from the manager configuration file, except
               when Type=oneshot is used, in which case the timeout is disabled by
                default (see systemd-system.conf(5)).

  3. #3

    Default Re: systemd - no effect change service start timeout configuration

    Hi TSU,

    I already try before , this values I show was just other test...
    At first time I've set I use "3min 30s"... and got same effect.
    Always the systemd "abort" the start of the service in 30 seconds...




    Quote Originally Posted by tsu2 View Post
    Maybe you should set a longer value, eg 3min ? According to the MAN pages, the service fails if your timeout isn't configured long enough and 95sec might be enough only for a very small database. If you database grows, you may want to consider a much larger number.

    TSU

    http://www.dsm.fordham.edu/cgi-bin/m...ystemd.service
    From the MAN pages
    Code:
     TimeoutStartSec=           
               Configures the time to wait for start-up. If a daemon service does
               not signal start-up completion within the configured time, the
               service will be considered failed and will be shut down again.
               Takes a unit-less value in seconds, or a time span value such as
               "5min 20s". Pass "0" to disable the timeout logic. Defaults to
               DefaultTimeoutStartSec= from the manager configuration file, except
               when Type=oneshot is used, in which case the timeout is disabled by
                default (see systemd-system.conf(5)).

  4. #4

    Default Re: systemd - no effect change service start timeout configuration

    TSU , just for sure I try again.
    This time I set to 15 and 20 minutes... keeps aborting the start after 30 seconds.

  5. #5
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,296
    Blog Entries
    2

    Default Re: systemd - no effect change service start timeout configuration

    Am wondering if your two custom commands might be conflicting with each other. Have you tried running only one or the other?

    JobTimeoutSec
    TimeoutStartSec

    Also, don't forget to run the following after any modifications to any Service Unit files else your changes won't take (forgetting this leads to a lot of lost time)
    Code:
    systemctl daemon-reload
    TSU

  6. #6

    Default Re: systemd - no effect change service start timeout configuration

    Yes, I try one first , the other...the both... then others (which isn't in this examples)...

    and yes to daemon-reload , I always execute it and check at journalctl if give some syntax error or invalid parameter...

  7. #7
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,296
    Blog Entries
    2

    Default Re: systemd - no effect change service start timeout configuration

    Without knowing why your systemd setting isn't working,

    Maybe you should consider splitting out the backup creation from your existing job (ie create 2 smaller jobs from the one large job). Since the backup creation is likely the only thing causing the job to run exceedingly long, when you actually try to startup the mysql service the default startup values should not be exceeded.

    It's probably better practice anyway to configure multiple small jobs rather than one large job anyway, it should also run more efficiently.

    TSU

  8. #8

    Default Re: systemd - no effect change service start timeout configuration

    Hi Tsu ,

    There is no way to split the job... since this is a snapshot and not a regular backup with incremental data.

    Is there some way to debug the systemctl command ?

    someting like verbose mode... : systemctl -v start mysql

  9. #9
    Join Date
    May 2012
    Location
    Finland
    Posts
    2,004

    Default Re: systemd - no effect change service start timeout configuration

    As I understand; you physically copy /var/lib/mysql/* database files back, then restart MariaDB in order for it to go through recovery cycle.

    Stupid question;
    Why don't you simply mysqldump with routines and then restore it as opposed to physically copying data? I don't really see the benefit in restoring the files physically.
    .: miuku #suse @ irc.freenode.net
    :: miuku@opensuse.org

    .: h​ttps://download.opensuse.org/repositories/home:/Miuku/

  10. #10

    Default Re: systemd - no effect change service start timeout configuration

    Hi Miuku ,

    Don't worry , isn't stupid question since I don't explain all context before. (my focus here is only systemctl , not mysql db)
    This database run a Zabbix Server (IT services monitor) , so it have high volume of transactions by second (collecting data from hosts) and not should have any fault or performance issue.
    To run mysqldump I need lock the database to keep the integrity of the data and it affect considerable the performance of MySQL and the server .
    I need keep simple and reduce any overhead to backup all data and avoid affect the server performance (or consequently it affect the Zabbix monitor and trigger false alarms to all IT staff)

    The best solution I got so far is lock the MySQL database , create a LVM snapshot of FS , release the database , this took 5 seconds top.
    Then later I use a rsync to copy from the snapshot only changed data of db files to other server.
    At this server I start the database (where I got the problem with systemctl), run the data check (this way I avoid any overhead at production and if get some problem then I act at our mysql production database) , then run a mysqldump... and at finish I keep a backup do db files + mysqldump...

    (other solution which I don't try yet is active a mysql replication... but now isn't the moment)


    Now, with the context explained... people , please , let keep the focus at systemctl.... it have a problem , not my MySQL database...


    Quote Originally Posted by Miuku View Post
    As I understand; you physically copy /var/lib/mysql/* database files back, then restart MariaDB in order for it to go through recovery cycle.

    Stupid question;
    Why don't you simply mysqldump with routines and then restore it as opposed to physically copying data? I don't really see the benefit in restoring the files physically.

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •