Page 1 of 6 123 ... LastLast
Results 1 to 10 of 51

Thread: Apache doesn't start correctly on boot (very often)

  1. #1

    Question Apache doesn't start correctly on boot (very often)

    Hi,

    I have been experiencing this so many times and I can't find a patter. The system boots normally but the few sites which I have on the local machine don't load (404). Running 'rcapache2 restart' makes everything work but I can't find why it sometimes happens and based on what factors.

    Today I looked at /var/log/apache2/error_log:

    Code:
    [Sun Feb 19 16:28:30.000427 2017] [:notice] [pid 1767] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
    [Sun Feb 19 16:28:30.469773 2017] [:notice] [pid 1767] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"
    [Sun Feb 19 16:28:30.469792 2017] [:notice] [pid 1767] ModSecurity: PCRE compiled version="8.33 "; loaded version="8.39 2016-06-14"
    [Sun Feb 19 16:28:30.469798 2017] [:warn] [pid 1767] ModSecurity: Loaded PCRE do not match with compiled!
    [Sun Feb 19 16:28:30.469802 2017] [:notice] [pid 1767] ModSecurity: LUA compiled version="Lua 5.2"
    [Sun Feb 19 16:28:30.469807 2017] [:notice] [pid 1767] ModSecurity: LIBXML compiled version="2.9.1"
    [Sun Feb 19 16:28:30.469811 2017] [:notice] [pid 1767] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Sun Feb 19 16:28:30.495914 2017] [core:error] [pid 1767] (EAI 2)Name or service not known: AH00547: Could not resolve host name site1.local -- ignoring!
    [Sun Feb 19 16:28:30.495979 2017] [core:error] [pid 1767] (EAI 2)Name or service not known: AH00547: Could not resolve host name site2.local -- ignoring!
    [Sun Feb 19 16:28:30.496037 2017] [core:error] [pid 1767] (EAI 2)Name or service not known: AH00547: Could not resolve host name site3.local -- ignoring!
    [Sun Feb 19 16:28:31.018255 2017] [mpm_prefork:notice] [pid 1767] AH00163: Apache/2.4.16 (Linux/SUSE) OpenSSL/1.0.1i-fips configured -- resuming normal operations
    [Sun Feb 19 16:28:31.018285 2017] [core:notice] [pid 1767] AH00094: Command line: '/usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -D SYSCONFIG -D SSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -D SYSTEMD -D FOREGROUND'
    Then I ran 'rcapache2 restart' and in the same log I see:

    Code:
    [Sun Feb 19 16:33:38.465418 2017] [mpm_prefork:notice] [pid 1767] AH00170: caught SIGWINCH, shutting down gracefully
    [Sun Feb 19 16:33:39.000468 2017] [:notice] [pid 3473] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
    [Sun Feb 19 16:33:39.000561 2017] [:notice] [pid 3473] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"
    [Sun Feb 19 16:33:39.000568 2017] [:notice] [pid 3473] ModSecurity: PCRE compiled version="8.33 "; loaded version="8.39 2016-06-14"
    [Sun Feb 19 16:33:39.000572 2017] [:warn] [pid 3473] ModSecurity: Loaded PCRE do not match with compiled!
    [Sun Feb 19 16:33:39.000575 2017] [:notice] [pid 3473] ModSecurity: LUA compiled version="Lua 5.2"
    [Sun Feb 19 16:33:39.000579 2017] [:notice] [pid 3473] ModSecurity: LIBXML compiled version="2.9.1"
    [Sun Feb 19 16:33:39.000582 2017] [:notice] [pid 3473] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Sun Feb 19 16:33:40.013885 2017] [mpm_prefork:notice] [pid 3473] AH00163: Apache/2.4.16 (Linux/SUSE) OpenSSL/1.0.1i-fips configured -- resuming normal operations
    [Sun Feb 19 16:33:40.013934 2017] [core:notice] [pid 3473] AH00094: Command line: '/usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -D SYSCONFIG -D SSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -D SYSTEMD -D FOREGROUND'
    [Sun Feb 19 16:33:45.022714 2017] [core:notice] [pid 3473] AH00052: child pid 3485 exit signal Segmentation fault (11)
    [Sun Feb 19 16:33:47.024813 2017] [core:notice] [pid 3473] AH00052: child pid 3486 exit signal Segmentation fault (11)
    [Sun Feb 19 16:33:52.029721 2017] [core:notice] [pid 3473] AH00052: child pid 3481 exit signal Segmentation fault (11)
    [Sun Feb 19 16:34:16.054527 2017] [core:notice] [pid 3473] AH00052: child pid 3505 exit signal Segmentation fault (11)
    [Sun Feb 19 16:34:17.056118 2017] [core:notice] [pid 3473] AH00052: child pid 3501 exit signal Segmentation fault (11)
    [Sun Feb 19 16:34:17.056145 2017] [core:notice] [pid 3473] AH00052: child pid 3502 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:13.176170 2017] [core:notice] [pid 3473] AH00052: child pid 3508 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:28.191208 2017] [core:notice] [pid 3473] AH00052: child pid 3506 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:28.191241 2017] [core:notice] [pid 3473] AH00052: child pid 3507 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:38.201731 2017] [core:notice] [pid 3473] AH00052: child pid 3617 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:40.204770 2017] [core:notice] [pid 3473] AH00052: child pid 3620 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:41.207193 2017] [core:notice] [pid 3473] AH00052: child pid 3621 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:51.219528 2017] [core:notice] [pid 3473] AH00052: child pid 3555 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:54.222686 2017] [core:notice] [pid 3473] AH00052: child pid 3480 exit signal Segmentation fault (11)
    [Sun Feb 19 16:36:54.222723 2017] [core:notice] [pid 3473] AH00052: child pid 3489 exit signal Segmentation fault (11)
    [Sun Feb 19 16:37:13.241850 2017] [core:notice] [pid 3473] AH00052: child pid 3642 exit signal Segmentation fault (11)
    and after that everything works as expected.

    site1.local, site2.local and site3.local are properly configured in /etc/hosts and point to 127.0.0.1

    Any idea what the problem/solution may be?

  2. #2
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    878

    Default Re: Apache doesn't start correctly on boot (very often)

    Taking the case of this 13.2 system with an Apache Server version 2.4.10, /var/log/apache2/error_log contains some intermittent AH00052 Segmentation faults when the Server is shut down (gracefully):
    • once 25th April last year;
    • 5 times (5 different child processes) during the shut down on the 24th of January this year.

    So, yes there seems to be an intermittent child process Segmentation fault issue with Apache version 2.x when the server is being shut down (gracefully).

    But, your machine dependent start up issue seems to be a AH00547 problem with resolving host names.

    And, as you have described, not all machines and, only at boot time (restarting the Apache Server resolves the issue).

    A shot in the dark:
    • Do the affected machines boot from a SSD or a hybrid (SSHD) disk?

    If so, please check the SMART health: " # smartctl --health /dev/sda".
    You could, also check the systemd Journal (the complete Journal) for increasing ECC and/or read error rates: " # journalctl | grep 'SMART' | grep -Ei 'ECC|read'".
    Even if the affected machines have a physically rotating drive, it may be worth checking the health of the thing.

  3. #3

    Default Re: Apache doesn't start correctly on boot (very often)

    This is Leap 42.1. What do you mean "this 13.2 system"? Also where do you get those dates from?

    Yes, the system boots from SSD (the logs are on HDD though) but what has hardware to do with Apache? All the hard drives are healthy.

    Code:
    # smartctl --health /dev/sda
    smartctl 6.2 2013-11-07 r3856 [x86_64-linux-4.1.38-47-default] (SUSE RPM)
    Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    # smartctl --health /dev/sdb
    smartctl 6.2 2013-11-07 r3856 [x86_64-linux-4.1.38-47-default] (SUSE RPM)
    Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    # smartctl --health /dev/sdc
    smartctl 6.2 2013-11-07 r3856 [x86_64-linux-4.1.38-47-default] (SUSE RPM)
    Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    # smartctl --health /dev/sdd
    smartctl 6.2 2013-11-07 r3856 [x86_64-linux-4.1.38-47-default] (SUSE RPM)
    Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED

  4. #4
    Join Date
    Nov 2009
    Location
    West Virginia Sector 9
    Posts
    13,871

    Default Re: Apache doesn't start correctly on boot (very often)

    Possible timing problem due to speed of drive and maybe a slow connect to NET. Also using Wicked or NetworkManager???

  5. #5

    Default Re: Apache doesn't start correctly on boot (very often)

    Quote Originally Posted by gogalthorp View Post
    Possible timing problem due to speed of drive and maybe a slow connect to NET.
    Could you explain a little more?
    Also using Wicked or NetworkManager???
    wicked

  6. #6
    Join Date
    Nov 2009
    Location
    West Virginia Sector 9
    Posts
    13,871

    Default Re: Apache doesn't start correctly on boot (very often)

    Simple the Web server tries to start before the network is completed. No net then web server fails. Network start is variable since it depend not only on your hardware but the ISP.

    Maybe systemd-analyze blame may give a clue.

  7. #7

    Default Re: Apache doesn't start correctly on boot (very often)

    But what ISP to do with Apache considering we are discussing sites hosted on the localhost? Are you saying that if I pull my cable Apache won't start? I don't quite understand what you mean.

    Code:
    [~]: systemd-analyze blame
             51.572s nut-driver.service
              2.146s mysql.service
              2.028s wicked.service
              1.288s apache2.service
              1.175s ModemManager.service
              1.123s SuSEfirewall2_init.service
               335ms php-fpm.service
               258ms libvirtd.service
               254ms avahi-daemon.service
               237ms SuSEfirewall2.service
               ...

  8. #8

    Default Re: Apache doesn't start correctly on boot (very often)

    I have rebooted after my previous post and the thing happened again. Here is the new output:

    Code:
    [~]: systemd-analyze blame                                                                                                                                  
              6.767s nut-driver.service                                                                                                                         
              2.222s apache2.service                                                                                                                            
              2.124s mysql.service                                                                                                                              
              2.027s wicked.service                                                                                                                             
              1.353s ModemManager.service                                                                                                                       
              1.292s SuSEfirewall2_init.service                                                                                                                 
               302ms php-fpm.service                                                                                                                            
               275ms avahi-daemon.service                                                                                                                       
               266ms libvirtd.service                                                                                                                           
               242ms SuSEfirewall2.service 
    ...

  9. #9
    Join Date
    Nov 2009
    Location
    West Virginia Sector 9
    Posts
    13,871

    Default Re: Apache doesn't start correctly on boot (very often)

    Web server still uses the network even if local

    Apache is starting before the network maybe

    try syatemctl-analyze plot to see

  10. #10

    Default Re: Apache doesn't start correctly on boot (very often)

    Quote Originally Posted by gogalthorp View Post
    Web server still uses the network even if local

    Apache is starting before the network maybe

    try syatemctl-analyze plot to see
    Ok, it obviously does (considering the second 'systemd-analyze blame' which I pasted when the failure showed.

    But why does this happen? The service says the unit should start after network.target, isn't that enough to make it wait for net first?

    Code:
    cat /usr/lib/systemd/system/apache2*
    [Unit]
    Description=The Apache Webserver
    After=network.target nss-lookup.target time-sync.target remote-fs.target
    Before=getty@tty1.service plymouth-quit.service xdm.service
    
    
    [Service]
    Type=notify
    PrivateTmp=true
    ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start
    ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful
    ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop
    KillMode=mixed
    
    
    [Install]
    WantedBy=multi-user.target
    Alias=httpd.service apache.service
    
    
    [Unit]
    Description=The Apache Webserver %I
    After=network.target nss-lookup.target time-sync.target
    Before=getty@tty1.service plymouth-quit.service xdm.service
    
    
    [Service]
    Type=notify
    PrivateTmp=true
    Environment="HTTPD_INSTANCE=%i"
    ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start
    ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful
    ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop
    KillMode=mixed
    
    
    [Install]
    WantedBy=multi-user.target
    Alias=httpd@.service apache@.service
    If the "After" is not the thing to make it wait correctly - what is the solution?

    BTW in /usr/lib/systemd/system I have 2 services: apache2.service and apache2@.service - is that normal and why are they 2?

Page 1 of 6 123 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •