Page 1 of 4 123 ... LastLast
Results 1 to 10 of 40

Thread: Headless server crash

  1. #1

    Default Headless server crash

    I have a headless server running a scanner and a webcam.
    Quite often (several times a day) it becomes unreachable (via http or telnet) and I have to do a forced poweroff.
    I have found lots of pages on how to troubleshoot a crash when you know what is wrong, but none that really seem to start from a position of ignorance. Strange, because I thought that was what troubleshooting was all about. Can anyone offer any suggestions as to where I should be looking. I have tried looking in /var/log for a crash or dump file, but there is nothing there. I have installed some crash dump utilities and I believe I have set up the kernel to do an auto-reboot on a kernel panic, but none of this seems to be getting me anywhere.

  2. #2
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    32,320
    Blog Entries
    15

    Default Re: Headless server crash

    On Thu, 24 Jul 2008 15:16:03 GMT
    nalexmay <nalexmay@no-mx.forums.opensuse.org> wrote:

    >
    > I have a headless server running a scanner and a webcam.
    > Quite often (several times a day) it becomes unreachable (via http or
    > telnet) and I have to do a forced poweroff.
    > I have found lots of pages on how to troubleshoot a crash when you
    > know what is wrong, but none that really seem to start from a
    > position of ignorance. Strange, because I thought that was what
    > troubleshooting was all about. Can anyone offer any suggestions as to
    > where I should be looking. I have tried looking in /var/log for a
    > crash or dump file, but there is nothing there. I have installed some
    > crash dump utilities and I believe I have set up the kernel to do an
    > auto-reboot on a kernel panic, but none of this seems to be getting
    > me anywhere.
    >
    >

    Hi
    Have you looked at the kernel and messages log.

    If you on a windows machine have a look at installing kiwisyslog and
    setup remote sys logging

    http://www.kiwisyslog.com/

    Else login via ssh and tail the logs and run top to see what is
    happening when the system stops. Maybe a process is just running and
    chewing up cpu.

    Hardware wise, power supply ok, disks ok (run smart), can you run
    memtest at boot to check the memory.

    Install hddtemp (pacman repos) and check out disk temperatures, setup
    lm_sensors to check temperatures of cpu and power supply voltages.

    --
    Cheers Malcolm (Linux Counter #276890)
    SLED 10 SP2 i586 Kernel 2.6.16.60-0.25-default
    up 4:33, 2 users, load average: 0.59, 0.46, 0.91
    GPU GeForce Go 6600 TE/6200 TE Version: 173.14.09


  3. #3
    Join Date
    Jun 2008
    Location
    Earth - Denmark
    Posts
    10,730

    Default Re: Headless server crash

    well, first you have to ask yourself: is it a hardware problem, a
    security problem or a network problem..

    HARDWARE:
    have you ruled out: heat (clean out the dog and cat hair, etc), failing
    power supply, run a memory checker? [and, how about the router? blow the
    dust out of it too..]

    SOFTWARE/SECURITY
    are you SURE a root kit has not been installed? is the box exposed to
    the net in any way? running tripwire or similar on that box? is it fully
    patched and updated? is it a still supported version of SuSE, or?

    NETWORK:
    you said you can telnet in...if YOU can, can someone from outside? your
    router have an airtight password? consider abandoning telnet and use the
    much safer SSH..

    there are lots of tools to diagnosis most any potential network
    problem...i'm not qualified to help you, but you might try posting in
    the network forum (where i'm not)..

    good luck,

    --
    DenverD (Linux Counter 282315)
    A Texan in Denmark

  4. #4

    Default Re: Headless server crash

    Thanks Malcolm and thanks DenverD.
    I have plenty to get on with there, but here are a few comments on stuff that I had already looked at.
    I tried looking for a kernel log, but didn't find one.
    I looked at messages, but couldn't see anything that meant anything useful to me. It's huge though so I could be missing something.
    I keep the hardware pretty clean.
    I ran a memory check (memtest86) and it found no errors.
    I think the router is ok as it is the device that other computers on the network are using without any problem.
    I don't know what a root kit is, so I suppose I don't know if one has been installed.
    I do know that the server is more or less completely cut off from the internet. Telnet port is certainly closed.
    I believe it is fully patched and updated.
    Yast2 seems to be saying it is anyway.
    I've set up a cron job which should tell me if the server is still running after it is inaccessable via the network which should narrow the problem down a bit.
    I will report back when I have tried some more of this stuff.
    In the meantime, thanks again for your help.

  5. #5

    Default Re: Headless server crash

    I'm not quite beat yet, but I'm getting close.
    I installed kiwi on my linux box, but it didn't help. I still can't make head or tail of any of the log files.
    I ran extensive tests on my hard drive and found no problems.
    Temperatures look okay and I am still working on installing automated temerature monitoring.
    So far I can't get lm_sensors to make all.
    Anything else I can try?

  6. #6
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    32,320
    Blog Entries
    15

    Default Re: Headless server crash

    On Thu, 24 Jul 2008 21:06:03 GMT
    nalexmay <nalexmay@no-mx.forums.opensuse.org> wrote:

    >
    > I'm not quite beat yet, but I'm getting close.
    > I installed kiwi on my linux box, but it didn't help. I still can't
    > make head or tail of any of the log files.
    > I ran extensive tests on my hard drive and found no problems.
    > Temperatures look okay and I am still working on installing automated
    > temerature monitoring.
    > So far I can't get lm_sensors to make all.
    > Anything else I can try?
    >

    Hi
    lm_sensors should be there already? Try;

    sensors-detect

    you may also need to activate if it finds sensors to use;

    chkconfig lm_sensors on

    I would look at running top and/or vmstat 5 (print every 5 seconds)
    just to see when what is happening.

    --
    Cheers Malcolm (Linux Counter #276890)
    SLED 10 SP2 i586 Kernel 2.6.16.60-0.25-default
    up 10:40, 1 user, load average: 0.37, 0.20, 0.14
    GPU GeForce Go 6600 TE/6200 TE Version: 173.14.09


  7. #7

    Default Re: Headless server crash

    Yes, you're right.
    I looked on Yast for lm_sensors, didn't find it and assumed I needed to install it!
    Sensors-detect ran ok, but it said:

    To load everything that is needed, add this to some /etc/rc* file:
    #----cut here----
    # Chip drivers
    modprobe w83627hf
    # sleep 2 # optional
    /usr/bin/sensors -s # recommended
    #----cut here----

    What am I supposed to do with that? Which file am I supposed to add that too?

    Other than that, sensors-detect finished.

    What now?

  8. #8

    Default Re: Headless server crash

    Sorry, I should also have said that I can run top and/or vmstat, but there is nothing in the response that means anything to me in terms of solving this issue. Can you explain what it is that I am looking for?

  9. #9
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    32,320
    Blog Entries
    15

    Default Re: Headless server crash

    On Fri, 25 Jul 2008 06:06:03 GMT
    nalexmay <nalexmay@no-mx.forums.opensuse.org> wrote:

    >
    > Yes, you're right.
    > I looked on Yast for lm_sensors, didn't find it and assumed I needed
    > to install it!
    > Sensors-detect ran ok, but it said:
    >
    > To load everything that is needed, add this to some /etc/rc* file:
    > #----cut here----
    > # Chip drivers
    > modprobe w83627hf
    > # sleep 2 # optional
    > /usr/bin/sensors -s # recommended
    > #----cut here----
    >
    > What am I supposed to do with that? Which file am I supposed to add
    > that too?
    >
    > Other than that, sensors-detect finished.
    >
    > What now?
    >
    >

    Hi
    As long as you accept the 'yes' to build the
    config /etc/sysconfig/lm_sensors the init script should start
    everything ok.

    You can test by running the command;

    /etc/init.d/lm_sensors start

    If that goes ok, you can check with the command;

    sensors

    Check that it's configured to start;

    chkconfig lm_sensors

    It should say on, if not just add on to the end of the above command

    --
    Cheers Malcolm (Linux Counter #276890)
    SLED 10 SP2 i586 Kernel 2.6.16.60-0.25-default
    up 0:33, 2 users, load average: 0.30, 0.15, 0.23
    GPU GeForce Go 6600 TE/6200 TE Version: 173.14.09


  10. #10
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    32,320
    Blog Entries
    15

    Default Re: Headless server crash

    On Fri, 25 Jul 2008 06:16:03 GMT
    nalexmay <nalexmay@no-mx.forums.opensuse.org> wrote:

    >
    > Sorry, I should also have said that I can run top and/or vmstat, but
    > there is nothing in the response that means anything to me in terms of
    > solving this issue. Can you explain what it is that I am looking for?
    >
    >

    Hi
    Ok for the top command, keep an eye on the information in the top
    section. It may be swap or cpu, especially if you see swap suddenly
    start to be used up. Then look at the current running processors in the
    list below.

    Vmstat will do the same sort of thing, but has information on the io
    for the disks, see man vmstat for more detail.

    Now I don't think I asked if your disk space is ok. What is the output
    from;

    df -k

    --
    Cheers Malcolm (Linux Counter #276890)
    SLED 10 SP2 i586 Kernel 2.6.16.60-0.25-default
    up 0:51, 2 users, load average: 0.05, 0.15, 0.17
    GPU GeForce Go 6600 TE/6200 TE Version: 173.14.09


Page 1 of 4 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •