Dear all, I have system running LAMP and acting as a regular webserver.
After running the setup for quite some months, I start having major issues:
- Applications do not respond neither from LAN nor WAN - SSH daemon, Apache, MySQL, FTP
- Network still seems to work for ping and port listeners
- Telnet is still successful for 21, 22, 80, 3306
- Server has to be restarted manually
Trying to find out the issue, I went through /var/log/ looking for major issues or warnings. But nothing seemed plausible to me to understand the issue - except I knew I was running out of disk space a few times.
Not being able to identify nor replicate the issue, I replaced the hardware running 24x7 since quite a few years. Doing this, I migrated at the same time from OpenSUSE 10.1 to 11.3.
The machine itself is behind a firewall and only the above mentioned standard ports are accessible.
Can somebody please advise me where to go into details to find the issue provoking the problem?
I am confused. Do you see the symptoms you describe on the fresh install of 11.3?
I am confused too - because it is the identical behavior for both installations!
As soon as I put them online they will get laid within 6 hours…
I already thought it was due to some php I am executing too often on crontab (doing SQL operations) - but removing did not help…
The only thing I was able to reproduce on the 10.2 box (11.3 is down again) is what I gues might be called a DoS attack:
When I refresh a page continuosly in my browser (60 request/1 minute), the server load will get up to 6 or higher. It will take quite a while until the system will get back to a normal state. As far I was able I also did check the server-tuning.conf which seemed fine to me along what people recommend.
I currently would like to try using mod_envasive - but this requires an online repository for the sources which are not available anymore. I think this will have to wait until monday having rebooted the 11.3 box.
Or do you have any other recommendations/suggestions?
A shortlist that comes to mind
- Is it possible to upgrade versions of PHP and MySQL? In particular, PHP circa 10.2 (PHP3?) has since been shown to be riddled with vulnerabilities, PHP has practically been re-built completely since then. If running a very old version of PHP, many possibilities exist from attacks to simple incompatibilities and inefficiencies.
- Have you run top or htop to determine (for starters) if CPU or mem resources are specific to an application/process? htop in particular can display your processes in treeview.
If you’re running out of disk space, it should be almost trivial to determine what is using up all your disk space unless you’ve been rootkit. File change programs like AIDE or Tripwire can assist in comparing/monitoring changes in directories and files. Remember also if you’re running “regular” ftp your security is thoroughly broken, Base64 encryption is next to useless today, you should instead be implementing https, sftp or similar. stunnel can encrypt existing ftp if you can’t change.
Tony, I can ensure all relevant binaries are more or less up to date (Apache 2.2, PHP 5.2.3, MySQL 5.0.26).
And yes, as mentioned earlier the server load goes up due to frequent requests. What will be on top of top are the apache-prefork processes.
Your tip for the file monitoring is interesting.