Complex problem with system hang (I/O) due to "disk sleep" process

Hello

I have a clean install of Opensuse 12.1 64 bit on a Flash drive (USB) with the proprietary Nvidia drivers installed (per one-click). The flash drive is a very fast USB 2.0 and my hardware is from 2011 with a Nvidia Quadro Card, 16 GB Ram and a very powerful Quadcore processor (able to do 8 Threads). Filesystem is ext4, kernel 3.1.x. I noticed ever since the first boot of the system that the systems hangs often when it comes to writing tasks for instance by installing software. If I watch the system monitor on the fresh booted system the usb-storage process is every few seconds at disk sleep state. If I install software the LED of the USB starts to shine permanently. Then many processes enter the disk sleep state among them plasma and everything freezes for a time. Same with copying big files. If I copy a big file everything freezes due to the disk sleep state. First I thought all this because of the fact the USB is too slow compared to a HDD, or the USB may be broken. So I copied everything with dd to another USB, but same problem. Then I tried other Linux distributions as Ubuntu or Chakra Linux and none of them had the same problem, at least KDE never froze. Some processes entered disk sleep too, but only a very short time and only one or two, so nothing froze, and I was able to work while installing software. So I compiled Linux kernel 3.0.x and installed it because I knew, I had never problems with this kernel on other distributions on flash drives in the past, but same problem. I also tried to make flash optimizations like noop and deadline, but same problem. I also tried a new install but same problem again. On Opensuse 11.04 I don’t have the problems. Nvidia driver can’t be the problem, because if I use the open source driver I see the same problem. What can cause the problem? All systems were tested on flash drives.
Updating to the latest available software in the repo didn’t help.
I find this really pitty because my Opensuse 12.1 setup is perfect in every aspect, everything works. Updating to kernel later than 3.1 won’t help because of this bug:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/836250
which is affecting my wifi card (3.1 is patchable). 11.4 is too old for my purposes. So 12.1 would be perfect to stay on the next time.
I do not know what I should do, I tried nearly everything, that has to do with I/O problems of flash drives.
In the logs I can’t see special I/O errors.

Please excuse my poor English

Thanks in advance

I figured out the processes which enter disk sleep while copying: jbd2/sdb2-8, flush-8:16 and kio_file (On Opensuse 11.4). On Chakra linux the same processes enter disk sleep. But on these distributions I can start other processes and work normal without a hang. On Opensuse 12.1 many other processes also enter disk sleep (plasma …) and I can’t start a new process even not konsole. Everything just freezes. I suppose the disk sleep problem is normal on flash drives, because it appears on different distributions with different kernels. Maybe someone could explain me a fix (for 12.1 and to prevent this problem in general) and what exactly is the problem.

Thank you

Same with btrfs. I noticed that the other distributions also start hanging after a time, 12.1 is the worst in this matter. In many other forums people are experiencing similar problems as here
Computer almost freezes while copying files (Page 1) / Hardware & Multimedia / ArchBang Forums
It seems the kernel isn’t able to handle this things proper, it doesn’t matter if it is 2.6 or 3.3. That is really pitty. It seems the only thing I can do is, to mount as many ramdisks as possible, turning off journaling and do ssd optimizations to reduce the I/O rate. Has anybody ideas?

Try booting with sysvinit: press F5 in GRUB, and pick System V.

Thanks for your reply!
It does not make any difference. Plasma still hangs while cooying and it is not possible to start any other process. Although it is much better on 11.4 and on other distributions, they also freeze. Is there no way to fix that? I wonder why 12.1 is so much worse compared to the others? Maybe because of the kernel? But all other kernels I tried behaved equally bad.

This problem pops up here and there on the Internet, with the prevalent gratuitous advice involving the purchase of new hardware. I ran into this myself, and am posting my experiences for future reference.

I had this problem on two systems, one desktop and one laptop, and suspectedly on a third system also (a server with two drives on RAID1), but I’ll leave that one out of consideration because it has speed issues for other reasons and I haven’t had a proper look at it yet.

The visible symptom is that, from time to time, the machine “freezes” all disk I/O operations, for a duration usually around 90 seconds, even when under low load. This happens approximately twice per hour. For example, the home desktop machine might be playing music on Amarok (and doing nothing else except routine background services), and about twice an hour the music will freeze in the middle of a song, then after little more than a minute playback resumes. The laptop showed the same problem playing video files, for example films: video and audio playback freeze, and then resume after the pause. In an effort to find something amusing about all this, I noticed that xine (kaffeine) resumes playback from “ninety seconds later”, as if it had been playing all along, so that you need to rewind; mplayer continues right from where it froze. Other symptoms include emails not being written to mailboxes, problems/delays in saving or moving or renaming any kind of file, etc.

When opening ksysguard, all processes attempting disk I/O show as “disk sleep” for the entire duration of the freeze. This usually includes the symptom process (amarok, xine, etc.), as well as background processes involved in I/O operations, such as the various jbd2, flush, and kio_file, as pointed out by Linuxmath above.

“Disk sleep” is generally understood to indicate a bottleneck disk I/O condition, i.e., the disk can’t keep up with the rate of data supplied to it or requested from it, so that the I/O must be frozen at the software end to allow the disk to catch up. In normal everyday use, this is not supposed to cause any problems; “disk sleep” either shouldn’t happen, or should be resolved in fractions of a second. Also, there is no reasonable explanation for an I/O bottleneck condition when playing reasonable multimedia formats (FLAC audio, films in xvid / dirac / theora / x264, etc., assuming the resolutions are “sensible”). Any 5400rpm hard disc can handle this… in its sleep, as it were.

When I turned to the Internet, the overwhelming initial advice I got was “dude, get a new disk” (– you buy me one, sucker!). Don’t blame your hardware before you’ve at least done the following, obviously as root:

e2fsck -Dfvy /dev/sda2
smartctl -a /dev/sda
smartctl -t long /dev/sda2
badblocks -nv -b 4096 -o /root/20140202.sda2.badblocks /dev/sda2

The first, e2fsck, is a check of your file system (n.b., assuming you’re using ext2/ext3/ext4), obviously referring to the partition you’re struggling to read from, in this case partition 2 on drive /dev/sda. You will need to unmount it first. If it’s your root partition, your best bet is to boot up a Live USB of openSUSE, then you’re guaranteed your disks have nothing to do. If it’s not your root partition (in my case it was “/home”), you can use umount, or sometimes it’s easier to comment out (i.e., prefix with #) its line in your filesystem table (/etc/fstab), and rebooting. You will need to remove the # to get your files back, of course.

The second command, smartctl, is a utility to “talk” to the hard disk’s S.M.A.R.T. capabilities (Self-Monitoring, Analysis, and Reporting Technology). With the “-a” option, smartctl simply spits out all the information it has on the disk in question. Look for anything that looks like a warning; some useful variables include “197 Current_Pending_Sector”, “198 Offline_Uncorrectable”, “199 UDMA_CRC_Error_Count” and “254 Free_Fall_Sensor”, all of which are bad news if their RAW_VALUE is anything other than zero. The “smartctl -t long” command tells the disk to start an extended self-test, which will run in the background, you can do other things in the meantime. The “smartctl -a” output should tell you how long the extended test will last (typically a few hours). After that duration of time, a new “smartctl -a” will show the outcomes of the test.

Smartctl also kindly pointed out that a firmware update was available for my drive. So I went to the manufacturer’s website, and downloaded & installed the new firmware. I’d suggest you check this with the manufacturer, too; smartctl will tell you what make/model disk you have, and which firmware version is installed.

The final command, badblocks, does some of the same stuff as smartctl, but more disruptively to the user. Meaning: it detects “bad blocks”, storage areas on the disk that have become unusable (indicative that your disk is “dying”). However, badblocks only works while the disk is unmounted (you cannot access your data in the meantime), and for a typical SATA-II 7200rpm disk today you might want to budget 24h for every 500GB you’d like to test, per pass (the example command above does a single pass, but higher degrees of paranoia are available). By the way: the -o option specifies an output file, where badblocks will send a list of any bad blocks found; this can be passed to e2fsck to ensure that those blocks are not used for data storage. This only works if your -b option to badblocks matches the block size for your file system; try something like: dumpe2fs -h /dev/sda2 | grep “Block size”.

My drives merrily passed all these tests. And if yours does too, and does not give you any other reason to doubt it (rattling, etc.), then don’t let anyone tell you to buy a new one.

By the way: if you did find any bad blocks, with either smartctl or badblocks, you should be able to find out which file they go with (you will have lost at least part of this file), and tell your filesystem to move that file someplace else, and not use the same blocks again. Here’s a fantastic article on how to do that. You can happily enough keep on using the same disk; however, bad blocks are usually a sign of old age, and having a few bad blocks is a good hint that more will go bad soon. Best to replace the disk in the near future!

After much poking around, I found that the following command fixes the disk sleep freeze issue for me on both machines:

hdparm -va0 /dev/sda

This instructs the disk to disable the “read ahead” feature of the disk.

This setting will always be reset when shutting down or restarting the computer, so it is useful to make sure the above command is executed automatically at every boot time. In older SysV-based systems, the file to look to was /etc/rc.local; on openSUSE 12.3 this is /etc/init.d/boot-local. Simply add a line at the bottom with your hdparm command.

How the “read ahead” feature relates to the symptoms described above is beyond me; I am also by no means convinced that I have yet found the root cause of this problem. The above fix worked for two different machines, with different hardware and software configurations. The common denominators are: openSUSE 12.3, KDE desktop environment, and hard disks manufactured by Seagate (but different product lines: one Barracuda and one Momentus).

If anyone else has any experience to share on the same issue, I’d look forward to reading.

— Warning: some rant lives below this line. —
I’d file a bug report, but I’m tired of filing bug reports for the current version of openSUSE only to get told that “it’s been fixed in Factory” and will be available in the next release. I don’t report bugs just to do the SLED/SLES people a favour. Also, I’m tired of upgrading from a stable openSUSE version to the newest, now 13.1, only to find that it’s horribly buggy and unstable because Attachmate is using openSUSE releases as bétas for community testing. I was once one of those people who had the latest openSUSE on my servers, and the next Release Candidate on my own machine, in order to report bugs… only to find that they’d get fixed AFTER final release, leaving me with buggy servers. So, what can I say… I’ll be upgrading to 13.1… later this year. :wink:

Correction: etc/init.d/boot.local

On 2014-02-02 21:36, Kalenz wrote:

> After much poking around, I found that the following command fixes the
> disk sleep freeze issue for me on both machines:
>
>
> Code:
> --------------------
> hdparm -va0 /dev/sda
> --------------------

Curious.

>
> This instructs the disk to disable the “read ahead” feature of the
> disk.

Right.

> How the “read ahead” feature relates to the symptoms described above is
> beyond me; I am also by no means convinced that I have yet found the
> root cause of this problem. The above fix worked for two different
> machines, with different hardware and software configurations. The
> common denominators are: openSUSE 12.3, KDE desktop environment, and
> hard disks manufactured by Seagate (but different product lines: one
> Barracuda and one Momentus).

Well, if the same combination happens to others, it is an interesting find.

> — Warning: some rant lives below this line. —
> I’d file a bug report, but I’m tired of filing bug reports for the
> current version of openSUSE only to get told that “it’s been fixed in
> Factory” and will be available in the next release. I don’t report bugs
> just to do the SLED/SLES people a favour. Also, I’m tired of upgrading
> from a stable openSUSE version to the newest, now 13.1, only to find
> that it’s horribly buggy and unstable because Attachmate is using
> openSUSE releases as bétas for community testing. I was once one of
> those people who had the latest openSUSE on my servers, and the next
> Release Candidate on my own machine, in order to report bugs… only to
> find that they’d get fixed AFTER final release, leaving me with buggy
> servers. So, what can I say… I’ll be upgrading to 13.1… later this
> year. :wink:

Well, you should look at it in another manner. SUSE has a number of
employees dedicated full time to openSUSE (quite a number of them). The
money to pay them has to come from somewhere… like SLES sales.

If your Bugzilla solution is not backported to the current distribution
you use… well, it can happen. Usually because it is difficult. It
certainly goes into factory (not SLES), and factory goes into the next
openSUSE release and SLES, directly or indirectly. And conversely, the
people working on SLES make developments that eventually get also to
openSUSE, and other distributions as well (because they go upstream).

But this is common to all Linux development (please have a read of “The
cathedral and the bazaar”). Bugs are corrected on the next version.
Linux tends to do fast development, with frequent releases. You get
those bug solutions on those next releases. Often upstream does not do
backporting (with exceptions).

In any way, those Bugzilla reports benefits us all.

Please, do report in Bugzilla. :slight_smile:

(I have many reports with no answer; yet I keep writing them).


Cheers / Saludos,

Carlos E. R.
(from 12.3 x86_64 “Dartmouth” at Telcontar)