Problems with buffered write to USB

BertBril · July 26, 2015, 3:31pm

Hi,

I’m using OpenSUSE 13.2 x64.

When I plug in any USB device, it will be auto-mounted. I do not know how to influence the options.

The problem is that my system almost stops responding when I do large data transfers. The reason is - I assume - that the writing is buffered. It seems to me that this is the case because the problem persists until I do a ‘sync’ after the write.

OK, so you say that I already have a solution: use’sync’. The things is, I often need to transfer large amounts of data, and during each transfer I can hardly use my computer. I’m juggling with scripts that copy each file and sync immediately afterwards. But with very large files (e.g when I’m zipping data) or many small files is hard to handle. On the other hand there is hardly every any advantage in buffering the data for non-system USB devices.

I guess that a simple solution would be to make writing unbuffered. My questions therefore are:

Am I wrong about the cause, or is there maybe another solution?
If I’m right, can I influence automount to use the ‘sync’ option whenever a USB drive is mounted? I did a lot of searching and can’t find any general way. All I can see is ways to explicitly specify something for certain explicit devices. But I don;t have just one device, I get many and need to make sure every disk/stick of ‘USB’ type is mounted unbuffered. The sad thing is I can see this is the default on M$ …

Thx, Bert

nrickert · July 26, 2015, 3:59pm

Yes, that used to happen here.

So what did I do to change it? I bought a new computer.

Actually, it still happens with the newer computer, but is a lot less of a problem. I was listening to music when I started a copy of the Leap milestone iso to a USB. And the the music playing hesitated twice as the copying began.

If your copying is started at the command line, you might be able to use “ionice” to reduce the I/O resources used during the copying.

robin_listas · July 26, 2015, 4:18pm

On 2015-07-26 15:36, BertBril wrote:

> I guess that a simple solution would be to make writing unbuffered. My
> questions therefore are:
>
> * Am I wrong about the cause, or is there maybe another solution?

No, you are right.

All writes to the filesystem (and reads) are cached by the kernel. A
mount option will not change this: with sync, normal writes are
immediately written, yes, but they go slowly. On a floppy disk it goes
terribly slowly.

The issue is that when you write big files to a slow device, the kernel
caches all it can in RAM, in order to allow the calling program to
continue with other things as soon as possible. As the files are big,
RAM is spent soon, and the rest of the system suffers because there is
no free ram.

A practical solution would be for the file browser to intentionally
write slowly.

With big, single files, like when burning an iso image to an usb stick,
you can use dd like this:

dd if=… of=… oflag=direct

dcache might also work.

But of course, you have to create a script to handle a bunch of files.
If you use midnight commander, you may call a script on a list of
selected files easily - but I have not stopped yet to think it over
carefully.

Maybe control groups could help, but I don’t know how to use them.

–
Cheers / Saludos,

Carlos E. R.

(from 13.1 x86_64 “Bottle” (Minas Tirith))

BertBril · July 26, 2015, 5:24pm

[QUOTE=nrickert;2721039]

> Yes, that used to happen here.
>
> So what did I do to change it? I bought a new computer.

Actually, I have a pretty new, heavy system, 8 cores, 32 G memory.

I think it’s not the problem, the symptoms say otherwise. I see that Carlos also replied, anyway thx for the view.

Cheers, Bert

gogalthorp · July 26, 2015, 5:31pm

Note that writing to MS type file systems consumes more resources then if writing to a Linux based file system.

You can use top to monitor memory usage but simnce most of the process happens in kernel space it will not show process usages. I run KDE an have a CPU monitor widgets set that shows actual CPU usage even if in kernel space

BertBril · July 26, 2015, 5:57pm

Carlos wrote:

> No, you are right.

Good. No need to buy a new computer then (yet).

> All writes to the filesystem (and reads) are cached by the kernel. A
> mount option will not change this: with sync, normal writes are
> immediately written, yes, but they go slowly. On a floppy disk it goes
> terribly slowly.

Yes, writing is the only problem. When reading a large file my computer isn’t bothered at all. I can imagine that the slowness of input will never make it a problem, but then you’d expect only a problem with writing during the filling of the cache. But it remains until you sync. Then it will clear.

> The issue is that when you write big files to a slow device, the kernel
> caches all it can in RAM, in order to allow the calling program to
> continue with other things as soon as possible. As the files are big,
> RAM is spent soon, and the rest of the system suffers because there is
> no free ram.

This indeed matches the symptoms perfectly.

> A practical solution would be for the file browser to intentionally
> write slowly.

I seldom use the file browser, and I wouldn’t know how to force it to do that either.

> dd if=… of=… oflag=direct

Ah! Really useful. When zipping a file to USB, that would be:

zip -r - directory_name | dd oflag=direct > /run/media/bert/101F-5B74/zipfilename.zip

That indeed seems to cure the problem.

> dcache might also work.

cnf dcache
dcache: command not found

All in all I find this a bit worrying. What you are basically saying is that the kernel is designed under the assumption that write caching is always a good idea. That sounds … weird. I was under the impression that there are situations in which write caching is simply unacceptable, and I would expect there to be ways to specify this. And this would be a very nice place to use such an option.

Thanks anyway, the dd trick will do wonders!

Cheers, Bert

gogalthorp · July 26, 2015, 6:15pm

It is always best practice to get the data out to disk as fast as possible So writes are generally given top priority. Use ionice to adjust the niceness of the process. Which can cause it to be a lower priority process thus freeing up some CPU time. USB’s are slow write and MS file systems also use a lot more processing time. USB sticks also take up CPU doing house keeping for the flash memory since thy don’t in general have a dedicated on-board controller that SSDs have.

BertBril · July 26, 2015, 6:18pm

gogalthorp wrote:

> Note that writing to MS type file systems consumes more resources
> then if writing to a Linux based file system.

Yeah, if I were the master of the universe, there would be no M$ type file systems. The thing is, I’m not.

> You can use top to monitor memory usage but simnce most of the
> process happens in kernel space it will not show process usages. I
> run KDE an have a CPU monitor widgets set that shows actual CPU usage
> even if in kernel space

Top and KSysguard are my friends but after you figure out what the problem is, you need to figure out what to do about it …

Cheers, Bert

robin_listas · July 26, 2015, 6:18pm

On 2015-07-26 17:36, gogalthorp wrote:
>
> Note that writing to MS type file systems consumes more resources then
> if writing to a Linux based file system.

Well, writing to ntfs-3g does, but writing to fat does not

–
Cheers / Saludos,

Carlos E. R.

(from 13.1 x86_64 “Bottle” (Minas Tirith))

robin_listas · July 26, 2015, 6:34pm

On 2015-07-26 18:06, BertBril wrote:
> Carlos wrote:

>> A practical solution would be for the file browser to intentionally
>> write slowly.
>
> I seldom use the file browser, and I wouldn’t know how to force it to do
> that either.

No, no, it would have to be done by that particular browser devs adding
such a feature or hack.

>> dd if=… of=… oflag=direct
>
> Ah! Really useful. When zipping a file to USB, that would be:
>
> zip -r - directory_name | dd oflag=direct >
> /run/media/bert/101F-5B74/zipfilename.zip
>
> That indeed seems to cure the problem.

Good!

>> dcache might also work.
>
> cnf dcache
> dcache: command not found

No, I meant “oflag=dcache”

> All in all I find this a bit worrying. What you are basically saying is
> that the kernel is designed under the assumption that write caching is
> always a good idea. That sounds … weird. I was under the impression
> that there are situations in which write caching is simply unacceptable,
> and I would expect there to be ways to specify this. And this would be a
> very nice place to use such an option.

Well, I know that they are aware of the problem, but I don’t know what
they intend to do about it and how. Perhaps current kernel behaves
better, dunno.

There was a thread in the mail list where we talked about this.

http://lists.opensuse.org/opensuse/2014-09/msg00414.html

–
Cheers / Saludos,

Carlos E. R.

(from 13.1 x86_64 “Bottle” (Minas Tirith))

robin_listas · July 26, 2015, 6:38pm

On 2015-07-26 18:16, gogalthorp wrote:
>
> It is always best practice to get the data out to disk as fast as
> possible So writes are generally given top priority. Use ionice to
> adjust the niceness of the process. Which can cause it to be a lower
> priority process thus freeing up some CPU time. USB’s are slow write and
> MS file systems also use a lot more processing time. USB sticks also
> take up CPU doing house keeping for the flash memory since thy don’t in
> general have a dedicated on-board controller that SSDs have.

No, no, if you watch the CPU load while this happens you will see that
it low.

–
Cheers / Saludos,

Carlos E. R.

(from 13.1 x86_64 “Bottle” (Minas Tirith))

BertBril · July 26, 2015, 6:47pm

gogalthorp wrote:

> It is always best practice to get the data out to disk as fast as
> possible So writes are generally given top priority. Use ionice to
> adjust the niceness of the process. Which can cause it to be a lower
> priority process thus freeing up some CPU time.

I tried:

ionice --class idle zip -r zipfile usbpath

… but the cursor still drags and jumps.

All in all I feel that the experts are saying ‘njet’.The direct writing does help a bit, but I have to admit that there must be an I/O component, because what I said previously, ‘cured’, well, not really.

I’ll accept that normal working and writing to USB (both sticks and HDDs) do not go together. At least not on my (pretty expensive) computer.

Cheers, Bert

gogalthorp · July 26, 2015, 7:02pm

Well when I do a backup to USB stick I see lots of CPU usage in CPU monitor widget on all 6 cores here. I don’t see drag down of cursor but it is a large load. It does not show in top though just the widget. Also last time was about a gig of updates (use luckybackup (rsync GUI)) process was done at 6 min but the processors ran high for another 6. Also the stick is formatted ext2. I wait until the CPU’s settle before removing.

robin_listas · July 27, 2015, 3:25am

On 2015-07-26 19:06, gogalthorp wrote:
> robin_listas;2721086 Wrote:

>> No, no, if you watch the CPU load while this happens you will see that
>> it low.

> Well when I do a backup to USB stick I see lots of CPU usage in CPU
> monitor widget on all 6 cores here. I don’t see drag down of cursor but
> it is a large load. It does not show in top though just the widget. Also
> last time was about a gig of updates (use luckybackup (rsync GUI))
> process was done at 6 min but the processors ran high for another 6.
> Also the stick is formatted ext2. I wait until the CPU’s settle before
> removing.

Well, I’m writing now a 3 GB file (Leap iso) to a stick (as file). Total
CPU load on the machine is about 1%. The stick is FAT.


cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time cp openSUSE-42.1-DVD-x86_64-Build0017-Media.iso /media/Transcend/

real    3m57.224s
user    0m0.025s
sys     0m6.824s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time sync

real    0m0.757s
user    0m0.000s
sys     0m0.010s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

What is important is the time used on user and sys loads, not the “real”
time. Now, same copy to the hard disk:


cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time cp openSUSE-42.1-DVD-x86_64-Build0017-Media.iso ~/

real    0m44.355s
user    0m0.027s
sys     0m5.700s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time sync

real    0m3.595s
user    0m0.000s
sys     0m0.040s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

You see, CPU time is about the same (no cache flushing between the
tests, that may matter).

[At this point, I had to leave, so I hibernated my machine - and it crashed. I don’t know if related to the tests or to recent updates]

Let’s try with dd.


cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=direct ; time sync
6266880+0 records in
6266880+0 records out
3208642560 bytes (3,2 GB) copied, 2569,03 s, 1,2 MB/s

real    42m49.054s
user    0m3.558s
sys     2m12.962s

real    0m0.597s
user    0m0.000s
sys     0m0.017s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=nocache ; time sync
6266880+0 records in
6266880+0 records out
3208642560 bytes (3,2 GB) copied, 220,407 s, 14,6 MB/s

real    3m40.431s
user    0m1.305s
sys     0m29.520s

real    0m1.295s
user    0m0.001s
sys     0m0.019s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>
cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=dsync ; time sync
^C223247+0 records in
223247+0 records out
114302464 bytes (114 MB) copied, 1659,24 s, 68,9 kB/s

real    27m39.681s
user    0m0.397s
sys     0m7.106s

cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

maybe playing with chunk size.


cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=dsync bs=32K; time sync
^C40976+0 records in
40976+0 records out
1342701568 bytes (1,3 GB) copied, 1064,36 s, 1,3 MB/s

real    17m44.420s
user    0m0.027s
sys     0m4.696s

cer@Telcontar:/data/storage_b/Isos/Leap/42.1>
cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=dsync bs=32M; time sync
95+1 records in
95+1 records out
3208642560 bytes (3,2 GB) copied, 202,266 s, 15,9 MB/s

real    3m22.860s
user    0m0.002s
sys     0m7.256s

real    0m0.771s
user    0m0.000s
sys     0m0.016s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

Chunk size does make a difference, more so as I’m disabling the cache.
However, notice that on all tests CPU load is very low, and the machine remains very responsive (I opened a LO writer document while testing).


cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=nocache bs=32K; time sync
97920+0 records in
97920+0 records out
3208642560 bytes (3,2 GB) copied, 203,151 s, 15,8 MB/s

real    3m24.145s
user    0m0.032s
sys     0m7.868s

real    0m1.406s
user    0m0.000s
sys     0m0.015s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=nocache bs=32M; time sync
95+1 records in
95+1 records out
3208642560 bytes (3,2 GB) copied, 197,229 s, 16,3 MB/s

real    3m17.809s
user    0m0.000s
sys     0m6.747s

real    0m1.284s
user    0m0.000s
sys     0m0.013s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=direct bs=32K; time sync
97920+0 records in
97920+0 records out
3208642560 bytes (3,2 GB) copied, 279,719 s, 11,5 MB/s

real    4m40.178s
user    0m0.042s
sys     0m8.434s

real    0m0.596s
user    0m0.000s
sys     0m0.012s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso oflag=direct bs=32M; time sync
95+1 records in
95+1 records out
3208642560 bytes (3,2 GB) copied, 241,07 s, 13,3 MB/s

real    4m1.098s
user    0m0.002s
sys     0m7.232s

real    0m0.514s
user    0m0.000s
sys     0m0.012s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso ; time sync
6266880+0 records in
6266880+0 records out
3208642560 bytes (3,2 GB) copied, 245,751 s, 13,1 MB/s

real    4m5.784s
user    0m1.178s
sys     0m16.470s

real    0m2.262s
user    0m0.000s
sys     0m0.013s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso  bs=32K; time sync
97920+0 records in
97920+0 records out
3208642560 bytes (3,2 GB) copied, 217,996 s, 14,7 MB/s

real    3m39.344s
user    0m0.035s
sys     0m8.587s

real    0m1.276s
user    0m0.000s
sys     0m0.011s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1

cer@Telcontar:/data/storage_b/Isos/Leap/42.1> time dd if=openSUSE-42.1-DVD-x86_64-Build0017-Media.iso of=/media/Transcend1/openSUSE-42.1-DVD-x86_64-Build0017-Media.iso  bs=32M; time sync
95+1 records in
95+1 records out
3208642560 bytes (3,2 GB) copied, 204,788 s, 15,7 MB/s

real    3m25.765s
user    0m0.000s
sys     0m8.877s

real    0m1.328s
user    0m0.000s
sys     0m0.010s
cer@Telcontar:/data/storage_b/Isos/Leap/42.1>

Table of results:


512B         32K          32M

dd          13,1 MB/s  14,7 MB/s   15,7 MB/s
dd direct    1.2 MB/s  11,5 MB/s   13,3 MB/s
dd nocache  14.6 MB/s  15,8 MB/s   16,3 MB/s
dd dsync    68,9 kB/s   1,3 MB/s   15,9 MB/s

dsync writes to device asap, leaves a copy in cache.
nocache uses the kernel cache, then flushes it.
direct eliminates kernel cache mechanism completely.

I would vote for using “nocache” with 32 MB chunk size. The usb sticks native blocks are 32 KB, I think, which is why I chose that chunk size for testing; increasing the chunk size beyond that makes little difference, except with the dsync flag.

Using a small chunk size with the kernel cache mechanism disabled, besides being slow, probably wears the usb stick more. I should have remembered this before doing some of the tests, which is why I aborted some of them.

Notice that the problem I describe (unresponsive system) arises only when the size of what is being written to the slow writing device is about or bigger than the amount of ram the kernel can dedicate to disk buffers/cache.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

brunomcl · July 28, 2015, 12:33am

Just guessing:

Can’t what the OP wants be done with a systemd rule? I’m supposing systemd gobbled the udev service for this, where you could specify sync in the mount rule.

As a workaround, it might be usable to define at least some of the external drives is fstab with sync and perhaps noauto options, although the OP doesn’t want to do this for all drives.

Or perhaps I’m getting this all wrong, if the copy is buffered anyway and the sync option only forces the writing to finish ASAP, as when I issue a sync from terminal after the copy started.

robin_listas · July 28, 2015, 3:44am

On 2015-07-28 00:36, brunomcl wrote:
>
> Just guessing:
>
> Can’t what the OP wants be done with a systemd rule? I’m supposing
> systemd gobbled the udev service for this, where you could specify sync
> in the mount rule.

I don’t know/remember how, but mount options for automatic mounts can be
changed “somewhere”, yes. But I think that would not work in this case.

On the past, when writing to a floppy, I remember that using the “sync”
mount was terribly slow. And in flash, if it does similarly to “dd
oflag=dsync”, it is terribly slow.

Anyways, I can test it some other day, and add to my table.

–
Cheers / Saludos,

Carlos E. R.

(from 13.1 x86_64 “Bottle” (Minas Tirith))

robin_listas · July 28, 2015, 9:58pm

On 2015-07-28 03:44, Carlos E. R. wrote:

> On the past, when writing to a floppy, I remember that using the “sync”
> mount was terribly slow. And in flash, if it does similarly to “dd
> oflag=dsync”, it is terribly slow.
>
> Anyways, I can test it some other day, and add to my table.

Tested. It is very slow.


512B         32K          32M

dd          13,1 MB/s  14,7 MB/s   15,7 MB/s
dd direct    1.2 MB/s  11,5 MB/s   13,3 MB/s
dd nocache  14.6 MB/s  15,8 MB/s   16,3 MB/s
dd dsync    68,9 kB/s   1,3 MB/s   15,9 MB/s

mounted sync
dd          59.3 kB/s   201 kB/s   704 kB/s

Even increasing the number of bytes written each time, it doesn’t get to
1 MB/s.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)