Slow External Hard Drive Help Plz

English Hardware
1

So I just reinstalled OpenSUSE 11.2 and I went from having 20-30 mbps speed to and from my 1 tb usb ntfs external hard drive to having about 1 mbps transfer speeds now. Is there any suggestions for how I can fix this? I know it can go faster, I just need to figure out how. and would setting acpi=off at boot affect this?

Opensuse 11.2
KDE 4.3
64 bit
EXT4 main disk
NTFS external

fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00022619

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          25      200781   83  Linux
/dev/sda2              26         281     2048000   82  Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3             282       19457   154031220   83  Linux

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xe8900690

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      121601   976760001    7  HPFS/NTFS
2

Have you run zypper up

There were some ntfs-3g updates IIRC
Though I’m not sure what they were fixing

3

yea everythings fully upgraded. should i change permissions of the external from root? its on root right now and Im wondering if changing it to user would do anything

4

It’s ntfs and usb, so you shouldn’t need to change anything.

In kde4 devices may show in Dolphin, but they are not mounted until you click it in the ‘places’. if it’s currently mounted, right click a safely remove/release/eject whatever… then

Try a manual mount

mount -t ntfs-3g /dev/sdb1 /path_to/mount_point

Eg; mount -t ntfs-3g /dev/sdb1 /media

5

the external is still going super slow and after getting suspicious and trying it on my ubuntu laptop its going really slow on it as well. It could possibly be the hard drive going bad but its only started going agonizingly slow after I reinstalled opensuse. Im stumped as to what to do to fix it.

6

Do you have access to a Windows PC?

7

Just wanted to confirm that I am having exactly the same problem with all the latest updates etc. It starts of at 25 to 50 MB per second on my i7-930 and the simple drops of to speeds of 1-12 MB/second. As I have e-SATA ports on my external Drive, I tried using them and the problem persists. Personally I think the driver(s) used for NTFS might not be up to scratch in OpenSUSE and probably also in other Linux versions.

The sure way to test if the problem is NTFS related is to install EXT3 or EXT4 file system on the external device and see if the same problems exist. If not, obviously it must be related to the NTFS drivers or a combination of factors like file system and system cache.

I will test this an hopefully report back on my findings. You could also do this yourself. The drive effected is a 2 TB Western Digital Green. I know their were some other issues with Western Digital Drives related to partitioning I have not seen any issues. Dolphin is also not to blame as the same problem is evident on rsync. It appears that it goes fast until the Linux memory cache has been filled and then it slows down and then never properly recovers. Perhaps the problem is related to system cache!

I need my drive with NTFS because I use it on an Western Digital media player which I don’t think is Linux compatible (At least not EXT4). Reading seems fine. I did format the drive with the latest Windows 7 has to offer. Not sure if new features have been implemented in NTFS as included in Windows 7. Microsoft is known to release updates that cause compatibility issues. If I did not need it, I would not use a MS based files system to start with as a matter of not supporting Microsoft. Formatting the drive with FAT32 is not an option for a 2 TB drive but one could try using XP to create NTFS partitions and see what happens. I assume that Windows XP is using an older version of NTFS which might effect drive operation. If so, perhaps the developers of the new NTFS drivers for Linux have missed something. It is working but not right and this usually only becomes evident if you start copying lots of files or large files. I also have a couple of Western Digital Black 2 TB drives which I will also test one of. The reason I picked the Green drive for my media player is because it does not get as hot, saves power, extra bandwidth capacity is not needed and the price is more realistic. Fact is that the problem does not appear drive related unless it has to do with drives exceeded. My bet is that it is the NTFS driver.

8

Tried to add this link which is related to the same issue in my opinion: USB copy too slow - openSUSE Forums

9

**More findings: **Later today I tried synchronizing over 100GB to my 2 TB Western Digital green drive using:

Command:
rsync --progress -avx source destination

As usual the problem re-appeared.
**
Hypothesis: **Try syncronizing the drives cache buffers

Command:
sync

Meaning of Sync Command:
Force changed blocks to disk, update the super block.

14.4 `sync’: Synchronize data on disk with memory

sync' writes any data buffered in memory out to disk. This can include (but is not limited to) modified superblocks, modified inodes, and delayed reads and writes. This must be implemented by the kernel; The sync’ program does nothing but exercise the `sync’ system call.

The kernel keeps data in memory to avoid doing (relatively slow) disk
reads and writes. This improves performance, but if the computer
crashes, data may be lost or the file system corrupted as a result.
The `sync’ command ensures everything in memory is written to disk.

Any arguments are ignored, except for a lone --help' or –version’
(*note Common options::).

An exit status of zero indicates success, and a nonzero value
indicates failure.

Result: This does appear to have an impact. Drive returns to normal operation shortly after but the speed problem re-appears at different transfer capacities (Re-appeared at 8 and then after flushing again with “rsync” again at over 36+ GB)

Hypothesis: Turn of the drives write caching feature like this:

Command:
hdparm -W 0 /dev/sdg1

/dev/sdg1:
setting drive write-caching to 0 (off)
write-caching = 0 (off)

Meaning of hdparm parameter:

  • W Get/set the IDE/SATA drive´s write-caching feature

Result: This did not do anything and I was back where I started after just 9 GB later. It appeared to have a similar impact as the “sync command”.

Hypothesis: Turn of read-lookahead - Did not think this would do anything when writing to the drive but heck, if you don’t try you will not know:

Command:
hdparm -A 0 /dev/sdg

/dev/sdg:
setting drive read-lookahead to 0 (off)
look-ahead = 0 (off)

Meaning of hdparm parameter:
-A Get/set the IDE drive´s read-lookahead feature (usually ON by default). Usage: -A0 (disable) or -A1 (enable).

Result: This did not do anything and I was back where I started after just 12 GB later. It appeared to have a similar impact as the “sync command”.

Findings: Problem appears to be system cache related. Still need to try using EXT4 filesystem etc. to see if this reproduces the result. I have experienced the problem with FAT32 and NTFS on memory stick (flash drives - Verbatim Store-n-Go 8 & 16GB) as well as USB, e-SATA Drives etc. Don’t have have any internal drives with NTFS or FAT32 but I suspect the same result as per e-SATA. At least I have a partial solution that works but I think the Kernel Programmers must review this problem. Perhaps we will stumble onto a more precise solution allowing Kernel programmers to figure out the problem.

10

Problems are definitely related to NTFS (Not sure about FAT32).

Formatted another 2 TB Western Digital Green Drive with EXT4 file system. Transferring data from exactly the same source to the newly setup drives transfers data at about 70MB/sec via e-SATA which is at least 2 times faster than with the NTFS and continuously transfered 200GB+ using rsync without any sign of slowdown or performance degradation during the transfer.

Stopped the transfer, ran sync and flushed the system cache. Ran “hdparm -f /dev/sdg” to make sure all data store in the drives onboard cache is flushed to the plattern and “hdpram -Y /dev/sdg” to shutdown the drive. Once completed, I disconnected the e-SATA cable and connected the drive via USB2 port and cable. Started drive. As this is USB, OpenSUSE detected the drive automatically. Using eSATA I need to run “rescan-scsi-bus.sh” for the system to rescan the SCSI bus which will then pick up SATA drives as well.

After the above, I continued the rsync transfer which keeps running at about 32MB/sec (Limited by USB2) which proves my point that the problem is either a NTFS problem or the way it uses the system cache.

Best way to flush caches is to run:

sync; echo 3 > /proc/sys/vm/drop_caches;

Here is the meaning for the echo command:

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

Just to mention, accessing the source drive during the transfer appeared (I am not 100% sure) to influence the problem (It seemed to happen more frequantly). The reason I am saying this is because my source (Internal)drive is not part of the root Linux system which means that it should not be influenced by disk writes somewhere in the root file system. It is mounted into a sub-folder of the root system. The root system is on /dev/sdax while the discussed source drive is on /dev/sdbx. My source drive is using EXT4 file system. Up to now the problem has not been resolved. Any suggestions we can test are welcome.

In case you are wondering, here are my software details:

OpenSUSE Linux 11.2 x86_64
uname -a
Linux ICS-LINUXSERVER 2.6.31.12-0.2-desktop #1 SMP PREEMPT 2010-03-16 21:25:39 +0100 x86_64 x86_64 x86_64 GNU/Linux

Latest Update yesterday for KDE 4.4 and OpenSUSE 11.2

11

Not sure about this. We have the same setup (nearly the same), with openSUSE 11.4 and three USB drives that we use for backup with rsync.
rsync starts at a good rate and then … slows… until 1 mb/s.

It may be the USB devices themselves or it may be the OS.

Reading the thread, it seems that formatting the drive fasters the write access, but in our case we can’t do that. We’re doing incremental rsyncs.

The file system are all EXT4.

Can somebody help us on this ? I’ll continue to post our work and related results here.

12

On 01/29/2013 05:26 PM, vdegrandpre wrote:
> openSUSE 11.4

that version has gone past its end of life and has had no security
updates for months…strongly advise you to move it to evergreen (a
simple process) which will get you patches to Jul 2013, read here:
http://tinyurl.com/4aflkpy

note: i am not saying this will solve your problem…unless you find
you machine has already been kracked, and sending out all that spam/etc
is slowing stuff down.

–
dd
http://tinyurl.com/DD-Caveat
http://tinyurl.com/DD-Software

13

On 2013-01-29 17:26, vdegrandpre wrote:

> Not sure about this. We have the same setup (nearly the same), with
> openSUSE 11.4 and three USB drives that we use for backup with rsync.
> rsync starts at a good rate and then … slows… until 1 mb/s.
>
> It may be the USB devices themselves or it may be the OS.

It is the device. A write speed of 1MB/s for flash devices is typical.
Some may be faster, I don’t think I have seen over 4 MB/s. Over that,
you get the USB bus limit.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

14

On 01/29/2013 08:15 PM, dd wrote:

> Jul 2013,

sorry, openSUSE 11.4 Evergreen is supported to July 2014…

–
dd

15

On Tue, 29 Jan 2013 16:26:01 +0000, vdegrandpre wrote:

> Not sure about this. We have the same setup (nearly the same), with
> openSUSE 11.4 and three USB drives that we use for backup with rsync.
> rsync starts at a good rate and then … slows… until 1 mb/s.
>
> It may be the USB devices themselves or it may be the OS.

It is the device, unless you’re using USB 3.0 (in which case, the answer
is “it may be the device”).

I/O to USB devices in general (and 1.1 and 2.0 devices specifically) is
relatively slow compared to a direct attached hard drive.

You see fast performance at the beginning because the OS is hiding the
fact that the device is slow by buffering the writes. Eventually,
though, you fill the buffer and the OS has to push back on the
application and tell it to wait before sending more data. (This is
referred to as “iowait” - ie, waiting on Input/Output).

When you push a lot of data, the application (rsync in this case) is
constantly told to stop sending data while the device flushes the data to
disk. You’ll see a slow effective throughput (which you do), and also
you’ll see (if you use iostat) that the process is being held up on I/O.
If you look at /proc/loadavg you’ll see that the averages are increasing.

This will happen regardless of the filesystem or the type of device
(flash drive, USB connected hard drive) because the bottleneck is the
bus, not the device.

The only solution is to get a faster interface for the device.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

16

On 2013-01-30 00:32, Jim Henderson wrote:
> This will happen regardless of the filesystem or the type of device
> (flash drive, USB connected hard drive) because the bottleneck is the
> bus, not the device.

There are two bottle necks. When writing to a usb flash stick, the neck
is the flash device itself. The bus can go much faster, as it is seen
when you read from the same device or you write to a hard disk via usb.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

17

On Wed, 30 Jan 2013 02:18:07 +0000, Carlos E. R. wrote:

> On 2013-01-30 00:32, Jim Henderson wrote:
>> This will happen regardless of the filesystem or the type of device
>> (flash drive, USB connected hard drive) because the bottleneck is the
>> bus, not the device.
>
> There are two bottle necks. When writing to a usb flash stick, the neck
> is the flash device itself. The bus can go much faster, as it is seen
> when you read from the same device or you write to a hard disk via usb.

Depends on the device - a USB hard drive and a USB stick drive are going
to have different performance characteristics. Similarly, read
performance and write performance are two different things and aren’t
directly comparable, especially with more complex devices that insulate
the read performance using read-ahead caching in the device and other
similar technologies.

And in most instances, from everything I’ve read and worked with
(including with low-level tools designed to optimize the throughput of
the bus in the Linux kernel itself), the bottleneck is generally the USB
bus itself, not the device.

But regardless of whether the bottleneck is the bus or the device, the
queuing mechanism is the same in the kernel, and particularly when
writing (which is the issue the OP here is having), that queuing
mechanism is the bottleneck from the kernel’s perspective, his iowait% is
going up, and his loadavg will be very high relative to normal
performance.

Jim

Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

18

On 2013-01-30 03:47, Jim Henderson wrote:
> On Wed, 30 Jan 2013 02:18:07 +0000, Carlos E. R. wrote:
>
>> On 2013-01-30 00:32, Jim Henderson wrote:
>>> This will happen regardless of the filesystem or the type of device
>>> (flash drive, USB connected hard drive) because the bottleneck is the
>>> bus, not the device.
>>
>> There are two bottle necks. When writing to a usb flash stick, the neck
>> is the flash device itself. The bus can go much faster, as it is seen
>> when you read from the same device or you write to a hard disk via usb.
>
> Depends on the device - a USB hard drive and a USB stick drive are going
> to have different performance characteristics. Similarly, read
> performance and write performance are two different things and aren’t
> directly comparable, especially with more complex devices that insulate
> the read performance using read-ahead caching in the device and other
> similar technologies.

Yes, absolutely.

> And in most instances, from everything I’ve read and worked with
> (including with low-level tools designed to optimize the throughput of
> the bus in the Linux kernel itself), the bottleneck is generally the USB
> bus itself, not the device.

My experience is different. The 1 MB/s write speed is the typical limit
of cheap usb sticks (flash). Better sticks can get 4 MBs. Hard disks on
the same bus and machine are way faster: I can tell you a figure later
on the day if you wish, I have none connected this minute.

> But regardless of whether the bottleneck is the bus or the device, the
> queuing mechanism is the same in the kernel, and particularly when
> writing (which is the issue the OP here is having), that queuing
> mechanism is the bottleneck from the kernel’s perspective, his iowait% is
> going up, and his loadavg will be very high relative to normal
> performance.

Yep.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

19

On Wed, 30 Jan 2013 11:38:06 +0000, Carlos E. R. wrote:

>> And in most instances, from everything I’ve read and worked with
>> (including with low-level tools designed to optimize the throughput of
>> the bus in the Linux kernel itself), the bottleneck is generally the
>> USB bus itself, not the device.
>
> My experience is different. The 1 MB/s write speed is the typical limit
> of cheap usb sticks (flash). Better sticks can get 4 MBs. Hard disks on
> the same bus and machine are way faster: I can tell you a figure later
> on the day if you wish, I have none connected this minute.

USB 2.0 can achieve speeds of up to 35 MB/s. Compare to SATA, which has
speeds measured in Gbit/s.

SATA 1.0 topped out at 1.5 Gbit/s.

You’re defining high-speed here (which, arguably, USB 2.0’s spec is
defined as) as being relatively slow compared to modern disk/bus speeds.

USB 3.0, OTOH, claims a theororetical max of 5 Gbit/s.

So, if you’re using rsync from a SATA drive - even SATA 1.0 - the data
read speed from the SATA device is going to saturate the USB bus in an
incredibly short amount of time.

If your source drive is SATA 3.2 (the latest spec, it seems), you’re
reading at 8 Gbit/s of data and still only writing via USB 2.0 at an
absolute max of 35 MB/s (= 480 Mbit/s). You’ll saturate the bus in under
a second if you’re reading at full speed from the source drive.

The bus is going to be a bottleneck in this case /no matter what/ is at
the other end of the USB bus.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

20

Jim, Carlos, now that the both of you are elaborating on this, here’s a question that’s been on my mind for a while now: using rsync to backup a couple of server folders to a USB-harddisk takes approx. double the time that luckybackup (rsync based) takes. IIRC I already took care that the backup script used the same rsync options that were (by default) configured in luckybackup. Both backup results are identical (taken on a dead moment, i.e. no other activity to the disk).