Fastest local file transfer protocol?

English Network/Internet
#1

Recently I resurrected an old laptop to act as a file & media server for my family. It was working swimmingly, and the only gripe I had was that file transfer speeds seemed slow. For example, copying 300MB of images could take 15 minutes.

At first I thought it was a limitation of Samba, since both Vista and XP systems all copy files slowly.

It’s not a hardware limitation. When I have the laptop pull down updates for OpenSuse I can get 300MB of data over the net in just a few minutes.

This weekend I installed ftpd, thinking I could work around “the samba problem”, but was surprised to find the transfer speeds were no faster compared to copying files through samba.

It seems odd to me that I can download files over the net faster than I can copy them over my local network.

Any thoughts / ideas / guesses / suggestions for the best way to set up a speedy local file xfer process?

#2

Probably the bottleneck is the laptop disk. Those disks are optimised for low power consumption at the expense of speed.

If you have enough spare RAM left over from program footprint, a lot of downloaded data is buffered in memory and written to the disk later, after the transfer appears to have completed.

#3

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

FTP should be fairly fast, but Samba shouldn’t be THAT slow at least
either. You coul dtest just the read speed with no speed hit on the
destination side due to writes like the following:

curl ftp://your.server.here/path/to/file/here.zip >/dev/null

If that goes any faster then it’s a write speed issue on the destination
side… though really that’s insanely slow even for a slow filesystem
like NTFS so I’m not sure. If nothing else perhaps get a LAN trace to
see what you can come up with there, though reading those can take some
time. Do other boxes providing files also work slowly? Do other boxes
reading files also read slowly? Narrow it down so you know for sure
it’s just something strange on this box… try ‘scp’ (secure copy…
file transfer over SSH) or ‘nfs’ if possible.

Good luck.

riddil wrote:
> Recently I resurrected an old laptop to act as a file & media server for
> my family. It was working swimmingly, and the only gripe I had was that
> file transfer speeds seemed slow. For example, copying 300MB of images
> could take 15 minutes.
>
> At first I thought it was a limitation of Samba, since both Vista and
> XP systems all copy files slowly.
>
> It’s not a hardware limitation. When I have the laptop pull down
> updates for OpenSuse I can get 300MB of data over the net in just a few
> minutes.
>
> This weekend I installed ftpd, thinking I could work around “the samba
> problem”, but was surprised to find the transfer speeds were no faster
> compared to copying files through samba.
>
> It seems odd to me that I can download files over the net faster than I
> can copy them over my local network.
>
> Any thoughts / ideas / guesses / suggestions for the best way to set up
> a speedy local file xfer process?
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJYWyP3s42bA80+9kRArpHAKCJRznnru8zXfqOTWHYAIlzf4OizwCff/I+
kvcRSzBuutM/fkG9qZ7x2VA=
=Mu9F
-----END PGP SIGNATURE-----

#4

Hi,

Maybe the TCP window of your windows box is to small. More information you get here:

Description of Windows 2000 and Windows Server 2003 TCP Features

and here:

Windows XP step by step TCP tuning

hth

Greetings

Erik

#5

if you get 9Mbits/sec transfer you cannot complain .
that is about the max transfer possible with vnc ssh ftp and lots of others.

#6

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

While 9 Mbit is good it’s nowhere near the maximum possible unless he’s
on a 100 Mbit connection. On a gigabit connection I can transfer at
forty to seventy megabits regularly (for sustained periods, not just
bursts) on my laptop using either FTP or scp (SSH) though obviously this
implies that the other machine is also on gigabit and there aren’t other
issues.

Good luck.

ramaswamyps wrote:
> if you get 9Mbits/sec transfer you cannot complain .
> that is about the max transfer possible with vnc ssh ftp and lots of
> others.
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJY0S83s42bA80+9kRAnEzAJ9ssYHDebCJXGm0P3z9OBqrYSkgBgCghFVm
duNEzVFCIUSjUv3B1mwRpp4=
=fbH5
-----END PGP SIGNATURE-----

#7

Hi,

Sorry, but 9Mbit/s is less than poor on a cabled 100Mbit-Network. I just copied a dvd imag file of 4.3 Gig in 9.30 mins. So now let’s calculate:

4.3gig = 4.3 x 1024 x 8 Mbit = 35,225.6 Mbit

35,255.6Mbit / 570 s = 61.85 Mbit/s

That’s the average netto data rate of that transmission. And that’s what you should reach on a cabled 100Mbit-Network. It’s about 1/3 of the brutto data rate what we need for headers and network managment. So in the best case you can reach 2/3 of the brutto data rate as netto data rate which means 66Mbit/s on a cabled 100Mbit-Network.

To go back to the original problem: It is for sure not the smb protocol which is fast and stable (not everything that comes from M$ is bad ;)). I did this transmission with samba from Suse 9.3 to Suse 11.1. No problem with this. So the reason for the problem must be placed on the layers under the samba protocol.

So I think the best thing to resolve it is first to check the hardware like cable, switch, nics. A problem like this that the network connection to one port works fine (in this case the internet connection which is faster than the local one) and to another lames can be caused by a broken port in the switch, a bad cable from the station that lames and the switch etc.

The first thing I would do is to use ping in various ways to make sure that the connection really works. All pings you should do from the linux machine cause I use the linux parameters:


ping ip_of_the_win_box

A normal ping. You should get round trip times less than 0.4ms. If it is like this go on.


ping -s 60000 ip_of_the_win_box

That pings the windows box with 60,000 bytes. That should give you a result of about 11 ms. For the next steps you must be root.


ping -f ip_of_the_win_box

This does a flood ping which means that the pings are sent out as fast as possible. Let it run for a while. You don’t see anything but one (or more dots). Well, on local network you should see only one. After some minutes hit ctrl-c to stop the pings. You should get an average of about 0.2 ms. So the last step is to do a flood ping with big package size.


ping -fs 60000 ip_of_the_win_box

Average should be about 11ms. If this all works fine you can be quite sure that it is not the cabling or the IP-Stack. If not you have to measure cable and perhaps reinstall or reconfigurate your IP installation.

Next step would be to play around with TCP window settings. Also it could be a wrong configured firewall software or a thing like windows defender or any other security software. To make sure that it is not switch all that stuff off and test again.

Ok, enough to try for the moment. If this doesn’t help we go on. :wink:

Greetings

Erik

#8

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sorry… I typo’d and put in an extra ‘0’ when I typed ‘100 Mbit’… I
meant 10 Mbit…

Good luck.

erikro wrote:
> Hi,
>
> ab@novell.com;1922564 Wrote:
>> While 9 Mbit is good it’s nowhere near the maximum possible unless
>> he’s
>> on a 100 Mbit connection.
>
> Sorry, but 9Mbit/s is less than poor on a cabled 100Mbit-Network. I
> just copied a dvd imag file of 4.3 Gig in 9.30 mins. So now let’s
> calculate:
>
> 4.3gig = 4.3 x 1024 x 8 Mbit = 35,225.6 Mbit
>
> 35,255.6Mbit / 570 s = 61.85 Mbit/s
>
> That’s the average netto data rate of that transmission. And that’s
> what you should reach on a cabled 100Mbit-Network. It’s about 1/3 of the
> brutto data rate what we need for headers and network managment. So in
> the best case you can reach 2/3 of the brutto data rate as netto data
> rate which means 66Mbit/s on a cabled 100Mbit-Network.
>
> To go back to the original problem: It is for sure not the smb protocol
> which is fast and stable (not everything that comes from M$ is bad ;)).
> I did this transmission with samba from Suse 9.3 to Suse 11.1. No
> problem with this. So the reason for the problem must be placed on the
> layers under the samba protocol.
>
> So I think the best thing to resolve it is first to check the hardware
> like cable, switch, nics. A problem like this that the network
> connection to one port works fine (in this case the internet connection
> which is faster than the local one) and to another lames can be caused
> by a broken port in the switch, a bad cable from the station that lames
> and the switch etc.
>
> The first thing I would do is to use ping in various ways to make sure
> that the connection really works. All pings you should do from the linux
> machine cause I use the linux parameters:
>
>
> Code:
> --------------------
>
> ping ip_of_the_win_box
>
> --------------------
>
>
> A normal ping. You should get round trip times less than 0.4ms. If it
> is like this go on.
>
>
> Code:
> --------------------
>
> ping -s 60000 ip_of_the_win_box
>
> --------------------
>
>
> That pings the windows box with 60,000 bytes. That should give you a
> result of about 11 ms. For the next steps you must be root.
>
>
> Code:
> --------------------
>
> ping -f ip_of_the_win_box
>
> --------------------
>
>
> This does a flood ping which means that the pings are sent out as fast
> as possible. Let it run for a while. You don’t see anything but one (or
> more dots). Well, on local network you should see only one. After some
> minutes hit ctrl-c to stop the pings. You should get an average of about
> 0.2 ms. So the last step is to do a flood ping with big package size.
>
>
> Code:
> --------------------
>
> ping -fs 60000 ip_of_the_win_box
>
> --------------------
>
>
> Average should be about 11ms. If this all works fine you can be quite
> sure that it is not the cabling or the IP-Stack. If not you have to
> measure cable and perhaps reinstall or reconfigurate your IP
> installation.
>
> Next step would be to play around with TCP window settings. Also it
> could be a wrong configured firewall software or a thing like windows
> defender or any other security software. To make sure that it is not
> switch all that stuff off and test again.
>
> Ok, enough to try for the moment. If this doesn’t help we go on. :wink:
>
> Greetings
>
> Erik
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJZLp83s42bA80+9kRAi3CAJ9w10TwpozErKyGI7SRzr1A6yNraACeL3lc
NgOdxcGn0bBcJCm1SwsR730=
=TQZG
-----END PGP SIGNATURE-----

#9

Howdy folks, thanks for the great replies! And apologies for my absence… I abhor folks who ask questions and then go incognito. This week I found out I’m taking a 50% pay cut, and I (and my pregnant wife) need to relocate to a new city several thousands of miles away. Joy!

So, apologies, I haven’t had time to verify / test any of the suggestions yet. But reading through the comments I think there’s some good ideas.

FYI, all the systems are connected over a 54Mb (802.11g) AP locally, with a 2M ADSL pipe to the outside world. What gets me is that I can have the two systems simultaneously download a 10MB file from a remote server faster than I can copy the same 10MB file between the two systems.

On a whim I loaded Ubuntu 8.10 since it’s so quick to load and configure to confirm the same slow-to-copy-but-fast-to-download behavior was occurring. (Answer: yes).

If I get some time this weekend I’ll tinker with the TCP settings, and also check the AP and make sure there’s not some odd QoS setting that’s throttling the data.

#10

Hmm… I think IBM are to thank initially. Rebranded as CIFS by M$. Admittedly M$ have added some bits themselves, very nice of them.