The fast & secure way to transfer large data

English Forums Feedback
#1

You might think, scp is easy, secure, and usually fast enough. But the truth is that for large files scp is slow.

Thankfully tunneling through ssh is fast. You might also think that scp is tunneling through ssh. But scp actually uses these nifty internal buffers that really slow things down.

Using tar and lzma compression in openSUSE 11.1 or gzip compression in less than openSUSE 11.1 you can significantly reduce the time to securely copy large files from one machine to another.

tar --lzma -cf - <source> | ssh <host> tar --lzma -xf - -C <destination>

For example:

tar --lzma -cf - /home/mdb/NeverwinterNights/NWN/ | ssh 192.168.1.2 tar --lzma -xf - -C /home/xemiu/
#2

If this on your LAN and you are not concerned about interlopers, you can do this:

At the receiving end run:

netcat -l -p 12345 | tar xvf -

Then go to the sending end and run:

tar cf - . | netcat recv 12345

where recv is the name or IP address of the other end.

This assumes you are in the right directories already, so adjust as necessary.

I’ve used a variant of this to clone a machine onto a disk on another, and as a last step I install the bootloader.