SOLVED: wget is hanging on transferring images

swerdna · June 25, 2009, 6:33am

Hi
AB has suggestted that I experiment with wget to back up websites remotely. The suggestion is here: SOLVED: How do I connect ftp in one command - Page 2 - openSUSE Forums

I have a test home website with a document root at /home/webname/public_html.
The test site is also an ftp server and is accessible via vsftpd.
I’m using wget to backup the files with a command like this from a remote client:

wget -r --ftp-user=username --ftp-password=password ftp://testdomain.blah.com

That nicely downloads the entire contents of the document root and subdirectories recursively EXCEPT it always hangs mid way through.
The process seems to just stop. Each time I try it the process stops on the same file. If I delete the file from the server (just to see what happens) and repeat the “wget” command from the client, the process proceeds further than the last time and then hangs on a different file.

Here’s an example of the dialogue when it stops:

100%===============================================================================================>] 16,912      --.-K/s   in 0.007s

2009-06-25 14:19:20 (2.25 MB/s) - `domain.name.com/webname/public_html/dynamic/snapshot3.png' saved [16912]

--2009-06-25 14:19:20--  ftp://domain.name.com/webname/public_html/dynamic/snapshot4.png
           =&gt; `domain.name.com/webname/public_html/dynamic/snapshot4.png'
==> CWD not required.
==> PASV ...

Can you advise what I might be doing wrong?

Thanks
Swerdna

ken_yap · June 25, 2009, 1:14pm

Does the ftp server have some kind of rate-limiting?

swerdna · June 25, 2009, 2:00pm

I don’t know how to answer than, not knowing much about ftp / vsftpd. I looked in vsftpd.conf and this is commented out: #anon_max_rate=7200. I don’t know if that implies some other default value is active. The actual contents are:

dirmessage_enable=YES
nopriv_user=ftpsecure
anonymous_enable=NO
anon_world_readable_only=YES
syslog_enable=NO
connect_from_port_20=YES
pam_service_name=vsftpd
listen=YES
ssl_enable=NO
pasv_min_port=30000
pasv_max_port=30100
anon_mkdir_write_enable=NO
anon_root=/srv/ftp
anon_upload_enable=NO
chroot_local_user=YES
ftpd_banner=Welcome message
idle_session_timeout=900
local_enable=YES
log_ftp_protocol=NO
max_clients=10
max_per_ip=10
pasv_enable=YES
ssl_sslv2=NO
ssl_sslv3=NO
ssl_tlsv1=YES
write_enable=YES
local_root=/home/
local_umask=022

ken_yap · June 25, 2009, 2:08pm

Also if you are running vsftpd under xinetd, xinetd will also have rate-limiting to protect the system from too many ftp requests in a given period. If so, you might want to run vsftpd standalone. Have a look in /var/log to see if there are any clues in any log file.

swerdna · June 25, 2009, 2:17pm

No it’s a standalone started by /etc/init.d/vsftpd at boot time.

ken_yap · June 25, 2009, 2:22pm

You may have to do a packet trace on both ends with tcpdump or ethereal to see which side was left waiting in the conversation.

Is there some kind of NAT gateway with port forwarding in between?

swerdna · June 25, 2009, 2:28pm

Log file is missing. I looked in vsftpd.conf. A log file is designated at /var/log/xferlog but it’s not there – strange.

ab · June 25, 2009, 5:26pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I add -T 3 to all of my wget commands these days in case the server is
stupid. See the man page for info but basically after three seconds of
stupidity wget tries again. I hit this a lot because of bad
configurations (now fixed I believe) with some NAM/iChain implementations
but that’s still useful for timeouts that can be pushed past with a retry.
Give it a shot.

Good luck.

swerdna wrote:
> Hi
> AB has suggestted that I experiment with wget to back up websites
> remotely. The suggestion is here: ‘SOLVED: How do I connect ftp in one
> command - Page 2 - openSUSE Forums’ (http://tinyurl.com/lt38lk)
>
> I have a test home website with a document root at
> /home/webname/public_html.
> The test site is also an ftp server and is accessible via vsftpd.
> I’m using wget to backup the files with a command like this from a
> remote client:
> Code:
> --------------------
> wget -r --ftp-user=username --ftp-password=password ftp://testdomain.blah.com
> --------------------
>
> That nicely downloads the entire contents of the document root and
> subdirectories recursively EXCEPT it always hangs mid way through.
> The process seems to just stop. Each time I try it the process stops on
> the same file. If I delete the file from the server (just to see what
> happens) and repeat the “wget” command from the client, the process
> proceeds further than the last time and then hangs on a different file.
>
> Here’s an example of the dialogue when it stops:
> 100%===============================================================================================>]
> 16,912 --.-K/s in 0.007s
>
> 2009-06-25 14:19:20 (2.25 MB/s) -
> `domain.name.com/webname/public_html/dynamic/snapshot3.png’ saved

[16912]

–2009-06-25 14:19:20–
ftp://domain.name.com/webname/public_html/dynamic/snapshot4.png
=>
`domain.name.com/webname/public_html/dynamic/snapshot4.png’
> ==> CWD not required.
> ==> PASV …
>
> Can you advise what I might be doing wrong?
>
>
> Thanks
> Swerdna
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIcBAEBAgAGBQJKQ5c8AAoJEF+XTK08PnB5H14QAI8/SFE29XDoX7g4JFcasEKT
bQhdMmijfV2z4hk/xtW7NLizuTqNGWdCwy6NCn7vg1Ti1zeGk0OkgZT4/ANAEl5t
Toilw+yYF/QBfQTcTggZJ3xX6mTlNjRAnzKkshHDYO96K/YI4qFpHG8dbMw3bRUn
hey5ClIc/PWWaYcFF//dn6HKcEqsh6YQEm2SH2WSFDWvQNzs3bhmHV0eIOJ2+nOz
7txc4H40MPGnve+ROq20AEuDSkqGhq2ZFxESpYWgdv3q2k5B1ajxe3UlPSQl2CGa
36cRXyk/1810v/IV058P8awfsW+bNLpIQkcdQh6jS7AQ2Y2eUXGeU86l9BAnRkML
XiGu9sTzuChRjvDfgsy2O8P9MqcTUYc7NAhjaDSJiO7wSzhI4GNtxm+ZXosa0dEr
b/0GzpIoJeHtT7AFq+vmLmUP1FBjnM51hZNUB5nVil+gGPJIrKSVg0iJH6M7o4Wr
DFmvRq+ln7Jlm9YlYWdprXw0r+VcP7npcQGQxoMZji8wgDzJNnaZlWrQzDCTYQKD
dwa7GznHn8eKp1OkWVkN7lon8vKO9+m6UnnJfkVdapT8H7bItrIbz9NqWhUNYbDT
skU9yavWidHM56NmZzXU7hv2w+6Gg8ISd/+VjUnTdeNlEqAG3MkqfB5D1i1hMM/e
YDovJLaPWukg42MGVc/W
=UipJ
-----END PGP SIGNATURE-----

ken_yap · June 25, 2009, 5:44pm

You may have to create it manually. It might be a server that doesn’t automatically create log files. Make sure it is writable by the account running vsftpd, but this would probably be root anyway.

swerdna · June 25, 2009, 10:28pm

That fixed it – and the prize goes to ab@novell.com
Many thanks

swerdna · June 25, 2009, 11:19pm

The log file was not recording because I omitted to switch it on in vsftpd.conf. Once I did that and restarted vsftpd, it recorded. But this didn’t throw any light on the problem.

However I found that by shortening the recursion depth the problem disappeared. So there are two ways to work around the problem:

use the -T 3 option
do not use long recursion depths

And thanks to you both very much for the help.

ken_yap · June 26, 2009, 2:34am

It may not be wget or your server. I’ve seen a situation where connections to public servers inside a enterprise LAN would break in the middle of transsfers (ssh, http, media streaming, you name it). The catch was that it only happened at high speed, so only the fraction of people with cable or ADSL2+ would experience the problem. I looked everywhere and finally concluded it could be some router dropping connections on congestion. I was wondering how to collect evidence of this when it came good and stayed good. Coincidentally infrastructure was moved/upgraded that weekend. So maybe not coincidence. I guess I’ll never know, but I’m not complaining.

So black magic workarounds like these are sometimes necessary.

swerdna · June 26, 2009, 3:33am

There’s a saying I’ve adapted that applies so well to Linux: “whatever turns your clockwork orange”

ab · June 26, 2009, 4:33am

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

That’s good to hear. Thank-you for posting back.

Good luck.

swerdna wrote:
> ab@novell.com;2004216 Wrote:
>> I add -T 3 to all of my wget commands these days in case the server is
>> stupid. See the man page for info but basically after three seconds
>> of
>> stupidity wget tries again. I hit this a lot because of bad
>> configurations (now fixed I believe) with some NAM/iChain
>> implementations
>> but that’s still useful for timeouts that can be pushed past with a
>> retry.
>> Give it a shot.
>>
>> Good luck.
> That fixed it – and the prize goes to ab@novell.com
> Many thanks
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIcBAEBAgAGBQJKRDNvAAoJEF+XTK08PnB5tvEQAIkf5iiTVLFvDkb6kBfYA8T6
EECN/2IZ6G9ct3lm0MWvvDCC52B7Q9mtMHXbx2o+Wu1caZX9kvX1M2lZu9xAawmT
BtrXBxosU4jwNTY6d4p01WqPqELoUK1X7k7gU+sKhhNshipXBoc+90UuGH1C8lYJ
yaFIP/tUb2p2JngdBluFvzSgcxaKCINi4GUn/o6JFCufpZPezz76T1dbioKgAzu2
JozaosG2gy72MGnuB7sWLdEQnl3PdeExMUlgtUQJKSHW9oq/04epxKnr9Jf5wT5z
7Z3vI+r5Uu4d0Tk1FRG/3p16JeAjGcnyoDzvO6jfFokdpBN79OkrGx73BdztSrHW
+iZv1pQncd+WfNlz/FEz2zjBnWdt2K2gyX1ZMNg6mnPgMMu94XjlrSCBYuC6VXao
O4yB6jjmqDDxQVncn3goNH+bz2c1CKevOjiWgfUCjCKbXA9hvK2wy9lnIKtTjrp3
v1x5cIWuttFu6bYtEE1IYCWkjZk487G+KdQVDlIbuEd+jfNjrwZ2CK7OjFO1VDmG
jQXOhrWuBmlmAoQF7hpRMlYu/Nyah1Xj81s++0QuOfNx3k9gZivOE2oMU2BKeyYf
ksJHd0MKqu3phsAYFr5Y2j2e0WPP7h4sNpLjd6yPg8/lJUKaGSaI2MaMWm+vmYDc
FVGqG7vx9U8dNEJHOPb3
=Rn/7
-----END PGP SIGNATURE-----