SOLVED: wget is hanging on transferring images

[swerdna]

Hi
AB has suggestted that I experiment with wget to back up websites remotely. The suggestion is here: SOLVED: How do I connect ftp in one command - Page 2 - openSUSE Forums

I have a test home website with a document root at /home/webname/public_html.
The test site is also an ftp server and is accessible via vsftpd.
I'm using wget to backup the files with a command like this from a remote client:

wget -r --ftp-user=username --ftp-password=password ftp://testdomain.blah.com

That nicely downloads the entire contents of the document root and subdirectories recursively EXCEPT it always hangs mid way through.
The process seems to just stop. Each time I try it the process stops on the same file. If I delete the file from the server (just to see what happens) and repeat the "wget" command from the client, the process proceeds further than the last time and then hangs on a different file.

Here's an example of the dialogue when it stops:

100%[===============================================================================================>] 16,912      --.-K/s   in 0.007s

2009-06-25 14:19:20 (2.25 MB/s) - `domain.name.com/webname/public_html/dynamic/snapshot3.png' saved [16912]

--2009-06-25 14:19:20--  ftp://domain.name.com/webname/public_html/dynamic/snapshot4.png
           => `domain.name.com/webname/public_html/dynamic/snapshot4.png'
==> CWD not required.
==> PASV ...

Can you advise what I might be doing wrong?


Thanks
Swerdna

[ken_yap]

Does the ftp server have some kind of rate-limiting?

[swerdna]

Originally Posted by ken_yap

Does the ftp server have some kind of rate-limiting?

I don't know how to answer than, not knowing much about ftp / vsftpd. I looked in vsftpd.conf and this is commented out: #anon_max_rate=7200. I don't know if that implies some other default value is active. The actual contents are:

dirmessage_enable=YES
nopriv_user=ftpsecure
anonymous_enable=NO
anon_world_readable_only=YES
syslog_enable=NO
connect_from_port_20=YES
pam_service_name=vsftpd
listen=YES
ssl_enable=NO
pasv_min_port=30000
pasv_max_port=30100
anon_mkdir_write_enable=NO
anon_root=/srv/ftp
anon_upload_enable=NO
chroot_local_user=YES
ftpd_banner=Welcome message
idle_session_timeout=900
local_enable=YES
log_ftp_protocol=NO
max_clients=10
max_per_ip=10
pasv_enable=YES
ssl_sslv2=NO
ssl_sslv3=NO
ssl_tlsv1=YES
write_enable=YES
local_root=/home/
local_umask=022

[ken_yap]

Also if you are running vsftpd under xinetd, xinetd will also have rate-limiting to protect the system from too many ftp requests in a given period. If so, you might want to run vsftpd standalone. Have a look in /var/log to see if there are any clues in any log file.

[swerdna]

Originally Posted by ken_yap

Also if you are running vsftpd under xinetd, xinetd will also have rate-limiting to protect the system from too many ftp requests in a given period. If so, you might want to run vsftpd standalone. Have a look in /var/log to see if there are any clues in any log file.

No it's a standalone started by /etc/init.d/vsftpd at boot time.

[ken_yap]

You may have to do a packet trace on both ends with tcpdump or ethereal to see which side was left waiting in the conversation.

Is there some kind of NAT gateway with port forwarding in between?

[swerdna]

Originally Posted by ken_yap

Have a look in /var/log to see if there are any clues in any log file.

Log file is missing. I looked in vsftpd.conf. A log file is designated at /var/log/xferlog but it's not there -- strange.

[ab@novell.com]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I add -T 3 to all of my wget commands these days in case the server is
stupid. See the man page for info but basically after three seconds of
stupidity wget tries again. I hit this a lot because of bad
configurations (now fixed I believe) with some NAM/iChain implementations
but that's still useful for timeouts that can be pushed past with a retry.
Give it a shot.

Good luck.





swerdna wrote:
> Hi
> AB has suggestted that I experiment with wget to back up websites
> remotely. The suggestion is here: 'SOLVED: How do I connect ftp in one
> command - Page 2 - openSUSE Forums' (http://tinyurl.com/lt38lk)
>
> I have a test home website with a document root at
> /home/webname/public_html.
> The test site is also an ftp server and is accessible via vsftpd.
> I'm using wget to backup the files with a command like this from a
> remote client:
> Code:
> --------------------
> wget -r --ftp-user=username --ftp-password=password ftp://testdomain.blah.com
> --------------------
>
> That nicely downloads the entire contents of the document root and
> subdirectories recursively EXCEPT it always hangs mid way through.
> The process seems to just stop. Each time I try it the process stops on
> the same file. If I delete the file from the server (just to see what
> happens) and repeat the "wget" command from the client, the process
> proceeds further than the last time and then hangs on a different file.
>
> Here's an example of the dialogue when it stops:
> 100%[================================================== =============================================>]
> 16,912 --.-K/s in 0.007s
>
> 2009-06-25 14:19:20 (2.25 MB/s) -
> `domain.name.com/webname/public_html/dynamic/snapshot3.png' saved
> [16912]
>
> --2009-06-25 14:19:20--
> ftp://domain.name.com/webname/public.../snapshot4.png
> =>
> `domain.name.com/webname/public_html/dynamic/snapshot4.png'
> ==> CWD not required.
> ==> PASV ...
>
> Can you advise what I might be doing wrong?
>
>
> Thanks
> Swerdna
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIcBAEBAgAGBQJKQ5c8AAoJEF+XTK08PnB5H14QAI8/SFE29XDoX7g4JFcasEKT
bQhdMmijfV2z4hk/xtW7NLizuTqNGWdCwy6NCn7vg1Ti1zeGk0OkgZT4/ANAEl5t
Toilw+yYF/QBfQTcTggZJ3xX6mTlNjRAnzKkshHDYO96K/YI4qFpHG8dbMw3bRUn
hey5ClIc/PWWaYcFF//dn6HKcEqsh6YQEm2SH2WSFDWvQNzs3bhmHV0eIOJ2+nOz
7txc4H40MPGnve+ROq20AEuDSkqGhq2ZFxESpYWgdv3q2k5B1a jxe3UlPSQl2CGa
36cRXyk/1810v/IV058P8awfsW+bNLpIQkcdQh6jS7AQ2Y2eUXGeU86l9BAnRkML
XiGu9sTzuChRjvDfgsy2O8P9MqcTUYc7NAhjaDSJiO7wSzhI4G Ntxm+ZXosa0dEr
b/0GzpIoJeHtT7AFq+vmLmUP1FBjnM51hZNUB5nVil+gGPJIrKSV g0iJH6M7o4Wr
DFmvRq+ln7Jlm9YlYWdprXw0r+VcP7npcQGQxoMZji8wgDzJNn aZlWrQzDCTYQKD
dwa7GznHn8eKp1OkWVkN7lon8vKO9+m6UnnJfkVdapT8H7bItr Ibz9NqWhUNYbDT
skU9yavWidHM56NmZzXU7hv2w+6Gg8ISd/+VjUnTdeNlEqAG3MkqfB5D1i1hMM/e
YDovJLaPWukg42MGVc/W
=UipJ
-----END PGP SIGNATURE-----

[ken_yap]

Originally Posted by swerdna

Log file is missing. I looked in vsftpd.conf. A log file is designated at /var/log/xferlog but it's not there -- strange.

You may have to create it manually. It might be a server that doesn't automatically create log files. Make sure it is writable by the account running vsftpd, but this would probably be root anyway.

[swerdna]

Originally Posted by ab@novell.com

I add -T 3 to all of my wget commands these days in case the server is
stupid. See the man page for info but basically after three seconds of
stupidity wget tries again. I hit this a lot because of bad
configurations (now fixed I believe) with some NAM/iChain implementations
but that's still useful for timeouts that can be pushed past with a retry.
Give it a shot.

Good luck.

That fixed it -- and the prize goes to ab@novell.com
Many thanks