Not sure if I already asked this exact question in the past. I wanna clone a folder from the internet (http path) to a local folder on my machine. This already works with wget. However, I want it to automatically delete files in my local path that no longer exist on the remote location when I update. Also to skip existing files.
I heard wget doesn’t have a cleanup function, so I was wondering if anything else can be done to clone and update properly. I tried rsync, and can already do this using rsync servers (with the --delete flag if I remember right). But I need this for pure http or ftp paths, and I’m not sure if rsync can handle those. When I tried it I ran into having to specify a module, since rsync servers have a tag you write with : in the URL whereas http paths just are.
Overall, I simply need to sync a remote location (NOT a rsync server) to a local folder. eg: http://www.website.com/the_folder to /home/the_folder. But that includes deleting files / folders from my drive that don’t exist on the web path any more, to avoid wasting drive space and have downdated files piling up. Also to not re-download files which already exist, so I don’t throw bandwidth away. Can anyone please explain how this is done, be it through wget, rsync or anything else?