I’m writing my first shell script ever. It’s supposed take my Work folder at /home/user/Documents/Work, copy it over to /home/user/Documents/Backup, then tar it into a tar.gz file.
My script does make a compressed tar file for me, but when I extract it, it has a folder inside a folder inside a folder. So I have to navigate through the folders home–>user–>Documents–>Backup–>Work just to see my backed-up files. Is there a way I can just not preserve the whole directory tree and just have the files all in one spot once I extract the tar file?
I’m going to put this script in /etc/cron.hourly so that it makes hourly backups of my Work folder. It won’t fill up my hard drive because it removes the old files and overwrites the backup tar file.
I don’t know if that would work or not. I’m still scratching the surface of shell scripting. I used your first idea, ken_yap, and now my script looks like this:
#!/bin/bash
current_date="`date '+%a'`"
cp -r /home/user/Documents /home/user/Backup
OF=/home/user/Backup/$current_date.backup.tar
cd /home/user/Backup
tar -cf $OF Documents/
rm -r /home/user/Backup/Documents
It backs up the entire Documents folder to my Backup folder. It adds the day of the week to the beginning of the backup.tar file, so that I end up with backup files such as Mon.backup.tar. I put this script in /etc/cron.daily. That way, it should keep 7 days’ worth of backups. Every day, it overwrites the last week’s backup, so it won’t fill up my hard drive, and I’ll always have a week’s worth of backups.
I encountered a problem, though. I can’t untar the file. It’s 15 GB in size, and when I try to untar it, it says “an error occurred while trying to open the archive”
Well, I hoped that by looking in what I provided you you what ask yourselg why you make a tar from a copy of thedata. Why not directly from the data?
The intermediate Backup file is useless imho.
And you better post exactly (by copy/past) what you do so we can see the tar statement and the output of the error message as it is. You have less to explain and we can decide for ourselves what is important in what you do.
if you are set on writing you own script instead of a free linux tool, which is very fun to do and makes one learn a lot you could choose to split your files to 2gb portions like so:
tar -cvzf - /path/to/backup | split -b2000m - backup.gz
of course you would have to change the stuff to match your setup and “cd” into /data/to/ to forgo the absolute paths.
Where did you find that? I’ve made bigger TAR files than just 2 GB on
several occasions and never had a problem with it. The biggest
limitation is usually the filesystem (fat32 if it’s anything small like
this).
Good luck.
BNG22908 wrote:
| Oh, ok. Apparently, tar only works with files that are 2 GB or smaller.
| I will have to find some other way to back up an entire 15 GB
| directory.
|
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
I read somewhere that if you have a certain version of tar (GNU tar?) that you can tar up to 4 GB files. I’m not sure if that’s accurate info, though. It’s still better to learn how to break it up into smaller pieces, because you might need to use that some day.
This article tells you how to split and compress on-the-fly, so that when it compresses your file or directory, it doesn’t require more hard drive space than the actual file or directory takes up.
Splitting big files into pieces is a common task. Another common task is to create a tar archive, and split it into smaller chunks that can be burned onto CD/DVD. The straightforward approach is to create the archive and then use ’split.’ To do this, you will need more free space on your disk. In fact, you’ll need space twice the size of the created archive. To avoid this limitation, split the archive as it is being created.
To create a tar archive that splits itself on the fly use the following set of commands:
First create the archive:
tar -czf /dev/stdout $(DIRECTORY_OR_FILE_TO_COMPRESS) | split -d -b $(CHUNK_SIZE_IN_BYTES) - $(FILE_NAME_PREFIX)
To extract the contents:
cat $(FILE_NAME_PREFIX)* >> /dev/stdout | tar -xzf /dev/stdin
The above shown set of commands works on the fly. You don’t need additional free space for temporary files.
GNU tar is almost definitely what you are using unless something radical
has happened w/opensuse 11 (doubtful). The wikipedia entry for Tar says
that 8GB is the size limit, which seems closer to reality. ‘split’ is a
great command but may be overkill unless you’re dealing with a lot of
files. Anyway, just wanted to be sure I wasn’t losing my mind about the
limitation there.
Good luck.
BNG22908 wrote:
| I read somewhere that if you have a certain version of tar (GNU tar?)
| that you can tar up to 4 GB files. I’m not sure if that’s accurate
| info, though. It’s still better to learn how to break it up into
| smaller pieces, because you might need to use that some day.
|
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
This seems a bit awkward… maybe it’s the best way though. Anyway to
reassemble you can use zcat. Assuming all the files in your split
archive are able to be sorted alphabetically…
cd /path/to/split/files
ls -1 | xargs zcat | tar -xvf
Give it a shot and remember that tar will extract wherever you run it
from so perhaps change to a temporary directory first… or tell tar to
put them somewhere else via the -C switch (I think that’s how it works
anyway).
Good luck.
BNG22908 wrote:
| I changed my script to include your recommendation:
|
| #!/bin/bash
| current_date="date '+%a'"
| cd /home/user/Backup
| tar -cvzf - /home/user/Documents/ | split -b2000m - backup.gz
|
|
| Now all I need to do is figure out how to extract all the 2GB files it
| created.
|
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
I must be using a different version of tar or something. I’m using Open Suse 10.3 also, whereas you’re using 11.0. I also haven’t run an update in a while, and I have an early 2.6 kernel.
BNG22908 wrote:
| I must be using a different version of tar or something. I’m using Open
| Suse 10.3 also, whereas you’re using 11.0. I also haven’t run an update
| in a while, and I have an early 2.6 kernel.
|
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
#!/bin/bash
current_date="`date '+%a'`"
cd /home/user/Backup
if -e $current_date ]
then
rm -r $current_date
fi
mkdir /home/user/Backup/$current_date
cd /home/user/Backup/$current_date
tar -czf /dev/stdout /home/user/Documents | split -d -b 2000m - $current_date.backup
It navigates to your Backup directory, then creates a directory with that day of the week as its name (Mon, Tues, Wed, etc). If it exists already (from 7 days ago) it deletes that directory and makes a new one. It then does the on-the-fly backup in that day’s folder. If you put this shell script in /etc/cron.daily , and run chmod 755 backupscript, it will be run daily by cron. You will then have daily backups of your Documents folder stored in your Backup folder. You can add to the shell script to make it write the backup to a CD, DVD, or flash drive, or even email or scp it somewhere too.
Name : tar Relocations: (not relocatable)
Version : 1.17 Vendor: SUSE LINUX Products GmbH, Nuernberg, Germany
Release : 21 Build Date: Fri 21 Sep 2007 03:26:44 PM EDT
Install Date: Sun 24 Feb 2008 09:15:49 PM EST Build Host: cara.suse.de
Group : System/Base Source RPM: tar-1.17-21.src.rpm
Size : 2042121 License: GPL v2 or later
Signature : DSA/SHA1, Fri 21 Sep 2007 03:31:16 PM EDT, Key ID a84edae89c800aca
Packager : Submitting Bug Reports - openSUSE
URL : Tar - GNU Project - Free Software Foundation (FSF)
Summary : GNU implementation of tar ((t)ape (ar)chiver)
Description :
This package normally also includes the program “rmt”, which provides
remote tape drive control. Since there are compatible versions of ‘rmt’
in either the ‘star’ package or the ‘dump’ package, we didn’t put ‘rmt’
into this package. If you are planning to use the remote tape features
provided by tar you have to also install the ‘dump’ or the ‘star’
package.
Why not rethink the whole thing? If you use rsync and some smart scripting to use links for unchanged files, you can avoid any size limitations and get space savings too. Something like this:
Say today is tuesday
make links in tuesday to all files in monday with the same paths
that is to say tuesday is now a forest of hard links
run rsync to update tuesday
tuesday will now have some new files with one link and unchanged ones with > 1 link
I believe there are some scripts to do this sort of thing. Alas the original idea wasn’t mine.