Please note the word ‘partition’ (not: disk) in the title. Most of my disks have multiple partitions, some Windows, some Linux, and can boot both OS. While I got reliable imaging software for Windows, it lacks btrfs support–what would you recommend?
Basically, for restore I’d like to point the tool to either an existing partition or to unallocated space on the disk and the tool should restore the partition from the image (on USB stick), i.e. create the partition on the fly as needed.
I tried Clonezilla / partclone. Creating the image was easy but restore failed. Clonezilla did not allow me to create a new partition, I could only select an existing disk or partition. Even after creating one from command line it didn’t work. It seems that partclone requires an existing partition with the exact same size of the original backupped partition. I think it should be sufficient that the data fits in the partition, it should be possible to restore to a smaller or bigger partition.
For better understanding what you want. What exactly do you mean with “imaging”?
I ask because I see incompatible texts in your post. On one side you seem to mention “cloning”, which is IMHO a byte for byte copy of the contents of something (could be a partition), but that is then independent of what that contents represents and thus it is irrelevant if it is a file system of whatever type.
OTOH you say you have problems with doing this with a btrfs file system.
Again, this is contrary to the “cloning” idea. When you copy a contents that is a file system to a smaller container, then that file system will be corrupt.
OTOH, when you copy the files from one file system (in a partition or elsewhere) to a new file system (of the same type or not, on a smaller container/partition, the that may fit, but I would not call that “imaging”.
So I assume we must first be clear about the terms used beafore a fruitful discussion can be started.
No. I never mentioned this. What I mentioned is the name of the software tool Clonezilla which also supports what I want to do:
a) Intelligently write a partition to an image file. Intelligently means only real data, not unused sectors
b) Restore a partition from a previously created image file, even on new hardware.
This is similar to snapshots, just on a partition level, not on a file system level. One partition may hold multiple volumes, as is normal for standard openSUSE btrfs layout with multiple subvolumes (which are being created to exclude them from snapshots if my understanding is correct).
So, this (or equivalent tools) writes the content “that matters” from a file system to a file (which you call a"Image file". I assume then that when you want to reverse the action by creating the file system from the image file, you need a partition (or other container) to put it there. Thus I do not understand that you complain that the partition is not created. IMHO the partition should be there. The tool can not read minds and thus decide which partition (or other container that it might find on the system) is to be used to restore the file system upon. It also can not guess which unallocated space on which device can be used to “create the partition on the fly”.
As far as I can see, this image file is created by the tool to it’s own specifications. There is no standard how unallocated space, that is not copied, is administered in the image file so that it can be restored in the correct place. I thus assume that only the same tool that was used for the creation of the image file, can interpret it at recreation of the file system.
Restoring in a smaller container would mean a reorganisation of the file system (moving files, inodes and what all), which is rather different from working on the unallocated gaps.
Restoring in a larger container is possible, but that leaves space at the end of the container, that then could be added to the file system, if the file system type allows enlarging.
I get 381,000,000 hits which proves that this is a common term. Everybody in professional IT knows imaging software.
The features that I am looking for in an imaging software with btrfs support are quite common in Windows imaging software. I am just avoiding the word standard here.
If a sector by sector copy would be fine, there would be no need for such software because every linux system comes with the dd command. However, if you use the dd tool to backup a 100 GB partition of which only 20% is used, you not only generate 80 GB garbage but also prolong the time needed to do the backup (and the restore) operation by 400%. The desire for small backup archives and quick backup/restore is one reason why people use imaging software.
Another reason is that you do not need to install the operating system on the target system first. The typical use case for an image restore is a faulty system disk that has been replaced.
If you haven’t already, check this Reddit thread:
though it does not fit your thoughts on “partition imaging software” - it offers 2-3 solutions:
. https://www.reddit.com/r/btrfs/comments/si0yo8/backup_strategy/
.
(Clonezilla does support BTRFS, so maybe your attempt to restore was hindered by some odd anomaly ).
Thank you. btrbk could be what I am looking for. Will check that out.
As we say in German: „Der Teufel steckt im Detail.“ Clonezilla does support btrfs, but it can only restore to an already existing partition of the exact same size. While partclone (the command line tool being used under the cover) supports a -B option to force restoring to a partition of different size, Clonezilla does not support it.
I admit that I am a bit careful about having the same definition about imaging (of file systems) as you have. And it also may be that you should not expect “Everybody in professional IT” here in a users (try to) help users forum.
But please take into account that you started with
and as partitions used on the system we handle here are just partitions and not Windows or Linux partitions and the idea you had that the tool you use should by clairvoyance know that you want to create a partition somewhere, made me trying to get more insight in what you mean to say and what you want.
And yes, when I want "cloning, I use dd. And when I want backups on the file level (or reorganise file systems), I use rsync , cp or even tar.
So, now understanding what sort of tool you need, sorry, never used some of those. So no advice.
And for the ability of a tool to not only recreate from an image, but also change size of a file system on the flight, better consult the documentation of the tool, or/and ask here in another thread with more info then just “didn’t work”.
My personal opinion (which doesn’t match Martin’s (OP) scheme) - I have a separate partition for /home (XFS) - that gets backed up weekly using rsync.
I could care less about the running system, i.e., the root / partition (as BTRFS)… if it becomes un-usable for corruption reasons, I will simply recreate / reinstall TW. Everyone has different thoughts about recovery
Typical imaging software will let you select the target, which may be a disk, an existing partition, or unallocated space on a disk. And if you select the latter, it will typically ask you if you want to use the whole space and give you a chance to create a new partition of the desired size if not. No magics included here.
Otherwise no worries. Better to figure out the real requirements than coming up with quick recommendations that do not help.
For many years, I am also creating separate partitions to hold my data files and using file synch software. This is perfect for documents. However, the situation is different when it comes to programs, program settings and data for all users (which lies typically not under /home).
That’s an alternative for desktop systems. However, if you made lots of customizations or spent many hours setting up a server to your specific needs, you may want to review this decision. In particular when you’re depending on that machine in your daily life and every hour of unavailability means lots of pain.
It conforms BTW why I stay away from such tools. They contradict the Unix rule: make a tool for one dedicated task and make it to do that task very well. In this case it would mean that the imaging tool should not try to do partitioning, nor file system managment (and that even for several different types of file system ).
But who am I, with my old-fashioned ideas about Unix?
I like the KISS principle as well since I started with Interactive Unix (SVR3.2) back in 1993. But that is mainly valid for the command line. GUI tools either use OS APIs or command line tools under the covers.
Most Linux GUI backup and imaging tools for the Desktop use rsync or other commandline tools for the real task. One of those tools is Timeshift which has been mentioned in the EndevaourOS thread that aggie posted above. Also looks interesting.
Just a quick update. I spent some time on research, what tools are available, supported features etc.
The following is a very short summary. Feel free to ask if you want more details.
Commercial tools do not support BTRFS
Macrium Reflect will create and restore images of BTRFS partitions but this will be 1:1 clones like dd would create them, including the unused sectors. Only benefit is that Macrium will compress the image file.
TeraByte Drive Image: same as Macrium
Acronis Cyber Protect Home Office, previously known as True Image, does only support Windows and Mac OS file systems.
Available freeware tools may support BTRFS such that they can restore a BTRFS partition. BUT all previously existing subvolumes will be lost and re-created as plain directories.
This is in particular true for FSArchiver and tools built upon it (SystemRescue, G4L aka Ghost for Linux, qt-fsarchiver)
All other programs either use partclone (Clonezilla, Rescuezilla, PartedMagic, Foxclone, Redo Rescue, formerly Redo Backup and Recovery) under the hood or partimage (FOG). Not sure about Mondo Rescue but it hasn’t been updated for years and like partimage seems to be abandoned for a long time.
The problems I faced restoring from an image created using partclone have been described above.
So ATM a custom procedure could look like this:
boot from a Rescue media with BTRFS tools on it
create the HDD partition to restore to from scratch in the desired size
re-create the subvolumes from former snapshot backups
boot the hard disk partition from Rescue media’s Grub2
re-install Grub to MBR or EFI partition on the HDD
Does this sound reasonable or am I overlooking something?
At least for me, the disk partitioning information is not that important. And here’s why:
my system disks (and imaging is only relevant for system disks, not for data disks) do all have multiple system partitions, at least one Windows (sometimes two) and at least one openSUSE (either Leap or TumbleWeed).
in the disaster case of a faulty disk - and that’s the only real use case besides hardware upgrades and corrupted file systems - the Windows partitions will be recovered first
so the situtation when it comes to restore a linux partition is always the same: a SSD with unallocated disk space at the end (besides perhaps a Windows restore partition at the very end).
It doesn’ really matter if the unallocated space is 50 GB or 170 GB. We need 2-4 GB for a swap partition and the rest can be allocated as needed. I usually do 50 GB for openSUSE and convert the rest to an ExFAT data partition that can be accessed by both Windows and Linux without requiring any extra drivers and comes with no performance penalty as NTFS-3g does.
Creating the partitions can easily be done using GParted from the rescue media, it’s easy to handle and will update the partition table.
Rsync has the same issue as other fs-independent tools: When it comes to restore, it will recreate subvolumes as plain directories, i.e. you will lose your subvolumes if you didn’t care to create them manually before starting the restore operation.
You may argue that this is fine because you don’t create any subvolumes besides those that the installer creates per default and those will be there.
And you are right, I was also thinking about that and really considering it.
However, there is one scenario where the re-install may not serve you: Your system is not up to date because you are using older hardware that is no longer supported or simply lacking driver support. In that case, the reinstall will bring you exactly into that situation that you wanted to avoid. Unless you were clever enough to consider that back in the days and keep the old installation media around and are aware today that you must not use the latest one.
Besides that, the rsynch approach sounds fine, too.
What about snapper snapshots being re-created as plain directories by rsync? Will these still be bootable? I’m not an expert here but I think that each snapshot is a subvolume.
@duise On my primary desktop (Tumbleweed), I run btrfs and xfs, no snapshots. My MicroOS installs are SelfInstall image with combustion so also rsync the flatpak info.
).