Disaster! Entire /var directory disappeared!

The entire /var directory of my work system has disappeared.

I don’t know how.

My backup was done after the disaster, so no /var in there.

I was working on a test partition, where I was trying 12.1 before upgrading my main partition from
11.4 to 12.1. As part of the procedure, I was doing a backup with dd for imaging partitions, and
rsync for files. I think I might have done an rsync line that erased things, but I was trying it on
an empty directory… the line contained a --del that I later removed. It might be that… :-?


> bombadillo:~ # grep -e "--del" .bash_history
> time nice rsync --archive --acls --xattrs --hard-links --del --stats --human-readable /other/main/usr copia_usr_main
> time nice rsync --archive --acls --xattrs --hard-links --del --stats --human-readable /other/main/usr copia_usr_main
> time nice rsync --archive --acls --xattrs --hard-links --del --stats --human-readable  copia_usr_main/usr/local/ /other/main/
> time nice rsync --archive --acls --xattrs --hard-links --del --stats --human-readable /other/main/opt/ copia_opt_main

I think it has been the third line. I stopped it because it was supposed to be a quick test of a
restore and it was taking a long time… and it deleted “/other/main/var” and who knows what more…
so damn fast!

I should have mounted the main system RO.

I may have an older backup. …] Yep, from June 2011… Better than nothing, I guess. No… it
would place an antique rpm database, which is what matters for the upgrade procedure. I will have to
recover the entire 2011 backup for consistency, then upgrade from there.

As the saying goes, the question is not whether disaster will strike, but when.
And as the Spanish saying says, the ironmonger is using a wooden spoon.

:frowning:


Cheers / Saludos,

Carlos E. R.
(from 12.1 “Asparagus” GM (bombadillo))

On 2012-08-11 04:38, Carlos E. R. wrote:
>
> The entire /var directory of my work system has disappeared.

Well, I managed to restore, with a lot of work, the last good backup was too old.

I found a good tool for undeleting files in ext3/4, but it was not good enough.

This post in the factory mail list (from Greg Freemyer) was useful:


The developer of ext4magic (a ext4 deleted file recovery tool) has
created a simple text based menu'ing front-end.  He's used SuseStudio
to create a simple rescue CD.

You can see some screenshots as well as a link to the downloadable image at:
http://openfacts2.berlios.de/wikien/index.php/BerliosProject:Ext4magic-e4m-tui

The author  made a post about it here:
http://developer.berlios.de/forum/forum.php?forum_id=35219

He's looking for feedback if anyone has time to try it out.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On Sun, 12 Aug 2012 22:23:06 GMT, “Carlos E. R.”
<robin_listas@no-mx.forums.opensuse.org> wrote:

>On 2012-08-11 04:38, Carlos E. R. wrote:
>>
>> The entire /var directory of my work system has disappeared.
>
>Well, I managed to restore, with a lot of work, the last good backup wastoo old.
>
>
>I found a good tool for undeleting files in ext3/4, but it was not good enough.
>
>This post in the factory mail list (from Greg Freemyer) was useful:
>
>


>The developer of ext4magic (a ext4 deleted file recovery tool) has
>created a simple text based menu'ing front-end.  He's used SuseStudio
>to create a simple rescue CD.
>
>You can see some screenshots as well as a link to the downloadable imageat:
>http://openfacts2.berlios.de/wikien/index.php/BerliosProject:Ext4magic-e4m-tui
>
>The author  made a post about it here:
>http://developer.berlios.de/forum/forum.php?forum_id=35219
>
>He's looking for feedback if anyone has time to try it out.
>

Please consider that the entire /var tree is very temporal and normally
transient in existence. And yes it gets abused a LOT. Salt shaker
optional.

?-)

On 2012-08-13 08:53, josephkk wrote:

> Please consider that the entire /var tree is very temporal and normally
> transient in existence. And yes it gets abused a LOT. Salt shaker
> optional.
>
> ?-)

It holds several crucial databases, like for example, the rpm database. Without that YaST
doesn’t know what is installed. I also lost my mysql database, the hylafax database… I could
have lost the entire nntp cache, except that it resides on a different partition and was safe.

But also several directories in /etc and /root were lost. Files everywhere were lost, but only
/var was evident.

System would not boot.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On Mon, 13 Aug 2012 16:48:06 GMT, “Carlos E. R.”
<robin_listas@no-mx.forums.opensuse.org> wrote:

>On 2012-08-13 08:53, josephkk wrote:
>
>> Please consider that the entire /var tree is very temporal and normally
>> transient in existence. And yes it gets abused a LOT. Salt shaker
>> optional.
>>
>> ?-)
>
>It holds several crucial databases, like for example, the rpm database. Without that YaST
>doesn’t know what is installed. I also lost my mysql database, the hylafax database… I could
>have lost the entire nntp cache, except that it resides on a different partition and was safe.
>
>But also several directories in /etc and /root were lost. Files everywhere were lost, but only
>/var was evident.
>
>System would not boot.

Big time ouch. Was fsck run at that point?

??-)

On 2012-08-14 02:54, josephkk wrote:
> On Mon, 13 Aug 2012 16:48:06 GMT, “Carlos E. R.” <> wrote:

>> It holds several crucial databases, like for example, the rpm database. Without that YaST
>> doesn’t know what is installed. I also lost my mysql database, the hylafax database… I could
>> have lost the entire nntp cache, except that it resides on a different partition and was safe.
>>
>> But also several directories in /etc and /root were lost. Files everywhere were lost, but only
>> /var was evident.
>>
>> System would not boot.
>
> Big time ouch. Was fsck run at that point?
>
> ??-)

Let me see…

I was working on 12.1 on an auxiliary partition. With that one running I mounted the main
partition(s), which is 11.4. I was attempting to do a backup, and when testing recovery I
destroyed part of /root (I’m keeping the description simple). /usr and /home and /boot were
safe because they were not mounted. Without knowing what I had done, I rebooted to the main
system, and it failed. I’m not sure if an fsck did run on it or not, I don’t remember now. I
think not.

The destroying part is because I attempted an “rsync … -del backup_usr /main_root”.
Everything that was not in the backup of /usr was going to be deleted in the real root
partition - what I was attempting was to check the command to create a copy of /usr. Be careful
with “–del” in rsync.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)