btrfs - Fail to recover the chunk tree

English Install/Boot/Login
#1

Had a single drive used for just data, added another drive but only made it a RAID1 for the metadata. Data was still single. It was like this for less than a day and I don’t recall much that was written to it. I removed the 2nd drive and now cannot access any of the original data.

$ sudo mount -o degraded /dev/sdc1 /media/Data/
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

$ dmesg | tail
[45353.869448] KBD BUG in
…/…/…/…/…/…/…/…/
drivers/2d/lnx/fgl/drm/kernel/gal.c at line:
304!
[45353.901511] KBD BUG in
…/…/…/…/…/…/…/…/drivers/2d/lnx/fgl/drm/kernel/gal.c at line:
304!
[45353.901666] KBD BUG in
…/…/…/…/…/…/…/…/drivers/2d/lnx/fgl/drm/kernel/gal.c at line:
304!
[45354.148488] KBD BUG in
…/…/…/…/…/…/…/…/drivers/2d/lnx/fgl/drm/kernel/gal.c at line:
304!
[45354.148573] KBD BUG in
…/…/…/…/…/…/…/…/drivers/2d/lnx/fgl/drm/kernel/gal.c at line:
304!
[46241.155350] btrfs: device fsid bd78815a-802b-43e2-8387-fc6ab4237d67
devid 1 transid 60944 /dev/sdc1
[46241.155923] btrfs: allowing degraded mounts
[46241.155927] btrfs: disk space caching is enabled
[46241.159436] btrfs: failed to read chunk root on sdc1
[46241.177815] btrfs: open_ctree failed

$ btrfs-show-super /dev/sdc1
superblock: bytenr=65536, device=/dev/sdc1

csum 0x93bcb1b5 [match]
bytenr 65536
flags 0x1
magic _BHRfS_M [match]
fsid bd78815a-802b-43e2-8387-fc6ab4237d67
label
generation 60944
root 909586694144
sys_array_size 97
chunk_root_generation 60938
root_level 1
chunk_root 911673917440
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 1115871535104
bytes_used 321833435136
sectorsize 4096
nodesize 4096
leafsize 4096
stripesize 4096
root_dir 6
num_devices 2
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x9
csum_type 0
csum_size 4
cache_generation 60944
uuid_tree_generation 60944
dev_item.uuid d82b2027-17b6-4513-a86d-9227a42d7ed1
dev_item.fsid bd78815a-802b-43e2-8387-fc6ab4237d67 [match]
dev_item.type 0
dev_item.total_bytes 615763673088
dev_item.bytes_used 324270030848
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

$ sudo btrfs device add -f /dev/sdh1 /dev/sdc1
ERROR: error adding the device ‘/dev/sdh1’ - Inappropriate ioctl for device

$ sudo btrfs device delete missing /dev/sdc1
ERROR: error removing the device ‘missing’ - Inappropriate ioctl for device

$ sudo mount -o degraded,defaults,compress=lzo /dev/sdc1 /media/Data/
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

$ dmesg | tail
[106991.655384] btrfs: device fsid
bd78815a-802b-43e2-8387-fc6ab4237d67 devid 1 transid 60944 /dev/sdc1
[106991.665066] btrfs: device fsid
bd78815a-802b-43e2-8387-fc6ab4237d67 devid 1 transid 60944 /dev/sdc1
[107019.954397] btrfs: device fsid
bd78815a-802b-43e2-8387-fc6ab4237d67 devid 1 transid 60944 /dev/sdc1
[107019.962009] btrfs: device fsid
bd78815a-802b-43e2-8387-fc6ab4237d67 devid 1 transid 60944 /dev/sdc1
[107070.124927] btrfs: device fsid
bd78815a-802b-43e2-8387-fc6ab4237d67 devid 1 transid 60944 /dev/sdc1
[107070.126475] btrfs: allowing degraded mounts
[107070.126479] btrfs: use lzo compression
[107070.126480] btrfs: disk space caching is enabled
[107070.127254] btrfs: failed to read chunk root on sdc1
[107070.142983] btrfs: open_ctree failed

$ sudo btrfs rescue super-recover -v /dev/sdc1
All Devices:
Device: id = 1, name = /dev/sdc1

Before Recovering:
[All good supers]:
device name = /dev/sdc1
superblock bytenr = 65536

            device name = /dev/sdc1
            superblock bytenr = 67108864

            device name = /dev/sdc1
            superblock bytenr = 274877906944

    [All bad supers]:

All supers are valid, no need to recover

$ sudo btrfs check /dev/sdc1
warning, device 2 is missing
Check tree block failed, want=911673917440, have=0
read block failed check_tree_block
Couldn’t read chunk root
Couldn’t open file system

$ sudo btrfs check --repair /dev/sdc1
enabling repair mode
warning, device 2 is missing
Check tree block failed, want=911673917440, have=0
read block failed check_tree_block
Couldn’t read chunk root
Couldn’t open file system

$ btrfs rescue chunk-recover -v /dev/sdc1
<<snipped>>
Chunk: start = 860100755456, len = 1073741824, type = 1, num_stripes = 1
Stripes list:
0] Stripe: devid = 1, offset = 26877100032
No block group.
No device extent.
Chunk: start = 861174497280, len = 1073741824, type = 1, num_stripes = 1
Stripes list:
0] Stripe: devid = 1, offset = 27950841856
No block group.
No device extent.

Total Chunks: 333
Heathy: 305
Bad: 28

Orphan Block Groups:
Block Group: start = 872985657344, len = 1073741824, flag = 4
Block Group: start = 911673917440, len = 33554432, flag = 2
Block Group: start = 911707471872, len = 1073741824, flag = 4

Orphan Device Extents:
Device extent: devid = 2, start = 2182086656, len = 33554432, chunk
offset = 911673917440
Device extent: devid = 2, start = 2215641088, len = 1073741824,
chunk offset = 911707471872
Fail to recover the chunk tree.
<</snipped>>

Right now I’ve moved from 13.1 to Tumbleweed to stay more current with the kernel and btrfs-progs, but it hasn’t made a difference. Any idea’s how I can get at those healthy chunks of data?

#2

On 2014-10-17 12:56, clickwir wrote:
>
> Had a single drive used for just data, added another drive but only made
> it a RAID1 for the metadata.

I don’t understand this. Could you expand, please?

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#3

Had a single drive using btrfs, was doing fine for months. Added another drive to the system, thought I’d see what doing RAID1 for just the metadata would look like. Did it and thought “gee, it’s RAID1. It won’t miss the drive now that I need it for something else”. So I took it out. When I booted back up, I could no longer access the original drive.

#4

On 2014-10-18 01:16, clickwir wrote:
>
> robin_listas;2669925 Wrote:
>> On 2014-10-17 12:56, clickwir wrote:
>> I don’t understand this. Could you expand, please?
>>
>
> Had a single drive using btrfs, was doing fine for months. Added another
> drive to the system, thought I’d see what doing RAID1 for just the
> metadata would look like.

I don’t understand how you could separate data and metadata on different
disks. I don’t know of a RAID implementation that does this, so you had
to code this yourself, and that is way above my pay level and of many of
the people here. It also means that your coding skills are very high, so
I fail to understand how we can help you.

Or, you are explaining it completely wrong.

> Did it and thought “gee, it’s RAID1. It won’t
> miss the drive now that I need it for something else”. So I took it out.
> When I booted back up, I could no longer access the original drive.

And now, this does not match with you being able to separate metadata on
a different disk… Obviously, if you did that, you can not break the
raid again. I would be able to even guess how to do it.

So… care to try again explaining what exactly you did?

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#5

I think my original wording of “data” is what threw this off. I meant data as in non-system or general storage. Not data as in btrfs data vs metadata. I haven’t coded anything, never have.

Hopefully that clears it up, just in case, I’ll try to recreate what I did via commands. Maybe that will tell the story better.


sudo mkfs.btrfs /dev/sdc1
sudo mkdir /Data
sudo mount -o compress=lzo /dev/sdc1 /Data

At this point I use the /Data location for general storage of files for a few months.
Then I add another drive to the system, intending it to become /Data2 but never do that.
Instead I do this:


sudo btrfs device add /dev/sde /Data
sudo btrfs balance start -mconvert=raid1 /Data

Once it completed, I made sure I could see everything ok. I could still access /Data and btrfs command properly showed that it was not using raid1 for data, only metadata. I may have added a few files to it, probably just to test and see that it worked.
Less than a day went by and I shutdown the system. I needed to use that new drive somewhere else. Since I was only using RAID1 for the metadata I figured it would be even less of a problem than if I was using RAID1 for data and metadata. So I pulled the drive out and booted the system. At this point I could no longer access the original drive.

I ran commands to try and fix and repair things but they always seemed to point back to a chunk problem. Running a chunk-recover failed, so I seem to be at a brick wall. chunk-recover said there’s lots of healthy chunks, but yet it still failed.

#6

put the drive back in recover the data and then you can recreate it. If that can’t happen then I’d say the data is effectively gone. As I understand it in BTRFS the meta data is the glue that hold the FS together. (I could be wrong) Remove it and now you just have data on random sectors with nothing to hold it together. As to RAID as I understand it that stuff was still being worked on in 13.1 and may or may not work as expected.

And no I don’t have a clue of why you did what you did and what you really expected to happen. Seems like you were experimenting with real data. It is a good idea to test out cutting edge stuff before you take it live.

#7

On 2014-10-18 14:46, clickwir wrote:
>
> I think my original wording of “data” is what threw this off. I meant
> data as in non-system or general storage. Not data as in btrfs data vs
> metadata. I haven’t coded anything, never have.
>
> Hopefully that clears it up, just in case, I’ll try to recreate what I
> did via commands. Maybe that will tell the story better.
>
>
> Code:
> --------------------
>
> sudo mkfs.btrfs /dev/sdc1
> sudo mkdir /Data
> sudo mount -o compress=lzo /dev/sdc1 /Data
>
> --------------------
>
> At this point I use the /Data location for general storage of files for
> a few months.

I see.

We normally speak of system area or system files, and data area or files.

Metadata is a very specific term, and it means the internal data
structures a filesystem keeps in order to know what sectors are occupied
by each file, what sectors are free, the name of each file and
directory, its permissions, ownership, timestamps and other attributes.
On ext2/3/4 metadata is a fixed percent at format time, on xfs it
dynamic. btrfs dynamic, I think.

It is not something a user (even root) can access. Only the kernel can
(inexact, but suffices). You can not create metadata, store metadata
here or there. You don’t control that.

> Then I add another drive to the system, intending it to become /Data2
> but never do that.
> Instead I do this:
>
> Code:
> --------------------
>
> sudo btrfs device add /dev/sde /Data
> sudo btrfs balance start -mconvert=raid1 /Data
>
> --------------------

So you are using btrfs own type of raid, not the normal or traditional
type of raid.

> Once it completed, I made sure I could see everything ok. I could still
> access /Data and btrfs command properly showed that it was not using
> raid1 for data, only metadata. I may have added a few files to it,
> probably just to test and see that it worked.
> Less than a day went by and I shutdown the system. I needed to use that
> new drive somewhere else. Since I was only using RAID1 for the metadata
> I figured it would be even less of a problem than if I was using RAID1
> for data and metadata. So I pulled the drive out and booted the system.
> At this point I could no longer access the original drive.

And here you again go off talking about metadata, and again I understand
nothing.

Anyway, being a problem related to btrfs own type of raid, I’m afraid I
can not help much. Perhaps somebody else can take it from here.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#8

The btrfs documentation states that you can choose different RAID levels for data and metadata. Since RAID1 is normally a mirror, I assumed that mirroring the metadata and then removing it would not affect the original drive.

So I kept the data as a single drive but the metadata as RAID1.

Most of the data was part of a backup, I’m not really worried about what may not be recoverable. However, I’ve not seen a good explanation as to why this didn’t work as intended. Dissapointing since OpenSuse is going to be using this as default.

#9

I hate to say it but not all the bells and whistles are working in the BTRFS world. But then BTRFS has lots and lots of bells and whistles LOL It does appear to be otherwise stable if you don’t push too hard at the envelope. You probably should do a buigzilla

My concern is that people won’t know that snapshot is turned on and is rather aggressive and will short the extra space need for the snaps. I think we will have lots of relative newbees showing up here with out of space problems using old recommendations of partition size.

#10

The Btrfs repair / recovery tools aren’t up to it yet, and they need reorganising. It’s not the first time it’s been mentioned : http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg37924.html

It’s not far off prime time but it’s not quite there yet. If any distro starts shipping it as default file system I’d immediately swap it out for something more boring and stable.

#11

On 2014-10-19 04:56, clickwir wrote:
>
> The btrfs documentation states that you can choose different RAID levels
> for data and metadata. Since RAID1 is normally a mirror, I assumed that
> mirroring the metadata and then removing it would not affect the
> original drive.

Removing the metadata of any filesystem type DESTROYS it.

No way to recover it, except by analyzing the entire partition, reading
all sectors, seeking for sectors that appear to start with a known
pattern, like “this seems to be a jog header”, then seeking which sector
is number two, etc. And no names, no directories, no attributes.

> So I kept the data as a single drive but the metadata as RAID1.

So, you put that metadata in raid1, then remove one side?

Well, finally I understand what you did!

I hope. Because that setup is intended, I understand, for three devices:
metadata 1, metadata 2, and data (the metadata on small but very fast
devices). If the second device for metadata was missing at the start
(you talk of two disks), the instant you remove the first device,
everything is destroyed.

Maybe I’m wrong.

Well, you will have to ask the btrfs developers themselves, not on a
users forum like this. Not many skilled users will try such a “feature”.
You are walking on unmapped land.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)