Unable to remove snapper snapshots - Config is in use

Hi,

I noticed that my BTRFS filesystem was getting full ( 220 GB SSD ) , which got me worried

# btrfs filesystem df /
Data, single: total=208.41GiB, used=207.81GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=9.01GiB, used=7.67GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

I noticed some old snapshots lying around for the root config from when I was hit with a ext4-btrfs conversion bug and I had a half-working btrfs partition. I proceeded to delete them, but snapper refused to comply:

# snapper -c root delete 274
Config is in use.

Cleanup also failed

# snapper -c root cleanup number
Config is in use.

Now I’m stuck with a lot of old snapshots and only a few GBs of free space. How can I delete these old snapshots?

Please post “snapper list” output.

# snapper list
Type   | #    | Pre # | Date                     | User | Cleanup  | Description                                     | Userdata     
-------+------+-------+--------------------------+------+----------+-------------------------------------------------+--------------
single | 0    |       |                          | root |          | current                                         |              
single | 1    |       | Thu Jul 30 17:03:50 2015 | root |          |                                                 |              
single | 4    |       | Mon Jun 22 01:15:01 2015 | root | timeline | timeline                                        |              
single | 274  |       | Tue Jul 21 00:15:01 2015 | root | timeline | timeline                                        |              
single | 302  |       | Thu Jul 30 16:21:47 2015 | root |          |                                                 |              
single | 303  |       | Thu Jul 30 16:21:49 2015 | root |          |                                                 |              
single | 304  |       | Thu Jul 30 17:04:51 2015 | root |          |                                                 |              
single | 305  |       | Thu Jul 30 17:04:54 2015 | root |          |                                                 |              
single | 306  |       | Thu Jul 30 17:47:35 2015 | root |          | debug: after zypper ref                         |              
single | 308  |       | Thu Jul 30 18:07:34 2015 | root |          |                                                 |              
single | 309  |       | Thu Jul 30 18:07:37 2015 | root |          |                                                 |              
single | 310  |       | Thu Jul 30 22:47:13 2015 | root |          | Mount /.snapshots                               |              
single | 311  |       | Thu Jul 30 23:58:03 2015 | root |          | Before "fatal" zypper up                        |              
single | 312  |       | Thu Jul 30 23:58:16 2015 | root |          | Before "fatal" zypper up with flash-player lock |              
single | 316  |       | Fri Jul 31 00:08:45 2015 | root |          |                                                 |              
single | 317  |       | Fri Jul 31 00:08:46 2015 | root |          |                                                 |              
single | 319  |       | Fri Jul 31 00:16:19 2015 | root |          | zypper up partially worked                      |              
single | 323  |       | Fri Jul 31 00:32:05 2015 | root |          |                                                 |              
single | 324  |       | Fri Jul 31 00:32:06 2015 | root |          |                                                 |              
single | 358  |       | Fri Jul 31 00:52:39 2015 | root |          |                                                 |              
single | 359  |       | Fri Jul 31 00:52:41 2015 | root |          |                                                 |              
single | 399  |       | Sat Aug  1 00:15:01 2015 | root | timeline | timeline                                        |              
single | 528  |       | Tue Sep  1 08:45:01 2015 | root | timeline | timeline                                        |              
single | 676  |       | Sat Oct  3 00:30:01 2015 | root | timeline | timeline                                        |              
single | 862  |       | Sun Nov  1 12:15:01 2015 | root | timeline | timeline                                        |              
single | 963  |       | Tue Dec  1 13:45:01 2015 | root | timeline | timeline                                        |              
single | 1115 |       | Fri Jan  1 22:15:01 2016 | root | timeline | timeline                                        |              
pre    | 1154 |       | Fri Jan 15 23:06:13 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1155 | 1154  | Fri Jan 15 23:06:21 2016 | root | number   |                                                 | important=no 
pre    | 1156 |       | Fri Jan 15 23:15:14 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1157 | 1156  | Fri Jan 15 23:15:21 2016 | root | number   |                                                 | important=no 
pre    | 1158 |       | Fri Jan 15 23:27:43 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1159 | 1158  | Fri Jan 15 23:27:49 2016 | root | number   |                                                 | important=no 
pre    | 1160 |       | Sat Jan 16 23:56:25 2016 | root | number   | zypp(zypper)                                    | important=yes
post   | 1161 | 1160  | Sun Jan 17 00:04:50 2016 | root | number   |                                                 | important=yes
pre    | 1162 |       | Mon Jan 18 00:48:04 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1163 | 1162  | Mon Jan 18 00:48:12 2016 | root | number   |                                                 | important=no 
pre    | 1170 |       | Wed Jan 20 08:13:50 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1171 | 1170  | Wed Jan 20 08:14:20 2016 | root | number   |                                                 | important=no 
pre    | 1172 |       | Thu Jan 21 22:54:33 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1173 | 1172  | Thu Jan 21 22:54:38 2016 | root | number   |                                                 | important=no 
pre    | 1174 |       | Thu Jan 21 22:56:05 2016 | root | number   | zypp(zypper)                                    | important=yes
post   | 1175 | 1174  | Thu Jan 21 23:37:30 2016 | root | number   |                                                 | important=yes
pre    | 1176 |       | Fri Jan 22 16:50:15 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1177 | 1176  | Fri Jan 22 16:54:08 2016 | root | number   |                                                 | important=no 
pre    | 1182 |       | Sat Jan 23 16:29:09 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1183 | 1182  | Sat Jan 23 16:32:05 2016 | root | number   |                                                 | important=no 
pre    | 1184 |       | Thu Jan 28 22:17:02 2016 | root | number   | zypp(zypper)                                    | important=yes
post   | 1185 | 1184  | Thu Jan 28 22:28:16 2016 | root | number   |                                                 | important=yes
pre    | 1187 |       | Sun Jan 31 13:18:35 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1188 | 1187  | Sun Jan 31 14:20:42 2016 | root | number   |                                                 | important=no 
pre    | 1189 |       | Wed Feb 10 16:15:06 2016 | root | number   | zypp(zypper)                                    | important=yes
post   | 1190 | 1189  | Wed Feb 10 16:24:32 2016 | root | number   |                                                 | important=yes
single | 1191 |       | Thu Feb 11 19:00:02 2016 | root | timeline | timeline                                        |              
pre    | 1204 |       | Sat Feb 13 11:10:29 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1205 | 1204  | Sat Feb 13 11:14:27 2016 | root | number   |                                                 | important=no 
pre    | 1206 |       | Wed Feb 17 08:29:37 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1207 | 1206  | Wed Feb 17 08:29:43 2016 | root | number   |                                                 | important=no 
pre    | 1209 |       | Wed Feb 17 23:59:50 2016 | root | number   | zypp(zypper)                                    | important=yes
single | 1210 |       | Thu Feb 18 00:00:01 2016 | root | timeline | timeline                                        |              
post   | 1211 | 1209  | Thu Feb 18 00:03:47 2016 | root | number   |                                                 | important=yes
pre    | 1212 |       | Thu Feb 18 00:04:28 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1213 | 1212  | Thu Feb 18 00:04:36 2016 | root | number   |                                                 | important=no 
single | 1216 |       | Fri Feb 19 00:00:01 2016 | root | timeline | timeline                                        |              
single | 1224 |       | Thu Feb 18 23:30:01 2016 | root | timeline | timeline                                        |              
single | 1225 |       | Fri Feb 19 00:30:01 2016 | root | timeline | timeline                                        |              
single | 1226 |       | Sat Feb 20 05:45:01 2016 | root | timeline | timeline                                        |              
pre    | 1227 |       | Sat Feb 20 06:02:28 2016 | root | number   | zypp(packagekitd)                               | important=yes
post   | 1228 | 1227  | Sat Feb 20 06:02:41 2016 | root | number   |                                                 | important=yes
single | 1229 |       | Sun Feb 21 07:30:01 2016 | root | timeline | timeline                                        |              
pre    | 1231 |       | Sun Feb 21 23:13:26 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1232 | 1231  | Sun Feb 21 23:13:34 2016 | root | number   |                                                 | important=no 
single | 1234 |       | Mon Feb 22 21:15:02 2016 | root | timeline | timeline                                        |              
pre    | 1236 |       | Mon Feb 22 23:13:23 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1237 | 1236  | Mon Feb 22 23:13:44 2016 | root | number   |                                                 | important=no 
single | 1239 |       | Tue Feb 23 00:15:01 2016 | root | timeline | timeline                                        |              
pre    | 1240 |       | Tue Feb 23 13:40:21 2016 | root | number   | zypp(zypper)                                    | important=yes
post   | 1242 | 1240  | Tue Feb 23 13:45:54 2016 | root | number   |                                                 | important=yes
pre    | 1243 |       | Tue Feb 23 14:01:39 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1244 | 1243  | Tue Feb 23 14:04:31 2016 | root | number   |                                                 | important=no 
pre    | 1247 |       | Tue Feb 23 16:13:35 2016 | root | number   | zypp(zypper)                                    | important=no 
single | 1248 |       | Tue Feb 23 16:45:01 2016 | root | timeline | timeline                                        |              
post   | 1249 | 1247  | Tue Feb 23 16:48:20 2016 | root | number   |                                                 | important=no 
single | 1250 |       | Tue Feb 23 17:45:01 2016 | root | timeline | timeline                                        |              
single | 1251 |       | Tue Feb 23 18:45:01 2016 | root | timeline | timeline                                        |              
single | 1252 |       | Tue Feb 23 19:45:01 2016 | root | timeline | timeline                                        |              
single | 1253 |       | Tue Feb 23 21:45:01 2016 | root | timeline | timeline                                        |              
single | 1254 |       | Tue Feb 23 22:45:01 2016 | root | timeline | timeline                                        |              
single | 1255 |       | Tue Feb 23 23:45:01 2016 | root | timeline | timeline                                        |              
pre    | 1256 |       | Thu Feb 25 21:36:01 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1257 | 1256  | Thu Feb 25 21:38:04 2016 | root | number   |                                                 | important=no 
pre    | 1258 |       | Sun Feb 28 13:54:40 2016 | root | number   | zypp(zypper)                                    | important=yes
post   | 1259 | 1258  | Sun Feb 28 14:01:46 2016 | root | number   |                                                 | important=yes
single | 1260 |       | Sun Feb 28 21:45:01 2016 | root | timeline | timeline                                        |              
single | 1261 |       | Sun Feb 28 22:45:02 2016 | root | timeline | timeline                                        |              
single | 1262 |       | Mon Feb 29 17:00:01 2016 | root | timeline | timeline                                        |              
pre    | 1263 |       | Tue Mar  1 20:57:10 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1264 | 1263  | Tue Mar  1 21:09:58 2016 | root | number   |                                                 | important=no 
pre    | 1265 |       | Thu Mar  3 20:14:39 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1266 | 1265  | Thu Mar  3 20:17:12 2016 | root | number   |                                                 | important=no 
pre    | 1267 |       | Thu Mar  3 22:12:39 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1268 | 1267  | Thu Mar  3 22:12:46 2016 | root | number   |                                                 | important=no 
pre    | 1269 |       | Fri Mar  4 15:20:13 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1270 | 1269  | Fri Mar  4 15:20:21 2016 | root | number   |                                                 | important=no 
pre    | 1271 |       | Fri Mar  4 15:21:23 2016 | root | number   | zypp(zypper)                                    | important=no 
post   | 1272 | 1271  | Fri Mar  4 15:21:29 2016 | root | number   |                                                 | important=no

Did you ever attempt boot and revert to previous snapshot? What “btrfs sub get-default /” says?

Yes, at some point I booted from a snapshot - my partition was unstable due to buggy ext4 -> btrfs conversion code. I managed to fix it eventually but I needed a custom build of btrfsprogs and a lot of help from snapper to find a right solution ( looking at the snapshots it happened at some point in July ).

btrfs sub get-default /
ID 1492 gen 92362 top level 493 path .snapshots/359/snapshot

You’ve posted a list of snapshots, why are you hung up on that one?
Besides, your syntax is incorrect.

Don’t “snapper -c root”,
Just su to root and

snapper delete* list_all_snapshot_numbers_separated_by_a_space_you_want_to_remove*

Unfortunately, I haven’t found a way to specify a range of snapshots, I’ve only been able to specify each and every snapshot one by one although all can be done in a single command as I described.

TSU

This fails for all snapshots that I’ve tested for in the root config.

Superfluous? Yes :slight_smile: Incorrect? No

The -c argument specifies the snapper config to use ( defaulting to root ) , see the snapper man page.

Thanks, but that’s not my issue. I an unable to delete any snapshot for the root config. I would be happy if snapper cleanup would work, it should remove a lot of old snapshots, but at the moment it can’t remove any.

AFAIK, specifying a config is not the same as actually switching to that User.
The error you’re seeing is entirely accurate but I think you’re mis-interpreting… In other words…

Snapper trying to use a default config which is called “root.” Can’t switch to a config called “root” because it already exists and is currently being used.

I’d be surprised if you followed the steps I described exactly and you still have a problem.

So, for example

  1. su to root. The result should be a root console
su
  1. Next, type the following exactly or paste (CTL-SHFT-V in a Bash terminal)
snapper delete 274

Or, try deleting a few snapshots in your list with obviously low priority

snapper delete 323 324 358 359
  1. You can verify your results by listing your snapshots again
snapper list

Have done the above many times on a number of systems without ever a problem.

TSU

Thanks for the detailed instructions, see my exact terminal output beloe


$ su 
Password: 
mars:/home/robert # id
uid=0(root) gid=0(root) groups=0(root)
mars:/home/robert # snapper delete 274
Config is in use.
mars:/home/robert # snapper delete 323 324 358 359
Config is in use.

I am sure of that :slight_smile: But something in my snapper config/setup is broken, and I’m not sure exactly what and how to fix it.

To me, that output looks like a vestige of your previous incorrect command is still in use.
Am almost certain your misconfiguration is only related to the console session, either close/open a new console and “su” again or reboot and everything should be cleared and returned to default.

TSU

Can you remove any snapshot created after this one (so far all your examples show earlier snapshots)?

And, BTW, you always can remove btrfs snapshots directly; I thought there was some snapper option to cross-check, but I do not see one. You will be able to manually clean up by removing dangling snapshot directories under /.snapshots though. But it would be helpful to find root cause for your problem first.

Well, after a reboot I was suddenly able to remove snapshots … both with -c root and without it. I was able to delete all ‘old’ snapshots except one (thanks to both for helping!), but one snapshot refuses to go:

# snapper delete 359
Deleting snapshot failed.

Snapper log shows a BTRFS error:

2016-03-06 10:36:57 MIL libsnapper(11762) Snapper.cc(Snapper):103 - subvolume:/ filesystem:btrfs
2016-03-06 10:36:57 MIL libsnapper(11762) Snapper.cc(loadIgnorePatterns):152 - number of ignore patterns:8
2016-03-06 10:36:57 ERR libsnapper(11762) Snapshot.cc(read):217 - loading 357 failed
2016-03-06 10:36:57 ERR libsnapper(11762) Snapshot.cc(read):217 - loading 393 failed
2016-03-06 10:36:57 ERR libsnapper(11762) Snapshot.cc(read):217 - loading 430 failed
2016-03-06 10:36:57 MIL libsnapper(11762) Snapshot.cc(read):223 - found 72 snapshots
2016-03-06 10:36:57 ERR libsnapper(11762) Snapshot.cc(check):276 - time shift detected at snapshot num 1224
2016-03-06 10:36:57 ERR libsnapper(11762) Btrfs.cc(deleteSnapshot):375 - delete snapshot failed, ioctl(BTRFS_IOC_SNAP_DESTROY) failed, errno:1 (Operation not permitted)
2016-03-06 10:36:57 WAR libsnapper(11762) Client.cc(dispatch):1518 - CAUGHT: delete snapshot failed

The btrfs mount seems fine, I scrubbed it immediately afterwards:

# /etc/cron.monthly/btrfs-scrub.sh 
Running scrub on /
scrub device /dev/sda1 (id 1) done
        scrub started at Sun Mar  6 10:43:29 2016 and finished after 00:08:40
        total bytes scrubbed: 186.21GiB with 0 errors

I see no problems with the snapshot directory

ll -a /.snapshots/359/
total 4
drwxr-xr-x 1 root root  32 Jul 31  2015 .
drwxr-x--- 1 root root 624 Mar  6 10:45 ..
-rw------- 1 root root 119 Jul 31  2015 info.xml
drwxr-xr-x 1 root root 390 Oct 26 18:45 snapshot

So, what’s the deal with that snapshot?

Ah, so that is the default subvolume ( see output from btrfs sub get-default / in a previous post ). A related bug was filed at https://bugzilla.suse.com/show_bug.cgi?id=936605 .

Does this have any ill side-effects, like preventing free space from being reclaimed since the blocks used in that snapshot will always be around? Should I alter the setup somehow?

Quoting the bug you mentioned:

the system is installed into a (read/write) snapshot.

If you look at newly installed system, your real root is actually /.snapshots/1/snapshot subvolume. When you restore from snapshot, you flip root to another snapshot; it does not change anything w.r.t. space consumption. I wonder whether it may confuse snapper though (as newer snapshots now have “older” numbers).

So we still do not know what caused your issue. Pity.

Whenever you do a rollback, you don’t actually revert to the earlier snapshot id. That earlier id is only used to reference the time/date when that snapshot was created. You can think of these snapshots as virtual indexing numbers only to identify and not the actual snapshots themselves.

When you do a rollback, a new snapshot with a new id is created, so even when rolling back you’re always moving forward numerically. the BTRFS snapshot functionality is supposed to track these changes automatically and resolve all snapshot incremental relationships like deleting, so as long as you use a BTRFS authorized tool like snapper everything should work without a problem.

TSU

Well, it happened again ( not sure if it’s related to suspend, I tend to do that quite a lot on my desktop ). The only entry in in /var/log/snapper.log is

2016-03-09 21:45:40 WAR libsnapper(21360) Client.cc(dispatch):1488 - CAUGHT: config in use