Snapshot cleanup not working anymore?

Cris70 · June 28, 2024, 10:18pm

Recently my snapshots have been apparently growing out of control.
I have number cleanup configured:

# run daily number cleanup
NUMBER_CLEANUP="yes"

# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="2-5"
NUMBER_LIMIT_IMPORTANT="4-6"

But I have presently 63 snapshots!
If I try to force a cleanup I get this:

  polarisuse  cris  ~  LANG=C sudo snapper cleanup number
Deleting snapshot failed.

And in /var/log/snapper.log I find:

2024-06-29 00:14:12 MIL libsnapper(37663) snapperd.cc(main):283 - Requesting DBus name
2024-06-29 00:14:12 MIL libsnapper(37663) snapperd.cc(main):298 - Loading snapper configs
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(getConfigs):355 - Snapper get-configs
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(getConfigs):356 - libsnapper version 0.11.0
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(reload):920 - loading file /etc/sysconfig/snapper
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(get_value):1076 - key:SNAPPER_CONFIGS value:root
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(reload):920 - loading file /etc/snapper/configs/root
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(get_value):1076 - key:SUBVOLUME value:/
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(get_value):1076 - key:ALLOW_USERS value:
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(get_value):1076 - key:ALLOW_GROUPS value:
2024-06-29 00:14:12 MIL libsnapper(37663) snapperd.cc(main):311 - Listening for method calls and signals
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(Snapper):97 - Snapper constructor
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(Snapper):98 - snapper version 0.11.0
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(Snapper):99 - libsnapper version 7.4.3
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(Snapper):100 - config_name:root root_prefix:/ disable_filters:false
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(reload):920 - loading file /etc/snapper/configs/root
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(get_value):1076 - key:SUBVOLUME value:/
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(get_value):1076 - key:FSTYPE value:btrfs
2024-06-29 00:14:12 MIL libsnapper(37663) Selinux.cc(_is_selinux_enabled):141 - SELinux support disabled
2024-06-29 00:14:12 MIL libsnapper(37663) AsciiFile.cc(get_value):1076 - key:SYNC_ACL value:no
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(Snapper):130 - subvolume:/ filesystem:btrfs
2024-06-29 00:14:12 MIL libsnapper(37663) Snapper.cc(loadIgnorePatterns):204 - number of ignore patterns:8
2024-06-29 00:14:12 MIL libsnapper(37663) Snapshot.cc(read):340 - found 63 snapshots
2024-06-29 00:14:12 ERR libsnapper(37663) Btrfs.cc(deleteSnapshot):430 - delete snapshot //.snapshots/1/snapshot failed, btrfs_util_delete_subvolume_fd() failed, errno:39 (Directory not empt
y)
2024-06-29 00:14:12 WAR libsnapper(37663) Btrfs.cc(deleteSnapshot):431 - THROW: delete snapshot failed
2024-06-29 00:14:12 WAR libsnapper(37663) Client.cc(dispatch):1996 - CAUGHT: delete snapshot failed

So it is apparently trying to delete snapshot 1… but snapshot 1 should never be deleted
AFAIK… or at least I have never seen it being deleted before.

  polarisuse  cris  ~  1  sudo snapper list
   # │ Type   │ Pre # │ Data                      │ Utente │ Pulitura │ Description             │ Dati utente
─────┼────────┼───────┼───────────────────────────┼────────┼──────────┼─────────────────────────┼──────────────
  0  │ single │       │                           │ root   │          │ current                 │
  1  │ single │       │ dom 11 nov 2018, 12:23:19 │ root   │ number   │ first root filesystem   │
220  │ pre    │       │ mar 4 giu 2024, 00:51:08  │ root   │ number   │ zypp(zypper)            │ important=yes
221  │ post   │   220 │ mar 4 giu 2024, 00:51:46  │ root   │ number   │                         │ important=yes
234  │ pre    │       │ ven 7 giu 2024, 00:11:05  │ root   │ number   │ zypp(zypper)            │ important=yes
235  │ post   │   234 │ ven 7 giu 2024, 00:14:45  │ root   │ number   │                         │ important=yes
240  │ pre    │       │ lun 10 giu 2024, 01:18:30 │ root   │ number   │ zypp(zypper)            │ important=yes
241  │ post   │   240 │ lun 10 giu 2024, 01:20:59 │ root   │ number   │                         │ important=yes
252  │ pre    │       │ dom 16 giu 2024, 02:51:51 │ root   │ number   │ zypp(zypper)            │ important=no
253  │ post   │   252 │ dom 16 giu 2024, 02:52:40 │ root   │ number   │                         │ important=no
254  │ pre    │       │ dom 16 giu 2024, 02:54:30 │ root   │ number   │ zypp(zypper)            │ important=no
255  │ post   │   254 │ dom 16 giu 2024, 02:56:20 │ root   │ number   │                         │ important=no
256  │ pre    │       │ lun 17 giu 2024, 15:13:29 │ root   │ number   │ zypp(zypper)            │ important=no
257  │ post   │   256 │ lun 17 giu 2024, 15:13:32 │ root   │ number   │                         │ important=no
258  │ pre    │       │ mar 18 giu 2024, 14:19:38 │ root   │ number   │ zypp(zypper)            │ important=yes
259  │ post   │   258 │ mar 18 giu 2024, 14:28:07 │ root   │ number   │                         │ important=yes
260  │ pre    │       │ mer 19 giu 2024, 10:15:30 │ root   │ number   │ zypp(zypper)            │ important=yes
261  │ post   │   260 │ mer 19 giu 2024, 10:15:43 │ root   │ number   │                         │ important=yes
262  │ pre    │       │ mer 19 giu 2024, 10:18:38 │ root   │ number   │ zypp(zypper)            │ important=no
263  │ post   │   262 │ mer 19 giu 2024, 10:21:26 │ root   │ number   │                         │ important=no
264  │ single │       │ mer 19 giu 2024, 11:43:44 │ root   │ number   │ rollback backup of #1   │ important=yes
265  │ single │       │ mer 19 giu 2024, 11:43:45 │ root   │ number   │ writable copy of #259   │
266  │ pre    │       │ mer 19 giu 2024, 11:45:15 │ root   │ number   │ zypp(zypper)            │ important=yes
267  │ post   │   266 │ mer 19 giu 2024, 11:45:29 │ root   │ number   │                         │ important=yes
268  │ single │       │ mer 19 giu 2024, 11:47:01 │ root   │ number   │ rollback backup of #265 │ important=yes
269* │ single │       │ mer 19 giu 2024, 11:47:02 │ root   │          │ writable copy of #258   │
270  │ pre    │       │ mer 19 giu 2024, 12:19:20 │ root   │ number   │ zypp(zypper)            │ important=yes
271  │ post   │   270 │ mer 19 giu 2024, 12:28:51 │ root   │ number   │                         │ important=yes
272  │ pre    │       │ mer 19 giu 2024, 17:06:11 │ root   │ number   │ zypp(zypper)            │ important=yes
273  │ post   │   272 │ mer 19 giu 2024, 17:06:25 │ root   │ number   │                         │ important=yes
274  │ pre    │       │ gio 20 giu 2024, 02:24:11 │ root   │ number   │ zypp(zypper)            │ important=yes
275  │ post   │   274 │ gio 20 giu 2024, 02:30:52 │ root   │ number   │                         │ important=yes
276  │ pre    │       │ gio 20 giu 2024, 09:18:20 │ root   │ number   │ zypp(zypper)            │ important=yes
277  │ post   │   276 │ gio 20 giu 2024, 09:18:59 │ root   │ number   │                         │ important=yes
278  │ pre    │       │ gio 20 giu 2024, 21:52:40 │ root   │ number   │ zypp(zypper)            │ important=no
279  │ post   │   278 │ gio 20 giu 2024, 21:55:10 │ root   │ number   │                         │ important=no
280  │ pre    │       │ ven 21 giu 2024, 02:19:31 │ root   │ number   │ zypp(zypper)            │ important=no
281  │ post   │   280 │ ven 21 giu 2024, 02:19:41 │ root   │ number   │                         │ important=no
282  │ pre    │       │ sab 22 giu 2024, 04:01:03 │ root   │ number   │ zypp(zypper)            │ important=no
283  │ post   │   282 │ sab 22 giu 2024, 04:02:25 │ root   │ number   │                         │ important=no
284  │ pre    │       │ dom 23 giu 2024, 00:35:02 │ root   │ number   │ zypp(zypper)            │ important=no
285  │ post   │   284 │ dom 23 giu 2024, 00:36:44 │ root   │ number   │                         │ important=no
286  │ pre    │       │ mar 25 giu 2024, 15:57:56 │ root   │ number   │ zypp(zypper)            │ important=no
287  │ pre    │       │ mar 25 giu 2024, 15:58:53 │ root   │ number   │ zypp(zypper)            │ important=no
288  │ post   │   287 │ mar 25 giu 2024, 16:00:53 │ root   │ number   │                         │ important=no
289  │ pre    │       │ mar 25 giu 2024, 23:42:15 │ root   │ number   │ zypp(zypper)            │ important=no
290  │ post   │   289 │ mar 25 giu 2024, 23:42:26 │ root   │ number   │                         │ important=no
291  │ pre    │       │ mer 26 giu 2024, 14:09:45 │ root   │ number   │ zypp(zypper)            │ important=no
292  │ post   │   291 │ mer 26 giu 2024, 14:17:51 │ root   │ number   │                         │ important=no
293  │ pre    │       │ gio 27 giu 2024, 10:05:48 │ root   │ number   │ zypp(zypper)            │ important=no
294  │ post   │   293 │ gio 27 giu 2024, 10:05:52 │ root   │ number   │                         │ important=no
295  │ pre    │       │ gio 27 giu 2024, 11:21:01 │ root   │ number   │ zypp(zypper)            │ important=no
296  │ post   │   295 │ gio 27 giu 2024, 11:21:10 │ root   │ number   │                         │ important=no
297  │ pre    │       │ gio 27 giu 2024, 12:56:26 │ root   │ number   │ zypp(zypper)            │ important=yes
298  │ post   │   297 │ gio 27 giu 2024, 12:59:29 │ root   │ number   │                         │ important=yes
299  │ pre    │       │ gio 27 giu 2024, 14:00:51 │ root   │ number   │ zypp(zypper)            │ important=no
300  │ post   │   299 │ gio 27 giu 2024, 14:01:15 │ root   │ number   │                         │ important=no
301  │ pre    │       │ ven 28 giu 2024, 00:01:18 │ root   │ number   │ zypp(zypper)            │ important=yes
302  │ post   │   301 │ ven 28 giu 2024, 00:02:04 │ root   │ number   │                         │ important=yes
303  │ pre    │       │ ven 28 giu 2024, 00:24:11 │ root   │ number   │ zypp(zypper)            │ important=no
304  │ post   │   303 │ ven 28 giu 2024, 00:25:05 │ root   │ number   │                         │ important=no
305  │ pre    │       │ ven 28 giu 2024, 22:55:47 │ root   │ number   │ zypp(zypper)            │ important=no
306  │ post   │   305 │ ven 28 giu 2024, 23:00:37 │ root   │ number   │                         │ important=no

What is happening here?
Anybody else noticing this?
I have plenty of disk space right now, but if snapshots are not cleaned it is going to be a big problem in a few days/weeks.

doscott · June 28, 2024, 10:31pm

Snapshot 1 is marked as “number”, which means it is subject to cleanup.

However, snapshot 1 will only change from an empty field to “number” after a rollback has been performed, which creates a new writable copy of with an empty field. That is what occurred with snapshot 269.

You should be able to clean up snapshot 1, but it may require manual intervention and I will leave it to someone more experienced to guide you (I only had to do this once many years ago).

Cris70 · June 28, 2024, 10:35pm

Thank you @doscott,
that explains why snapshot 1 was never deleted in the past: I have never needed a rollback until very recently when the Mesa disaster stroke.
I’ll wait for someone else to try and help me understand why snapshot 1 cannot be deleted.

doscott · June 28, 2024, 10:38pm

In the mean time, you can use yast control center “Filesystem Snapshots”, click on snapshot 1, press the modify button and change the Cleanup algorithm to blank, which should clear up your error condition for now.

Cris70 · June 28, 2024, 10:47pm

Worked like a charm, thank you!

arvidjaar · June 29, 2024, 4:49am

Cris70:

2024-06-29 00:14:12 ERR libsnapper(37663) Btrfs.cc(deleteSnapshot):430 - delete snapshot //.snapshots/1/snapshot failed, btrfs_util_delete_subvolume_fd() failed, errno:39 (Directory not empty)

Show output of

btrfs subvolume list /
findmnt -u -t btrfs

Cris70 · June 29, 2024, 10:22am

Hi @arvidjaar,
here it comes:

  polarisuse  cris  ~  sudo btrfs subvolume list /
[sudo] password di root:
ID 257 gen 27 top level 5 path @
ID 258 gen 1522727 top level 257 path @/var
ID 259 gen 1522667 top level 257 path @/usr/local
ID 260 gen 1519903 top level 257 path @/srv
ID 261 gen 1522656 top level 257 path @/root
ID 262 gen 1455886 top level 257 path @/boot/grub2/x86_64-efi
ID 263 gen 1503265 top level 257 path @/boot/grub2/i386-pc
ID 264 gen 1522666 top level 257 path @/.snapshots
ID 265 gen 1522477 top level 264 path @/.snapshots/1/snapshot
ID 308 gen 1455887 top level 258 path @/var/lib/machines
ID 433 gen 1364828 top level 265 path @/.snapshots/1/snapshot/var/lib/machines
ID 1163 gen 1480158 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/f4815b19de8520d14e2dada906a121cba456c652c2322214f770c9485d6233c6
ID 1164 gen 1480158 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/25f1253294d76613fd07843b73c65b1696517105c35edfc3dc85d7885e219de2
ID 1165 gen 1480158 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/90dc89e1e289dd2e65d516e9560c94169dfe3376e63431c19ea73a01fd3b5b19
ID 1166 gen 1480158 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/1ee6ad054006d511ac702e26d179962f8c917b298d99e1f8f28e4190cbb7f378
ID 1173 gen 1480158 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/dd9e91d72dc9052e0a0f2962f056b83ba1edc6374981d226609638277ff4736f-init
ID 1174 gen 1480158 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/dd9e91d72dc9052e0a0f2962f056b83ba1edc6374981d226609638277ff4736f
ID 4157 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/ecfac849db45409d2ca6aebed3bf2045dc09a0ec4708e75ebaec82bc06796f4f
ID 4158 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/cd496ee9a0825bb8ae58f2c899b2e625bee4e203eb63ac24da2e01de89768af5
ID 4159 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/3438b0288ecda994a9139eaaae358642a2af69ad8457f97538d245eec90e673a
ID 4160 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/b161ab44f77a01fb9ebff177f8cf4b6268fc773d2cedb9c96aab4327b3d8891d
ID 4161 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/906964687b543826cdba86ffeba1031c6cf280ea65a7bfe2af895846a46d1269
ID 4162 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/e144ed903b5db9e5c71327825612267767e9fc2a51856f7a1e7b497a45ca6adf
ID 4163 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/d827f6fcf3a9afe0d472c38925caa5de88848371aaff282eb9e325872b6a0636
ID 4164 gen 1455887 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/fed112b807faffb34e383c9112e5a35971051178dc3ee5cea72a1cbdeaeef56e
ID 4165 gen 1480149 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/1f9ac2f2e961ba82324040ea7ce68a58489f060b531d20a835506ca610cd2dda
ID 4166 gen 1480149 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/d931ed7f916b823e6b0219ced4996815ee6b9f5c35b7fc7330b3be8fba2f8cd4
ID 4167 gen 1480149 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/d5d5d0b08c93cb6953feff33ab50f8b91182109cbb97c2bdf5869f169c172890
ID 4168 gen 1480149 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/49ef9fadc79b299728c0a680ec66d93b34f2b854bd668ff97fc24c2e5c4ed80c-init
ID 4169 gen 1480149 top level 258 path @/var/lib/docker.bk/btrfs/subvolumes/49ef9fadc79b299728c0a680ec66d93b34f2b854bd668ff97fc24c2e5c4ed80c
ID 5391 gen 1522725 top level 264 path @/.snapshots/269/snapshot
ID 5400 gen 1513088 top level 264 path @/.snapshots/276/snapshot
ID 5401 gen 1513094 top level 264 path @/.snapshots/277/snapshot
ID 5427 gen 1520247 top level 264 path @/.snapshots/297/snapshot
ID 5429 gen 1520330 top level 264 path @/.snapshots/298/snapshot
ID 5435 gen 1520722 top level 264 path @/.snapshots/301/snapshot
ID 5436 gen 1520728 top level 264 path @/.snapshots/302/snapshot
ID 5447 gen 1522626 top level 264 path @/.snapshots/313/snapshot
ID 5448 gen 1522639 top level 264 path @/.snapshots/314/snapshot

and

  polarisuse  cris  ~  sudo findmnt -u -t btrfs
TARGET                   SOURCE                                FSTYPE OPTIONS
/                        /dev/sdc1[/@/.snapshots/269/snapshot] btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=5391,subvol=/@/.snapshots/269/snapshot
├─/.snapshots            /dev/sdc1[/@/.snapshots]              btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=264,subvol=/@/.snapshots
├─/boot/grub2/i386-pc    /dev/sdc1[/@/boot/grub2/i386-pc]      btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=263,subvol=/@/boot/grub2/i386-pc
├─/boot/grub2/x86_64-efi /dev/sdc1[/@/boot/grub2/x86_64-efi]   btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=262,subvol=/@/boot/grub2/x86_64-efi
├─/usr/local             /dev/sdc1[/@/usr/local]               btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=259,subvol=/@/usr/local
├─/root                  /dev/sdc1[/@/root]                    btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=261,subvol=/@/root
├─/srv                   /dev/sdc1[/@/srv]                     btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=260,subvol=/@/srv
└─/var                   /dev/sdc1[/@/var]                     btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=258,subvol=/@/var

Thank you in advance
Cris

arvidjaar · June 29, 2024, 10:35am

That is the reason why removing of the snapshot 1 fails. There is nested subvolume. You need to delete it manually. If it has useful content, you need to copy it.

btrfs subvolume delete /.snapshots/1/snapshot/var/lib/machines

Cris70 · June 29, 2024, 9:39pm

Thank you. It is empty.
That directory name rings a bell, but I cannot remember why it was there.
How do I remove it?
Is deleting the directory enough?

Cris70 · June 29, 2024, 9:47pm

Now I know why that name ringed a bell: it was mentioned in a thread by me in 2017!!
Here it is: /var/lib/machines error during update

system · July 29, 2024, 9:48pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.