I made a reddit post about this recently, but figured this is the place.
I looked in my logs and saw that snapper-cleanup.service failed to remove snapshot 72. Snapshot 72 is currently my oldest snapshot out of almost 100
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: Running cleanup for 'root'.
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: Running number cleanup for 'root'.
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: Removing snapshot from root:
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: 72
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: Could not remove snapshot
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: number cleanup for 'root' failed.
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: Running timeline cleanup for 'root'.
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: Running empty-pre-post cleanup for 'root'.
jan 28 20:33:56 localhost.localdomain systemd-helper[10012]: Running 'btrfs qgroup clear-stale /.snapshots'.
My snapper config is the default Aeon one (“root”). I have no other configs
# subvolume to snapshot
SUBVOLUME="/"
# filesystem type
FSTYPE="btrfs"
# btrfs qgroup for space aware cleanup algorithms
QGROUP="1/0"
# fraction or absolute size of the filesystems space the snapshots may use
SPACE_LIMIT="0.5"
# fraction or absolute size of the filesystems space that should be free
FREE_LIMIT="0.2"
# users and groups allowed to work with config
ALLOW_USERS=""
ALLOW_GROUPS=""
# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="no"
# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"
# run daily number cleanup
NUMBER_CLEANUP="yes"
# limit for number cleanup
NUMBER_MIN_AGE="3600"
NUMBER_LIMIT="2-10"
NUMBER_LIMIT_IMPORTANT="4-10"
# create hourly snapshots
TIMELINE_CREATE="no"
# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"
# limits for timeline cleanup
TIMELINE_MIN_AGE="3600"
TIMELINE_LIMIT_HOURLY="10"
TIMELINE_LIMIT_DAILY="10"
TIMELINE_LIMIT_WEEKLY="0"
TIMELINE_LIMIT_MONTHLY="10"
TIMELINE_LIMIT_QUARTERLY="0"
TIMELINE_LIMIT_YEARLY="10"
# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"
# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="3600"
This is the output of snapper list (shortened):
localhost:/.snapshots # snapper list
# │ Type │ Pre # │ Date │ User │ Used Space │ Cleanup │ Description │ Userdata
─────┼────────┼───────┼──────────────────────────┼──────┼────────────┼─────────┼─────────────────────────┼─────────────────────────────────────
0 │ single │ │ │ root │ │ │ current │
72 │ single │ │ Thu Nov 7 21:21:10 2024 │ root │ 2.77 MiB │ number │ Snapshot Update of #71 │ important=yes
73 │ single │ │ Thu Nov 7 21:32:28 2024 │ root │ 2.80 MiB │ number │ Snapshot Update of #72 │
74 │ single │ │ Fri Nov 8 12:57:54 2024 │ root │ 424.00 KiB │ number │ Snapshot Update of #68 │
75 │ single │ │ Fri Nov 8 12:58:17 2024 │ root │ 500.00 KiB │ number │ Snapshot Update of #74 │
76 │ single │ │ Sat Nov 9 14:31:04 2024 │ root │ 5.03 MiB │ number │ Snapshot Update of #75 │ important=yes
77 │ single │ │ Sun Nov 10 21:41:59 2024 │ root │ 464.00 KiB │ number │ Snapshot Update of #76 │
78 │ single │ │ Sun Nov 10 21:42:47 2024 │ root │ 516.00 KiB │ number │ Snapshot Update of #77 │ important=yes
79 │ single │ │ Fri Nov 15 13:58:26 2024 │ root │ 3.43 MiB │ number │ Snapshot Update of #78 │ important=yes
... ... ... ...
150 │ single │ │ Sun Jan 12 11:58:09 2025 │ root │ 3.91 MiB │ number │ Snapshot Update of #149 │ important=yes
151 │ single │ │ Thu Jan 16 19:36:49 2025 │ root │ 29.67 MiB │ number │ Snapshot Update of #150 │
152 │ single │ │ Thu Jan 16 19:43:50 2025 │ root │ 8.93 MiB │ number │ Snapshot Update of #151 │ important=yes
153 │ single │ │ Mon Jan 20 20:23:57 2025 │ root │ 592.00 KiB │ number │ Snapshot Update of #152 │
154 │ single │ │ Mon Jan 20 20:24:34 2025 │ root │ 508.00 KiB │ number │ Snapshot Update of #153 │ important=yes
155 │ single │ │ Tue Jan 21 09:51:54 2025 │ root │ 320.00 KiB │ number │ Snapshot Update of #154 │
156 │ single │ │ Tue Jan 21 09:52:03 2025 │ root │ 440.00 KiB │ number │ Snapshot Update of #155 │ important=yes
157 │ single │ │ Mon Jan 27 19:59:07 2025 │ root │ 14.27 MiB │ number │ Snapshot Update of #156 │
158 │ single │ │ Mon Jan 27 20:01:38 2025 │ root │ 2.61 MiB │ number │ Snapshot Update of #157 │ important=yes
159 │ single │ │ Tue Jan 28 19:11:52 2025 │ root │ 3.37 MiB │ number │ Snapshot Update of #158 │ important=yes
160 │ single │ │ Tue Jan 28 20:23:03 2025 │ root │ 3.38 MiB │ number │ Snapshot Update of #159 │
161 │ single │ │ Wed Jan 29 07:34:25 2025 │ root │ 376.00 KiB │ number │ Snapshot Update of #160 │
162* │ single │ │ Wed Jan 29 07:34:35 2025 │ root │ 436.00 KiB │ number │ Snapshot Update of #161 │
There are no messages in dmesg. My drive is only 25% full.
The only thing wonky about my setup is a @gnu subvolume for gnu guix stuff.
Removing the snapshot manually (with the verbose flag) prints the same error messages as above (minus the things that snapper-cleanup does extra).
I think I see the problem. I have gnu dir in the 72nd snapshot:
ID 256 gen 29903 top level 5 path @
ID 257 gen 51816 top level 256 path @/.snapshots
ID 259 gen 51832 top level 256 path @/home
ID 260 gen 51265 top level 256 path @/opt
ID 261 gen 51829 top level 256 path @/root
ID 262 gen 51425 top level 256 path @/srv
ID 263 gen 51832 top level 256 path @/var
ID 264 gen 51829 top level 256 path @/usr/local
ID 332 gen 51265 top level 256 path @/boot/writable
ID 355 gen 30997 top level 257 path @/.snapshots/72/snapshot
ID 356 gen 21585 top level 355 path @/.snapshots/72/snapshot/gnu
ID 357 gen 30997 top level 257 path @/.snapshots/73/snapshot
ID 362 gen 21564 top level 357 path @/.snapshots/73/snapshot/gnu
ID 363 gen 30997 top level 257 path @/.snapshots/74/snapshot
ID 364 gen 30997 top level 257 path @/.snapshots/75/snapshot
ID 366 gen 30997 top level 257 path @/.snapshots/76/snapshot
ID 367 gen 30997 top level 257 path @/.snapshots/77/snapshot
ID 368 gen 30997 top level 257 path @/.snapshots/78/snapshot
ID 370 gen 30997 top level 257 path @/.snapshots/79/snapshot
ID 371 gen 30871 top level 257 path @/.snapshots/80/snapshot
ID 372 gen 30871 top level 257 path @/.snapshots/81/snapshot
ID 373 gen 24146 top level 257 path @/.snapshots/82/snapshot
ID 374 gen 24155 top level 257 path @/.snapshots/83/snapshot
ID 375 gen 30997 top level 257 path @/.snapshots/84/snapshot
ID 376 gen 30997 top level 257 path @/.snapshots/85/snapshot
ID 377 gen 30997 top level 257 path @/.snapshots/86/snapshot
ID 378 gen 30997 top level 257 path @/.snapshots/87/snapshot
ID 379 gen 26127 top level 257 path @/.snapshots/88/snapshot
ID 380 gen 26136 top level 257 path @/.snapshots/89/snapshot
ID 381 gen 30997 top level 257 path @/.snapshots/90/snapshot
ID 382 gen 30997 top level 257 path @/.snapshots/91/snapshot
ID 383 gen 30997 top level 257 path @/.snapshots/92/snapshot
ID 384 gen 30997 top level 257 path @/.snapshots/93/snapshot
ID 385 gen 30997 top level 257 path @/.snapshots/94/snapshot
ID 386 gen 30997 top level 257 path @/.snapshots/95/snapshot
ID 387 gen 51816 top level 257 path @/.snapshots/96/snapshot
ID 388 gen 30997 top level 257 path @/.snapshots/97/snapshot
ID 389 gen 30997 top level 257 path @/.snapshots/98/snapshot
ID 390 gen 30997 top level 257 path @/.snapshots/99/snapshot
ID 392 gen 29836 top level 390 path @/.snapshots/99/snapshot/gnu
ID 393 gen 30997 top level 257 path @/.snapshots/100/snapshot
ID 394 gen 30997 top level 257 path @/.snapshots/101/snapshot
ID 395 gen 51832 top level 256 path @/gnu
ID 396 gen 43783 top level 257 path @/.snapshots/102/snapshot
ID 397 gen 30997 top level 257 path @/.snapshots/103/snapshot
ID 398 gen 30997 top level 257 path @/.snapshots/104/snapshot
ID 399 gen 31448 top level 257 path @/.snapshots/105/snapshot
ID 400 gen 31457 top level 257 path @/.snapshots/106/snapshot
ID 401 gen 32550 top level 257 path @/.snapshots/107/snapshot
ID 402 gen 32559 top level 257 path @/.snapshots/108/snapshot
ID 403 gen 33482 top level 257 path @/.snapshots/109/snapshot
ID 404 gen 33491 top level 257 path @/.snapshots/110/snapshot
ID 405 gen 34905 top level 257 path @/.snapshots/111/snapshot
ID 406 gen 34914 top level 257 path @/.snapshots/112/snapshot
ID 407 gen 36429 top level 257 path @/.snapshots/113/snapshot
ID 408 gen 36438 top level 257 path @/.snapshots/114/snapshot
ID 409 gen 37553 top level 257 path @/.snapshots/115/snapshot
ID 410 gen 37562 top level 257 path @/.snapshots/116/snapshot
ID 411 gen 38143 top level 257 path @/.snapshots/117/snapshot
ID 412 gen 38152 top level 257 path @/.snapshots/118/snapshot
ID 413 gen 38835 top level 257 path @/.snapshots/119/snapshot
ID 414 gen 38844 top level 257 path @/.snapshots/120/snapshot
ID 415 gen 40087 top level 257 path @/.snapshots/121/snapshot
ID 416 gen 40096 top level 257 path @/.snapshots/122/snapshot
ID 417 gen 40972 top level 257 path @/.snapshots/123/snapshot
ID 418 gen 40981 top level 257 path @/.snapshots/124/snapshot
ID 419 gen 42575 top level 257 path @/.snapshots/125/snapshot
ID 420 gen 42588 top level 257 path @/.snapshots/126/snapshot
ID 422 gen 43200 top level 257 path @/.snapshots/127/snapshot
ID 423 gen 43213 top level 257 path @/.snapshots/128/snapshot
ID 424 gen 43758 top level 257 path @/.snapshots/129/snapshot
ID 425 gen 43771 top level 257 path @/.snapshots/130/snapshot
ID 426 gen 44392 top level 257 path @/.snapshots/131/snapshot
ID 427 gen 51816 top level 257 path @/.snapshots/132/snapshot
ID 428 gen 44378 top level 257 path @/.snapshots/133/snapshot
ID 429 gen 44412 top level 257 path @/.snapshots/134/snapshot
ID 430 gen 44928 top level 257 path @/.snapshots/135/snapshot
ID 431 gen 44963 top level 257 path @/.snapshots/136/snapshot
ID 432 gen 45291 top level 257 path @/.snapshots/137/snapshot
ID 433 gen 45323 top level 257 path @/.snapshots/138/snapshot
ID 434 gen 45471 top level 257 path @/.snapshots/139/snapshot
ID 435 gen 45996 top level 257 path @/.snapshots/140/snapshot
ID 436 gen 45996 top level 257 path @/.snapshots/141/snapshot
ID 437 gen 45996 top level 257 path @/.snapshots/142/snapshot
ID 438 gen 45996 top level 257 path @/.snapshots/143/snapshot
ID 441 gen 46240 top level 257 path @/.snapshots/144/snapshot
ID 444 gen 46238 top level 257 path @/.snapshots/145/snapshot
ID 445 gen 46369 top level 257 path @/.snapshots/146/snapshot
ID 446 gen 46644 top level 257 path @/.snapshots/147/snapshot
ID 447 gen 46700 top level 257 path @/.snapshots/148/snapshot
ID 448 gen 48208 top level 257 path @/.snapshots/149/snapshot
ID 450 gen 48467 top level 257 path @/.snapshots/150/snapshot
ID 451 gen 49740 top level 257 path @/.snapshots/151/snapshot
ID 452 gen 50099 top level 257 path @/.snapshots/152/snapshot
ID 453 gen 50206 top level 257 path @/.snapshots/153/snapshot
ID 454 gen 50513 top level 257 path @/.snapshots/154/snapshot
ID 455 gen 50528 top level 257 path @/.snapshots/155/snapshot
ID 456 gen 50896 top level 257 path @/.snapshots/156/snapshot
ID 457 gen 51107 top level 257 path @/.snapshots/157/snapshot
ID 458 gen 51321 top level 257 path @/.snapshots/158/snapshot
ID 461 gen 51580 top level 257 path @/.snapshots/159/snapshot
ID 462 gen 51829 top level 263 path @/var/lib/machines
ID 464 gen 51724 top level 257 path @/.snapshots/160/snapshot
ID 465 gen 51740 top level 257 path @/.snapshots/161/snapshot
ID 466 gen 51749 top level 257 path @/.snapshots/162/snapshot
I don’t really know what that means, but I suspect that might be the problem.
Thank you for helping me track this down. I know it is not a dir, but it sure does look and behave like one in everyday use I should probably learn to make better use of all btrfs features.
That was the first thing I tried, which didnt work since I’m on Aeon. I tried the same thing in a transactional-update shell which also didn’t work. Walking to the bus I realized it is probably because /.snapshots/72/snapshot is marked RO.
I am not marking this as solved just yet, but whenever I am using my computer next time I will make sure to try to mark 72 as RW and then remove the gnu subvolume.
I should probably learn to make better use of all btrfs features.