aggie
September 18, 2023, 8:50pm
#1
I was reviewing journalctl log and I see this - specifically the line (see more of log below)
backup-rpmdb[3738]: ERROR!! can not backup RPM Database
It suggests “Maybe there is not enough disk space”, but there’s plenty.
---cut out of Kate editor
Sep 18 14:13:54 backup-rpmdb[3742]: cat: /usr/lib/sysimage/rpm/Packages.db: Input/output error
Sep 18 14:13:54 kernel: BTRFS warning (device nvme0n1p2): csum failed root 533 ino 9541247
Sep 18 14:14:03 backup-rpmdb[3746]: gzip: stdin: Input/output error
Sep 18 14:14:03 backup-rpmdb[3738]: ERROR!! can not backup RPM Database to /var/adm/backup/rpmdb.
Sep 18 14:14:03 backup-rpmdb[3738]: Maybe there is not enough disk space.
Sep 18 14:14:03 systemd[1]: backup-rpmdb.service: Deactivated successfully.
Sep 18 14:14:03 systemd[1]: Finished Backup RPM database.
Sep 18 14:14:03 systemd[1]: backup-rpmdb.service: Consumed 9.960s CPU time.
Sep 18 14:18:53 systemd[1]: Starting Backup /etc/sysconfig directory...
Sep 18 14:18:53 systemd[1]: backup-sysconfig.service: Deactivated successfully.
.
# ll /var/adm/backup/rpmdb/
total 803908
-rw-r--r-- 1 root root 118431384 Mar 10 2020 Packages-20200310.gz
-rw-r--r-- 1 root root 118515459 Mar 11 2020 Packages-20200311.gz
-rw-r--r-- 1 root root 124330983 Mar 13 2020 Packages-20200313.gz
-rw-r--r-- 1 root root 124509203 Mar 20 2020 Packages-20200320.gz
-rw-r--r-- 1 root root 130 Mar 23 2020 Packages-20200323.gz
-rw-r--r-- 1 root root 67664577 Apr 24 00:55 Packages.db-20230424.gz
-rw-r--r-- 1 root root 67670187 Apr 25 08:28 Packages.db-20230425.gz
-rw-r--r-- 1 root root 67317574 Apr 26 08:31 Packages.db-20230426.gz
-rw-r--r-- 1 root root 67356778 May 2 13:13 Packages.db-20230502.gz
-rw-r--r-- 1 root root 67379562 May 3 16:13 Packages.db-20230503.gz
-rw-r--r-- 1 root root 36 May 3 16:13 rpmdb_recent_md5
#
.
# systemd-analyze blame
9.868s backup-rpmdb.service <=== yep - it runs
7.707s wicked.service
[...]
#
.
Are there too much snapshots?
snapper list
aggie
September 18, 2023, 9:35pm
#3
Thanks … nope, not too many snapshots, and disk space is adequate:
# snapper list
# | Type | Pre # | Date | User | Used Space | Cleanup | Description | Userdata
------+--------+-------+--------------------------+------+------------+---------+--------------+--------------
0 | single | | | root | | | current |
151* | single | | Fri Jun 28 10:45:02 2019 | root | 14.55 MiB | | |
2479 | pre | | Fri Sep 15 20:16:41 2023 | root | 1.03 GiB | number | zypp(zypper) | important=yes
2480 | post | 2479 | Fri Sep 15 20:23:00 2023 | root | 3.39 MiB | number | | important=yes
2481 | pre | | Fri Sep 15 20:53:35 2023 | root | 480.00 KiB | number | zypp(zypper) | important=yes
2482 | post | 2481 | Fri Sep 15 20:53:44 2023 | root | 108.26 MiB | number | | important=yes
2495 | pre | | Mon Sep 18 14:11:47 2023 | root | 187.86 MiB | number | zypp(zypper) | important=no
2496 | post | 2495 | Mon Sep 18 14:11:52 2023 | root | 2.16 MiB | number | | important=no
#
.
# btrfs filesystem usage -T /
Overall:
Device size: 40.00GiB
Device allocated: 20.99GiB
Device unallocated: 19.01GiB
Device missing: 0.00B
Device slack: 0.00B
Used: 16.50GiB
Free (estimated): 23.08GiB (min: 23.08GiB)
Free (statfs, df): 23.08GiB
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 46.34MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single single single Unallocated Total Slack
-- -------------- -------- --------- -------- ----------- -------- -----
1 /dev/nvme0n1p2 19.96GiB 1.00GiB 32.00MiB 19.01GiB 40.00GiB -
-- -------------- -------- --------- -------- ----------- -------- -----
Total 19.96GiB 1.00GiB 32.00MiB 19.01GiB 40.00GiB 0.00B
Used 15.89GiB 622.34MiB 16.00KiB
.
.
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p2 40G 17G 24G 42% /
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 32G 49M 32G 1% /dev/shm
efivarfs 128K 40K 84K 33% /sys/firmware/efi/efivars
tmpfs 13G 2.1M 13G 1% /run
/dev/nvme0n1p2 40G 17G 24G 42% /.snapshots
/dev/nvme0n1p2 40G 17G 24G 42% /boot/grub2/i386-pc
/dev/nvme0n1p2 40G 17G 24G 42% /boot/grub2/x86_64-efi
/dev/nvme0n1p2 40G 17G 24G 42% /opt
/dev/nvme0n1p2 40G 17G 24G 42% /root
/dev/nvme0n1p2 40G 17G 24G 42% /srv
/dev/nvme0n1p2 40G 17G 24G 42% /tmp
/dev/nvme0n1p2 40G 17G 24G 42% /usr/local
/dev/nvme0n1p2 40G 17G 24G 42% /var
/dev/nvme0n1p1 500M 5.9M 494M 2% /boot/efi
tmpfs 6.3G 504K 6.3G 1% /run/user/1000
/dev/nvme0n1p3 394G 47G 347G 12% /home
#
What about as root:rpm --rebuilddb
Please use su -l to switch to root, no sudo
Also/usr/lib/base-scripts/backup-rpmdb
aggie
September 18, 2023, 10:15pm
#5
Well, that might have fixed it!! Thanks for the hints.
# /usr/lib/base-scripts/backup-rpmdb
cat: /usr/lib/sysimage/rpm/Packages.db: Input/output error
gzip: stdin: Input/output error
ERROR!! can not backup RPM Database to /var/adm/backup/rpmdb.
Maybe there is not enough disk space.
#
.
# rpm --rebuilddb
error: rpmdbNextIterator: skipping h# 55226
Header SHA256 digest: BAD (Expected 1f6a4da9bea03d2d4c3e16a9xxxxxxxxxxxxx25a404b4620d38a231e191bea7dd != c6d62241c719967a1727ef90xxxxxxxxxxxxxxd521344b83e04d04ef43b125d214)
Header SHA1 digest: BAD (Expected 9bef4bdc820dc9xxxxxxxxxxxx08dbf4f1482be2 != 01d85f98cff6345xxxxxxxxxxc2319427ae8124150)
#
# /usr/lib/base-scripts/backup-rpmdb
#
# ll /var/adm/backup/rpmdb/
total 299612
-rw-r--r-- 1 root root 67670187 Apr 25 08:28 Packages.db-20230425.gz
-rw-r--r-- 1 root root 67317574 Apr 26 08:31 Packages.db-20230426.gz
-rw-r--r-- 1 root root 67356778 May 2 13:13 Packages.db-20230502.gz
-rw-r--r-- 1 root root 67379562 May 3 16:13 Packages.db-20230503.gz
-rw-r--r-- 1 root root 37062023 Sep 18 17:07 Packages.db-20230918.gz <=== new one!
-rw-r--r-- 1 root root 36 Sep 18 17:07 rpmdb_recent_md5
#
This is file-system corruption. You can try moving away this file and copy from backup.
You may try filter warnings:
erlangen:~ # journalctl -q -g 'BTRFS warning'
Sep 10 06:30:04 erlangen kernel: BTRFS warning (device nvme1n1p2): qgroup rescan init failed, qgroup is not enabled
Sep 14 06:55:58 erlangen kernel: BTRFS warning (device nvme0n1p2): qgroup rescan init failed, qgroup is not enabled
erlangen:~ #
Presumably backup-rpmdb fails due to file system issues. Ignoring warnings is asking for more trouble.
I initially ran the two commands out of sequence, but seems it now works, manually.
Will it work tomorrow, next month … ?