I had a problem with this last week the btrfs-balance ran for two days. I think I know why it was i problem. The disk space was low, so I have tidied up the disk, and this time the balance ran in five minutes and was finished. I would like to see the log so I can understand a little more about what have been done and so on. But when i run:
dagr@opensuse:~> systemctl status btrfs-balance.timer
● btrfs-balance.timer - Balance block groups on a btrfs filesystem
Loaded: loaded (/usr/lib/systemd/system/btrfs-balance.timer; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/btrfs-balance.timer.d
└─schedule.conf
Active: active (waiting) since Sun 2020-01-12 16:04:31 CET; 16h ago
Trigger: Mon 2020-01-20 00:00:00 CET; 6 days left
Docs: man:btrfs-balance
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
It seems the log output is not possible to read. Is it anything to do to get those logs?
Timers only start other services. You probably want output of btrfs-balance.service instead.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
By default Leap 15.1 does not use persistent journal, so you only get information since last boot (and even this may be truncated if there is not enough space). You need to check /var/log/messages* that get copy of journal output.
If you execute “systemctl status xxx” with a “normal” user then, you’ll often see the message“Warning: Journal has been rotated … ”.
If, however, you execute the command with either the user “root” or, by means of “sudo”, the warning message, due to a lack of system privilege, will not occur.
**@dagring:
**
With the user “root”, execute “journalctl | grep ‘btrfs-balance’ ” to inspect for all the entries related to the systemd btrfs-balance service and timer in the systemd journal.
# journalctl | grep 'btrfs-balance'
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: Dumping filters: flags 0x6, state 0x0, force is off
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: METADATA (flags 0x2): balancing, usage=30
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: SYSTEM (flags 0x2): balancing, usage=30
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: Done, had to relocate 2 out of 24 chunks
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: After balance of /
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: Data, single: total=20.01GiB, used=19.66GiB
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: System, DUP: total=32.00MiB, used=16.00KiB
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: Metadata, DUP: total=1.00GiB, used=558.12MiB
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: GlobalReserve, single: total=54.86MiB, used=0.00B
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
Dez 23 21:55:46 xxx btrfs-balance.sh[1463]: /dev/sda3 86G 23G 63G 27% /
.
.
.
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: Dumping filters: flags 0x6, state 0x0, force is off
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: METADATA (flags 0x2): balancing, usage=30
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: SYSTEM (flags 0x2): balancing, usage=30
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: Done, had to relocate 2 out of 26 chunks
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: After balance of /
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: Data, single: total=22.01GiB, used=21.05GiB
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: System, DUP: total=32.00MiB, used=16.00KiB
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: Metadata, DUP: total=1.00GiB, used=614.95MiB
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: GlobalReserve, single: total=58.36MiB, used=0.00B
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
Jan 13 19:43:23 xxx btrfs-balance.sh[14036]: /dev/sda3 86G 24G 62G 29% /
#
Whenever possible,
The usual parsing tools like grep, tail and head should be avoided because when invoked, the entire journal file(s) need to be loaded and only then parsed.
When you use the “journalctl” commands, the operation is performed as the log file(s) are streamed.
So, for instance if you wanted to search for all log entries that are generated by btrfs-balance.service during the previous (not current) boot, the following can be run
journalctl -u btrfs-balance.service -b -1
And of course, if you wanted entries specific to the btrfs-balance.timer service, you would run the expected modified command.
You might also like to know that the journalctl command is applied to all accessible log file(s, so as long as you haven’t moved any log files you won’t have to be concerned that they’ve been rotated.
sudo journalctl -u btrfs-balance.service -b -1
[sudo] passord for root:
-- Logs begin at Wed 2020-01-08 01:06:28 CET, end at Sun 2020-01-19 17:53:19 CET. --
-- No entries --
It seems the logs is not there. Or I have misunderstood how to run the code? I have reboooted av couple of times because I thought the box was hanging in loop some how, when the btrfs-balance was running more than a day.
I had not noticed the change arvidjaar describes(My main system is an upgrade which seems to have retained the older setting), with a little research it looks like the change was made upstream in the main branch of systemd. I also don’t remember this being an issue with 15.1 when it first launched, I suspect this is a very recent new “feature.”
Boy, would I like to somehow find whatever discussion led to changing logging to be non-persistent, IMO it violates very common and well-known principles of both security and regular maintenance. I can’t even remember in my personal experience if I’ve ever come across any discussion that didn’t highlight the importance of collecting historical system data.
Anyway,
It looks like current LEAP documentation describes how to fix this, which I consider very, very bad setting
The documentation is a little hard to understand because IMO it’s written poorly(why all that unnecessary chaff above and below the setting that needs to be changed, and incidentally includes the next line which is not relevant?),
Just do as described which is to uncheck the following line
#Storage=auto
which can then be modified to the following instead
Storage=persistent
And then restart systemd-journald.service
systemctl restart systemd-journald.service
If you only have access to the current boot log,
Then just run the command without the “-b -1” option.
The “journalctl” option ‘-b -1’ means “Read the journal for the boot with the index -1”.
Either, omit the “-b [ID]±offset]” {–boot=[ID]±offset]} option, to search the entire systemd Journal or, list the boots in the systemd Journal with “journalctl --list-boots” and then, choose the boot, via ID or offset, to be inspected.
BTW, the option “-u UNIT or PATTERN” {–unit=UNIT|PATTERN} means “Show messages for the specified systemd unit UNIT (such as a service unit), or for any of the units matched by PATTERN.”
[HR][/HR]Also, despite the systemd Journal automatically taking care of it’s disk usage by means of system parameters, it pays to occasionally execute “journalctl --disk-usage” and “journalctl --verify” to check the Journal is behaving as expected – if not then, the option “–vacuum-time=” is useful cleaning up any misbehaviour which may have occurred …
Thanks for all help on this topic. I have learned a lot on how to check in on this process. I have changed the setting in journald.conf to get persistent logs. Btrfs-balance now runs some minutes, so it seems to function as it should.