systemd - journald Settings likey need your attention

I upgraded to 13.2/KDE recently, then some time later noticed that my system log files were not rotating properly.
After spending time in these two forum items
https://forums.opensuse.org/showthread.php/503896-What-runs-logrotate-in-OpenSUSE-13-2https://forums.opensuse.org/showthread.php/506799-systemd-and-journald-Looking-for-setup-suggestions

I have concluded that for those running systemd journaling, whether you know it or not, your setup might need attention.

In my case, I have upgraded since 11.x using zypperdup method. As a result, my installation has rsyslog installed.
As of 13.2, if your do a clean install, rsyslog is no longer installed unless you manually install it. System logging is now handled by systemd-journald.
If you are like me and update, logging is still managed by systemd-journald, but if rsyslog is installed in your system log messages are also forwarded to /var/log/messages in the ‘traditional’ manner.
Systemd-journald accumulates messaging in a database, rather than a text file. By default, the journal database is volatile and deleted on shutdown. This volatility can be overridden and the database will accumulate in /var/log/journal/ if that directory exists.
These man pages are helpful,

man journalctl 
man journald.conf

as is this Arch Linux systemd item.

At issue for me is the default configuration for journald, found in /etc/systemd/journald.conf

# cat /etc/systemd/journald.conf.oldcjm
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=login
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#TTYPath=/dev/tty10
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info

I assume the format of this file shows the default values in effect, although each of these settings are commented in this file.
Key settings for this discussion include


#SystemMaxUse=         Sets the maximum disk usage for the folder /var/log/journal - NOT SET
#MaxRetentionSec=      Sets the maximum 'time-depth' for the journal database - NOT SET
#MaxFileSec=1month    Sets the maximum time span for each database component file
#ForwardToSyslog=yes  As discussed, rsyslog was installed in my system when systemd was updated so the pass on to /var/log/messages is ON

With current installs, systemd persistent journaling (/var/log/journal/ exists) is not enabled by default, but i found it was installed and running on my system.
What I found was that this directory was now consuming 1.3GB, had monthly database component files all the way back to 2012 and when I ran journalctl the first time, I had system journaling all the way back to late 2011, likely when systemd was first introduced (? it does not seem that long ago, but).

I have now modified /etc/systemd/journald.conf to have

SystemMaxUse=100M
MaxRetentionSec=3month


and then executed

systemctl daemon-reexec

to restart systemd( a reboot would work as well).

After about 6 hours runtime, I have not observed any changes to the old database files in /var/log/journal/. It is unclear from the available man pages if in fact systemd will ‘clean-up and remove’ those files older than 3 months. I plan to give it 24 hours, then begin to manually delete the old database files.

My initial usage of journalctl to review logs finds it quite comprehensive, with a bit of a learning curve.
I highly recommend users review journald and journalctl and the new capabilities they introduce.

I also suggest a check for the existence and size of the /var/log/journal/ directory and take appropriate action.
As I read and understand the documentation, simply deleting the /var/log/journal/ directory will stop systemd persistent journaling.

On 2015-04-21 22:06, cmcgrath5035 wrote:

> My initial usage of journalctl to review logs finds it quite
> comprehensive, with a bit of a learning curve.
> I highly recommend users review journald and journalctl and the new
> capabilities they introduce.

It also has new problems that syslog solved decades ago.

For example, the non-persistent journal in this laptop uses 200 Mb after an uptime of 20 days. A lot of that are lines from “leafnode” (nntp local server).

Look:


minas-tirith:~ # date --rfc-3339=seconds ; journalctl | grep leafnode | wc -l ; date --rfc-3339=seconds
2015-04-22 01:50:02+02:00
66248
2015-04-22 01:50:46+02:00

Notice how slow the command is, just to tell me that the log has 66000 matching lines. Compare with the traditional method:


minas-tirith:~ # date --rfc-3339=seconds ; zgrep leafnode /var/log/news/*bz2 | wc -l ; date --rfc-3339=seconds
2015-04-22 01:53:37+02:00
48966
2015-04-22 01:53:39+02:00
minas-tirith:~ #

This is because leafnode is (intentionally) reporting debug output. On a traditional syslog setup, I can route those messages to a different file (or files), and I can rotate, or simply delete, the old logs, with different settings for different things. Say, keep mail warning entries forever, delete mail debug entries after two weeks.

This fine adjustment can’t be done with systemd journal.

I would like a command to purge the journal database of entries matching a pattern, to make it smaller. I could run this from cron.

Notice that I don’t have persistent systemd journal, just for the current uptime; yet it is “huge”.


Cheers / Saludos,

Carlos E. R.

(from 13.1 x86_64 “Bottle” (Minas Tirith))

Carlos - Yes, I failed to add to my comment that I have zero real time experience with journalctl. Clearly some aspects may not be right for seasoned users such as yourself.

Status update - I waited 24+ hours and no changes were obvious in the /var/log/journal/ directory, so I manually deleted the existing database, the restarted with

# systemctl restart systemd-journald

then

# systemctl status systemd-journald
systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
   Active: active (running) since Thu 2015-04-23 06:54:58 EDT; 5s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 32004 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─32004 /usr/lib/systemd/systemd-journald

Apr 23 06:54:58 PVE-LinuxSRV5 systemd-journal[32004]: Permanent journal is using 8.0M (max allowed 100.0M, trying to leave 4.0G free of 4.9G available → current limit 100.0M).
Apr 23 06:54:58 PVE-LinuxSRV5 systemd-journal[32004]: Journal started

The highlighted message shows that my modifications to /etc/systemd/journalctl.d are now in effect.

Alas, I fat fingered cleaning old database files, removed the entire journal database so won’t be able to see the 3 month retention rule kick in for, well, 3 months.

It is likely I improperly restarted journald when chnages to the config were made, the proper way appears to be

# systemctl restart systemd-journald

An additional update.

Although systemctl list-times showed logrotate.timer scheduled to run each day at 00:00:00 (12Midnight) and with AccuracySec=12h (12hours of ‘slop’ in the trigger event) I found that logrotate was not consistently running each day, when the PC awoke from overnight sleep (STR).

Not sure why.

So I modified /etc/systemd/system/timers.target.wants/logrotate.conf and restarted the logrotate.timer as well as the daemon

/etc/systemd/system/timers.target.wants # cat logrotate.timer
[Unit]
Description=Daily rotation of log files
Documentation=man:logrotate(8) man:logrotate.conf(5)

[Timer]
#cjm OnCalendar=daily
#cjm This should run logrotate at 10AM daily
OnCalendar=*-*-* 10:00:00
AccuracySec=12h
# not supported by our systemd (210) yet
#Persistent=true

[Install]
WantedBy=timers.target

systemctl restart logrotate.timer
systemctl daemon-reload


systemctl list-timers
NEXT                         LEFT     UNIT                         ACTIVATES
Sun 2015-04-26 10:00:00 EDT  11h left logrotate.timer              logrotate.service
Sun 2015-04-26 11:03:23 EDT  12h left systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
n/a                          n/a      systemd-readahead-done.timer systemd-readahead-done.service

The timer now triggers at 10AM, when computer is normally UP,
the logrotate process is being run and is following directives in
/etc/logrotate.d/syslog that affect log files such as /var/log/messages.

Hello,

Very interesting thread. I had for whatever reason a disturbed journalctl that showed only with the -b option journals three month old. Let me summarise what I did from the messages above:

  • changed /etc/systemd/journald.conf to have


SystemMaxUse=100M
MaxRetentionSec=3month 


and


systemctl restart systemd-journald

this set the journal with a normal way and removed the old entries

jdd