systemd and journald - Looking for setup suggestions

My system is 13.2/KDE, upgraded from 13.1.
In monitoring an issue with logrotate not running properly in 13.2 for my setup (here), I was introduced to fact that systemd - journalctl is running and is becoming the default logging application.
Since I migrated from 13.1, rsyslog was still operational as well so I just plain missed it.

In experimenting with journalctl, I found that I had some very old log info.
Then looking in /var/log/journal I found that in fact systemd has been logging (journaling) away since 2013 and it is all still there!

I guess it is time to get with the times and start managing this a bit more proactively.
After reviewing ‘man journald.conf’, I located my configuration

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=login
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#TTYPath=/dev/tty10
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info

Somewhat to my surprise, it appears that no size or time (retention) limits are set by default.
As I read it, the only directive active limits the duration of a journal file to 1 month.

Before I try editing the file to have

#MaxRetentionSec=3month

which should be more than adequate for the way I use log file, are the more comprehensive suggestions
on how journald should be configured?

Because I haven’t watched, the directory /var/log/journal is now 1.2GB, of which at least 1.1GB is likely useless information

Any best practices to review?

Maybe not answering your question directly, but
There have been a few presentations I’ve attended at the annual local regional Linux conference which have described collecting and organizing system logs for enterprise purposes, with the side effect of minimizing the size of logs stored on individual machines.

So, maybe you might think your old syslog data is “useless” but for many Network Admins it might actually be quite valuable if it can be analyzed. At the very least it provides historical baseline data, but analyzed differently might provide useful information.

The Basic Idea
System logs are one of the most useful unmined sources of information for the Network (and System) Administrator because of the variety and volume of events that are logged. But, utilizing that information is difficult. For Enterprise purposes, you generally want to be able not only to extract useful information from any machine in your network quickly using a single tool, but for various purposes you may even want to correlate searches against multiple machines to identify problems that may be affecting more than one machine simultaneously.

The Solution (Framework)

  • Collect and centralize the data. A centralized repository for all your machines should be set up, and logfiles of various types including syslogs should be copied or streamed to your Server. *NIX including openSUSE contain various native tools in their repositories for log shipping besides well-known general purpose utilities like rsync. Intermediate tools like Logstash might transform and tag your data, and key-value nodes like Redis can be used to not only transform but buffer your network streams improving performance and providing fault tolernance.
  • Various types of log analytical tools. There is a wide variety of possible tools which in part depends on the type of repository you set up. If still in their original syslog formats, you could apply the same tools you might use locally like syslog-ng and sysdig. Leading edge solutions might store your logfiles in a single database like MySql or Postgres (if you have SQL skills) or Hadoop style clusters where you can use special search tools.

The bottom line for each of your individual machines is that you can minimize the size of your locally stored syslog data while also improving your ability to look at all the syslogs in your network.

TSU