Results 1 to 5 of 5

Thread: systemd - journald Settings likey need your attention

  1. #1
    Join Date
    Jun 2008
    Location
    Florida, USA
    Posts
    970

    Default systemd - journald Settings likey need your attention

    I upgraded to 13.2/KDE recently, then some time later noticed that my system log files were not rotating properly.
    After spending time in these two forum items
    https://forums.opensuse.org/showthread.php/503896-What-runs-logrotate-in-OpenSUSE-13-2

    https://forums.opensuse.org/showthre...up-suggestions

    I have concluded that for those running systemd journaling, whether you know it or not, your setup might need attention.

    In my case, I have upgraded since 11.x using zypperdup method. As a result, my installation has rsyslog installed.
    As of 13.2, if your do a clean install, rsyslog is no longer installed unless you manually install it. System logging is now handled by systemd-journald.
    If you are like me and update, logging is still managed by systemd-journald, but if rsyslog is installed in your system log messages are also forwarded to /var/log/messages in the 'traditional' manner.
    Systemd-journald accumulates messaging in a database, rather than a text file. By default, the journal database is volatile and deleted on shutdown. This volatility can be overridden and the database will accumulate in /var/log/journal/ if that directory exists.
    These man pages are helpful,
    Code:
    man journalctl 
    man journald.conf
    as is this Arch Linux systemd item.

    At issue for me is the default configuration for journald, found in /etc/systemd/journald.conf
    Code:
    # cat /etc/systemd/journald.conf.oldcjm
    #  This file is part of systemd.
    #
    #  systemd is free software; you can redistribute it and/or modify it
    #  under the terms of the GNU Lesser General Public License as published by
    #  the Free Software Foundation; either version 2.1 of the License, or
    #  (at your option) any later version.
    #
    # See journald.conf(5) for details
    
    [Journal]
    #Storage=auto
    #Compress=yes
    #Seal=yes
    #SplitMode=login
    #SyncIntervalSec=5m
    #RateLimitInterval=30s
    #RateLimitBurst=1000
    #SystemMaxUse=
    #SystemKeepFree=
    #SystemMaxFileSize=
    #RuntimeMaxUse=
    #RuntimeKeepFree=
    #RuntimeMaxFileSize=
    #MaxRetentionSec=
    #MaxFileSec=1month
    #ForwardToSyslog=yes
    #ForwardToKMsg=no
    #ForwardToConsole=no
    #TTYPath=/dev/tty10
    #MaxLevelStore=debug
    #MaxLevelSyslog=debug
    #MaxLevelKMsg=notice
    #MaxLevelConsole=info
    I assume the format of this file shows the default values in effect, although each of these settings are commented in this file.
    Key settings for this discussion include
    Code:
    #SystemMaxUse=         Sets the maximum disk usage for the folder /var/log/journal - NOT SET
    #MaxRetentionSec=      Sets the maximum 'time-depth' for the journal database - NOT SET
    #MaxFileSec=1month    Sets the maximum time span for each database component file
    #ForwardToSyslog=yes  As discussed, rsyslog was installed in my system when systemd was updated so the pass on to /var/log/messages is ON
    With current installs, systemd persistent journaling (/var/log/journal/ exists) is not enabled by default, but i found it was installed and running on my system.
    What I found was that this directory was now consuming 1.3GB, had monthly database component files all the way back to 2012 and when I ran journalctl the first time, I had system journaling all the way back to late 2011, likely when systemd was first introduced (? it does not seem that long ago, but).

    I have now modified /etc/systemd/journald.conf to have
    Code:
    SystemMaxUse=100M
    MaxRetentionSec=3month
    and then executed
    Code:
    systemctl daemon-reexec
    to restart systemd( a reboot would work as well).

    After about 6 hours runtime, I have not observed any changes to the old database files in /var/log/journal/. It is unclear from the available man pages if in fact systemd will 'clean-up and remove' those files older than 3 months. I plan to give it 24 hours, then begin to manually delete the old database files.

    My initial usage of journalctl to review logs finds it quite comprehensive, with a bit of a learning curve.
    I highly recommend users review journald and journalctl and the new capabilities they introduce.

    I also suggest a check for the existence and size of the /var/log/journal/ directory and take appropriate action.
    As I read and understand the documentation, simply deleting the /var/log/journal/ directory will stop systemd persistent journaling.
    Desk: i7-4790K Leap 15.1(x86_64)4.12.14-lp151.28.7-default KF5 59.0 Plasma 5.14.4 Qt 5.13.0
    Lap: HPDV7T i7 Leap 15.0(x86_64)4.12.14-lp151.28.7-default KF5 59.0 Plasma 5.14.4 Qt 5.13.0

  2. #2
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: systemd - journald Settings likey need your attention

    On 2015-04-21 22:06, cmcgrath5035 wrote:

    > My initial usage of journalctl to review logs finds it quite
    > comprehensive, with a bit of a learning curve.
    > I highly recommend users review journald and journalctl and the new
    > capabilities they introduce.


    It also has new problems that syslog solved decades ago.

    For example, the non-persistent journal in this laptop uses 200 Mb after an uptime of 20 days. A lot of that are lines from "leafnode" (nntp local server).

    Look:

    Code:
    minas-tirith:~ # date --rfc-3339=seconds ; journalctl | grep leafnode | wc -l ; date --rfc-3339=seconds
    2015-04-22 01:50:02+02:00
    66248
    2015-04-22 01:50:46+02:00
    Notice how slow the command is, just to tell me that the log has 66000 matching lines. Compare with the traditional method:

    Code:
    minas-tirith:~ # date --rfc-3339=seconds ; zgrep leafnode /var/log/news/*bz2 | wc -l ; date --rfc-3339=seconds
    2015-04-22 01:53:37+02:00
    48966
    2015-04-22 01:53:39+02:00
    minas-tirith:~ #


    This is because leafnode is (intentionally) reporting debug output. On a traditional syslog setup, I can route those messages to a different file (or files), and I can rotate, or simply delete, the old logs, with different settings for different things. Say, keep mail warning entries forever, delete mail debug entries after two weeks.

    This fine adjustment can't be done with systemd journal.

    I would like a command to purge the journal database of entries matching a pattern, to make it smaller. I could run this from cron.

    Notice that I don't have persistent systemd journal, just for the current uptime; yet it is "huge".

    --
    Cheers / Saludos,

    Carlos E. R.

    (from 13.1 x86_64 "Bottle" (Minas Tirith))

  3. #3
    Join Date
    Jun 2008
    Location
    Florida, USA
    Posts
    970

    Default Re: systemd - journald Settings likey need your attention

    Carlos - Yes, I failed to add to my comment that I have zero real time experience with journalctl. Clearly some aspects may not be right for seasoned users such as yourself.

    Status update - I waited 24+ hours and no changes were obvious in the /var/log/journal/ directory, so I manually deleted the existing database, the restarted with
    Code:
    # systemctl restart systemd-journald
    
    then
    
    # systemctl status systemd-journald
    systemd-journald.service - Journal Service
       Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
       Active: active (running) since Thu 2015-04-23 06:54:58 EDT; 5s ago
         Docs: man:systemd-journald.service(8)
               man:journald.conf(5)
     Main PID: 32004 (systemd-journal)
       Status: "Processing requests..."
       CGroup: /system.slice/systemd-journald.service
               └─32004 /usr/lib/systemd/systemd-journald
    
    Apr 23 06:54:58 PVE-LinuxSRV5 systemd-journal[32004]: Permanent journal is using 8.0M (max allowed 100.0M, trying to leave 4.0G free of 4.9G available → current limit 100.0M).
    Apr 23 06:54:58 PVE-LinuxSRV5 systemd-journal[32004]: Journal started
    The highlighted message shows that my modifications to /etc/systemd/journalctl.d are now in effect.

    Alas, I fat fingered cleaning old database files, removed the entire journal database so won't be able to see the 3 month retention rule kick in for, well, 3 months.

    It is likely I improperly restarted journald when chnages to the config were made, the proper way appears to be
    Code:
    # systemctl restart systemd-journald
    Desk: i7-4790K Leap 15.1(x86_64)4.12.14-lp151.28.7-default KF5 59.0 Plasma 5.14.4 Qt 5.13.0
    Lap: HPDV7T i7 Leap 15.0(x86_64)4.12.14-lp151.28.7-default KF5 59.0 Plasma 5.14.4 Qt 5.13.0

  4. #4
    Join Date
    Jun 2008
    Location
    Florida, USA
    Posts
    970

    Default Re: systemd - journald Settings likey need your attention

    An additional update.

    Although systemctl list-times showed logrotate.timer scheduled to run each day at 00:00:00 (12Midnight) and with AccuracySec=12h (12hours of 'slop' in the trigger event) I found that logrotate was not consistently running each day, when the PC awoke from overnight sleep (STR).

    Not sure why.

    So I modified /etc/systemd/system/timers.target.wants/logrotate.conf and restarted the logrotate.timer as well as the daemon

    Code:
    /etc/systemd/system/timers.target.wants # cat logrotate.timer
    [Unit]
    Description=Daily rotation of log files
    Documentation=man:logrotate(8) man:logrotate.conf(5)
    
    [Timer]
    #cjm OnCalendar=daily
    #cjm This should run logrotate at 10AM daily
    OnCalendar=*-*-* 10:00:00
    AccuracySec=12h
    # not supported by our systemd (210) yet
    #Persistent=true
    
    [Install]
    WantedBy=timers.target
    
    systemctl restart logrotate.timer
    systemctl daemon-reload
    
    
    systemctl list-timers
    NEXT                         LEFT     UNIT                         ACTIVATES
    Sun 2015-04-26 10:00:00 EDT  11h left logrotate.timer              logrotate.service
    Sun 2015-04-26 11:03:23 EDT  12h left systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
    n/a                          n/a      systemd-readahead-done.timer systemd-readahead-done.service
    The timer now triggers at 10AM, when computer is normally UP,
    the logrotate process is being run and is following directives in
    /etc/logrotate.d/syslog that affect log files such as /var/log/messages.
    Desk: i7-4790K Leap 15.1(x86_64)4.12.14-lp151.28.7-default KF5 59.0 Plasma 5.14.4 Qt 5.13.0
    Lap: HPDV7T i7 Leap 15.0(x86_64)4.12.14-lp151.28.7-default KF5 59.0 Plasma 5.14.4 Qt 5.13.0

  5. #5
    Join Date
    Jun 2015
    Location
    Toulouse
    Posts
    129

    Default Re: systemd - journald Settings likey need your attention

    Hello,

    Very interesting thread. I had for whatever reason a disturbed journalctl that showed only with the -b option journals three month old. Let me summarise what I did from the messages above:

    * changed /etc/systemd/journald.conf to have
    Code:
    SystemMaxUse=100M
    MaxRetentionSec=3month
    and
    Code:
    systemctl restart systemd-journald
    this set the journal with a normal way and removed the old entries

    jdd

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •