Journalctl - some details missing.

Hi,

Having been recently introduced to Journalctl (ty Jay at Learnlinux TV et al) I have been monitoring /var/log/messages via the CLI with

sudo journalctl -f -a

This is set to follow the log and show me ‘all’ entries. This is great as it has helped me sort out a few non-fatal issues as well as learn more how the system works.

However, I am not seeing “all” entries as I would expect. For example, if, as non-sudo since sudo is logged, I run the mpv media play against a file then that is not logged. If I open leafpad and edit/save a file that is not logged. I found that in some cases I have to activate the application’s own log file file to capture logs.

I have scanned the man page but nothing is obvious to me about what “all” means. Is what is currently logged in /var/log/messages all there is to work with or are there other system tools (besides app’s own log files)?

Thanks.

I can not inform you about the details, but in general I would expect that with journalctl you see system logs. And a normal user that runs some application is not normally creating much logs on the system level. E.g. what you show as a user that tries to open a file that does not exist, that is simply a user error.

Imagine that all hundreds of users of a system use it extensive, how many logs do you expect then to be created? That would fill all disk space in due time.

And when a user application has the feature of creating logs in a log file (to be used by the user e.g. when he encounters some problem), that file will be in the users realm (somewhere in his/her home directory) and also most probably not in a format that journalctl will understand.

And I assume that “all” in this case means all the log entries that are available are listed, thus no filtering. Logs that are not available will obvious never be listed

I assume you are confused here and think that the -a option means “all entries”. This is not the case. Reading

man journalctl

you will find at the very beginning:

If called without parameters, it will show the full contents of the journal, starting with the oldest entry collected.

Thus it will show “all” by default.

And about the -a option:

-a, --all
Show all fields in full, even if they include unprintable characters or are very long.

which has nothing to do with the selection of entries to be shown.

Did you tinker with logging? My machine never had /var/log/messages. Everything here goes to /var/log/journal/ See ‘man journald.conf’.

Modern Linux gave up on System V logging more than a few years ago …

  • Currently, all
    system (and user) logging is handled by the systemd Journal – located in a subdirectory of ‘/var/log/journal/’ and are directly accessible only be the user “root”. - The only access allowed is via the systemd “journalctl” command – also applies to “normal” (non-system) users …
  • There are some user logging relics in ‘~/.local/share/’ but, they’re often antiquated and unreliable …

The default Journal time stamps displayed by the systemd tools are (accurate) local time – which are usually sufficient for most purposes – with the exception of working out why the system boot isn’t as fast as it should be …

I assume the OP has rsyslog active concurrently. That logs to /var/log/messages by default.

systemctl status rsyslog

Some well-known non-systemd distros would argue with that statement. >:)

Text-based logging (rsyslog, syslog-ng) still exists for those who want it.

Thanks for the clarification. I read the man page too fast and assumed too much :shame:

No, I did not tinker. Rsyslog is set by default on any 15.1 or 15.2 install I have done with LXDE, LXQt and KDE.

Yes.

chris@asus-roc:~> systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-03-03 07:48:10 AWST; 8min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 1265 (rsyslogd)
    Tasks: 5
   CGroup: /system.slice/rsyslog.service
           └─1265 /usr/sbin/rsyslogd -n -iNONE
chris@asus-roc:~> systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-03-03 07:48:10 AWST; 2h 30min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 1265 (rsyslogd)
    Tasks: 5
   CGroup: /system.slice/rsyslog.service
           └─1265 /usr/sbin/rsyslogd -n -iNONE

I guess I just need to disable rsyslog.service and the binary logs start in either /run/log/journal or /var/log/journal depending on journald.conf Storage setting.

Storage=
           Controls where to store journal data. One of "volatile", "persistent", "auto" and "none". If "volatile", journal log data will be stored only in memory, i.e.
           below the /run/log/journal hierarchy (which is created if needed). If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal
           hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable.
           "auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes.  "none"
           turns off all storage, all log data received will be dropped. Forwarding to other targets, such as the console, the kernel log buffer, or a syslog socket
           will still work however. Defaults to "auto".


I like BSD too :wink:

Now, do I want text based or binary logs?

Thanks all.

Correct.

I like BSD too :wink:

Now, do I want text based or binary logs?

Thanks all.

Choices, choices… :slight_smile:

Not so fast, please.

  • Also here – fairly new, fresh, Leap 15.2 installation on new hardware –

 > systemctl list-unit-files | grep -i 'syslog'
rsyslog.service                                                  enabled        
syslog.service                                                   enabled        
syslog.socket                                                    static         
 > 

We’ll need to check why the syslog services are enabled by default – something is relying on them …
[HR][/HR]Back soon – please, brew a perfect porcelain cup of tea in a perfect porcelain teapot …


 > apropos syslog
ipmievd (8)          - IPMI event daemon for sending events to syslog
pam_syslog (3)       - send messages to the system logger
pam_vsyslog (3)      - send messages to the system logger
rsyslog.conf (5)     - rsyslogd(8) configuration file
rsyslogd (8)         - reliable and extended syslogd
syslog (2)           - read and/or clear kernel message ring buffer; set console_loglevel
syslog (3)           - send messages to the system logger
syslog (3p)          - log a message
syslog (8)           - the system logging service
syslog.h (0p)        - definitions for system error logging
vsyslog (3)          - send messages to the system logger
 > 

  • rsyslogd
    ” is only vaguely related to “remote” – it’s “reliable and extended” with the following capability –

Support of both internet and unix domain sockets enables this utility to support both local and remote logging.

Digging deeper reveals that, syslog is still needed deeper down in the system logging architecture – the systemd Journal can be viewed as being merely a “syslog based management tool” …

  • The systemd Journal stores the system’s logs or messages in binary format …

[HR][/HR]So, the 64 dollar question –

  • Will the systemd Journal continue to function correctly if the syslog and rsyslog services are disabled
    ?

My personal view is – possibly

@kitman:

Please take a look at the man (8) syslog page – written by SUSE staff …

  • Pursuing the systemd services –

 # systemctl status syslog.service 
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-03-03 12:18:21 CET; 3h 49min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
  Process: 1175 ExecStartPre=/usr/sbin/rsyslog-service-prepare (code=exited, status=0/SUCCESS)
 Main PID: 1255 (rsyslogd)
    Tasks: 5
   CGroup: /system.slice/rsyslog.service
           └─1255 /usr/sbin/rsyslogd -n -iNONE

Mär 03 12:18:21 xxx systemd[1]: Starting System Logging Service...
Mär 03 12:18:21 xxx rsyslogd[1255]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.39.0]
Mär 03 12:18:21 xxx rsyslogd[1255]:  [origin software="rsyslogd" swVersion="8.39.0" x-pid="1255" x-info="http://www.rsyslog.com">
Mär 03 12:18:21 xxx systemd[1]: Started System Logging Service.
Mär 03 13:18:21 xxx rsyslogd[1255]: -- MARK --
 # 
 # systemctl status rsyslog.service 
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-03-03 12:18:21 CET; 3h 50min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
  Process: 1175 ExecStartPre=/usr/sbin/rsyslog-service-prepare (code=exited, status=0/SUCCESS)
 Main PID: 1255 (rsyslogd)
    Tasks: 5
   CGroup: /system.slice/rsyslog.service
           └─1255 /usr/sbin/rsyslogd -n -iNONE

Mär 03 12:18:21 xxx systemd[1]: Starting System Logging Service...
Mär 03 12:18:21 xxx rsyslogd[1255]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.39.0]
Mär 03 12:18:21 xxx rsyslogd[1255]:  [origin software="rsyslogd" swVersion="8.39.0" x-pid="1255" x-info="http://www.rsyslog.com">
Mär 03 12:18:21 xxx systemd[1]: Started System Logging Service.
Mär 03 13:18:21 xxx rsyslogd[1255]: -- MARK --
 # 
 # systemctl status syslog.socket
● syslog.socket - Syslog Socket
   Loaded: loaded (/usr/lib/systemd/system/syslog.socket; static; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 12:18:19 CET; 3h 54min ago
     Docs: man:systemd.special(7)
           https://www.freedesktop.org/wiki/Software/systemd/syslog
   Listen: /run/systemd/journal/syslog (Datagram)
 # 

Personally, I’m leaving the syslog services as they are – per default …

The syslog.service is an alias for rsyslog (ie same service)…and it is enabled for legacy purposes. You can happily stop and disable it if that is your wish.

Um…on my 15.2 which is an upgrade from 15.1, I don;t have a ‘syslog.service’ at all.

chris@asus-roc:~> sudo systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/


chris@asus-roc:~> systemctl list-unit-files | grep -i 'syslog'
rsyslog.service                                                        disabled 
syslog.socket                                                          static   


chris@asus-roc:~> sudo systemctl status syslog
Unit syslog.service could not be found.
chris@asus-roc:~>

Regardless, I disabled ‘rsyslog.service’ yesterday and the binary log file is being populated.

Regards.

That’s why you don’t have syslog. :wink: (You disabled rsyslog.service already)