Hello,
RE: Suse 12.2 64 bit. A bit of a puzzle, during boot I’m catching a glimpse of an error message that goes by too quickly for me to get the gist of it. Looking at the system logs I’m finding numerous failure messages, 148391, in systemd-journalctl -a to be exact and 22030 in /var/log/messages. Dealing with failures during recent updates:
Sample search on “WebYaST/df-root/df_complex-free.rrd” |grep -i illegal |wc"
“Mar 3 16:39:36 jjjhome collectd[2876]: rrdtool plugin: rrd_update_r (WebYaST/df-root/df_complex-used.rrd) failed: WebYaST/df-root/df_complex-used.rrd: illegal attempt to update using time 1362357576 when last update time is 1362357576 (minimum one second step)”
Question(s):
Should these logs not be cleared out routinely/regularly? I can’t see the forest for the trees and need to thin things down so I can see what is happening.
What is the mechanism for clearing these logs and can it be done by hand if not automagically?
Does the update error being found mean anything? I’m not a total newbie but it looks like the cited failure(s) have to do with the update times being the same.
Well… Thank you. That helps my understanding greatly. Doesn’t fully educate me as to what is needed and what is not.
So as nearly as I can tell there are ::
dmesg
systemd-journalctl -a
to retrieve messages which may or may not be related to
/var/log/messages
/run/log/*/system.journal
Well anyway I seem to have fixed the original problem about spurious records of update failures by simply removing web-yast. But now I’m wading through the entire logging system. Any pointers to explain that?
Leave the logging as is. The various logfiles contain pointers to what is going wrong when things do go wrong. No way to save/rescue a borked system without them.
On 2013-03-05 05:56, jefferies wrote:
>
> vazhavandan;2532064 Wrote:
>> “Web YAST” is not installed by default. Do we need it ?
> Good point and I just removed it. But the question(s) remain about how
> to prune the logs to remove extraneous reports over the past several
> months.
Logs are not pruned.
For syslog type logs, they are “rotated”. Files are compressed as they
grow large, and they are later removed after so many days (months). This
is done by the logrotate cron job.
systemd implements its own log, and there have been reports of huge
growth. I don’t know how this one should be controlled.
–
Cheers / Saludos,
Carlos E. R.
(from 11.4, with Evergreen, x86_64 “Celadon” (Minas Tirith))
On 2013-03-05 17:43, dd wrote:
> On 03/05/2013 05:06 PM, vazhavandan wrote:
>> systemd logs don’t rollover by default
>
> then the adept system administrator will want to add that log to the
> cron /etc/cron.daily/logrotate
No, I think that’s not possible, different structure. IIRC, systemd logs
are binary. There was talk of a database, so that you can make queries
based on fields and such.
> and, it might be good if someone put a request into FATE for a
> reasonable default (if there is none already–i have no 12.x system
> running to check)…
I have a 12.1 using systemv mode, so I don’t know about this.
But the people that have noticed this problem should ASAP create a bugzilla.
–
Cheers / Saludos,
Carlos E. R.
(from 11.4, with Evergreen, x86_64 “Celadon” (Minas Tirith))
On 03/05/2013 05:54 PM, Carlos E. R. wrote:
> I think that’s not possible, different structure. IIRC, systemd logs
> are binary.
oh! sounds complicated…and bloated.
> There was talk of a database
at least they didn’t name it “registry”
me thinks we are having more and more ‘contributors’ with a fresh IT
degree whose school and professors have far more non-BSD/*nix
experience than all others put together…and, i do not think that
is a good thing.
On 2013-03-05 18:56, dd wrote:
> On 03/05/2013 05:54 PM, Carlos E. R. wrote:
>> I think that’s not possible, different structure. IIRC, systemd logs
>> are binary.
>
> oh! sounds complicated…and bloated.
It has advantages and disadvantages… as all in engineering.
If you are interested in a particular type of event you can select that
class and see only them, months back if saved. If you want to find
something that you don’t know how it is exactly classified… you are in
trouble.
I have not seen it (in systemd, mind). I have seen it elsewhere. I don’t
know if the tools are ready or finished - I guess not.
>> There was talk of a database
>
> at least they didn’t name it “registry”
Indeed :-))
But you are right, that is how Windows does logging, and it is a
nightmare, IMHO. I have seen four or five engineers simultaneously
looking on their machines where Windows had logged a particular event -
and failing. It took some days to find out… and not all of them did.
And it was relatively simple task: we were playing with directory
permissions per groups with AD and trying to find out why it worked or
not…
> me thinks we are having more and more ‘contributors’ with a fresh IT
> degree whose school and professors have far more non-BSD/*nix
> experience than all others put together…and, i do not think that is a
> good thing.
Me neither.
–
Cheers / Saludos,
Carlos E. R.
(from 11.4, with Evergreen, x86_64 “Celadon” (Minas Tirith))
On 03/05/2013 07:14 PM, Carlos E. R. wrote:
> If you are interested in a particular type of event you can select that
> class and see only them, months back if saved.
same with a (smaller) flat text file and grep (quicker, i bet).
On 2013-03-05 20:34, dd wrote:
> On 03/05/2013 07:14 PM, Carlos E. R. wrote:
>> If you are interested in a particular type of event you can select that
>> class and see only them, months back if saved.
>
> same with a (smaller) flat text file and grep (quicker, i bet).
Yes, that is the tool that unix/linux admins use. It was my job once,
locating problems in logs. There are some proprietary, some very
expensive, tools that grep unix type of text logs and put the contents
into searchable databases (and more, like generating alarms).
There are organizations that do want logs stored in databases. For them,
having the logs that way instead of converting them makes a lot of sense.
–
Cheers / Saludos,
Carlos E. R.
(from 11.4, with Evergreen, x86_64 “Celadon” (Minas Tirith))
systemd-journald.conf(5) says that by default the several journals can occupy 10% of the size of the respective file system.
On a terabyte file system that’s rather much to try and make use of:
SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMinFileSize=, RuntimeMaxUse=, RuntimeKeepFree=,
RuntimeMaxFileSize=, RuntimeMinFileSize=
Enforce size limits on the journal files stored. The options prefixed with System apply to the journal
files when stored on a persistant file system, more specifically /var/log/journal. The options prefixed
with Runtime apply to the journal files when stored on a volatile in-memory file system, more
specifically /run/log/journal. The former is used only when /var is mounted, writable and the directory
/var/log/journal exists. Otherwise only the latter applies. Note that this means that during early boot
and if the administrator disabled persistant logging only the latter options apply, while the former
apply if persistant logging is enabled and the system is fully booted up. SystemMaxUse= and
RuntimeMaxUse= control how much disk space the journal may use up at maximum. Defaults to 10% of the
size of the respective file system. SystemKeepFree= and RuntimeKeepFree= control how much disk space
the journal shall always leave free for other uses if less than the disk space configured in
SystemMaxUse= and RuntimeMaxUse= is available. Defaults to 5% of the size of the respective file system.
SystemMaxFileSize= and RuntimeMaxFileSize= control how large individual journal files may grow at
maximum. This influences the granularity in which disk space is made available through rotation, i.e.
deletion of historic data. Defaults to one eigth of the values configured with SystemMaxUse= and
RuntimeMaxUse=, so that usually seven rotated journal files are kept as history. SystemMinFileSize= and
RuntimeMinFileSize= control how large individual journal files grow at minimum. Defaults to 64K. Specify
values in bytes or use K, M, G, T, P, E as units for the specified sizes. Note that size limits are
enforced synchronously to journal files as they are extended, and need no explicit rotation step
triggered by time.
Sort of wondering what a rational max size might be.
It was not the file that i was too worried about but it was most probably causing my system to boot slower and slower by the day.
When i got rid of the 100MB file there was significant improvement in boot speed.
Currently i set it at SystemMaxFileSize=10M. I will need to investigate later as to how many days of data this 10M will hold.
Thanks, I’ll give it a try. The current file covers the last 2 months.
But one further question on these issues. Apparently there is also the /run/log/journal which my casual reading indicates is in an in-memory file system. If so would it not make sense to reduce the size? A very quick glance shows it as
ls -l /run/log/journal//
-rw-r----- 1 root root 14962688 Mar 7 13:21 /run/log/journal/013ee6618c4c24ade9c0f6e300000843/system.journal
which if it’s on disk is no biggie. But if it’s actually occupying memory would be an irritant. Ok, so I learned on a PDP-11 with 128K. and memory wastage was a capital sin. I do admit to being a curmudgeon about such.
From what i have read about systemd /run is generally used only when it is not able to write into /var/log/journal/ etc…
It does write logs into that folder for me thankfully.
Here it is
Enforce size limits on the journal files stored. The options prefixed with System apply to the journal files when stored on a persistent file system, more specifically /var/log/journal. The options prefixed with Runtime apply to the journal files when stored on a volatile in-memory file system, more specifically /run/log/journal. The former is used only when /var is mounted, writable and the directory /var/log/journal exists. Otherwise only the latter applies.
