PDA

View Full Version : Display systemd journal's archived logs



pfedort
29th June 2015, 11:32 PM
I have systemd journal's archived logs that I need to view.
How can I do it?

This command is not worthwhile:

$ journalctl --since=yesterday

It showed me today's logs.

Peter H.S.
30th June 2015, 12:25 AM
If you want to display older messages in the _current_ journal, try:


journalctl --since -5d

The above shows the last five days worth of logs. Use "-1w" for the one week ago. "-2h" for two hours ago. etc.

The systemd time format is documented in:

man systemd.time
Not an easy read since all time related stuff tends to be complex. But the time abbreviations should be obvious.

The above assumes that you are using a recent Fedora version, since they have persistent journald storage enabled.

pfedort
30th June 2015, 01:30 AM
Did not worked, I tried this:


journalctl --since -5d


journalctl --since -1w

Now, shows today's and yesterday's log. Nothing else.

I have more logs. I have not remove them. I think they are archived:


$ ls -l /var/log/*/

system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001c5ba-000519acc5b3571d.journal
system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001fa65-000519ad0ff00bd2.journal
system.journal
user-1000@ee13e6436422421da906549125cdc0fa-000000000001c5b9-000519acc5b355e5.journal
user-1000.journal

Peter H.S.
30th June 2015, 11:30 AM
"systemctl" will automatically read and collate all files in the journal directory.

So this sounds very much like a log corruption problem that makes it skip old dates. Since it collates all files, a single file may "block" things for the rest. Moving that file may "unblock" systemctl's ability to read the rest of the journals.

This may find the problematic journal:

journalctl --verify

Try to move the corrupted file to another directory, and then try to use "systemctl" again.

Another option is to query specific files directly, using "--file".


journalctl --file /var/log/journal/MACHINE-ID*/<journal>

Corrupted journals may still be queried by "systemctl" and may still contain most data. In this case it may only be some index or data fields that are corrupted.

*machine-id is an unique number that is issued for your pc. You can find it by issuing a
hostnamectl

pfedort
1st July 2015, 11:29 PM
This may find the problematic journal:

journalctl --verify


journalctl --verify
PASS: /var/log/journal/MACHINE-ID/system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001c5ba-000519acc5b3571d.journal
PASS: /var/log/journal/MACHINE-ID/system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001fa65-000519ad0ff00bd2.journal
PASS: /var/log/journal/MACHINE-ID/user-1000@ee13e6436422421da906549125cdc0fa-000000000001c5b9-000519acc5b355e5.journal
PASS: /var/log/journal/MACHINE-ID/user-1000.journal
PASS: /var/log/journal/MACHINE-ID/system.journal

MACHINE-ID is the correct one.
Everything is allright!



Another option is to query specific files directly, using "--file".


journalctl --file /var/log/journal/MACHINE-ID*/<journal>


Systemd journal log files:


$ ls -1s /var/log/journal/*/
total 57344
8192 system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001c5ba-000519acc5b3571d.journal
8192 system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001fa65-000519ad0ff00bd2.journal
8192 system.journal
16384 user-1000@ee13e6436422421da906549125cdc0fa-000000000001c5b9-000519acc5b355e5.journal
16384 user-1000.journal


journalctl --file=system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001c5ba-000519acc5b3571d.journal
-- Logs begin at Seg 2015-06-29 20:05:25 WEST, end at Seg 2015-06-29 20:25:40 WEST. --
.................................................. .....................................
.................................................. .....................................
.................................................. .....................................

$ journalctl --file=system@bcaa2f2178c7450d9ab5bb5d1c496892-000000000001fa65-000519ad0ff00bd2.journal
-- Logs begin at Seg 2015-06-29 20:26:10 WEST, end at Seg 2015-06-29 23:02:30 WEST. --
.................................................. .....................................
.................................................. .....................................
.................................................. .....................................

$ journalctl --file=system.journal
-- Logs begin at Seg 2015-06-29 23:04:13 WEST, end at Qua 2015-07-01 23:05:39 WEST. --
.................................................. .....................................
.................................................. .....................................
.................................................. .....................................

$ journalctl --file=user-1000@ee13e6436422421da906549125cdc0fa-000000000001c5b9-000519acc5b355e5.journal
-- Logs begin at Seg 2015-06-29 20:05:25 WEST, end at Seg 2015-06-29 20:25:51 WEST. --
.................................................. .....................................
.................................................. .....................................
.................................................. .....................................

$ journalctl --file=user-1000.journal
-- Logs begin at Seg 2015-06-29 23:03:16 WEST, end at Qua 2015-07-01 23:11:15 WEST. --
.................................................. .....................................
.................................................. .....................................
.................................................. .....................................

I think systemd journal logs since June 26 to June 29 are gone...

Here is my journald.conf:


$ grep -v "#" /etc/systemd/journald.conf

Storage=persistent
SystemMaxUse=50M
SystemMaxFileSize=16M
RuntimeMaxUse=32M
RuntimeMaxFileSize=12M

Another question: I set SystemMaxUse to 50M but actually systemd journal log occupy 57M(57344). Is it normal?

Peter H.S.
2nd July 2015, 12:15 AM
Ok. This seems to simply be a matter of too small storage values in journald.conf. The older log files have simply been purged because you have chosen a size based log rotation scheme with very small values.

On my Fedora 22 the journald.conf is empty, relying on the systemd default value of 10% of the file-system for log use. So I assume you inserted those values.

If you want to keep logs for any reasonable length of time, you should really increase those values.

Another thing is, that I find it strange that the UID1000 logs are larger than the system logs, and that you managed two system journal log rotates withing a day. You should check the journals to see if some service are spamming the log files.

pfedort
2nd July 2015, 05:08 PM
Ok. This seems to simply be a matter of too small storage values in journald.conf. The older log files have simply been purged because you have chosen a size based log rotation scheme with very small values.

Maybe, you're right...


On my Fedora 22 the journald.conf is empty, relying on the systemd default value of 10% of the file-system for log use. So I assume you inserted those values.

You've assumed correctly.


If you want to keep logs for any reasonable length of time, you should really increase those values.

Perhaps I will increase those values but not too much. I want to keep systemd journal log's files the smallest possible.


Another thing is, that I find it strange that the UID1000 logs are larger than the system logs, and that you managed two system journal log rotates withing a day. You should check the journals to see if some service are spamming the log files.

I see the user's logs files are at the limit that I set: 16M. The system's logs files are at the limit of 8M which I did not set. Maybe the last one is the default value.

rccharles
2nd July 2015, 05:37 PM
Why have small size log file? What is file space for except to store data?

Robert

pfedort
2nd July 2015, 06:02 PM
Why have small size log file? What is file space for except to store data?

Robert

I'm using btrfs and I make snapshots regularly which occupies a lot of space.
I rather use space in snapshots than on logs.

Peter H.S.
3rd July 2015, 12:42 AM
I'm using btrfs and I make snapshots regularly which occupies a lot of space.
I rather use space in snapshots than on logs.

Personally I would raise the log size considerably, since I find it useful so. But I am not you, and I don't know your setup.

Here is my advice for running a system with small log file sizes:

1. In "journald.conf" :
Raise the "SystemMaxUse=" limit somewhat to 128M. Keep the SystemMaxFileSize=16M" since that will give you a 8-1 ratio between the two, which is a ratio journald defaults to otherwise. Perhaps combine with "MaxRetentionSec=1month" or whatever time-limit you find acceptable. That way the log files may never grow to the whole 128M, and you still have an acceptable amount of log info.

Set "Compress=true"
Set "SplitMode=none" This is because there will be some overhead in splitting your user logs into a separate log.


2. Only store logs with syslog level "warning" and above in severity level. (MaxLevelStore=warning).

When needing more logging-info for debugging a problem, just turn it on with:

systemd-analyze set-log-level debug

and turn it off afterwards by:

systemd-analyze set-log-level warning

Still set "SplitMode=none" and use compress.

pfedort
3rd July 2015, 08:17 PM
Personally I would raise the log size considerably, since I find it useful so. But I am not you, and I don't know your setup.

Here is my advice for running a system with small log file sizes:

1. In "journald.conf" :
Raise the "SystemMaxUse=" limit somewhat to 128M. Keep the SystemMaxFileSize=16M" since that will give you a 8-1 ratio between the two, which is a ratio journald defaults to otherwise. Perhaps combine with "MaxRetentionSec=1month" or whatever time-limit you find acceptable. That way the log files may never grow to the whole 128M, and you still have an acceptable amount of log info.

Set "Compress=true"
Set "SplitMode=none" This is because there will be some overhead in splitting your user logs into a separate log.

For a starting, I decided to limit the logs only by date. However, day to day, I'm going to evaluate their growth.


2. Only store logs with syslog level "warning" and above in severity level. (MaxLevelStore=warning).

When needing more logging-info for debugging a problem, just turn it on with:

systemd-analyze set-log-level debug

and turn it off afterwards by:

systemd-analyze set-log-level warning

I'm not going to follow this advice for now because I need info on the ssh connection that I set occasionally with my mother, to help her.

Here is my current journald configuration:


$ grep -v "#" /etc/systemd/journald.conf

[Journal]
Storage=persistent
Compress=yes
SplitMode=none
MaxRetentionSec=3week

I will be watching the logs to adjust its configuration if needed.

Thanks for your help!