Reading System Logs
Programs on a Linux computer write down what happens in logs, the files in /var/log and systemd's journal; journalctl -u shows one service's lines, -f watches new ones arrive, and on a Mac log show and the Console app do the same job.
Journeyman · Unix · Linux · Updated
After this lesson you can
- Find a service's log lines with journalctl -u and in /var/log, and read each line's time, program and message
- Watch a log as it's written with journalctl -f, and stop watching with Ctrl+C
Key takeaways
- A log is a computer's diary: each line has a time, the program that wrote it and what happened. /var/log holds the log files, and syslog has a bit of everything.
- Old logs are renamed and zipped (syslog.1, syslog.2.gz), so a log never grows forever. That's log rotation.
- journalctl reads systemd's journal: -u cron for one service, -n 5 for the last five lines, --since "1 hour ago" for recent ones.
- journalctl -f, like tail -f, prints each new line as it's written. Ctrl+C stops watching.
- Reading the system's logs needs the adm group, or sudo. On a Mac, log show reads the logs and log stream watches them.
The book club's website stopped answering last night. Sam restarted the server this morning and it came back, but nobody knows why it stopped, so it could happen again tonight. There was no error on any screen: a server has no screen. What it does have is logs. Every program on it writes a line whenever something happens, like a ship's logbook, and reading them is how you find out why. It's the first thing an administrator, a help-desk tech or a web developer checks when something went wrong while nobody was watching.
Where the logs are: /var/log
Log files live in /var/log. Yours will be a little different:
ls /var/log
README dist-upgrade kern.log.2.gz
alternatives.log dmesg kern.log.3.gz
alternatives.log.1 dmesg.0 kern.log.4.gz
alternatives.log.2.gz dmesg.1.gz landscape
alternatives.log.3.gz dmesg.2.gz lastlog
alternatives.log.4.gz dmesg.3.gz private
alternatives.log.5.gz dmesg.4.gz syslog
apt dpkg.log syslog.1
auth.log dpkg.log.1 syslog.2.gz
auth.log.1 dpkg.log.2.gz syslog.3.gz
auth.log.2.gz dpkg.log.3.gz syslog.4.gz
auth.log.3.gz dpkg.log.4.gz ubuntu-advantage-apt-hook.log
auth.log.4.gz dpkg.log.5.gz ubuntu-advantage.log
bootstrap.log fontconfig.log ubuntu-advantage.log.1
btmp journal unattended-upgrades
btmp.1 kern.log wtmp
chrony kern.log.1
A few worth knowing: syslog (read "sys-log") has a bit of everything, auth.log has every sign-in and every sudo, and dpkg.log and the apt folder have every install and update. A web server keeps its own folder here once it's installed, such as /var/log/nginx.
Why syslog.1 and syslog.2.gz? Every week, syslog is renamed syslog.1, a fresh one starts, and older copies are zipped (.gz) and finally deleted. That's log rotation, and it keeps logs from filling the disk. syslog itself is always the newest.
The system's logs aren't for everyone:
ls -l /var/log/syslog
-rw-r----- 1 syslog adm 3511751 Nov 9 14:34 /var/log/syslog
Its owner, syslog, writes it, the group adm may read it, and nobody else may. Sam is in adm (Users and Groups), so tail and grep work on it as on any text file: grep -i cron /var/log/syslog | tail -n 3 prints the last three lines about cron. Not in adm? Put sudo in front.
The journal: journalctl -u
Most Linux systems also keep one big log for every program, systemd's journal. systemd is the program that starts and looks after the others (Services: systemctl), and journalctl reads its journal. On its own, it shows thousands of lines from the oldest, so narrow it down: -u picks one service (systemd calls each one a unit), and -n 5 the last five lines.
journalctl -u cron -n 5
Nov 09 14:30:59 laptop systemd[1]: Stopped cron.service - Regular background pr>
Nov 09 14:31:11 laptop systemd[1]: Started cron.service - Regular background pr>
Nov 09 14:31:11 laptop (cron)[309]: cron.service: Referenced but unset environm>
Nov 09 14:31:11 laptop cron[309]: (CRON) INFO (pidfile fd = 3)
Nov 09 14:31:11 laptop cron[309]: (CRON) INFO (Running @reboot jobs)
lines 1-5/5 (END)
Read a line from the left: the date and time; laptop, the computer's name; cron[309], the program that wrote it and its process ID; then, after the colon, what happened. journalctl shows its lines in a pager, as less does: a > at the end means the line goes on (the right arrow key scrolls), and q quits.
These five lines tell a story: cron was stopped, then started again 12 seconds later. On this computer, that's WSL: Ubuntu in WSL stops a little while after you close its last window, and starts again when you open one. On the book club's server, a stop and a start at 3 a.m. would be the first clue. There, the web server's lines from last night are journalctl -u nginx --since yesterday; --since "1 hour ago" works too, and -p err shows only errors.
Ask for a service that never wrote anything, and the answer is short:
journalctl -u nginx
-- No entries --
nginx isn't installed on Sam's laptop. A misspelled name gives the same answer, so check the spelling first.
Watching live: journalctl -f
-f (follow) works like tail -f from The Start and End of a File: head and tail: it prints the newest lines, then waits and prints each new one as it's written. Try it with lines of your own. logger writes a line into the journal, and -t gives it a tag, a name to find it by. In one window:
journalctl -t bookclub-site -f
In a second window:
logger -t bookclub-site "Maya signed up for November."
logger -t bookclub-site "Diego posted a book review."
The first window shows each line the moment it's written:
Nov 09 14:34:57 laptop bookclub-site[3365]: Maya signed up for November.
Nov 09 14:34:58 laptop bookclub-site[3369]: Diego posted a book review.
Press Ctrl+C in the first window to stop watching. On the server, journalctl -u nginx -f while a member tries to sign in shows the problem as it happens.
On a Mac
A Mac keeps most of its logs in one unified log, read with the log command:
log show --last 10m
log stream
log show --last 10m prints the last ten minutes, and log stream watches like -f, until Ctrl+C. The Console app shows the same logs with a search box. /var/log is there too, with fewer files.
Check yourself
2 questions to try on the lesson page.
So what
When something went wrong and nobody saw it, the logs did: journalctl -u and the files in /var/log tell you what happened and when, and -f lets you watch it happen again.