Logs & log management

When something goes wrong on a Linux machine (a service won't start, a disk fills up, someone tries to brute-force SSH) the answer is almost always in the logs. In this lesson you'll learn where logs live, how to read and search them efficiently, how to write your own entries, and how logrotate stops logs from filling the disk.

Two logging systems#

Modern distributions run two logging systems side by side:

1. The systemd journal (journald) collects everything: kernel messages, service stdout/stderr and syslog messages. It stores them in a binary, indexed format, and you read it with journalctl.
2. Text files in /var/log/, written by rsyslog (which receives messages from the journal) or by applications directly (nginx, MySQL, apt).

Ubuntu and Debian servers typically have both. Fedora and recent Debian desktop installs may be journal-only (no rsyslog), so journalctl is the one tool that works everywhere.

Important files in /var/log#

Most of these are readable only by root or the adm group. Use sudo, or add yourself to adm:

sudo tail -n 20 /var/log/auth.log
sudo tail -f /var/log/syslog                 # follow live; Ctrl+C to stop
sudo less +F /var/log/nginx/error.log        # follow inside less; Ctrl+C then q
last -n 5                                    # recent logins

Reading the journal#

You met journalctl in the systemd lesson. These are the options you'll use most:

journalctl -f                        # follow everything
journalctl -u nginx --since "10 min ago"
journalctl -p warning -b             # warnings and worse since boot
journalctl _COMM=sudo --since today  # filter by command name
journalctl -t CRON                   # filter by syslog identifier (tag)
journalctl --since "2026-10-01 09:00" --until "2026-10-01 10:00"
journalctl -u myapp -o json-pretty -n 1   # structured fields
journalctl -k -b -1                  # kernel log of the previous boot

Priorities, from most to least severe, are emerg, alert, crit, err, warning, notice, info and debug. -p err means "err and worse".

Persistence and size

If /var/log/journal/ exists, the journal survives reboots; otherwise it lives only in /run (RAM). Debian and Ubuntu create it by default; if journalctl -b -1 says there's no previous boot, create it with sudo mkdir -p /var/log/journal and run sudo systemctl restart systemd-journald. Control the size:

journalctl --disk-usage
sudo journalctl --vacuum-size=500M    # shrink the archive to 500 MB
sudo journalctl --vacuum-time=14d     # drop entries older than 14 days

Or set a permanent limit in a drop-in file:

[Journal]
SystemMaxUse=500M

Searching text logs#

Logs are just text, so everything from the text-processing lessons applies. Let's work on a sample web server access log:

cat > access.log <<'EOF'
203.0.113.5 - - [01/Oct/2026:10:00:01 +0530] "GET / HTTP/1.1" 200 5120
198.51.100.7 - - [01/Oct/2026:10:00:03 +0530] "GET /login HTTP/1.1" 200 1830
203.0.113.5 - - [01/Oct/2026:10:00:09 +0530] "POST /login HTTP/1.1" 302 0
192.0.2.44 - - [01/Oct/2026:10:01:15 +0530] "GET /wp-admin HTTP/1.1" 404 512
192.0.2.44 - - [01/Oct/2026:10:01:16 +0530] "GET /.env HTTP/1.1" 404 512
198.51.100.7 - - [01/Oct/2026:10:02:40 +0530] "GET /api/orders HTTP/1.1" 500 220
203.0.113.5 - - [01/Oct/2026:10:03:02 +0530] "GET /api/orders HTTP/1.1" 200 4096
192.0.2.44 - - [01/Oct/2026:10:03:05 +0530] "GET /phpmyadmin HTTP/1.1" 404 512
EOF
echo "requests: $(wc -l < access.log)"
echo "--- status codes"
awk '{print $9}' access.log | sort | uniq -c | sort -rn
echo "--- top clients"
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -3
echo "--- server errors"
grep '" 5[0-9][0-9] ' access.log

requests: 8
--- status codes
      3 404
      3 200
      1 500
      1 302
--- top clients
      3 203.0.113.5
      3 192.0.2.44
      2 198.51.100.7
--- server errors
198.51.100.7 - - [01/Oct/2026:10:02:40 +0530] "GET /api/orders HTTP/1.1" 500 220

The 192.0.2.44 client is probing for common vulnerable paths, a very typical sight on any public server. Other useful moves:

grep -c ' 404 ' access.log                      # count 404s
grep '10:0[12]:' access.log | wc -l             # requests in a time window
grep -v '^203\.0\.113\.5 ' access.log | head -2 # exclude one IP

3
3
198.51.100.7 - - [01/Oct/2026:10:00:03 +0530] "GET /login HTTP/1.1" 200 1830
192.0.2.44 - - [01/Oct/2026:10:01:15 +0530] "GET /wp-admin HTTP/1.1" 404 512

For a whole directory of logs, sudo grep -r "Out of memory" /var/log/ is often the fastest first step. grep -i, -C 3 (show context) and -E with alternation ('error|fatal|panic') are your best friends.

Compressed and rotated logs#

Old logs are rotated and compressed: syslog, syslog.1, syslog.2.gz, syslog.3.gz... The z* tools read .gz files directly:

gzip -k access.log            # -k keeps the original
zcat access.log.gz | head -2
zgrep -c ' 404 ' access.log.gz

203.0.113.5 - - [01/Oct/2026:10:00:01 +0530] "GET / HTTP/1.1" 200 5120
198.51.100.7 - - [01/Oct/2026:10:00:03 +0530] "GET /login HTTP/1.1" 200 1830
3

zless pages through compressed logs. For .xz files use xzcat and xzgrep, and for .zst files zstdcat.

Writing to the log: logger#

Your scripts and cron jobs can log to syslog/the journal with logger:

logger -t backup "Nightly backup finished: 1.2 GB in 4m10s"
logger -t backup -p user.err "Backup FAILED: disk full"
journalctl -t backup --since today

Oct 01 02:04:10 web1 backup[3121]: Nightly backup finished: 1.2 GB in 4m10s
Oct 01 02:04:10 web1 backup[3125]: Backup FAILED: disk full

That's much better than scattering ad-hoc files around, because everything lands in one searchable place, with timestamps and priorities.

logrotate: keeping logs under control#

logrotate runs daily (from a systemd timer, or cron on older systems), renames logs, compresses old ones, and deletes the oldest. Global defaults are in /etc/logrotate.conf, and each package drops its own rules into /etc/logrotate.d/. Here's a typical rule for your own app:

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp adm
    sharedscripts
    postrotate
        systemctl kill -s HUP myapp.service >/dev/null 2>&1 || true
    endscript
}

You can try logrotate safely as a normal user with your own state file. Here we force a rotation on a test log:

mkdir -p rot && printf 'line 1\nline 2\n' > rot/app.log
cat > rot/test.conf <<EOF
$HOME/rot/app.log {
    rotate 3
    compress
    missingok
    create 0640
}
EOF
/usr/sbin/logrotate -s rot/state -f rot/test.conf
ls rot
zcat rot/app.log.1.gz

app.log  app.log.1.gz  state  test.conf
line 1
line 2

To check a real config without changing anything, use debug mode: sudo logrotate -d /etc/logrotate.d/myapp. To force a rotation now, run sudo logrotate -f /etc/logrotate.d/myapp.

Centralised logging#

With more than a couple of servers, logging into each one to grep doesn't scale. Teams ship logs to a central system: rsyslog forwarding, Grafana Loki, the ELK/OpenSearch stack, or a cloud service (CloudWatch, Google Cloud Logging, Datadog). The concepts are the same (timestamps, priorities, fields), but the search is faster and covers every machine.

Common mistakes#

• Not checking the logs first. "It doesn't work" is usually explained in one line of journalctl -u service -n 50.
• Deleting big logs with rm while the app runs. The space isn't freed (see the disks lesson). Truncate the log, or fix rotation.
• Rotating without telling the app. It keeps writing to the renamed file. Use postrotate with a reload/HUP, or copytruncate.
• Logging secrets. Passwords, tokens and full card numbers in logs are a security incident waiting to happen.
• Unbounded journal or debug logging in production quietly filling the disk. Set SystemMaxUse= and sensible log levels.

What's next#

You can now run and troubleshoot the system. Time to automate it: the Advanced section starts with bash scripting basics.