systemd services & journalctl

On a server, the important programs (web servers, databases, SSH, your own API) run as services: background processes that start at boot, restart if they crash, and log somewhere sensible. On almost every modern distribution (Ubuntu, Debian, Fedora, RHEL, Arch) that job belongs to systemd, the first process the kernel starts (PID 1). You talk to it with systemctl, and you read its logs with journalctl.

💡 These commands need a real systemd system: a VM, a cloud server or a desktop install. Most Docker containers and WSL setups without systemd enabled will say "System has not been booted with systemd as init system". On WSL, add systemd=true under [boot] in /etc/wsl.conf and restart WSL.

Units#

systemd manages units, each described by a small text file. The most common types:

You can usually omit .service: systemctl status nginx means nginx.service.

Checking a service#

systemctl status ssh

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-10-01 09:12:44 IST; 1h 48min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 812 (sshd)
      Tasks: 1 (limit: 4558)
     Memory: 6.1M (peak: 7.9M)
        CPU: 212ms
     CGroup: /system.slice/ssh.service
             └─812 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Oct 01 09:12:44 web1 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Oct 01 09:12:44 web1 sshd[812]: Server listening on 0.0.0.0 port 22.
Oct 01 09:12:44 web1 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.

Read it like this:

• Loaded shows where the unit file is, and whether it's enabled (starts at boot).
• Active is the current state: active (running), inactive (dead), failed, or activating.
• Main PID, memory, CPU and the CGroup show every process belonging to the service.
• The last lines are the most recent log entries.

(The service is called ssh on Debian/Ubuntu and sshd on Fedora/RHEL.) Quick yes/no checks, which are handy in scripts:

systemctl is-active nginx      # active / inactive / failed
systemctl is-enabled nginx     # enabled / disabled / static / masked
systemctl is-failed nginx

Controlling services#

sudo systemctl start nginx       # start now
sudo systemctl stop nginx        # stop now
sudo systemctl restart nginx     # stop + start
sudo systemctl reload nginx      # re-read config without dropping connections (if supported)
sudo systemctl enable nginx      # start at boot
sudo systemctl disable nginx     # don't start at boot
sudo systemctl enable --now nginx    # enable AND start
sudo systemctl mask nginx        # make it impossible to start (even manually)
sudo systemctl unmask nginx

💡 Prefer reload after editing configs when the service supports it, and test the config first: sudo nginx -t, sudo sshd -t, sudo apachectl configtest. A typo plus restart equals downtime.

Note the distro difference: on Debian/Ubuntu, installing a service package usually starts and enables it right away. On Fedora/RHEL it doesn't, so you run sudo systemctl enable --now nginx yourself.

Listing units#

systemctl list-units --type=service              # loaded and active services
systemctl list-units --type=service --state=running
systemctl --failed                               # anything that failed
systemctl list-unit-files --type=service         # installed units and enabled state
systemctl list-timers                            # scheduled timers
systemctl cat nginx                              # show the unit file(s)

Writing your own service#

Say you have a small Python web app in /opt/myapp. Rather than leaving it in tmux or behind nohup, give it a unit file:

[Unit]
Description=My example web app
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Environment=PORT=8080
EnvironmentFile=-/etc/myapp/env
ExecStart=/opt/myapp/venv/bin/python app.py
Restart=on-failure
RestartSec=5

# A little hardening
NoNewPrivileges=true
ProtectSystem=full
PrivateTmp=true

[Install]
WantedBy=multi-user.target

What each part does:

• [Unit]: a description and ordering. After= sets ordering only; Wants=/Requires= set dependencies.
• [Service]:
  • ExecStart must be an absolute path. It isn't run through a shell, so there's no ~, pipes or &&. If you need those, use ExecStart=/bin/bash -c '...'.
  • User= runs the app as an unprivileged account (create it with sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp).
  • Environment= and EnvironmentFile= set variables. The - means "ignore if the file is missing".
  • Restart=on-failure restarts it if it crashes. always restarts it even after a clean exit.
  • Type=simple (the default) suits programs that stay in the foreground, which is what you want. Don't daemonise yourself.
• [Install]: WantedBy=multi-user.target means "start during normal boot" when enabled.

Load and start it:

sudo systemctl daemon-reload            # re-read unit files after any change
sudo systemctl enable --now myapp
systemctl status myapp

Never edit files in /usr/lib/systemd/system/ (or /lib/systemd/system/); package upgrades overwrite them. Put your own units in /etc/systemd/system/, and to tweak a packaged unit, use a drop-in override:

sudo systemctl edit nginx

That opens an editor for /etc/systemd/system/nginx.service.d/override.conf, where you add only the lines you want to change:

[Service]
Restart=always
LimitNOFILE=65535

systemctl edit runs daemon-reload for you; then sudo systemctl restart nginx. Use systemctl cat nginx to see the original plus overrides, and sudo systemd-analyze verify /etc/systemd/system/myapp.service to lint a unit.

Reading logs with journalctl#

systemd collects the stdout/stderr of every service, plus kernel and system messages, into the journal:

journalctl -u nginx                 # all logs for one unit (opens in a pager)
journalctl -u nginx -f              # follow live, like tail -f
journalctl -u nginx -n 50           # last 50 lines
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since today --until "15:00"
journalctl -p err -b                # priority error or worse, this boot
journalctl -b -1                    # previous boot (needs persistent journal)
journalctl -k                       # kernel messages (like dmesg)
journalctl -u myapp -o cat          # just the messages, no timestamps
journalctl --disk-usage

A typical failure investigation:

$ systemctl status myapp
× myapp.service - My example web app
     Active: failed (Result: exit-code) since Thu 2026-10-01 11:02:10 IST; 4s ago
    Process: 2210 ExecStart=/opt/myapp/venv/bin/python app.py (code=exited, status=1/FAILURE)

$ journalctl -u myapp -n 5 --no-pager
Oct 01 11:02:10 web1 python[2210]: Traceback (most recent call last):
Oct 01 11:02:10 web1 python[2210]:   File "/opt/myapp/app.py", line 3, in <module>
Oct 01 11:02:10 web1 python[2210]: ModuleNotFoundError: No module named 'flask'
Oct 01 11:02:10 web1 systemd[1]: myapp.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 11:02:10 web1 systemd[1]: myapp.service: Failed with result 'exit-code'.

The fix: install flask into the app's venv, then sudo systemctl restart myapp. If a unit restarts too often it hits the start limit. In that case, run sudo systemctl reset-failed myapp after fixing the cause.

Reading other users' and system logs requires sudo or membership of the adm or systemd-journal group. You'll learn more about the journal and /var/log in the logs lesson.

Boot, targets and power#

systemctl get-default                  # graphical.target (desktop) or multi-user.target (server)
sudo systemctl set-default multi-user.target
systemd-analyze blame | head           # which units slowed the boot
sudo systemctl reboot                  # or: sudo reboot
sudo systemctl poweroff

Common mistakes#

• Forgetting daemon-reload after editing a unit, so systemd keeps using the old version (it warns you in status).
• start without enable. The service works until the next reboot, then it's gone.
• Relative paths or shell syntax in ExecStart. Use absolute paths, or wrap the command in /bin/bash -c.
• Running your app as root. Set User= to a dedicated system user.
• Editing vendor units in /usr/lib/systemd/system. Use systemctl edit drop-ins instead.
• Daemonising inside the service (&, nohup, --daemon) with Type=simple. systemd thinks the program exited. Keep it in the foreground.

What's next#

Services store data and logs on disk. Next: disks and storage, covering how much space you have, what fills it, and how filesystems are mounted.