Performance monitoring

"The server is slow" is one of the most common tickets an engineer gets. Slowness almost always comes down to one of four resources: CPU, memory, disk I/O or network. In this lesson you'll learn a systematic, tool-by-tool way to find which one is the bottleneck, and which process is responsible.

The first 60 seconds#

When you log in to a slow machine, these commands give you the big picture quickly:

uptime                # load averages
dmesg -T | tail       # kernel errors: OOM kills, disk errors (needs sudo on some systems)
vmstat 1 5            # CPU, memory, swap, I/O over 5 seconds
free -h               # memory overview
iostat -xz 1 3        # per-disk utilisation and latency (package sysstat)
df -h                 # is a disk full?
top                   # who is using it all?

Let's look at what each one tells you.

CPU and load average#

uptime
nproc

 11:02:31 up 12 days,  3:47,  2 users,  load average: 3.80, 3.65, 3.70
4

The three numbers are the load average over the last 1, 5 and 15 minutes: the average number of tasks running or waiting to run (plus those stuck in uninterruptible I/O). Interpret it relative to the number of CPU cores (nproc):

Compare the three values to see the trend: 8.0, 4.0, 1.0 means load is rising right now, while 1.0, 4.0, 8.0 means it's calming down.

top

top

top - 11:02:31 up 12 days,  3:47,  2 users,  load average: 3.80, 3.65, 3.70
Tasks: 214 total,   3 running, 211 sleeping,   0 stopped,   0 zombie
%Cpu(s): 78.1 us,  6.2 sy,  0.0 ni, 14.9 id,  0.4 wa,  0.0 hi,  0.4 si,  0.0 st
MiB Mem :   7940.2 total,    412.6 free,   4120.8 used,   3406.8 buff/cache
MiB Swap:   2048.0 total,   2010.3 free,     37.7 used.   3819.4 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   2814 www-data  20   0  812344 210440  18236 R 182.4   2.6  41:12.08 php-fpm8.3
    912 mysql     20   0 2512880 980112  36412 S  95.0  12.1 310:45.77 mysqld
   1488 ada       20   0   18412   5120   3200 R   1.3   0.1   0:00.21 top

The %Cpu(s) line is the key:

A process can show more than 100% CPU: 182.4 means it's using almost two cores. Interactive keys: P sorts by CPU, M by memory, 1 shows per-core usage, c shows full command lines, k kills, q quits.

htop (sudo apt install htop / sudo dnf install htop) shows the same information with per-core bars, a tree view (F5) and easy filtering (F4). mpstat -P ALL 1 (sysstat) shows usage per core, which is useful for spotting a single-threaded program maxing out one core.

Memory#

free -h

               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       4.0Gi       412Mi        96Mi       3.3Gi       3.7Gi
Swap:          2.0Gi        37Mi       2.0Gi

Don't panic about a low free value. Linux deliberately uses spare RAM as page cache (buff/cache) to speed up disk access, and hands it back when programs need it. The number that matters is available: how much memory applications can still get without swapping.

Signs of real memory pressure:

• available close to zero;
• swap activity: non-zero si/so columns in vmstat (swap used on its own is fine; constant swapping in and out is not);
• the OOM killer: when memory runs out, the kernel kills a process (often your database). Check with sudo dmesg -T | grep -i -E 'killed process|out of memory' or journalctl -k | grep -i oom.

Find the memory hogs with ps aux --sort=-%mem | head or top (press M). RES (resident memory) is what a process actually uses; VIRT includes reserved-but-unused address space and is usually misleading.

vmstat: everything at a glance#

vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 3  0  38604 422508  91232 3394112    0    0    12    85 1210 2304 78  6 15  1  0  0
 4  0  38604 420112  91232 3394200    0    0     0    64 1388 2511 80  5 15  0  0  0
 5  2  38604 418960  91240 3394260    0    0  9120  4400 1502 2790 61  6  9 24  0  0

The first line is an average since boot; read the later ones.

• r: runnable tasks. Consistently greater than the core count means CPU saturation.
• b: tasks blocked on I/O.
• si/so: swap in/out per second. Should be ~0.
• bi/bo: blocks read from / written to disk.
• us sy id wa st: as in top. In the last line, wa jumped to 24 while bi spiked, which points to disk.

Disk I/O#

iostat -xz 1 3

Device            r/s     rkB/s   r_await     w/s     wkB/s   w_await  aqu-sz  %util
nvme0n1        812.00  51200.00     4.10    96.00   4400.00     9.80    4.10   97.60

(Abridged: the real output has more columns.) Focus on:

• %util: the share of time the device was busy. Near 100% on a single HDD means saturated. Fast NVMe/SSD drives handle parallel requests, so also check the latency.
• r_await / w_await: average milliseconds per read/write, including queueing. Single-digit ms is fine on SSD; tens to hundreds means trouble.
• aqu-sz: the average queue length.

To find which process is doing the I/O:

sudo iotop -o          # only processes doing I/O (sudo apt install iotop)
pidstat -d 1           # per-process disk reads/writes (sysstat)

And don't forget the simplest disk problem of all: a full filesystem (df -h, df -i). See the disks lesson.

Network#

ss -s                          # socket summary: how many connections?
ss -tn state established | wc -l
sar -n DEV 1 3                 # per-interface throughput (sysstat)
nload                          # live bandwidth per interface (sudo apt install nload)
sudo iftop -i eth0             # bandwidth per connection (sudo apt install iftop)

Watch for an interface close to its bandwidth limit, huge numbers of connections in TIME-WAIT or CLOSE-WAIT (an app not closing connections), and retransmissions (nstat -az | grep -i retrans).

Pressure stall information (PSI)#

Modern kernels report how much time tasks spend waiting for CPU, memory or I/O, which is a very direct "are we starved?" signal:

cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

some avg10=0.08 avg60=0.44 avg300=1.33 total=1945172559
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=0.00 avg60=0.01 avg300=0.00 total=48211
full avg10=0.00 avg60=0.00 avg300=0.00 total=41120

The avg10/avg60/avg300 values are percentages of time over 10 seconds, 1 minute and 5 minutes. some means at least one task was stalled; full means all non-idle tasks were. Sustained double-digit values are a real problem.

History: sar#

The tools above show now. For "what happened at 3 a.m.?", enable sysstat's collector, which records stats every 10 minutes:

sudo apt install sysstat
sudo systemctl enable --now sysstat     # on Debian/Ubuntu, also set ENABLED="true" in /etc/default/sysstat
sar -u              # CPU history for today
sar -r              # memory
sar -d              # disks
sar -q -f /var/log/sysstat/sa15     # load/run queue on the 15th (Fedora: /var/log/sa/sa15)

For fleets of servers, teams use a monitoring stack such as Prometheus + node_exporter + Grafana, Netdata, or a hosted service, with alerts on CPU, memory, disk space and latency.

A troubleshooting walkthrough#

1. uptime: load is 9 on a 4-core box, so something's queueing.
2. top: CPU is 30% us, 45% wa, so it's not compute; something is waiting on disk.
3. iostat -xz 1: nvme0n1 is at 99% %util with 80 ms w_await.
4. sudo iotop -o: a nightly mysqldump and a log-compression job are both hammering the disk.
5. Fix: reschedule one of them, and run the backup with nice -n 19 ionice -c3.

The pattern is always the same: load → which resource → which process → why.

Common mistakes#

• Panicking about "free" memory. Read "available".
• Reading load average without the core count. A load of 8 is idle for a 64-core server and terrible for a 2-core one.
• Only looking at CPU when the real problem is I/O wait, swapping, or a full disk.
• Trusting the first line of vmstat/iostat: it's the average since boot.
• Ignoring st (steal) on cloud VMs. Burstable instances (t-series) get throttled when they run out of CPU credits.
• No history. Install sysstat or monitoring before the incident.

What's next#

You can now watch a server's health. The final lesson brings everything together into a security and hardening checklist for any Linux server you run.