Disks & storage

"No space left on device" is one of the most common production incidents. It breaks databases, deploys and even logins. In this lesson you'll learn how Linux sees disks, partitions and filesystems, how to check and investigate disk usage, and how to add and mount a new disk, including making it permanent with /etc/fstab.

Disks, partitions and filesystems#

• A block device is a disk: /dev/sda (SATA/SCSI/virtio-scsi), /dev/nvme0n1 (NVMe), /dev/vda (virtio, common in cloud VMs), or /dev/xvda (older AWS).
• A disk is split into partitions: /dev/sda1, /dev/sda2, or /dev/nvme0n1p1, /dev/nvme0n1p2 (NVMe adds a p). Modern systems use a GPT partition table.
• Each partition holds a filesystem: ext4 (the Debian/Ubuntu default), xfs (the RHEL default), btrfs (the Fedora Workstation default, with snapshots), or vfat (the EFI boot partition).
• A filesystem is attached to the directory tree at a mount point. There are no drive letters: a second disk might appear at /data or /mnt/backup.

Layers like LVM (logical volumes you can resize) and LUKS (encryption) can sit between partitions and filesystems.

lsblk: what disks do I have?#

lsblk

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda           8:0    0   40G  0 disk
├─sda1        8:1    0    1M  0 part
├─sda2        8:2    0    2G  0 part /boot
└─sda3        8:3    0   38G  0 part
  └─ubuntu--vg-ubuntu--lv
            252:0    0   38G  0 lvm  /
sdb           8:16   0  100G  0 disk

Here sdb is a new, empty 100 GB disk with no partitions or mount point yet. Add -f to see filesystems and UUIDs:

lsblk -f

NAME                      FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1
├─sda2                    ext4        1.0            1b9c3f6e-5d2a-4c1e-9a77-0e4c1d2f8a10      1.6G    11% /boot
└─sda3                    LVM2_member LVM2 001       Jm2xQe-...
  └─ubuntu--vg-ubuntu--lv ext4        1.0            7f0e2d51-8c3b-4f6a-b1d2-3e9a5c7b4d21     21.4G    38% /
sdb

sudo blkid also prints UUIDs, and sudo fdisk -l shows detailed partition tables.

df: how full are my filesystems?#

df -h

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              393M  1.2M  392M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   38G   14G   22G  38% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
/dev/sda2                          2.0G  180M  1.6G  11% /boot
tmpfs                              393M   12K  393M   1% /run/user/1000

• -h gives human-readable sizes; -T adds the filesystem type; df -h /var reports just the filesystem that holds /var.
• tmpfs filesystems live in RAM and vanish at reboot.
• Ext4 reserves about 5% for root by default, so Used + Avail can be less than Size.

Inodes can run out too. Millions of tiny files (cache, sessions, mail queues) can exhaust inodes while plenty of bytes are free. You'll still get "No space left on device":

df -i /

du: what is using the space?#

df tells you that a disk is full; du tells you what filled it. Let's create some test data:

mkdir -p project/{logs,src,cache}
head -c 5M /dev/zero > project/logs/app.log
head -c 2M /dev/zero > project/cache/blob.bin
head -c 300K /dev/zero > project/src/main.py
du -sh project
du -h --max-depth=1 project | sort -h

7.4M	project
304K	project/src
2.1M	project/cache
5.1M	project/logs
7.4M	project

• -s gives a summary (one total), -h human-readable sizes, and --max-depth=1 (or -d 1) one level of subdirectories.
• sort -h understands K, M, G suffixes, so the biggest items end up at the bottom.

The classic "who's eating my disk?" hunt on a server:

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10   # -x: stay on this filesystem
sudo du -xh --max-depth=1 /var | sort -h | tail -5            # then drill into the biggest one

Usual suspects are /var/log (logs), /var/lib/docker (images and containers), /var/lib/mysql or /var/lib/postgresql (databases), /var/cache/apt (sudo apt clean), old kernels (sudo apt autoremove), ~/.cache, and core dumps.

Find large individual files with find:

find . -type f -size +1M -exec ls -lh {} + | awk '{print $5, $9}'

2.0M ./project/cache/blob.bin
5.0M ./project/logs/app.log

💡 ncdu (sudo apt install ncdu) is an interactive, navigable du, the fastest way to explore a full disk.

The deleted-but-open file trap

If df says the disk is full but du doesn't add up, a process may still be writing to a file you deleted (commonly a huge log removed with rm). The space is only freed when the process closes it:

sudo lsof +L1          # open files with a link count of 0 (deleted)

Restart that service to release the space. Next time, truncate instead of deleting a live log: sudo truncate -s 0 /var/log/app.log (or : > file).

Adding a new disk#

Suppose you attached the 100 GB sdb from above (a cloud volume, say). The steps are partition, format, mount, then make it permanent.

⚠️ Double-check the device name with lsblk first. These commands destroy any data on the device you point them at.

1. Partition it

sudo parted /dev/sdb --script mklabel gpt mkpart data ext4 0% 100%
lsblk /dev/sdb

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sdb      8:16   0  100G  0 disk
└─sdb1   8:17   0  100G  0 part

(sudo fdisk /dev/sdb and sudo cfdisk /dev/sdb are interactive alternatives.)

2. Create a filesystem

sudo mkfs.ext4 -L data /dev/sdb1      # or: sudo mkfs.xfs -L data /dev/sdb1

3. Mount it

sudo mkdir -p /data
sudo mount /dev/sdb1 /data
df -h /data
sudo chown ada:ada /data             # a fresh filesystem is owned by root

findmnt /data or plain mount | grep sdb shows what's mounted where. Unmount with sudo umount /data. If you get "target is busy", something is using it: cd out of it, or find the culprit with sudo lsof +f -- /data or fuser -vm /data.

4. Make it permanent with /etc/fstab

/etc/fstab lists filesystems to mount at boot. Get the UUID with sudo blkid /dev/sdb1, then add a line:

# <filesystem>                            <mount point> <type> <options>        <dump> <pass>
UUID=3e1f9b2c-7a4d-4e58-9c61-2b8d0f5a7c93 /data         ext4   defaults,nofail  0      2

Test before you reboot. A broken fstab can drop the machine into emergency mode:

sudo umount /data
sudo systemctl daemon-reload     # systemd generates mount units from fstab
sudo mount -a                    # mount everything in fstab; errors appear now, not at boot
sudo findmnt --verify            # sanity-check fstab syntax
df -h /data

Swap#

Swap is disk space used as overflow when RAM is full. It's slow, but it prevents sudden out-of-memory kills. Check it with swapon --show and free -h. To add a 2 GB swap file:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

(On btrfs, swap files need extra steps. Fedora uses compressed RAM swap, zram, by default.)

Health and maintenance#

• sudo smartctl -a /dev/sda (package smartmontools) shows a physical disk's SMART health and error counters.
• sudo fsck -f /dev/sdb1 checks and repairs an unmounted filesystem. Never run it on a mounted one.
• Growing a cloud disk: resize the volume in the provider's console, then run sudo growpart /dev/sda 3 and sudo resize2fs /dev/sda3 (ext4) or sudo xfs_growfs / (xfs). With LVM, use sudo lvextend -r -l +100%FREE /dev/ubuntu-vg/ubuntu-lv.

Common mistakes#

• Running mkfs, parted or dd on the wrong device. Always check lsblk first; /dev/sda is often your system disk.
• Using /dev/sdX names in fstab. Use UUID=.
• Rebooting without mount -a after editing fstab, and finding an unbootable server.
• Deleting a live log file with rm and wondering why space didn't come back. Truncate it instead, or restart the writer.
• Forgetting inodes. Check df -i when "disk full" makes no sense.
• Mounting over a non-empty directory. The old files are hidden (not deleted) until you unmount.

What's next#

A lot of disk space goes to logs, and they're also your best debugging tool. Next: where logs live, how to search them, and how logrotate keeps them under control.