Debugging Real World Issues and Best Practices

Every tool in this section works exactly as documented in isolation. Real storage problems tend to show up as something that doesn’t add up — a disk reported full when nothing seems to explain it, a filesystem that’s suddenly read-only, a volume group that vanishes after a reboot. This chapter walks through four genuinely common real-world scenarios, each pulling together tools from across this entire section, then closes with a consolidated checklist of the habits worth carrying forward.

Scenario 1: “No Space Left On Device” — But df Shows Plenty Free

You try to create a file, get a “no space left on device” error, and check df -h — which reports the filesystem is nowhere near full. This looks contradictory, and it’s a genuinely common trap: df by default reports space usage, not the other finite resource a filesystem has — inodes.

Recall from the Filesystems chapter: every file needs an inode, and a filesystem is created with a fixed number of them, decided at mkfs time. It’s entirely possible to run out of available inodes — meaning no new file can be created, ever, regardless of how much raw space remains — long before the space itself runs out, if a filesystem accumulates an enormous number of very small files.

Check inode usage specifically:

df -i

Filesystem     Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb1     3276800 3276800      0  100% /mnt/data

IUse% 100% with plenty of raw space free elsewhere is the exact signature of this problem. The usual cause is some directory accumulating a very large number of tiny files — a cache directory, session files, log rotation gone wrong — and the fix is finding and cleaning up whatever’s generating them, not adding more disk space, which wouldn’t help at all here.

Hunting for the culprit combines tools you already know if you are following entire course:

find /mnt/data -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head

This prints every file’s containing directory, counts how many files came from each one, and sorts the result — the directory with the highest count is very likely where the inode exhaustion is coming from.

Scenario 2: The Disk-Full Mystery, Revisited

The df And du chapter earlier in this section already covered this in depth: df reports a filesystem as full, but summing everything with du comes up far short, because a deleted file is still held open by a running process. Worth repeating here only as a reminder that this is one of the most common real-world storage tickets you’ll encounter, and the fix — lsof +L1 to find the offending open-but-deleted file, then dealing with (often restarting) the process holding it — is exactly the sequence covered there.

Scenario 3: A Filesystem Suddenly Mounted Read-Only

A service starts failing to write anything, and checking the mount reveals the filesystem has somehow become read-only, despite no one explicitly setting it that way:

mount | grep sdb1

/dev/sdb1 on /mnt/data type ext4 (ro,relatime)

That ro where you’d expect rw is the tell. This isn’t usually something anyone did on purpose — ext4 (and most journaling filesystems) will automatically force itself read-only when it detects internal corruption, specifically to prevent further writes from making the underlying problem worse. Confirm this is what happened by checking the kernel log, exactly as covered in the Mounting chapter’s dmesg addition:

dmesg -T | grep -i sdb1

A message resembling EXT4-fs error or EXT4-fs (sdb1): Remounting filesystem read-only confirms the kernel itself triggered this defensively.

The correct fix is running a filesystem check — fsck — which inspects and repairs filesystem-level inconsistencies, conceptually similar to what the journal is meant to prevent in the first place, but for damage the journal alone couldn’t fully resolve. Critically, fsck needs the filesystem unmounted first — running it against a mounted, active filesystem risks making things worse, not better:

sudo umount /mnt/data
sudo fsck /dev/sdb1

Follow its prompts — it will report and typically offer to fix any inconsistencies it finds. Once it completes cleanly, remount normally:

sudo mount /mnt/data

Scenario 4: An LVM Volume Group Missing After Reboot

Occasionally, after a reboot, a logical volume that was working perfectly before simply isn’t there — lsblk doesn’t show it, and mounting it fails because the device path doesn’t even exist. Before assuming data loss, check whether the volume group was actually activated at boot:

sudo vgs

If this shows nothing at all, or shows the volume group but not its logical volumes, the issue is very often that the volume group simply wasn’t activated automatically — a real but generally recoverable situation, not necessarily lost data:

sudo vgchange -ay

-ay activates all inactive volume groups the system can find. Follow up with lvs and lsblk to confirm the logical volume reappears, then mount it normally. If this becomes a recurring issue on a specific system, it’s usually worth checking that the disks holding the physical volumes are being detected reliably and consistently at boot time in the first place, rather than treating vgchange -ay as a permanent workaround to run manually after every restart.

A Troubleshooting Flow For Storage Issues

    flowchart TD
    A["Something's wrong with storage"] --> B{"df -h shows full?"}
    B -->|"Yes"| C{"du adds up to the same total?"}
    C -->|"No, way less"| D["lsof +L1<br/>deleted-but-open file"]
    C -->|"Yes, matches"| E["Genuinely full —<br/>clean up or extend"]
    B -->|"No, space looks fine"| F{"df -i shows inodes full?"}
    F -->|"Yes"| G["Hunt for a directory<br/>with too many small files"]
    F -->|"No"| H{"Mounted read-only<br/>unexpectedly?"}
    H -->|"Yes"| I["dmesg for errors,<br/>then fsck (unmounted)"]
    H -->|"No"| J["Check dmesg regardless —<br/>often the fastest real answer"]
  

Best Practices Recap

A few habits from across this section are worth restating together, since they’re the ones that actually prevent problems rather than just diagnosing them after the fact:

• lsblk before every destructive command — partitioning, mkfs, dd — no exceptions, every time, even when you’re confident.
• UUID over device path in /etc/fstab and anywhere else a disk needs referencing long-term — device letters aren’t guaranteed stable.
• Test with mount -a before rebooting on any /etc/fstab change — never assume an edit is correct just because it looks right.
• Check df -i alongside df -h as a matter of routine, not just when something’s already broken — inode exhaustion gives no other warning sign in advance.
• Choose xfs vs. ext4 deliberately, considering in advance whether you might ever need to shrink a volume — not something to reconsider after the fact.
• Keep /boot off LVM on any system using LVM for its main storage, per the reasoning in the LVM Concepts chapter.
• sync before disconnecting any removable media after a dd operation.
• dmesg -T is often the fastest real answer to “why did this just happen” — worth checking early in almost any storage investigation, not as a last resort.

What’s Next

With both the toolkit and the real-world troubleshooting instincts in place, Storage & Disks is genuinely complete. From here, the course moves to Services & Boot — systemd, the boot process, and journalctl — picking up several threads this section already referenced without fully covering, including /boot itself and the read-only-remount behavior just covered above.