How a 7-year-old PCIe 3.0 M.2 SSD can be 57× faster than a brand-new PCIe 4.0 M.2 SSD in real-world use cases

The journal stops mid-entry. No kernel panic, no OOM dump, no shutdown message. The box is gone, and shortly after, a hardware watchdog resets it. Between June 15 and August 31, this happened to my desktop sixteen times: June 15, June 26, a cluster of three around July 21–22, August 1, 3, 4, 9 (twice), 11 (twice), 22 (twice), 24, and 31. Most ended with the watchdog; the rest ended with my finger on the power button.

The machine is a GMKtec EVO-X2 — Ryzen AI Max+ 395, 128 GB of unified memory, and one 2 TB Lexar NQ790: QLC, the cheapest per-gigabyte NAND on the shelf. SMART never flagged any of this. The drive is still in the machine today. This is the post-mortem of the summer, and of what came after: the QLC kept its job, a PCIe 3.0 SSD came out of a drawer to do the work that matters, and the migration came within one boot menu of undoing itself twice.

This post has two halves, because the story has two halves. The hardware half explains why two drives in the same M.2 slot behave like different species: four specs decide your real-world speed, and none of them is the one printed on the box. The software half is everything I learned and reviewed about filesystems and Linux while living on the slow one: a mount option that froze the kernel, TRIM cadence, Btrfs hygiene, zram sizing, kernel dials, and a scheduler choice most guides get wrong for this class of drive.

Part 1 — the hardware: same slot, different physics

M.2 is a shape, not a capability. Both my drives fit the same slot, both stickers promise thousands of megabytes per second, and under real workloads one is fifty-seven times faster than the other. Four factors did that.

Four bits per cell

Flash stores bits as voltage levels in a cell. SLC puts one bit there, TLC three, QLC four. Four bits means sixteen voltage levels, and telling them apart is slow: QLC cells tolerate roughly 1,000 erase cycles against 3,000 for TLC, and they cannot be rewritten in small increments at speed. Nobody sells you slow flash outright. The controller hides it: writes land in a portion of the NAND run in fast single-bit mode — a pseudo-SLC write cache — and get relocated to the cheap cells later.

A QLC drive is two drives. Warm cache, a write takes ~20 µs. Empty cache, every write lands on QLC directly at ~253 ms. That is the same drive, on the same day, reporting itself perfectly healthy to SMART.

The cache in front of the cells

The pseudo-SLC cache is the drive’s real performance, and it degrades from two directions. It shrinks as the drive fills — the fast region is partly dynamic, converted from capacity you are already using. And it clogs: the controller only knows a block is free when something tells it, which is TRIM’s job.

Btrfs writes copy-on-write: every write allocates a fresh block and frees the old one. My fstrim timer sat on the distro default, weekly. So CoW churned out freed blocks faster than TRIM reported them, the stale entries piled up in the cache, and under a desktop-plus-services load the cache emptied in 22 to 47 hours. Then every write costs 253 ms, the I/O queue grows faster than it drains — I watched it go from 450 to 6,192 requests deep over three hours — journald starves, timers stop firing, and the watchdog collects the machine. A btrfs snapshot that should take a millisecond takes 20 seconds. The I/O pressure baseline on this box sat at 42%, chronic, and sitting in the metrics the whole time.

The sticker numbers never see any of this. The QLC’s box says 7 GB/s. Under live load the same day, the drive moved 13 MB/s; during the storm that froze the box, 0.3 MB/s. A hard drive does 150 MB/s on a bad day. A quiet QLC still reads sequentially at 1–3 GB/s, which is exactly the job I eventually left it.

A drive without DRAM keeps its map in your RAM

Flash cannot be overwritten in place, so the controller maintains a map from logical blocks to physical cells. Drives with DRAM keep that map in their own memory: the Samsung 970 EVO Plus from 2019 carries 1 GB of it. The Lexar has none — Maxio MAP1602, DRAM-less — so it borrows 10 MB of host RAM through HMB, the Host Memory Buffer.

On this machine that means the drive’s translation map competes with everything else for unified memory, including a GPU that can claim 51 GiB of it. This is the budget stack: QLC cells, no DRAM, a borrowed map. The 2019 premium stack beats it on everything except price per gigabyte and the sequential reads nobody’s morning consists of. Seven years of progress went into capacity per dollar, and the rest of the stack quietly got cheaper.

The bus is the least important spec

PCIe 3.0 x4 tops out near 4 GB/s, PCIe 4.0 x4 near 8. The Samsung is Gen3-native and runs at Gen3 speed in the EVO-X2’s Gen4 slot, and not once has it mattered, because the media, not the bus, is the ceiling. What the sticker hides better than the bus is queue depth: spec-sheet throughput is measured with 32 commands in flight, streaming. A developer’s day is queue depth one — a keystroke, a shell prompt, a git status, a database wanting its write acknowledged.

Same box, same afternoon, live load, fio:

Fifty-seven times the synchronous random reads, 256× on fsync. A fifth of a second collapses to under a millisecond when a database wants its write acknowledged. The newer drive, on the newer bus, loses by two orders of magnitude to a drive old enough to have a drawer for a home.

That is the hardware half. Here is what it did to my summer.

The freezes, sorted

Sixteen freezes. After the fact they sort into four classes and a pile of self-inflicted wounds: nine taught me how the parts interact, seven have my fingerprints on them.

The write cache is a battery

The August 4 post-mortem connected three crashes in three days to a drive that had never erred. The weekly TRIM between the first crash and the second moved 446 GiB of stale blocks and took 74 minutes. On a healthy schedule that backlog never forms — my daily trims now move 50–100 GiB in 10–15 minutes.

August 11 was the same class wearing a disguise: a browser-history service had been crash-looping for 40 hours — 3,677 server restarts and 1,335 agent restarts per boot — and every restart hammered a drive that was 90% full. Two freezes in one day. The bug underneath was beautiful: the service passed mattn/go-sqlite3 parameters like _journal_mode=WAL to modernc.org/sqlite, which ignores them silently. No WAL, no busy timeout, SQLITE_BUSY storms, restart loop, dead SLC cache, frozen kernel. A crash loop is a storage workload.

The fix stack since then: daily fstrim at idle I/O priority, commit=300 on the QLC mounts — metadata commits every five minutes instead of every 30 seconds, roughly 10× less metadata write amplification — and a capped journal, because its writes compete for the same cache blocks.

Page cache is memory

August 9, twice in one night. My auto-commit daemon reads 260+ git repos during discovery, and the kernel charges ~16 GB of page cache to its cgroup. The cgroup has a 16 GB ceiling, so the kernel starts reclaiming — page cache, politely, with no OOM kill; the kill counter stayed at zero the whole time. The daemon immediately re-reads what was just evicted. Repeat: 27,312 boundary hits, 91% CPU, and system-wide memory pressure pinned at 95% until the watchdog fired.

Nothing ever ran out of memory. The memory was full of cache serving a loop that refuses to stop reading. The fix was boring: a soft limit below the hard one, so the kernel throttles before the cliff, and fewer discovery workers.

An mmap is a swap contract

August 22, twice in one night. I run a local LLM server that mmaps 21.6 GB of model weights — shared memory, and the kernel can evict shmem in exactly one way: swap. My swap is zram. At 22:25 the kernel’s OOM dump already showed zram at 100% of its 29.5 GB, which made the model permanently unevictable. Monitoring had alerted at 23:44 and 23:50. To Discord. Where nobody was. The journal stops at 00:27.

The forensics from the oomd era explain why this kept happening: under sustained pressure, a flat 60-second restart re-read 22 GB of weights every 2–3 minutes. Eight kills, roughly 180 GB of NAND reads in 20 minutes, the drive doing punishment work for a config decision. The model now restarts with exponential backoff instead of re-reading 21.6 GB every few minutes.

The averages hid it

August 31, the humbling one. The box froze with zram empty, memory available, and zero OOM kills — every threshold I monitored was comfortable. What actually happened: after a nine-day backup outage (the same freeze night had also killed my external disk shelf), every persistent timer caught up at boot at once. A terabyte-class backup re-send, two weekly scrubs, and an LLM that core-dumped on every startup attempt — three crashes, each re-paying a 21.6 GB cold read at 27–43 minutes apiece, the fourth still loading when the journal cut. Four full-disk readers on one QLC drive.

Memory pressure arrived in bursts: PSI avg10 over 50%, repeatedly, for two hours. My newest guard threshold watched the one-minute average, which never exceeded 3.93% the whole boot; the last sample before the freeze read 0.49%. Here is the part I am least proud of: I shipped that threshold after this exact class of incident, tested it against a synthetic 55% average, and never checked it against the real telemetry — which never produced anything close. Phantom protection, calibrated against a scenario I invented. I replaced it with a leaky bucket over the episodic spikes, because the bursts were the signal, and the bursts were there all along.

The ones with my fingerprints

• Two metadata-exhaustion freezes in June. Once for running btrfs balance and nix-collect-garbage concurrently on a 97% full filesystem. Once because my nightly GC had no idea BTRFS chunks existed: deleting files frees extents inside already-allocated chunks, and when the device hit 100% allocated with metadata at 91%, the transactions for the deletions themselves had nowhere to go.
• Three freezes in July were my own processes eating RAM: a nine-hour Crush coding session that peaked at 40.2 GB and wrote 118.6 GB, and a runaway bun test at 61 GB. The kernel I/O stack froze under my own user slice, both times.
• My emergency guard tripped seven times in eighty minutes on August 22 because it kept re-arming the load it was defending against. It stopped the LLM service but left its socket up, and systemd happily re-paid the 21.6 GB cold load on the next connection. The guard stops both now. A guard’s threshold has to be re-satisfiable by the thing it gates; mine was not.
• And one morning with the device at 0% unallocated and zram at 85%, I ran btrfs balance by hand. The scheduled balances had correctly skipped themselves that morning. The kernel died in two and a half minutes.

Part 2 — the software: everything above the drive

The hardware sets the ceiling; the software decides where between 7 GB/s and 0.3 MB/s you actually live. My git log from that summer reads like a fever chart: “disable NVMe discards at block layer to prevent data corruption”, “mitigate QLC NAND SLC cache exhaustion causing WDT crashes”, “force fstrim daily schedule to prevent SLC cache exhaustion regression”. This section is that log, explained.

The mount option that froze the kernel

My mounts carried discard variants — continuous TRIM, the textbook recommendation. On this drive it was poison. The iostat capture from July 8 shows why: 86 discards per second, each taking 253 ms at the device, the queue 71 deep, and Btrfs commit times of 17.8 seconds while the filesystem tried to push metadata through a controller busy doing NAND arithmetic. The corruption scan afterwards counted 91,561 checksum failures. Discards were the I/O workload starving everything else.

The fix sounds like heresy: nodiscard on every mount, TRIM moved to a daily fstrim timer running at idle I/O priority. Batch the work, schedule it when nothing else wants the drive, and keep it small enough that it never becomes a backlog: the weekly default had let 446 GiB of stale blocks pile up, and clearing that took 74 minutes. Daily runs trim the churn of one day — 50–100 GiB in 10–15 minutes. After the remount alone, with no file changed, the same sequential read went from 14.4 to 1,276 MiB/s. Eighty-nine times, from a mount option.

Three more options earn their place on the QLC mounts. commit=300: metadata commits every five minutes instead of every 30 seconds, roughly 10× less metadata write amplification, and a five-minute data-loss window I accept with daily snapshots. noatime, because every read updating a write timestamp is pure churn. And space_cache=v2, the modern free-space cache. Two things deliberately stay off: compress-force — upstream docs say the heuristics are supposed to decide, and plain compress=zstd already halves my nix store — and qgroups, which tax every transaction commit for accounting I do not consume.

Filesystems are policies, not speeds

Before moving /nix I benchmarked ext4, XFS, and Btrfs on the actual Samsung, under live load. The result was a wash: 4K random reads between 17.0k and 18.1k IOPS across all three, fsync within 3 ms of each other, run-to-run variance bigger than any filesystem difference. Anyone quoting a filesystem choice as a speed win on a workload like mine is quoting noise.

So I chose by policy. /nix is Btrfs for compression: zstd stores a 629-path sample at 1.89×, the nix project disabled store preallocation years ago specifically so compression works, and the store moved at half the space. ClickHouse’s telemetry — 180 GB a day of async writes nobody waits on — got a dedicated XFS partition: append-heavy, no copy-on-write, no snapshots to pay for. The hot databases are moving to a nodatacow subvol on the Samsung: in-place writes, no per-block checksums, fsync in the 1–2 ms range. That one carries a documented landmine — no-CoW holds only for extents not shared with a snapshot, so a single scheduled snapshot silently reverts the subvol to CoW. The policy says “never snapshot it”. I write policies for the version of me that clicks things at 2 a.m.

One Btrfs hygiene set rounds this out, because two June freezes came from the allocator, not the drive: df lies on Btrfs — free space lives in chunks, per type, and deleting files frees extents inside allocated chunks without returning anything to the device. At 100% allocation with metadata at 91%, the transactions for deletions themselves had nowhere to go. The counters now watch chunk allocation directly; garbage collection refuses to run below 5 GiB unallocated or above 90% metadata; a weekly bounded balance returns empty chunks in bites of ten; and a 10 GiB emergency reserve sits on the filesystem so a full disk never wedges the fix for being full.

zram is a memory strategy, and I had it backwards

There is no disk swap on this machine, on purpose: swapping to the QLC would punish RAM with the slowest writes in the box. Swap is zram — compressed pages in RAM — and everything about tuning it is counterintuitive.

I resized it three times before understanding the gauge. At 30% of RAM, the device sat at 97% fill as a steady state: 27.4 GiB of swapped pages compressing to 10.7 GiB physical at 2.6×, while 53% of memory was available and memory pressure read 0.26%. A perfectly healthy machine whose swap gauge said critical. Fill percentage is only a cliff signal with headroom. At 50% (~47 GiB) the same load reads 58%, the shmem cliff moves ~19 GiB further out, and the idle cost is zero — zram only allocates physical pages for data actually stored.

The compression level is a benchmark, not a belief: zstd level 1 gives 2.85× at 373 MiB/s, the kernel-default level 3 gives 2.90× at 334 MiB/s, level 5 gives 2.96× at 172 MiB/s — 48% slower for 2% more ratio. On 4 KiB pages the higher levels cannot find patterns to pay for their CPU. Level 1 it is.

vm.swappiness I got spectacularly wrong twice before reading the kernel docs. At 1 (“never swap”), the box OOM-crashed in May, because the kernel recycled page cache into disk I/O instead of compressing RAM. At 10 it still favored reclaiming file pages over zram, and August brought Btrfs writeback storms. At 150 the kernel prefers zram — ~370 MiB/s of RAM compression — over evicting page cache that faults back in at ~253 ms per QLC write. The docs are blunt: for zram, use 100+. That one number was the largest single I/O reduction on the box.

Then the confession. A few days ago I checked every current NixOS, Arch, and distro guide against this config: 20 of 23 domains aligned or better, most deviations deliberate and documented — and two zram sysctls had been quietly wrong the entire time. page-cluster defaults to 3, which pre-reads eight pages per swap fault: a seek optimization for spinning disks that burns CPU decompressing pages nobody asked for. watermark_boost_factor defaults to 15000 and force-reclaims on fragmentation hints, a known stutter source. Fedora, ChromeOS, and Pop!_OS ship zeros for both. My config beat the wikis on storage and was still running a 1990s readahead over compressed RAM.

The kernel dials that mattered

The rest of the vm.* block, each line with a freeze class behind it: dirty_ratio=5 with dirty_background_ratio=1 start writeback at ~940 MB and finish by ~4.7 GB, so flushes happen in small steady bites instead of bursts that monopolize the SLC cache. vfs_cache_pressure=150 prefers reclaiming dentry and inode cache — cheap, no disk I/O — over page cache. watermark_scale_factor=100, because the old value of 10 caused panic reclaim: nothing, nothing, then a sudden I/O storm. min_free_kbytes=2 GB keeps headroom for kernel and GPU allocations. And MGLRU runs with a 1,000 ms minimum TTL, the documented sweet spot for protecting hot pages under pressure.

The scheduler and the gauges

The conventional advice is scheduler none for NVMe — let the drive sort it out. That advice assumes a drive with one personality. A QLC that answers in 20 µs or 253 ms depending on cache state breaks the assumption so badly that the blk-cgroup cost model cannot calibrate against it at all. This box runs BFQ on the NVMe, deliberately: it prioritizes interactive I/O over bulk writers, and every service is tagged into tiers — sshd at the interactive tier, desktop next, services, databases, background, builds, and maintenance jobs like fstrim at idle priority with nice 10. When the drive chokes, the shell prompt is still first in line.

Then the gauges, because none of the fixes would have been findable without them. /proc/pressure (PSI) reports how much of the last 10 and 60 seconds some tasks — or all tasks — spent stalled on I/O or memory. The chronic 42% I/O baseline was in those files the entire summer. Two reading rules survived the post-mortems: watch some avg10 for bursts, because my 60-second averages never crossed 3.93% in the boot that froze, with the last sample at 0.49% four minutes before death; and act on episodes, not levels — the replacement guard counts avg10 spikes in a leaky bucket and trips on the count. The deploy script now refuses to ship when memory pressure passes 20% or zram sits above 95% with pressure rising. And one counter in SMART is worth more than any dashboard: unsafe shutdowns. Mine reached 58 by early August — 46% of 126 power cycles. Every freeze the watchdog collected added one. The drive was logging what the dashboards missed.

Reboot-ready is a boot-menu claim

With 4,414 of 4,414 store paths proven on the Samsung, I wrote “reboot-ready” in my notes. Then I checked the boot menu. The deploy before it had activated the new system without writing a boot entry: the build tool had aborted at test-activation over two failing services, and my deploy script’s rescue path reads that exact error as “activated anyway, failures cosmetic”. A reboot at that moment would have silently booted the previous day’s generation, undone the whole migration, and every health check would have stayed green through all of it. Two commands repaired it — point the system profile at the activated system, write the boot entry — and then the init path verified on the new drive.

The same failure class came back two days later, angrier. A parallel session deployed fourteen generations in the window before the reboot, the boot menu’s default advanced onto store paths that existed only on the QLC, and the 2 a.m. reboot hung before journald could even start: no logs, no panic, a machine waiting for a kernel that was no longer mounted. Twelve hours later I picked an “older” entry by hand, and it turned out to be the flip generation itself — the machine had been migrated all along. Generation numbers lie across a store swap. All fourteen dead entries are pruned now, and every future build lands directly on the Samsung store, so the drift class is structurally gone.

Every fix in this half is a line in a NixOS config. The diff is the documentation, the revert is one generation back, and the mount options are validated at eval time — an invalid combination dies in the build, not in an emergency shell at 2 a.m.

The fix was in a drawer

An old Windows laptop died a few years ago. Its Samsung 970 EVO Plus — 1 TB, TLC, PCIe 3.0 — sat in a USB-C enclosure ever since, BitLocker and all. In August I finally dealt with it: dislocker unlocked it read-only, rsync moved 178 GiB — 550,573 entries — onto the RAID1 pool at 148 MB/s, a second dry run verified zero differences, and the drive got wiped and installed into the EVO-X2’s free M.2 slot with 98% of its life left. Total hardware spend: one screwdriver. The receipts for that purchase are the table above.

Before moving anything I asked one question of every workload: when this I/O happens, who is waiting on it, synchronously? A keystroke, a shell prompt, a git push, a database fsync — someone is waiting. A model load, a backup stream, a telemetry write — nobody is. That splits the machine into three tiers: RAM takes the hot working sets, the Samsung takes every synchronous cold path, the QLC and the HDD pool take bulk streaming. A second question matters as much: does the workload push the stuff above it out of RAM? A 21 GB model load that evicts your shell’s page cache belongs on the bottom tier, next to everything else nobody waits on.

Moving the store

/nix went first: 129 GB of store that every process spawn queues behind, now served at 2.4 GB/s and 27 µs per cache miss instead of queuing on the QLC. The filesystem policy was decided by the benchmarks above; the store moved onto Btrfs at half the space, and the new pool was formatted with block-group-tree — a problem you solve once at mkfs time and never again.

The migration order has one step that bit: after the deploy, sync again. The deploy realizes new store paths onto the old disk after your initial rsync; skip the second sync and you reboot into an incomplete closure. Stage-2 init lives in /nix/store, which makes the mount neededForBoot — a flag I now spell from memory.

What the QLC keeps

Everything nobody waits on. 589 GB of AI models — cold loads are bandwidth, and the retry storm that used to re-read them is fixed with backoff and socket activation, not a faster disk. Steam, 106 GB. And the ClickHouse partition: 180 GB a day of async telemetry writes, going to the cheapest cells in the box on purpose.

One gap, written down: everything above backs up to a RAID1 pool in the same chassis. That survives a dead NVMe, not a house fire.

What runs now

Your turn

Six commands, five minutes:

sudo smartctl -a /dev/nvme0 | grep -i unsafe
btrfs filesystem df /
compsize /nix
cat /proc/pressure/io
cat /sys/block/nvme0n1/queue/scheduler
zramctl

The first prints your unsafe-shutdown count — mine had reached 58 by early August, and every hard reset since has added one. The second shows chunk allocation; metadata above 90% is the precursor that killed the June freezes. The third prints what your compression mount option actually saves; near 1.0, the option is decoration. The fourth: read some avg10, not just the minute averages — the bursts are the signal. The fifth tells you who arbitrates when the drive chokes; if it says none and your drive has two personalities, you now know where the dial is. The sixth shows your zram fill and compression ratio — a full zram with headroom is a gauge, a full zram without headroom is a countdown. And after any deploy that touches /nix, check the boot menu — the activation and the boot entry are two different things.

The full archive — a post-mortem for every one of the sixteen, plus every guard — lives in SystemNix. Your drive is probably fine. Check what you’re asking it to do.

And if your machines misbehave while every dashboard stays green, that is the kind of mess I get hired to untangle. Tell me about yours.