Context

A storage bug had quietly caused a database behind one of the homelab's internal tools to lose its data on a pod restart — the kind of failure that's invisible until the exact moment something restarts. The proper fix wasn't just patching the immediate cause; it was replacing the underlying storage mechanism with something built for redundancy in the first place.

The problem, and the honest part of this story

An earlier attempt to fix a very similar symptom had been applied — and never actually verified by doing the one test that would have proven it worked: deliberately killing the pod and confirming the data survived. That's the real root cause of this whole episode. A fix that's never tested under the exact failure condition it claims to solve isn't really a fix yet, just a hypothesis.

The solution

A dedicated, replicated storage layer was added across the cluster's worker machines, with two copies of every piece of data kept on separate physical hosts — enough that losing one host still leaves a healthy copy, without demanding a third full replica of everything. A separate storage class, configured to never automatically delete its underlying data even if the higher-level object referencing it is deleted, was used specifically for the database — a deliberate extra safety margin for the one dataset that actually matters here.

The migration itself followed a strict discipline: take a fresh, verified backup; only then remove the old storage claim; let the system recreate it on the new, replicated storage; restore; and verify the restored content itself — table counts, a known identifier — not just a "command exited successfully" status.

The test that mattered most

After migrating, the database's pod was deliberately, forcefully deleted — on purpose, to prove the point. It came back in well under a minute, on the same replicated volume, with every bit of data intact. That's the test that had never been run the first time around, and it's the only thing that actually proves a storage fix works, as opposed to merely looking correct.

A smaller lesson along the way

A restore command that exits successfully doesn't guarantee anything actually got restored — one attempt reported success while silently restoring nothing at all, caught only by manually checking the database afterward rather than trusting the exit code. Worth remembering any time a "success" status is the only thing being checked after a data-recovery step.

Alternatives considered

• Expanding the existing storage mechanism's redundancy instead of replacing it outright — not viable; the mechanism in use had no redundancy model to expand in the first place.
• Three replicas instead of two — rejected for this cluster's size: with only two machines actually equipped for this new storage layer, a third replica wouldn't have added real resilience, just overhead.