A power outage and a provider switch — what broke, and why

What happened

A power outage hit while I was, coincidentally, also switching internet providers and replacing the home router/modem at the edge of the network. The timing made diagnosis harder than it needed to be — two unrelated events landing on the same day, each capable of causing exactly the symptoms observed.

After the restart, the Kubernetes cluster and its automation platform were visibly unhealthy, and once the new router was reconfigured, the portal itself started returning server errors on login.

Problem 1 — a hardcoded default gateway

The new router briefly came up on a different local subnet than the old one. That alone wouldn't have mattered, except the "default gateway" address was hardcoded, independently, in about a dozen different virtual machines across two separate infrastructure-as-code repositories — instead of being a single, centrally defined value referenced everywhere.

Traffic within the same subnet (cluster heartbeat traffic, direct machine-to-machine connections) was never affected — only traffic that actually needed to leave the subnet, like time synchronization and one authentication service doing an external lookup. The Kubernetes control plane refuses to start core components until its clock is properly synchronized — a deliberate safety behavior, not data corruption, but confusing to diagnose under pressure if you don't already know that's how it behaves.

Fix: reconfigure the new router to keep the same local address the old one had. Once time sync and DNS resolution came back, the cluster self-healed completely — no VM configuration had to be touched.

Lesson: a hardcoded gateway address, repeated across many resources instead of centralized in one place, has a much bigger blast radius than it looks like on any single day it works fine.

Problem 2 — a static DNS record pointing at a dynamic address

This second problem surfaced a few days after the initial power outage, once the internet provider switch itself actually took place. Once the cluster was healthy again, the portal's login page was still broken. The root cause: the firewall's outward-facing address, obtained dynamically from the home router's DHCP, had been hardcoded as a static internal DNS record — because it hadn't changed in months, that fragility was invisible until an actual provider switch forced a new address negotiation.

Fix: rather than updating DNS and re-checking every rule that might implicitly depend on that address, the simpler and more contained fix was to reconfigure the firewall to reclaim its previous address on the new connection. No DNS change ended up being necessary at all.

Lesson: the real problem wasn't a dynamic public IP address — it was an internal DHCP lease, mistakenly treated as if it were a stable, fixed address. A DNS record for something whose address isn't actually guaranteed stable is a fragility that "works" purely by coincidence, right up until the coincidence ends. The right long-term fix — a reserved address lease, or dynamic DNS — was noted as follow-up work rather than rushed through mid-incident.

A smaller, easily missed lesson

A ping that fails on a public-facing interface doesn't necessarily mean anything is down — many firewalls intentionally drop such probes by default as a security posture. A different, lower-level check (confirming the device actually responds on the local network segment) turned out to be the reliable way to verify presence, independent of whether ICMP happens to be allowed.

Takeaway

Infrastructure-as-code doesn't remove the fragility that comes from relying on a real, physical network at the edge of your control (an ISP-assigned address, a router that resets to factory defaults) — it just makes that fragility traceable and fixable in minutes instead of being a mystery.