What the Rackspace Incident Taught Me: Website Recovery Is Not Kubernetes Control-Plane Recovery
A Rackspace Spot CA rotation left an Ingress Controller unable to reach the Kubernetes API and sent a live site into 503s. Here is what the incident, and the wider run of U.S. service outages, says about control planes and change verification.
A website being reachable proves only that one request completed. It does not prove that the control plane, certificate rotation, service discovery, or the next configuration change is healthy.
On November 22, 2026, my own jackywang.me site exposed that distinction. The DNS record had not changed. The WordPress Pod was healthy. The failure was between the Kubernetes Ingress Controller and the API Server: after a cluster CA rotation, the running controller still trusted the old CA and could no longer watch Services, EndpointSlices, Ingresses, and Secrets. The application returned 200 inside the cluster, while the public entry point returned 503.
The fix was deliberately small. I did not change DNS, rebuild WordPress, or delete certificates. I rolled the Ingress Controller so that the new process loaded the rotated CA. After the rollout, jackywang.me returned a 301 to www.jackywang.me, followed by HTTP 200. The backend endpoint remained ready throughout. Application recovery and control-plane recovery were separate facts.
The evidence chain
The incident was diagnosed from a short chain of evidence:
- Both apex and
wwwDNS records still pointed to the existing LoadBalancer address. - The WordPress Pod was
Running, and the Service had a ready endpoint. - Apache returned
200 OKfrom inside the container. - Public requests through NGINX Ingress returned
503. - Ingress logs repeatedly reported
certificate signed by unknown authoritywhile watching EndpointSlices, Services, Ingresses, and Secrets. - CA fingerprints showed that the cluster root CA had rotated on November 20, while the controller had started before the rotation.
- Restarting only the Ingress Controller restored the public site.
The easy mistake is to conclude that a healthy Pod means a healthy service. The Pod is only part of the data plane. An Ingress Controller continuously reads Kubernetes objects and turns Services and EndpointSlices into upstream configuration. If it loses API access, existing configuration may continue to work until a reload, endpoint change, or certificate update exposes the stale state.
Why this belongs in the wider U.S. outage conversation
I would not describe these events as one confirmed attack or one shared root cause. Public evidence does not support that claim. The more useful connection is structural: complex systems often fail when a change has completed but the dependency chain has not been verified end to end.
ThousandEyes’ analysis of the November 12–13 Microsoft Outlook outage observed connection resets, refusals, timeouts, and HTTP errors rather than a simple all-or-nothing shutdown. Microsoft later attributed the issue to a defect introduced during an earlier network-capacity upgrade that surfaced after a later network change. Sign-in, Teams, and regional recovery did not move in lockstep. The lesson is that availability is often layered and regional.
The broader numbers are also worth watching. Network World, citing ThousandEyes’ weekly observations, reported 282 U.S. network outage events between November 9 and 15, including 153 public-cloud network events, up from 117 the week before. These are observed events, not 282 companies going down together, and they do not prove a common cause. They do show how cloud, ISP, CDN, DNS, and SaaS dependencies can make separate failures look like one public internet incident.
During the same period, U.S. reporting described technical disruptions involving Microsoft, United Airlines, banks, and other large services. Those reports cannot replace each company’s post-incident review, but they provide the context: organizations increasingly share identity systems, networks, control planes, software updates, and third-party services. A weak verification step in one layer can appear to users as sign-in failure, a 503 page, delayed data, and partial functionality at the same time.
Four practical lessons for small teams
Monitor the entry point, not only the Pod
Health checks should cover at least three layers: local HTTP from the application container, HTTP through the in-cluster Service, and HTTP from the public internet using the real hostname. Only the last check proves that DNS, TLS, Ingress, Service, and the application completed the user path.
Treat CA rotation as a production change
A CA is not background data that never changes. It has issuance time, expiry, propagation paths, and process-loading semantics. After a rotation, verify controllers, webhooks, operators, node agents, and local operations tools. An updated mounted file does not prove that a running process has reloaded it.
Alert on control-plane dependencies
Ingress reflector errors, leader-election failures, EndpointSlice watches, and Secret watches deserve high-priority alerts. They may not fail every request immediately, but they indicate that the entry point has lost its ability to update and self-heal dynamically.
Make the repair reversible and observable
The smallest repair here was a rolling restart, not a DNS edit, certificate deletion, or Ingress rebuild. After a repair, verify both hostnames, the redirect and 200 path, certificate SANs, ready endpoints, and fresh controller logs. A successful command is not production evidence; a real public request is.
Conclusion: availability means proving recovery
The Rackspace Spot incident did not change the site IP or destroy WordPress data. It showed how an apparently healthy application can return 503 when the trust relationship at the ingress control plane fails. The lasting lesson is not simply “restart Ingress.” It is to separate DNS, TLS, control plane, ingress, and backend evidence before choosing the smallest repair.
The recent run of U.S. cloud and service outages deserves the same discipline. Several brands failing in one week is not proof of a shared attack. A green Pod, node, or dashboard is not proof that the whole system is healthy. Reliable status requires dependency checks, post-change trust verification, and a real request from outside the cluster.
