Single-fault tolerance, measured rather than assumed

domains nodes gaps

“Highly available” is the easiest claim in infrastructure to make and the hardest to keep true. This page is the audit: which failure domains survive losing one machine, which do not, which are deliberately not replicated, and what happened when the untested assumptions were exercised in anger. Every status below was read off the live cluster with a query where a cluster can answer, not off an architecture diagram; the rows for the public edge, the switch and power, and the operations agent are not cluster-measured and say so. The score today is 7 of 13, the badge figures are rendered from the measured snapshot of 2026-10-27, and the replica counts and spread rules in the table were read back from the cluster on 2026-10-09. Anything below with a date on it is history, not a reading.

Retraction, 2026-10-09. This page said four of thirteen failure domains were tolerant, and in one place nine gaps open against fifty-one in two others. Both were wrong. The platform’s own roadmap and the estate page said seven of thirteen; this page was the one that had not been re-read. It also said the application databases replicate synchronously, which was untrue from 2026-09-16 to 2026-10-09 (below). An audit page that is itself unaudited is the failure it describes.

The inventory

Thirteen failure domains. 7 are genuinely single-fault tolerant, 2 are partly so, and 4 are not, all of them tracked in the open. A domain is only marked tolerant once the running replica count and the placement rules have been read back from the live API, never from the manifest that was supposed to produce them. The exceptions are the rows marked as not cluster-measured.

DomainVerdictShape (cluster replica counts read 2026-10-21)
Control planeTolerant3-member embedded quorum with all three members on the fastest disks in the fleet, and a floating API address; the address-holding service runs on all three members and the address fails over between them (shipped 2026-08-16, below)
Relational dataTolerant13 clusters under an operator, every one archiving its write-ahead log continuously (13 of 13). The four application and identity clusters commit with quorum-synchronous replication again since 2026-10-09 (they had been asynchronous for 23 days). The two-instance clusters are asynchronous by design
Block storageTolerantEvery block is stored 3 times over 4 disk daemons, one per host, read back from the live cluster
Stateless applicationsTolerant2 replicas each, hard topology spread, and a disruption budget on all but one (the missing one is tracked)
IdentityTolerant2 replicas (read 2026-10-21) with database-backed cluster discovery. The cross-pod login exchange that proves it was last run 2026-07-03, not today
IngressTolerant2 of 2 replicas ready
Cluster DNSTolerant2 of 2 replicas ready, with a disruption budget
Object storePartlyOne pod. Its volume sits on replicated block storage, so the data survives a node and the service does not
MonitoringPartlyAlert routing has 2 replicas. The metrics store, the dashboards and the log store are one replica each
Image registrySingleOne pod on one read-write-once volume; see below
Public edgeSingleOne small machine (not cluster-measured). Automatic failover needs a DNS or tunnel-level answer that does not exist yet
Physical layerSingleOne switch, one power feed (not cluster-measured). An explicit accept, written down as such
Operations agentSingleOne instance (not cluster-measured). Losing it degrades autonomy, not availability

The physical layer is an accepted gap, not an oversight. One switch and one power feed are genuinely hard problems at 6 machines in a house, and pretending otherwise would be the worst kind of documentation. The honest score is 7 of 13, and it is the number that makes the rest fixable.

Two things that are deliberately not replicated

Replication is not automatically an improvement, and a register that treats “1 replica” as a defect everywhere teaches you nothing.

The backup agent runs single-instance on purpose. The databases it protects are replicated across two or three instances; the agent that ships their write-ahead logs to object storage is not. Two agents writing to one backup destination would collide on log archiving and on which base backup is current, and the failure mode is a corrupted recovery point, which is strictly worse than a backup agent that is briefly absent. High availability of the data and concurrency of the backup are different properties, and only one of them is wanted here.

Low-stakes internal dashboards run one replica each, and the register says so rather than scoring them as gaps. Being honest about what does not need to be redundant is what makes the rest of the register credible.

The replication that would have been theatre

The image registry is the last stateful single point in the delivery path, and it cannot be replicated as built: one pod, one read-write-once volume.

The obvious fix, two registry replicas over shared object storage, was rejected after being written down, because the object store underneath is itself one pod:

flowchart TD
  subgraph WRONG["Tempting — and still a SPOF"]
    R1["registry replica 1"] --> M1[("object store<br/>1 pod · replicated volume")]
    R2["registry replica 2"] --> M1
  end
  subgraph RIGHT["Actual sequence"]
    S1["1 · add a storage host<br/>(done, 2026-09-03)"] --> S2["2 · make the object service redundant"]
    S2 --> S3["3 · move the registry onto it"]
  end

Two replicas over one dependency is a relocated single point of failure wearing the badge of a fixed one, and it is worse than the honest version because it stops anyone looking. The register carries it as blocked-on-prerequisite rather than as done. (The object store’s data is no longer a single copy, since its volume moved onto replicated block storage. The service still is one pod, which is what the registry would be leaning on.)

The ordering used to be driven by a harder constraint. Until early September the block storage ran on exactly three hosts, so a drained node could not be re-replicated onto anything and planned maintenance consumed all storage redundancy. A fourth host joined on 2026-09-03 and the constraint is history. The distinction it taught is still true: the small quorum of monitors rebuilds itself on another host in minutes, the way etcd does, while data replicas need a distinct host to go to. A second failure in that window used to mean blocked writes, which is why a maintenance runbook exists for this layer and nowhere else on the storage side.

Scheduling by fact, not by hope

Machines spanning a decade of hardware generations (chosen as a handicap, on purpose: if it runs here, it runs anywhere) do not schedule safely under a single “a node is a node” model, so nodes carry labels for disk speed and CPU class, and workloads are steered toward suitable nodes with a soft preference, never a hard requirement: the ideal node being full or cordoned should degrade placement, not block it.

Correction, 2026-10-09. This page used to say those labels were measured, along with thermal headroom. They were assigned by eye at inventory time and have never been checked against a benchmark; that check is an open gap. The labels for disk speed and CPU class are on all 6 nodes. A thermal label is on one.

One label is different in kind: a single node is labelled by role as the deliberate fault-injection target, so that chaos tooling can be pinned to it specifically.

That distinction was learned the expensive way. A cluster-management controller had been scheduled onto the fault-injection node alongside the workload it was supposed to help guard, so powering that node off for a real test took the controller down with the very target it existed to supervise. The fix became a standing rule: hard-pin only deliberate targets, never anything the cluster depends on to recover.

A second, quieter defect came from the same audit: the labels lived in a provisioning playbook and in the inventory it was supposed to read from, and the two drifted apart. The fix was to delete one of the two sources, so labels are derived from the inventory alone. Two sources of truth is a bug with a delay on it, not a redundancy.

Restore is drilled, not assumed

A backup nobody has restored is a belief, not a control. So restoring one is a standing, non-destructive drill: any database can be restored from object storage into a scratch namespace, fingerprinted against the live one, and torn down again, on demand.

What was measured, and when. Write-ahead log shipping is configured to bound the recovery point at five minutes, and archiving is healthy on 13 of 13 clusters as of 2026-10-27. A real timed restore came back in about two minutes on 2026-07-02, on the previous major version and before the September rebuild. It has not been re-timed since, and the drill itself was found reading a path the cluster no longer writes to, and fixed on 2026-09-29, which is how a drill rots when nobody runs it. Until it is re-timed, the two minutes is a July figure.

The whole platform is rebuilt from its own declarative state too: once in July 2026 by tearing the live cluster down and rebuilding it from nothing, and again in September, when the fleet went from nine machines to six. What has not been exercised is a rebuild onto a different set of machines with the data restored into it. That drill exists and is still an open row. A wrong foundational decision caught by a rebuild costs an afternoon; caught in production it costs a migration project.

The rule is the same beyond the databases, and only for data nobody else holds: caches and proxies are deliberately not backed up, because they re-download.

The cluster must not need itself to start

The home network’s DNS resolver moved onto the cluster. Two failure modes came with it, and one of them happened.

It happened: the load-balancer integration publishes a port by capturing traffic to that port on every local address of every node, including the node’s own local resolver on port

  1. For about ten minutes the nodes could not resolve anything, because their own lookups were being answered by the new service’s plumbing. The fix was to publish the resolver on the one shared virtual address only, and the deploy now asserts that every node can still resolve a public name.

It would have happened next: once the network hands out the cluster-hosted resolver, a node that uses it needs the cluster running in order to pull the images the cluster needs to start. A cold start would deadlock on itself. So the nodes are pinned to public resolvers, ignore what the network hands them, and the provisioning run checks that none of them resolves through the cluster. The rest of the house does; the cluster itself does not.

Any service a cluster hosts for others is a dependency the cluster must be able to start without.

The monitoring stack was the single point that mattered most

(History, 2026-08-14. The placement below has since been fixed.)

Metrics, alerting, dashboards and logs all ran on one node, with their volumes pinned to that host’s disk. When that node was cordoned for planned maintenance, alerting went dark for about nine minutes: the alert router was evicted mid-reschedule, and creating a silence failed outright while it moved.

Nothing was lost. Quorum held, the databases recovered unattended, the workloads rescheduled. The operator was blind while it did it, during precisely the window that needed visibility most.

An observability stack that shares a failure domain with the thing it observes is not observability. It is a report that stops arriving at the moment it becomes interesting.

The mitigation was deliberately unglamorous: a retention cap so the metrics store cannot fill its own disk (today 30 days or 8 GiB, whichever comes first), alert-router discovery instead of a hardcoded address, and a scheduled capacity check that bypasses the metrics stack entirely and posts into the alert router’s own API. If the primary path is what broke, the check that finds it must not run through the primary path. The components have since been spread across nodes with their volumes on replicated storage. They are still one replica each, apart from alert routing, which is why monitoring stays partly tolerant in the table.

The drain that removed its own control surface

This is the incident worth the whole page.

The cluster’s data had been redundant for months. Its control surface had not. Every tool pointed at one machine by name: the API endpoint in every kubeconfig, the alert router URL baked into scripts, the registry hostname, and a partially-populated set of local host entries that were stale on most machines.

Cordoning that one node for maintenance therefore removed the tools required to manage the cordon. What should have been a runbook became forty minutes of improvisation.

flowchart LR
  D["cordon one control-plane node"] --> A["kubeconfig points at it — API unreachable"]
  D --> B["alert router URL points at it — silence creation fails"]
  D --> C["registry hostname points at it — pulls stall"]
  A --> X["the drain removed the means of managing the drain"]
  B --> X
  C --> X

Four things came out of it, and only one is the obvious one:

There is a general shape here. Redundancy in the data plane does not imply redundancy in the control plane, and the control plane is what you need during the failure.

The patching loop is the chaos experiment

Automated OS patching usually gets filed under hygiene. On a cluster designed to survive losing any one machine, it is more valuable than that. The design claim is: lose one node and nothing stops. A reboot is losing one node. So every automated patch cycle (cordon, drain honouring disruption budgets, reboot, rejoin, one machine at a time) is an unattended, production-workload single-fault experiment that nobody had to schedule.

flowchart LR
  P["security patch lands"] --> S["sentinel: reboot required"]
  S --> L["cluster-wide lock · one node at a time"]
  L --> G{"health gate<br/>is the cluster quiet?"}
  G -- no --> W["wait"]
  W --> G
  G -- yes --> DR["cordon + drain, honouring PDBs"]
  DR --> RB["reboot"]
  RB --> RJ["rejoin + uncordon"]
  RJ --> EV["one more single-fault scenario, survived"]

The reboot window was deleted. It was originally restricted to the small hours. That was replaced by a health gate: reboot whenever the cluster is quiet, not whenever the clock says it is safe. A maintenance window is a claim that availability is conditional on nobody watching. If the HA property is real, it does not need the cover of darkness. And pausing the loop to run a chaos experiment would be backwards: the moment you pause patching because you are “in the middle of a resilience test,” you have admitted the resilience claim is conditional.

The same argument, applied to the database tier

Deleting the reboot window is easy to accept for stateless things. The harder case is the one with the data in it, and a major PostgreSQL version upgrade is the case people most readily concede a window for, because the on-disk format changes and there is no in-place path.

In September 2026 the application databases went from 16 to 18 without one. The method was to stop treating the upgrade as an event: build the replacement cluster alongside the live one, keep it in sync with logical replication, and make the cutover a pause rather than a stop. The connection pooler holds client connections open and drains the queries in flight, so applications block for seconds instead of receiving errors. Each database took between 87 and 142 seconds. One of them was the identity provider, which every other service authenticates against, so it was also the one with the least room to be wrong.

What that cost, stated plainly. The five old clusters are still running as the rollback path, idle, as of 2026-10-27. A “temporary” double footprint that is still there weeks later is a gap; a decommission runbook is drafted and blocked. Logical replication ran old to new only and was retired on 2026-09-28, so nothing flows back: by construction (not by a rollback test) the old copies are frozen at the cutover, and undoing the migration now is a data-loss decision, not a connection-string change. Retiring them is the real deadline.

Dedicated fault injection still exists and is worth having, since it covers fault classes the reboot loop never produces. But the most representative single-fault test on this platform is the one that was already running.

The upgrade that kept the uptime and lost the guarantee

The cutover above kept the applications up. It also, silently, kept less than it replaced.

The four application and identity databases had commit-time replication to a standby (any one of two), resource limits, a connection ceiling and failover tolerations. The PG18 clusters were built from manifests that never carried any of them, and the cutover swapped the new clusters in without anything comparing the two. From the cutover on 2026-09-16 until 2026-10-09 the live databases were asynchronous while this page and the roadmap said nothing committed could be lost. Nobody noticed, because a database that is asynchronous behaves exactly like one that is synchronous until the day it fails over.

It was found the way the rest of this page’s findings were: by asking each primary what its replication state actually was instead of reading the configuration or the docs. On the morning of 2026-10-09 the four clusters were given quorum-synchronous commit (any one of two standbys), their limits, max_connections of 200 and their failover tolerations back, and the quorum state was read from each primary afterwards. A guard now fails when a successor cluster is missing something its predecessor had, which is the check the cutover lacked. The two-instance clusters (the fleet database and the security tools’) are asynchronous by design, and are not part of this regression.

The stats page carries both questions side by side, what the configuration asks for and what each primary says now, so the next disagreement is a number rather than a discovery.

…except it had never rebooted anything

Which is the second half of the story, and the better half.

The reboot daemon’s sentinel path was configured against the host’s path, while inside its container that path resolved to its own empty runtime directory, because the chart mounts the host filesystem somewhere else entirely. So the daemon read an empty directory, concluded nothing needed rebooting, and logged that conclusion, hourly, on every node, for weeks. Meanwhile the patching half worked perfectly.

As found in August 2026, when the estate had nine nodes:

What was true What was reported
four of nine nodes flagged as needing a reboot, one for over two weeks “Reboot not required”
16,002 host findings, every one of them with a fix already available patching automation: shipped ✅
four different kernel series live simultaneously across nine nodes no alert, no dashboard anomaly

Every one of those findings was fixed on disk. None was fixed in memory, because nothing had rebooted.

The register entry for this now says: “shipped” was recorded from deployed, never from observed to have acted.

That sentence is the most expensive lesson on the platform, and it keeps recurring:

None of them errored. Every one reported success. The correction is procedural: a cluster-privileged daemon is not done when it is deployed; it is done when it has been observed to act. Every one now has a non-destructive drill that exercises the behaviour on demand.

Two smaller findings surfaced in the same sweep:

Once the fix was in place

A later, ordinary kernel patch (2026-08-19) is the cleanest proof that the loop does what it claims. Overnight, and without anyone scheduling it, the daemon drained and restarted five of the nine nodes, including one of the three that hold cluster quorum, one at a time. It surfaced three real things, exactly the way an unannounced fault should:

It is happening again as this is written. On 2026-10-09 the audit behind this rewrite found five of the six nodes flagged for reboot within about an hour and the daemon already working through them, and the nodes were then running three kernel builds: two on the 6.8 series, four on the 7.0 series in two builds. The wave was mid-flight at read-back, which is the loop working, not failing.

The newest instance: a replica that was there and did not stream

Redundancy you believe you have and do not is the page’s whole subject, and 2026-10-09 added a quiet version. A replica of one of the application databases had diverged from its primary at a failover on 2026-09-25. It stayed in the cluster, counted as an instance, looked present, and did not stream for fourteen days. Nothing alerted, because the replication-lag metric describes how far behind a streaming replica is, and was blind to one that was not streaming at all. It was found only when the replica was restarted and crash-looped. Two earlier diagnoses of that crash, a corrupt checkpoint and a stray timeline file, were wrong and are retracted; the cause was the divergence at the failover.

The property to assert is not “the lag is low” but “every replica is streaming.” Three alerts now cover that class: a replica not streaming, a container crash-looping, and an instance not ready.

The alert that was correct, deployed, and could not fire

One more in the same family, and the most instructive to read as code, because nothing about it looks wrong.

A node ran 23.5% of every 24 hours above 90 °C, 10.3% above 95 °C, touching the 100 °C throttle limit. Two alerts existed for exactly this. Neither fired, once, in a week. They were written the way everyone writes them:

- alert: NodeCoreTempHigh
  expr: node_hwmon_temp_celsius > 90
  for: 10m

for: requires the condition to hold continuously. The temperature crossed the line and fell back every couple of minutes, so the ten-minute timer reset every time it dipped. The worse the oscillation, the blinder the rule. The fix is to stop measuring time continuously above and start measuring fraction of time above:

- alert: NodeCoreTempExcursions
  expr: |
    avg_over_time(
      (max by (instance) (node_hwmon_temp_celsius{chip="platform_coretemp_0"}) > bool 90)[1h:1m]
    ) > 0.10
  for: 15m

> bool 90 is the whole mechanic: it turns each sample into 1 or 0 rather than filtering the series, so avg_over_time of that is the duty cycle. This generalises well beyond temperature: error rates, queue depth, saturation, p99 latency, anything bursty. If a signal sawtooths across its threshold, a for:-based rule is structurally incapable of seeing it.

Two things worth knowing before you deploy one:

And the punchline, which is the actual reliability lesson: the cause was not dust. The alert’s own description listed dust, airflow, fan bearings and thermal paste, so both machines were opened and physically cleaned first, and the temperature did not move. It was the CPU frequency governor, sitting at performance, pinning every core near max turbo at 4% CPU. Switching it took the node from 85 °C at 4332 MHz to 51-54 °C at 800 MHz within fifteen seconds, and package power from 8.23 W to 3.35 W.

An alert description that lists the wrong causes will send you to do the expensive ones first. The description is part of the control, not documentation attached to it.

The runnable rule, with a second worked example applying the same shape to HTTP error rates, is in the examples directory.

The estate was shut down on purpose for four days

Added 2026-10-27. 14 to 18 October 2026.

I took the whole estate down on purpose while I was away, and brought it back on the 18th. This was a planned shutdown, not an outage, and an earlier version of this page treated it as an unexplained one. It is on the page anyway because it was the first cold start the platform has had, and what the estate did on the way back is the same lesson as the rest of the page.

The pattern is the one this page keeps finding: jobs that were configured, scheduled and green while quietly not running. Catch-up jobs that fail within a minute of a restart, with no alert, are that class exactly. A planned shutdown was the cheapest way to find them.

Fault injection is paused by its own safety gate

Added 2026-10-27.

Two Chaos Mesh schedules (a pod kill and a 25 percent packet loss, hourly on weekdays in the daytime) run against a sandbox workload, behind a safety controller that pauses them whenever the cluster is not in steady state. That controller is the part of this page worth copying, and on 2026-10-27 it is also the reason the injection is not running.

The gap register, and why closing things you will never do matters

Gaps are tracked in one register, ordered by risk × leverage rather than by technology area:

Tier Meaning
P0 Launch gate, legal, or irreversible data loss
P1 Reliability and trust — material operational or user-trust risk
P2 Leverage — cash in an existing strength for disproportionate value
P3 Breadth and polish

Each row carries what the gap is, why it matters and an effort estimate. Since 2026-09-30 a closed row collapses to a single line and its full story, with how it was verified, moves to a frozen archive; nothing is deleted, because the reasoning is the valuable part.

The register holds 87 open items and 44 closed ones as of 2026-10-27, counted by the stats script from the register itself. That number is published deliberately, and it is meant to be read as a positive figure rather than a confession. 87 known weaknesses in a system this size does not mean 87 things are broken; it means they have been found, written down, argued about and prioritised instead of being discovered later by someone else, or never. A project with a short register has usually not looked. Its length is a measure of attention, not of decay.

The reason closing matters is a clean-up on 2026-08-10, which took 59 open items to 25 on a different counting basis from today’s. The 34 that closed break down like this:

Outcome Count
verifiably done, checked against the live cluster or the repo 17
obsolete — superseded by an architecture change, or referring to something that no longer exists 8
won’t do — deliberately declined, with the reason recorded 9

Half of the closures were work. The rest was admitting what was never going to happen; a backlog that only grows is not a plan, it is a wish list that quietly makes every real item look less urgent.

The count was wrong for about an hour on 25 August, and how it was wrong is worth keeping. The first attempt counted a row as closed if the line contained a tick, but the table has a second column that also uses a tick, to mark “closing this deserves its own decision record”. Open items were therefore counted as closed, and the published figure was flattering by exactly the amount the mistake allowed. It was caught by re-deriving the number a different way, which is the only reason this paragraph exists rather than a quietly wrong badge, and it is the same failure this whole page is about: a plausible measurement, produced by a method nobody checked. This page has since done it again, with a hand-typed nine, so the figure above is now generated.

The register grew by eight on 25 August 2026, and that is the mechanism working. Those eight came from writing down, for every tool in the stack, which parts of it are deliberately not used, a question nobody normally asks. The answers included: the only tier of the stack with no static security analysis at all is the mobile one; a scanner already deployed on every node has a whole category of its checks switched off; the aggregator collecting every security finding was being used as a bucket rather than a queue; and a hand-applied fix for a cluster-wide outage was living outside version control with no check able to see it. Asking a documentation question found more real gaps than the last several weeks of operating did.

That is also the honest description of the working method here: run it, inspect it, correct it, write down what was learned, run it again. Most entries exist because something was deliberately poked until it admitted a weakness. The mistakes are the input, not the accident, which only works where being wrong is cheap and nobody is defending a reputation instead of a system.

A representative sample of what the register has caught, all found by operating the system rather than by reading it:

What is still open, stated plainly

A reliability page that only lists wins is marketing. The 4 single domains and 2 partial ones are in the table and need no second list here. The ones that matter most: the registry is blocked behind the object service, the public edge needs an answer at the DNS or tunnel level, and monitoring is one replica deep apart from alert routing. The physical layer is accepted, in writing.

7 of 13 is not a perfect score. It is an honest one, and the rest of this page is about why the previous scores were not.

Written for publication. Machines, addresses, hostnames and credential locations are absent by construction and enforced by a guard that fails the build.