A training system, on purpose: the latest platform technology on the weakest hardware I could put to use, AI on every side, real apps on a real running system, to learn as much as possible before the day a killer app needs it.

What it is for. Training: principles, technologies and everything around them, learned as deeply and as hands-on as possible, on a system that is actually running. Real apps behind real identity, real incidents, a real public entrance, and no paying customers yet. The hardware is the handicap, chosen on purpose: retired laptops and small desktops (the ones audited date from 2011 to 2019), running the current stack. If it runs here, it runs anywhere, and any other hardware is an upgrade. AI is everywhere, also on purpose: a frontier cloud model helps build it, a small local model helps run it, and a cloud model grades the local one’s posts, with a person approving every action.

Why: two things at once. Learn as much as possible until the day a killer app appears, and be ready to take that opportunity when it does. Until then everything here is learning, and nothing is protected for uptime’s sake. On that day the direction changes on purpose: production becomes the main line and the experiments move to a branch.

None of this is a plan or a diagram of an intention. It is running. 6 machines, live services behind real identity flows, and scheduled chains that fire without anyone watching. Which ones I have seen fire is dated further down; the restore drill and the alert-to-agent chain I have not re-verified this month. A frontier model helped build it; a local model helps run it.

There are no paying customers yet, and no organic users. The load is synthetic: a generator hitting the public path from one address. What is real is what the platform did in response to itself (see the corrections), and the figures below are measured, not remembered.

Where to start

You have Read
2 minutes The In short block above, then Recently below
5 minutes Add the measurement traps, seven controls that were configured, green, and inert
20 minutes Add reliability, audited, the single-fault inventory and the patching loop that never rebooted anything, and what outside reviews found
One story One incident, in full, a SEV1 quorum loss nothing detected, published because omitting it would be more impressive and less true
An hour Add the delivery chain, including how a change reaches the cluster, and the operations agent; AI in development is the cloud half of the AI story
Code examples/, four extracted, runnable, commented artifacts, one with a self-test that demonstrates its own bug
Tooling arguments Why these tools, and what got rejected
Hiring What I’d do differently is probably the most informative page here
The rest Quality and self-graded maturity, legal and regulatory posture, the embedded side, the mechanical side, live status

Recently

What changed, newest first. Each line is dated, the full account is on the page it links to, and a check fails the build if this list stops being recent.

The three badges below are graded by referees that sit outside the system they grade.

Alerting alive Public endpoint 7d Public endpoint 30d

Those are live. The first is a dead-man’s-switch: the alert pipeline proves itself end to end on a schedule, and the badge stops being green if the heartbeat stops. The others are an off-site probe of the public entrance, so they still report when the power or the internet is what failed. None of them lives inside the system it grades, which is the entire point of a referee.

The uptime numbers are not the goal, and the 30-day one shows why. I shut the whole estate down on purpose from 2026-10-14 to 2026-10-18, while I was away, so that window carries roughly four and a half days of “downtime” until it scrolls out (about mid-November). The badges cannot be told that a shutdown was planned. Three more notes on reading these badges, and the arithmetic of why there is no 90-, 180- or 365-day figure, are on the status page.

Right now

Measured 2026-10-27T06:10:37Z by a script that queries the running estate, not typed in. How each figure is counted is on the status page. This is a snapshot: between runs the numbers stand still while the estate does not.

nodes ready control plane pods scrape storage ceph postgres signed fleet

Also in that run: 356 container restarts, cumulative across the pods that exist (reboot waves inflate it); 258674 head series, an order of magnitude and not a count to quote; 5 alerts firing, of which 3 is not a meta-alert; 13 database backups completed in the last 24 hours, 5 failed among 348 on record, and WAL archiving on 13 of 13 clusters. Metric history held is only 3.8 days, so I publish no weekly figure from it. Of 10 scheduled jobs one is suspended on purpose (the print job ships suspended), checked read-only on 2026-10-09.

The scrape figure used to be the interesting one. In September I published 67 of 70 and explained it: the three down were the etcd metrics endpoints, because a rebuild had not restored the drop-in that exposes them. etcd was fine; what had broken was the ability to see it. Rounding it up would have removed the only pressure that closed it. The same arc applies to alerts: an earlier snapshot listed eight, two of them firing on absent data rather than on a sick database, which is the more dangerous failure because an alert that fires for the wrong reason cannot tell you anything when the right reason arrives. Those were fixed, not silenced. Alerts are listed rather than waited out, because a snapshot published only when everything is green is a marketing asset, not a measurement.

The Postgres version count has a cost. The clusters went from 8 to 13 because the application databases were migrated from PostgreSQL 16 to 18 with no maintenance window: a parallel cluster kept in sync by logical replication, then a cut-over by pausing the connection pooler so clients blocked for a few seconds instead of erroring. Measured per database: 87 to 142 seconds, zero errors, at the time. 5 PostgreSQL 16 predecessors are still running as the rollback path and the soak has not ended; the operator is what made this a scripted procedure instead of a project.

A correction on those databases, 2026-10-09

Until 2026-10-09 the page said the application and identity databases ran synchronous replication. They did not. The 18 cutover on 2026-09-16 silently dropped the synchronous setting, along with resource limits, the connection cap and failover tolerations, so for 23 days the live databases were asynchronous while the documentation said an RPO of zero. I found it with an audit, not an alert. At about 09:21 CEST on 2026-10-09 quorum-synchronous commit (any one of two standbys) was restored on the four three-instance application and identity clusters, and sync_state=quorum was read back on each primary. The two-instance clusters (the fleet service and the security tools’ databases) are asynchronous by design. A guard now fails when a successor cluster does not mirror its predecessor’s settings.

The same day, a replica that had silently diverged at a failover on 2026-09-25 sat 14 days not streaming and crash-looped on its first restart. Alerts for that class (a replica not streaming, a container crash-looping, an instance not ready) now exist. The account is in the reliability page and the lessons page.

Corrections, 2026-10-27

A second read of this site against the estate, nine days after the first, found statements that had turned stale in the pessimistic direction (things that have since been fixed) and two that were false the whole time. They are retracted here rather than quietly edited:

Corrections, 2026-10-09

A fact audit measured the live system against this page. Some claims were wrong, and they are retracted here rather than quietly edited:

None of this is a demo

The repository is 309 days old, with 1943 commits and 257 of them in the last 30 days. The current cluster is 52 days old: the one before it ran for weeks and was torn down on purpose. That is not a hole in the record, it is the record. Nodes reboot because unattended patching reboots them, and the cluster is young because it was wiped and rebuilt to prove that a wrong foundational decision costs an afternoon here, not a migration project. A machine with a year of uptime is a machine that has not been patched in a year. What has persisted is the practice around the cluster, not the cluster: it was wiped, so neither the service nor its parts ran unbroken. Components are meant to be disposable, and several have been disposed of.

The chains here (commit to signed image to SBOM to CVE to ticket; alert to agent to notification; backup to off-site to restore drill) are wired end to end by design. I have verified, as of 2026-10-27, that backups are taken and archived (see the figures above) and that the patching reboot waves ran on 2026-09-24 and 2026-10-09. I have not re-verified the restore drill or the alert-to-agent chain, and the satellite is not running. It is a home-built platform run like a production one: 6 bare-metal Kubernetes nodes from retired laptops and small desktops, the services on them, the embedded devices that talk to them, and the delivery chain that ties it together.

A testing ground that is also load-bearing

Both are true at once. The applications run around the clock behind real identity, but this is a place to take chances. These pages describe a way through, written down with the alternatives passed over, and each rejection names the condition that would change the answer, because a rejection without a reopen-trigger is just an opinion. The damage would be getting stuck: settling into caution before there is a reason to, so that nothing more is learned. The reason, when it comes, is a killer app; on that day production becomes the main line on purpose.

The whole thing, on one screen

flowchart TB
  subgraph DEV["Development — off-cluster"]
    WS["workstation<br/>+ cloud LLM as reviewed peer"]
    CI["self-hosted runners<br/>lint · SAST · tests · scan<br/>build · sign · verify · rollout"]
    RV["daily reader: cloud LLM<br/>grades the ops agent's posts<br/>no tools · drafts only"]
  end

  subgraph EDGE["Public edge — outside the cluster on purpose"]
    FUN["relay + reverse proxy<br/>default-deny allowlist"]
  end

  subgraph K3S["Six bare-metal nodes"]
    CP["3 × control plane<br/>embedded etcd quorum"]
    APPS["Spring Boot services<br/>Flutter clients"]
    DATA["Postgres clusters<br/>replica-3 block storage<br/>object store"]
    OBS["metrics · logs · dashboards<br/>alert routing"]
    SEC["SBOM + CVE tracker<br/>findings aggregator"]
    CHAOS["scheduled fault injector<br/>(sandbox target)"]
  end

  subgraph OFF["Off-cluster, deliberately"]
    AGENT["ops agent + local LLM<br/>proposes; guardrails dispose"]
    REF["external referees<br/>heartbeat · blackbox probe"]
  end

  DEV_DEVICES["embedded devices<br/>signed A/B OTA"]

  WS --> CI --> K3S
  FUN --> APPS
  DEV_DEVICES -->|"heartbeat + SBOM"| K3S
  APPS --> DATA
  K3S --> OBS
  OBS --> AGENT
  SEC --> TRACK["work tracker<br/>auto-open · auto-close"]
  AGENT --> TRACK
  AGENT -.->|"narrow, allowlisted<br/>write path"| K3S
  AGENT -.->|"its posts, read daily"| RV
  REF -.->|"grades from outside"| K3S

  style OFF fill:#f5f5f5
  style EDGE fill:#ffe9e9

The two dotted lines are the ones that matter. The agent’s write path is deliberately narrow and allowlisted; the referees are deliberately outside everything they grade. A monitor that dies with the thing it monitors is not a monitor.

Why build it this way

Why build it this way instead of stopping at a tutorial: a system nobody has to operate teaches a smaller skill than one that is running, with real incidents and a real pager. The scars (a nine-day silent outage, a drain that removed its own control surface, a reboot daemon that quietly never rebooted anything) exist only because the thing they happened to was live. The load is synthetic; the incidents are not. Power cuts, thermal throttling, disks reporting FAILING_NOW, a carrier-grade-NAT boundary, an ISP, a public entrance that has actually been down, and kernel upgrades rolling across the machines unattended, one node at a time, at whatever hour the kernel lands. The failures were never the part that needed simulating.

What it cost

   
Hardware 6 machines, all retired or second-hand: laptops and small desktops. It was nine once; two were retired for reasons written down before they left, the two all-in-ones followed them, and one small desktop joined. Old on purpose: the handicap is the point, because if it runs here it runs on anything better. It also cost next to nothing.
Software licences Zero. Every tool is open source, a community edition, or a free tier. That is a standing constraint, and a filter: a tool that cannot be self-hosted for nothing does not get evaluated on features.
Power Measured per package with the CPUs’ own energy counters, not a spec sheet. Fixing the frequency governor on the two busiest nodes cut 13.1 W continuously, about 115 kWh/year (13.1 W × 8,760 h = 114.8 kWh; an earlier version of this page said 120, which rounded up). Measured at the time, not re-measured since.
Time Evenings and weekends, over months. The up-front cost was real: replicating storage on hardware that didn’t deserve it, giving up packing density for failure isolation, 41 decision records, refusing shortcuts that would have worked in the short run.
Cloud spend Effectively zero, and that is a constraint. GitHub-hosted runners were refused when Actions billing lapsed, so all 77 CI jobs that declare a runner (80 in 39 workflows; the rest call a reusable workflow) run on the estate’s own runners, 0 on GitHub’s. The earlier cloud/self-hosted split no longer exists. The price is that CI and delivery share the fate of one LAN. No inbound webhook is allowed anywhere.

One thing is not free. The frontier AI used as a development peer is a paid subscription; it has no access to the running system. The AI that operates the platform is a local model on hardware already counted, and costs nothing to run. The two are kept on opposite sides of the network boundary; see AI, twice. The largest cost of all was re-deriving context, which is why the working agreement, the decision records and the rolling state note exist.

The apps

The load the platform proves itself against is three of the repository’s five Flutter clients: Loc8 shares location inside invite-only groups, Warn maps road hazards people report, and Talk is a messenger with live events. They are at 1.6.1+2, 1.1.1+2 and 0.3.1+4. They are not a product: not in any app store, built to talk to a private homelab, with the unfinished parts listed. The apps page has screenshots, how they sign in, and the bug that signed people out for the wrong reason.

Repositories

Nine repositories show active work. The platform itself, the embedded Linux layer, a satellite application and the mechanical repository get attention on this site; the rest are a mowing robot, a drone swarm, a rover platform, a video rover and an i.MX8 layer. Commit counts, as of 2026-10-27: platform 1943, embedded Linux 150, mechanical 79, mower 246, rover platform 33, video rover 3, i.MX8

  1. They share the registry, signing keys and vulnerability tracker. The second and third projects were mostly assembly, not invention, because the seams (a heartbeat contract, an SBOM format, an identity token) were made explicit the first time.

The satellite is the interesting proof that an application does not have to live in the platform’s repository to run on it. It brings two files, a Dockerfile and a conformant manifest, and the platform supplies registry, signing chain, runtime, ingress, storage, identity, metrics and logs. It is deliberately trivial (one endpoint, no database, no identity) because the variable under test is the contract. What it surfaced is worth more than the deployment: the contract’s silences are permissions, and this one took every single one. As of 2026-10-09 it is not running: the cluster was rebuilt on 2026-09-04 and I have not redeployed it, so “deploys from outside” is a statement about the past until I do. A second repository, the drone swarm, also calls the platform’s deploy workflow.

The platform’s repository name is a fossil: it began on Docker Swarm and has run on k3s since. Renaming breaks every inbound link, so the name stayed.

What is actually running

Failure domains: the self-graded maturity assessment of 2026-09-09 puts seven of thirteen single-fault tolerant, up from four in early July. That is a self-grade I did not re-derive, and it contradicts the four on other pages dated earlier; the later number is this one. The other six are named, ranked and tracked in the audit.

Security is structural, not a layer at the end, but not magic either: branch protection cannot be enforced on a private repository under GitHub’s free plan, so “gates” here are CI workflows and pre-commit hooks that one maintainer is bound to follow, and GitHub-native code scanning and secret scanning are off. Details are in DevSecOps.

Concrete addresses, machines and topology stay in the private repository. What is here is the shape and the reasoning.

AI, twice

AI on every side, on purpose, in three roles: a cloud model that helps build it, a small local model that helps run it, and a cloud model that grades the local one. Each is held to the same bar as everything else: assert the property, do not trust the report.

Development time: a cloud model, in the loop. Every repository, including the words on this page, is built with a cloud LLM as an engineering peer, working from a written agreement and persistent memory of decisions made. It gets no exemption: its changes go through the same CI workflows and hooks as any other, enforced by convention rather than branch protection (see above), and I have not audited that every change was reviewed. The fuller account is on its own page, including the time it confidently told a reviewer two things about this platform that were not true.

Run time: a separate, local model, with a narrow write path. An operations agent off-cluster watches the cluster, deliberately not the cloud model, so the thing watching the cluster has no dependency on someone else’s API. The live work runs on a local model, so no cluster data goes to a model vendor in the alert path; the agent is not air-gapped, as it posts to chat and syncs findings with a ticket tracker. Nothing the model says is executed directly: the model proposes; a fixed, allowlisted guardrail layer disposes. Every action except the two auto-apply backup rules below is a small closed vocabulary, shown as a dry run, and waits for a human’s approval.

A reader between them (since 2026-10-28): the cloud model grades the local one. Once a day a cloud model reads up to twelve of the agent’s recent posts and drafts a grade for each, on whether it was right and whether it was usable. It is outside the alert path, has no tools, and its grades are drafts a person adopts or ignores; a guard that demands a human review counts only a human’s. What it is sent is alert text and node names, trimmed, with the local model’s wording: so the earlier statement on this page that no cluster data leaves the house was true until that date and is not now. Alerting, the hourly post and every approval work without it. The whole loop, with the numbers.

The honest answer to “what does it do on its own”:

This is not a self-healing cluster. It is a cluster where one narrow case is automated and everything else becomes a proposal. That is a smaller claim than the industry’s favourite phrase, and the one that is true.

What changed on 2026-10-26, because it had been measured. A review of the agent’s first 96 hourly summaries found “critical” in 79 of them with no critical alert firing in 69, and a data source that had been blind for fourteen days without one mention. The same review found that a model-supplied name was being placed into a command run as root. Both are fixed: arguments are validated, only named people can approve, every executed action is written to an audit file, the severity line is now written by code and not by the model, and every call is recorded so a monthly graded sample and a set of golden test cases can say whether a change helped. The account, with the numbers, is on the agent’s page.

The habit the whole thing is built on

Assert the property. Do not trust the report.

The recurring defect is not the crash. It is the thing configured, plausible-looking and quietly inert. All found by checking rather than reading:

Four more, including a kill switch that covered half the system and a documentation workflow that ran 89 times and never succeeded, are in the measurement traps. None of them errored. So: run the thing, read the value back off the live object, and make a test fail before trusting that it passes. A green status line is a claim, not evidence.

The stack

Standard, open-source tooling, so each choice maps onto what you would reach for in a datacentre. Deliberately a table, not a badge wall: badges overclaim, and several that used to be here did.

Layer What runs Qualification
Platform k3s, containerd, embedded etcd, CoreDNS, Traefik, Helm, Ansible The “retired laptops” part is literal
Data and identity Rook-Ceph (replica 3), CloudNativePG, MinIO object store, Keycloak Postgres on 16 and 18 during the soak
Change and secrets Git as the only way in (daily manifest-drift guard, staged GitOps), SOPS + age for Secrets in git Stage 3, rolling out from git on merge, is not built; no encryption at rest in the cluster
Supply chain cosign, SLSA provenance, Syft (CycloneDX 1.6), Trivy, Dependency-Track, DefectDojo, gitleaks, ZAP, Renovate, Dependabot, 47 pre-commit hooks CodeQL’s last push run failed on 2026-10-08, cause not read. SonarQube is advisory, not a gate. GitHub-native scanning is off
Observability Prometheus, Grafana, Loki, Alertmanager, Chaos Mesh, kured Retention is a ceiling, history held is 3.8 days
Applications Java 17, Spring Boot 3.5, Flutter, OpenAPI, Schemathesis, k6 Load is synthetic
Embedded Yocto, RAUC signed A/B OTA, Raspberry Pi  
Deliberately absent Admission-time policy A named gap

Workflow-status badges are deliberately absent: the project repositories are private, so their badges return 404 to anyone but me, and a badge nobody else can verify is decoration.

Why go to this much trouble

Quality isn’t a tax on speed. Past a very early point, it is the only source of speed.

Each of these happened on this fleet: two nodes powered off on a whim with the applications staying up; a database primary failing over during a planned drain; node objects deleted and rejoining on their own because the cluster is a derivative of git; the whole cluster wiped and rebuilt; a topology decision reversed the same day it proved wrong.

That up-front cost is what makes the list take minutes instead of weekends. It is also what the shortcuts cost, each with a foreseeable ending:

The shortcut Where it ends
Single-node volumes, because it’s only a homelab Every reboot risks data. You can’t drain, so you can’t patch.
One database per app, no operator Every upgrade is a maintenance window; failover is a human at 3am.
Backups configured but never restored You find out they don’t work on the day you need them.
No decision records The same arguments recur forever.
No contracts, no drills The public edge silently widens; “it rebuilds from git” stays a belief.
Hand-configured nodes The fleet diverges until nothing is reproducible.

The second row stopped being hypothetical: in September the application databases moved from PostgreSQL 16 to 18 while serving, between 87 and 142 seconds each, no errors, nobody woken. Then the same migration dropped the replication guarantee and I found out three weeks late. Both facts belong together. The operator made the migration a script; it did not make the script right.

Two things assumed expensive were not: high availability cost nothing in hardware (loopback-file storage daemons gave real replication on a single-disk fleet), and an elite supply chain was mostly wiring, since every component is open source and the work was connecting them once.

The condition that made it possible: no pressure

There was no commercial pressure of any kind: no runway, no board, no launch date, no customer waiting. Every “properly instead of quickly” decision was made in the absence of the force that normally makes that choice impossible. The industry default is not stupidity. Under real pressure the order inverts: find product-market fit first, ship what proves it, pay for the engineering later if there is a later. Anyone reading the table above and thinking “sure, but I have a deadline” is describing the actual problem, and this site is not evidence against them. So the honest reading is not “this is how it should be done”. It is “this is what it looks like when the usual forcing function is absent”. Customers import the pressure, and that is why the order here is deliberate: learn it the expensive way while nothing forces the shortcut, so that on the day a killer app arrives the platform, the patterns and the habits are already in place, and a shortcut is a choice rather than the only option.

Rigor did not transfer to every domain at once: open items remain for rotating the secrets that sat in git in plaintext, encryption at rest in the cluster, and a hosted privacy policy (gaps, legal posture).

So what happens to it now?

     
Can it? capability The only one that has to be earned.
Should it? judgement Answerable only once can is real.
Want to? intent A choice, and the easiest one to mistake for a plan.
Done? past tense Evidence. The rest of this site is the answer.

An unearned can makes the whole chain weightless.

Can it? Very nearly. Replicated storage under every volume, three control-plane members, databases that fail over during a drain, backups with a timed restore, automated patching with coordinated reboots, an incident practice with real postmortems. What is missing is written down and mostly unglamorous.

Should it? Not yet. Legal and regulatory sits well below its target in the self-assessment, secrets are encrypted in git but not yet rotated or encrypted at rest in the cluster, the public entrance is one small machine and one relay, and product analytics is at zero. Nothing here is one weekend from being a business.

And the plainest reason is the least often admitted: there is no killer app. The applications were chosen to be a demanding, realistic load, and none is a product anyone is waiting for. A platform without something worth deploying on it is infrastructure looking for a reason, and that gap is not solved by more engineering. Being ready when one appears is the other half of what the training is for.

Want to? Not yet, and that is a choice. The platform is portable and the instance is not. The pattern can travel: the contract, the pipeline, the assertions, the gap register as a habit. Without a killer app this cluster stays a laboratory, doing what nothing in production is allowed to do: get powered off on a whim, wiped, deliberately broken. On the day one is chosen the direction changes on purpose: production becomes the main line and the experiments move to a branch. The moment something here matters to someone else, two nodes can no longer be switched off to see what happens, and the loop that produced every finding here stops. Let the pattern travel, and keep the laboratory a laboratory.

Never done, by design

Nothing here describes a finished artifact. The loop is the same on every page:

ship → observe → learn → harden → prove → repeat.

It has no last iteration, and this site does not claim one.

The age of agents has brought us endless execution. Every idea, every hunch, every experiment is now within immediate reach. Working with agents truly is magic — in the best possible sense of the word. What a time to be alive. Nay, what a blessing. — DHH, Endless execution (August 2026)

Written for this public site rather than copied from the private repository, and checked on every commit by a guard that fails the build on hostnames, addresses and credential paths. Starting clean is cheaper than scrubbing.