Engineering validation
Seven failure scenarios on four real hosts
hostcd is not a slideware control plane. Its recovery paths are exercised by a live failure-injection drill on real Linux hosts — not simulate mode. This is what that drill is, and what it isn't.
The drill runs against one control plane and four real Ubuntu 24.04 hosts (Gcore, Frankfurt-2). The agents apply real packages, files and services — nothing is faked in memory. Each scenario injects a concrete failure and asserts that the fleet recovers to the state declared in Git.
Results
| # | Scenario | What was injected → expected recovery | Result |
|---|---|---|---|
| S1 | Baseline converges | Fresh fleet → all four hosts online, app base Synced/Healthy | PASS |
| S2 | Control-plane crash / restart | Kill the control plane → hosts hold their config; on restart the apps reload and the fleet reconverges | PASS |
| S3 | Corrupted state → snapshot recovery | Corrupt state.json → the server recovers applications from its .bak snapshot | PASS |
| S4 | Agent crash + drift self-heal | Kill the agent and hand-edit /etc/motd → on agent restart the managed content is restored | PASS |
| S5 | Network partition tolerated | Partition a host → it goes offline on the control plane but keeps its config, then reconnects and re-Syncs after the heal | PASS |
| S6 | Host reboot auto-recovers | Reboot a host → systemd restarts the agent; the host comes back online and Synced | PASS |
| S7 | Load: 50 extra managed files | Add 50 managed files → the fleet re-Syncs in ~3s with the files present on every host | PASS |
Result: 7 passed, 0 failed. The drill also covers load and rolling/HA upgrades; the scripts and committed reports live in the repository.
What this is — and what it isn't
Be clear-eyed about the claim. This is a point-in-time chaos run against a handful of hosts, authored and run by the maintainer. It is real evidence that the recovery paths work, on real machines, not in a mock. It is not a continuously-green CI gate, a large-scale soak test, or third-party validation. Those come with a paying pilot, not a demo.
Read the long version
The narrative — how each failure was injected and what survived — is written up here: I broke my GitOps control plane on purpose. Here's what survived. ↗
See it on your own hosts. hostcd is self-hosted and free for up to 10 hosts. Install the control plane → or read how it works.