From the engineering log · November 19, 2026
Ansible ran successfully. So why are my Linux hosts still different?
By Danylo Kochetov
I have trusted a lot of green Ansible runs.
The playbook completed. No failed hosts. The CI job was green. I closed the tab and moved on.
Then, hours later, one machine was different.
Someone had stopped a service while troubleshooting. A package had been upgraded manually. A configuration file had been edited directly because the incident was already ten minutes old and nobody wanted to wait for the next pipeline.
None of this made the earlier Ansible run false. It had succeeded. It just described the fleet at one point in time.
That distinction matters more than it sounds.
A successful run is a historical fact
Ansible is good at applying an intended change to a known set of machines. It is readable, widely understood and backed by a huge ecosystem. I still use it, and I am not trying to replace every playbook with a new abstraction.
The problem begins when a scheduled run is also expected to answer a different question:
Does every host still match the intended state right now?
A cron job cannot answer that between executions. A green CI result at 02:00 does not tell you whether postgresql is still running at 09:17. Running the same job every five minutes narrows the blind interval, but it does not automatically give you a durable model for drift, health, rollout history and recovery.
You can build those things around Ansible. Many teams do. Eventually, however, the surrounding system becomes the larger part of the problem.
What about ansible-playbook --check --diff?
A reader raised the most direct counterpoint: schedule Ansible in check mode and use --diff. That is a valid way to detect drift. Modules that support check mode can report what they would change, and --diff can show the predicted difference without applying it.
For a small environment, a scheduled check may be the right solution. It is familiar, inexpensive and does not require another control plane. If it gives your team enough visibility, there is no reason to replace it.
The tradeoff is that it remains a point-in-time inspection. Check mode support varies between modules, predicted changes do not necessarily describe whether a resource is operationally healthy, and the surrounding workflow still has to store results, notify an operator and connect a finding to remediation history.
The distinction is therefore not that Ansible cannot detect drift. It can. The distinction is between periodic drift checks and a long-running model that keeps drift, health, reconciliation and history in one control loop.
A small, concrete failure
I used the public hostcd demo to inspect a deliberately broken PostgreSQL application. The demo is read-only and its data resets every 30 minutes, so this is evidence of how state is represented, not a production benchmark.
The application declares three resources for one host:
- the
postgresql-16package; /etc/postgresql/16/main/conf.d/tuning.conf;- the
postgresqlservice.
The package and configuration file are both shown as Synced and Healthy. The service is not. It is Out of sync, Degraded, and carries the reason service failed.
This is the useful level of detail. “The host is broken” would be too broad. “The last automation job failed” would not tell me what remains correct. Here, the failed resource is isolated without hiding the two resources that still match the desired state.
The rollout result also preserves the actual failure instead of flattening it into a badge:
rollout halted: 0 of 1 hosts succeeded
db-1: service:postgresql: Job for postgresql failed because
the control process exited with error code
The history view records the failed attempt, the actor and the affected host count.
This does not make the incident disappear. It makes the current state and the attempted transition explicit.
Desired state and health are not the same thing
One detail in the demo is easy to miss: an application can be out of sync while still healthy.
The production Nginx application is behind the target Git revision, but its current resources are healthy. PostgreSQL is both out of sync and degraded because its service has actually failed.
- Out of sync, healthy: the host is not at the intended revision, but the currently deployed resources are functioning.
- Out of sync, degraded: the host differs from the intended state and at least one managed resource is broken.
- Synced, healthy: the managed resources match the target revision and are functioning.
If all three situations collapse into “the last job was green or red,” the operator loses useful information.
The application overview makes the distinction visible across the demo fleet: four applications are synced, two are out of sync, five are healthy and one is degraded.
The blind interval in scheduled automation
Imagine a simple nightly playbook:
- name: Keep PostgreSQL running
ansible.builtin.service:
name: postgresql
state: started
enabled: true
At 02:00 it succeeds. At 11:10 an engineer stops PostgreSQL while investigating disk pressure and forgets to start it again. The next scheduled run is at 02:00 the following day.
For almost fifteen hours, the desired state exists in Git and the actual state has diverged. The pipeline remains green because it has not run again.
This is not an Ansible bug. It is a property of the execution model.
The usual fixes are familiar:
- run the playbook more often;
- add a monitoring alert;
- trigger remediation from the alert;
- store the outputs somewhere;
- add locking so two runs do not overlap;
- build a view showing which machines converged;
- decide what to do with partial failures.
Each step is reasonable. Together they are the beginning of a control plane.
What continuous reconciliation changes
With reconciliation, the system repeatedly compares the declared state with the state reported by each agent. The agent pulls work, applies the necessary resource operations and reports the result. The control plane keeps the difference visible until the host converges or the operation fails with a concrete reason.
For a Linux fleet outside Kubernetes, I wanted the same basic operating idea that made GitOps useful in clusters:
- Git contains the intended state.
- Hosts do not need inbound management ports.
- Agents pull the desired work from the control plane.
- The UI separates revision drift from resource health.
- Failed operations retain enough detail to investigate them.
That is why I built hostcd.
It is not “Ansible, but with another UI.” The useful difference is the long-running reconciliation loop and the state model around it. Existing playbooks can still handle provisioning and complex one-off changes. hostcd is aimed at the part that starts after the playbook finishes: keeping the declared host state true.
What this demo proves — and what it does not
Observed in the v0.3.21 public demo: application-level and resource-level drift, health separated from sync status, a partially healthy resource tree, a target Git revision, and a failed rollout with the underlying service error.
Not proven by this demo: production recovery time, large-fleet performance, or automatic repair under every failure mode.
The demo resets seeded data and does not allow visitors to execute a sync. The next useful measurement is therefore specific: stop a non-critical service on a writable pilot host, record the time of the state change, then record when hostcd detects and repairs it. Publish both timestamps and the configuration used.
I would rather show that measurement than turn a demo fixture into a benchmark.
Try the failure, not just the happy path
If your Linux fleet is currently managed by scheduled Ansible, ask one practical question:
What tells us that a host changed after the last successful run?
If the answer is “the next run,” there is a blind interval. If the answer is monitoring, ask whether it only alerts or also connects the failure back to the desired Git state and the remediation history.
Open the read-only hostcd demo, select the PostgreSQL application, compare the package, file and service resources, and then inspect Pending changes and History.
If you manage Linux hosts outside Kubernetes and want to test the same failure on non-critical machines, write to [email protected]. I am looking for a small number of design partners and will help with the first deployment personally.
For a practical migration path, see From scheduled Ansible to continuous reconciliation.