The automation reported success and changed nothing

Green does not mean done

The command printed a healthy-looking line — eight batches, 144 inputs, OK — and exited cleanly. Then we looked in the queue those 144 units were supposed to be waiting in, and it was empty.

Nothing had gone wrong. The command had done exactly what it was built to do.

We were refreshing part of our own content system and needed to stage 144 pieces of work. The tool assembled its list, grouped it into eight batches, generated an identifier for each batch, and wrote them to the database. The identifiers were derived from the name of the job — so on a second run, it produced the same eight identifiers it had produced the first time.

The database already held all eight. And the write instruction the tool used said, in effect: if a record with this identifier already exists, skip it, and do not treat that as an error. That is a sensible instruction. It is what stops a job from creating duplicates when someone runs it twice.

So all eight were skipped. None was an error. The tool then counted the 144 items it had prepared in its own memory, printed that number, found no fatal problem, and reported success.

The command did not lie. We had asked it the wrong question.

What “success” was actually claiming

“OK” was an honest answer to a specific question: did you complete without hitting a fatal error? It was true. We read it as an answer to a different question: are there now 144 new units of work waiting to be processed? It had never claimed that, and it could not have — the number it printed came from a list it built in memory, before the database ever saw it.

Those two questions are usually correlated, which is exactly why the gap is easy to miss. Most of the time a job that finishes cleanly has also done the thing. The correlation holds until the day it doesn’t, and on that day nothing announces the difference.

The same shape, somewhere else entirely

A week earlier we had hit the identical pattern in a completely different system. Ten commits were reported as pushed to our code repository. None of them arrived.

The repository was in a state where the push updated a reference that had not moved — so there was genuinely nothing to transfer, and the tool correctly reported that it had succeeded. It had. The work was still only on the local machine.

The remedy was the same shape as the first: stop reading the status, and instead query the remote and confirm that what is there matches what is here. That check now runs after every push we make, and it has caught the situation since.

Two different systems, two different failure mechanisms, one interpretation error. That is what makes it a principle rather than a bug.

Process truth is not business truth

A process can report on one thing: whether it completed according to its own execution contract. That is a real and useful answer. Exit codes are not decoration, and I am not arguing that you should ignore them.

But the contract belongs to the process, not to your business. “I followed my instructions and encountered no fatal error” is a claim about the machine. “The lead moved to the next stage” is a claim about the company. The first does not establish the second, and the more reliable your automations are, the more comfortable you will become treating one as evidence for the other.

The genuinely dangerous case here is not a broken automation. A visible failure is easier to reason about: something refused, crashed or raised an error, and there is a specific thing to go and look at. The harder case is when nothing is technically wrong at all — the process completes exactly as implemented, and the state you cared about still does not change. There is no error to find, because there is no error. There is only a question that was never asked.

That is also why this creates confidence rather than concern. A green result is not neutral. It actively reassures you, and it will keep reassuring you for as long as you let it stand in for a fact it was never reporting.

What this looks like away from our systems

Our own evidence comes from software we build, so I want to be precise about what follows: these are the shapes to check, not findings we have measured in anyone’s business.

A CRM workflow marked completed — did the lead actually change stage, or did the rule match nothing? A scheduled report that ran — did a file reach a person, or did it generate into a folder nobody opens? An integration that reports a successful sync — did the destination system gain the record, or accept and discard it? A backup marked complete — has anyone ever restored from one?

That last one is the clearest, and it is the one we changed in our own practice. A backup process reporting complete is not evidence that anything can be recovered from it. Where the data matters, the only answer that counts is a restore: bring it back somewhere safe and confirm that the recovered state is what you expected it to be.

The operating principle

Before trusting an automation, decide what would be different in the business if it had worked. Then check that thing.

Not the log. Not the status. Not the summary line the tool prints about its own behaviour — that summary is generated by the same process whose work you are trying to verify, which is the whole problem in one sentence.

The question is short enough to use in a meeting:

“What state should be different if this actually worked — and who has looked at it?”

For a lead-capture rule, the state is a record in the right stage with an owner. For a report, it is a file in someone’s hands. For a refresh like ours, it is a row count in a queue. In every case it is something you can point at that exists outside the system reporting on itself.

Sometimes you will go looking and find the tool does not expose that state at all. That is useful information rather than a dead end — it tells you the outcome needs to surface somewhere you can inspect independently, rather than living only inside a product that grades its own work.

Most of the time you will look and everything will be exactly as advertised. That is fine. The check is cheap and the answer is not the point — the habit is. You are not trying to catch your tools lying. They are not lying. You are making sure that the question you are relying on is the one you actually asked.

Green tells you the machine finished. It does not tell you the business moved.

Related: Would Your SOP Survive a Full Inbox and a Missing Manager?