When the System Gets It Wrong: What Happens Next

Every system we have ever shipped has been wrong about something. That is not a confession, it is the normal condition of software, and it is doubly true once a language model is involved. The useful question is not whether a system will make a mistake. It is what happens on the day it does.

These are the questions clients actually ask, and our answers.

Does it go out before I see it?

Not if it matters. The default we build to is that anything leaving the business gets staged for a person to approve. A reply is drafted and waits. An invoice is prepared and waits. A page is written and waits.

Plenty of things do not need that. Moving a record between two systems, tagging an incoming lead, generating an internal summary: those run unattended because the cost of being wrong is low and visible.

The line is worth drawing explicitly at the start of a project, workflow by workflow, rather than assuming. It is usually the most useful conversation in discovery.

How would I find out it was wrong?

Three ways, in descending order of how much you want to rely on them.

The system tells you. Failures are logged and surfaced somewhere you already look, not in a dashboard you would have to remember to open.

You notice at the approval step. This is the main one, and it is why the approval step exists.

A customer tells you. This is the one we are trying to prevent, and the reason the first two matter.

Will the same mistake happen again?

This is the part that separates a system you can live with from one you cannot.

When you correct something, that correction has to be written down somewhere the system reads next time. Not remembered by a model, written down. In practice that means a file describing how your business does that particular thing, which grows as you correct it.

We run our own business this way. Early on, the system got the sign-off format wrong on customer replies. It was corrected once, that correction went into the instructions, and it has been right since. That is the whole mechanism, and it is far less magical than the marketing around it suggests.

Who fixes it, and how fast?

For anything on a retainer, we do, and the response depends on what broke. A workflow that has stopped entirely gets attention the same day. Something producing slightly wrong output gets a fix in the normal course of the week, with the workflow paused if the output is going somewhere it matters.

For project work outside a retainer, that is a conversation at scoping rather than something to discover afterwards.

Does more automation mean more things going wrong?

Yes, in the sense that there are more moving parts, and more places where a tool you do not control can change underneath you. An API changes, a login expires, a vendor renames a field.

What automation reliably removes is the other kind of error: the transcription mistake, the missed follow-up, the step someone forgot on a busy Thursday. Those are quieter and considerably more common, and they are usually what prompted the conversation in the first place.

The honest trade is that you exchange a lot of small invisible errors for fewer, larger, more visible ones, and you get a system that tells you when it is confused instead of guessing silently.

What if the tool has no API?

Then that workflow either needs a browser automation, which is more fragile and we will say so, or it stays manual. We would rather tell you that during scoping than build something brittle and let you discover it in month three.

---

If you are weighing up whether to automate something and the failure modes are what is holding you back, that is the right thing to be thinking about. Get in touch and we can walk through what could go wrong with your specific process before anyone builds anything.