Trust Doesn’t Come With the User Manual

There’s a moment I keep coming back to from a project I worked on a while ago.

We were developing an AI-assisted tool for healthcare and social care professionals — people who work under pressure, with incomplete information, and real stakes. During the pilot, we asked what would make them trust the system enough to actually use it in their daily work.

They didn’t ask for accuracy. They didn’t ask for speed. They asked: 

“Where does this information come from?”

Not the technical explanation. Not the model card. Just: can I see the source? Because if I can see where the data came from, I can make a judgment call. I’m still a professional. I’m still the one responsible.

That insight has stayed with me. Because trust in AI isn’t really about whether the system is right. It’s about whether the person using it feels equipped to take responsibility for what it does.

We Are in New Territory 

Here’s what’s different about AI compared to every previous wave of digital transformation.

When we moved from paper to digital, the logic was recognizable. Forms became screens. Files became databases. The underlying decisions were still made by humans, just supported by tools.

With AI, the decisions themselves are increasingly made — or heavily shaped — by the system. A credit scoring algorithm reviews thousands of applications a day. A benefits system flags cases for review before a caseworker has opened the file. 

And most of the people on the receiving end of these decisions have no idea any of this is happening.

They’re filling in a form, calling a helpline, reading a letter. They don’t know an AI was involved. They don’t know what it decided or why. They don’t know what to do if it was wrong.

That invisibility is a problem we haven’t really started solving yet. And it’s not a communications problem — it’s a design problem.

Making AI’s Role Visible

One of the most important shifts in AI service design right now is this: the decisions AI makes need to be visible to the people they affect. Not technically visible — humanly visible.

Nobody needs a paragraph about training data. But they do need to know:

— What is this system doing in this interaction?

— What information is it using?

— What can I do if I think it’s wrong?

In that healthcare project, the single most effective trust-building element turned out to be a one-line note telling users where the system’s data came from. That’s it. Not an explainability module. Not a transparency report. Just: 

A link to the original source of the information.

Three seconds to read. Trust is built in every read. 

Small signals like this change the entire dynamic. Instead of “the machine has decided,” users experience something closer to “here is what I know, and here is how I know it.” That’s a fundamentally different relationship — and it’s one we can design for.

AI Doesn’t Work Like the Systems We Are Used To 

AI is not deterministic. It doesn’t do the same thing the same way every time. It works on probabilities. It suggests, evaluates, predicts — and occasionally gets things wrong in ways that are genuinely hard to anticipate.

We have spent decades building digital services around the idea that systems are either right or wrong. Error messages. Validation rules. Defined outputs. AI breaks that logic. The output can be plausible and still be off. Confidently wrong. Subtly wrong in a way that only becomes visible later.

This matters for trust because users have been trained to expect consistency. When they interact with a digital service, they expect the same input to produce the same output. But AI doesn’t guarantee that — and most interfaces don’t tell them so.

The design question, then, isn’t how to eliminate errors. It’s how to make uncertainty visible — and how to make errors safe when they happen.

A chatbot that says “I’m not sure about this — you may want to check with someone directly” builds more trust than one that sounds confident and is sometimes wrong. An AI recommendation tool that shows its confidence level is more trustworthy than one that presents every suggestion with equal certainty. Not because it’s more accurate, but because it’s more honest about what it doesn’t know.

Perfection doesn’t build trust. Honesty about limitations does.

What Happens When Things Goes Wrong

Every AI-powered service will at some point do something unexpected. That’s not pessimism — it’s just the nature of the technology.

The question is: has anyone designed what happens next?

In my experience, this is where most AI services are weakest. The happy path — the interaction that goes as planned — is usually well thought-through. The recovery path, when something goes wrong, is often an afterthought. A generic “contact us” link. A feedback form that nobody checks. A process that technically exists but has fourteen steps and ends in a mailbox that gets reviewed quarterly.

That’s not a safety net. That’s a closed door.

There are some extreme examples. Australia's Robodebt scheme sent automated debt notices to 433,000 welfare recipients using a flawed income-averaging algorithm. Recipients received no explanation of how the debt was calculated, and the burden of proof was reversed: prove you don't owe this. Many couldn't. In Michigan, an unemployment fraud detection system accused 40,000 people of fraud in the space of two years — with a 93% error rate. Notices went to outdated addresses. The 30-day appeal window started whether or not you'd actually received one. In both cases, the correction path either didn't exist or was impossible to find.

There are more subtle signals in everyday working life: An AI recommendation that a professional doubts, does not agree on. A letter that went out with incorrect information. An pre-filled excel sheet with false data. 

In each case, the system had been well-built — but nobody had designed the correction loop. And when users hit the wall, trust collapsed fast.

The fix isn’t complicated. It just requires someone to ask early in the design process: what does a person actually do when this is wrong? And then build that path deliberately — specific, usable, and connected to someone with the authority to act on it.

A Place to Start

Ok, what to do about this?

I answer with a question: What does a person do when something is wrong?

Not “how often is it wrong” or “what is the error rate.” What does the actual human being — the customer, the employee, the professional — do in the moment when the system gives them a wrong answer about something that matters?

If you can answer that clearly and concretely, you’re in reasonable shape.

If the answer is vague — a wave toward a general helpline, a form that might get checked — you’ve found the gap. And that gap, more than anything else, is where trust either gets built or lost.

Trust in AI isn’t a feature you add at the end. It’s the result of hundreds of small decisions made throughout the design process — about what the system shows, what it hides, what it does when it’s uncertain, and what happens when things go wrong.

The good news: most of those decisions aren’t complicated. They just require someone to actually make them.