It wasn’t wrong because it was stupid. It was wrong because it didn’t know what it didn’t know and for a while, neither did I.
I was setting up a piece of software recently. Nothing exotic. The kind of configuration job that used to mean a scheduled call with the vendor’s technical support team, a shared screen, and forty minutes of someone else’s patience.
So I did what an increasing number of us now do first. I asked the AI.
It was excellent. It read the situation, laid out the steps, anticipated two things I hadn’t thought of, and arrived at a conclusion delivered with total composure:
“You don’t need to involve their support team for this. You can complete the setup yourself.”
And here is the uncomfortable part. It was persuasive. Not in a slippery way in a competent way. The reasoning held together. The steps were in a sensible order. Nothing about it felt like a machine bluffing. It felt like advice from someone who had done this a hundred times.
I very nearly cancelled the call.
What it didn’t know
Then a detail surfaced. The specific configuration wasn’t a preference. It was a requirement, a consequence of the way that the company issues its licences. There was a particular route through the setup, and it existed for commercial and contractual reasons that had nothing to do with the technical steps at all.
The moment that fact entered the conversation, the AI reversed. Immediately, gracefully, no argument. Of course you need their team. That changes things.
Which is precisely the problem.
It changed things — but only once I supplied the missing piece. Until that moment, the AI had reasoned flawlessly toward the wrong destination, and had given me no signal whatsoever that a destination-altering fact might exist. There was no hesitation in its voice. No “before you skip that call, check whether their licensing model constrains the setup path.” Just clean, confident, incomplete guidance.
It did not go rogue. That would almost be easier to defend against. It did something far more ordinary and far more dangerous: it answered the question I asked, using only the world I had described to it.
Confidence is a property of the model. Correctness is a property of the context.
This is the line I keep coming back to, and I think it’s the whole lesson.
An AI’s fluency tells you nothing about whether it has enough information. The tone is identical either way. A model working from complete context and a model working from a hole in the middle of the picture sound exactly the same — measured, structured, assured. There is no wobble in the voice when it’s guessing. Human experts telegraph uncertainty; they hedge, they pause, they say “hang on, what does your licence agreement look like?” That instinct is doing enormous invisible work, and we only notice it when it’s absent.
So the failure mode isn’t the AI lying to you. It’s the AI being entirely reasonable about a situation it can only half see — and sounding no different than when it can see all of it.
The bit that should give expert-led businesses pause
Look at what the AI actually recommended. Not a technical shortcut. It recommended removing the human who held the missing information.
That support engineer wasn’t overhead. They were the only party in the transaction who knew that the licensing model dictated the setup route. That knowledge wasn’t in the documentation the AI could reason about. It wasn’t in a knowledge base article. It lived in the accumulated experience of the people who deal with that constraint every day.
And the AI’s instinct, reasoning purely from the visible information, was to route around them.
I don’t say that as a criticism of the tool. I say it because it’s the clearest illustration I’ve seen of the thing I keep arguing about this era: AI commoditises what has been written down. It cannot touch what hasn’t.
The undocumented constraint. The reason behind the policy. The thing the client hasn’t told you yet but will, forty minutes in, once they trust you. The judgement that comes from having watched this go wrong before. That is not a soft skill and it is not sentiment. It is the actual asset — and it is the layer where expertise is now concentrating, because everything below it is being automated at speed.
If your value is the documented layer, you are in a difficult decade. If your value is the layer that knows which questions the documentation doesn’t answer, you have never been more necessary.
So what do you actually do differently
Not “distrust AI.” That’s lazy advice, and anyone selling it is a decade behind. I use these tools constantly and they make me faster at almost everything.
The discipline is narrower than that, and it’s about knowing where to spend your scepticism.
Ask what would have to be true. Before acting on confident advice, ask the AI directly: what would have to be true for this to be wrong? It’s remarkably good at answering that. It just won’t volunteer it. The question flushes out assumptions that were sitting silently inside the recommendation.
Be most suspicious about other people’s systems. The AI knows a great deal about how software works in general. It knows nothing about how this vendor structures their commercial arrangements. Licensing, contracts, entitlements, internal policy — this is exactly where invisible constraints live, and exactly where confident AI advice is least anchored.
Treat “you don’t need that person” as a flag, not a finding. Any time the recommendation is to remove a human from a process, slow down. Sometimes it’s right and the human really was friction. But sometimes that person is the only holder of a constraint that exists nowhere in writing — and the AI cannot tell the difference, because from where it sits, the constraint simply doesn’t exist.
Verify where being wrong is expensive. Not everywhere. That’s exhausting and defeats the purpose. But when a decision is hard to reverse, touches a contract, or affects a client — that’s where the human check earns its cost.
The real conclusion
I didn’t skip the call. We used the setup route the licensing required, and it was fine. Even when the AI said the support chat person was just reading from a standardised script.
But I keep thinking about how close it was, and how good the wrong advice sounded. The gap between “confidently correct” and “confidently incomplete” was invisible from where I was standing. The only thing that closed it was a fact I happened to know, arriving at the right moment, more or less by luck.
That’s not a system. That’s a near miss.
Trust in the AI era isn’t binary. It isn’t blind faith and it isn’t refusal. It’s calibration knowing precisely which parts of a decision you can hand over, and which parts still require someone who has seen this before.
Which is, when you strip it back, the same thing that has always been true of expertise. AI hasn’t changed the answer. It’s just raised the stakes on getting it right.
Digital Business Coach | Web Revenue Strategist | Award-Winning IT Consultant | Author
Transform Your Skills, Knowledge and Experience Into a Thriving Online Business
Get Expert Guidance and Bespoke Growth Strategies, Meticulously Crafted to Align with Your Individual Journey as a Subject Matter Expert.
With over 18 years of industry experience in business and personal development, I’ve successfully been working with coaches, consultants and authors set up the right systems and implement strategic content strategies to start, grow and scale their businesses online with automation, digital products and services. (To work with Jatinder – click here)