I ship production work with Claude almost every day. I also know, precisely, where it will lie to my face — confidently, fluently, in complete sentences, with perfect grammar and zero hesitation.

Those two facts aren’t in tension. They’re the whole job.

The mistake I see teams make isn’t using AI. It’s trusting it like a calculator — a tool that’s either right or throws an error. Claude isn’t a calculator. It’s the most capable, most confident, most plausible intern you’ve ever hired, and it will never once tell you it’s unsure unless you build the moment for it to. If you know where it breaks, it’s the best leverage available right now. If you don’t, it’ll embarrass you on a timeline you don’t control.

Here’s where it breaks. None of this is random — every failure below has the same root cause, which I’ll get to.

It invents sources

Ask for citations and you’ll get them: authors, titles, journals, years, sometimes a DOI. Beautifully formatted. A meaningful share of them won’t exist. Not “slightly wrong page number” — the paper is not real.

This one burns people because the format is flawless, and we’re trained to read a well-formatted citation as a real one. I’ll be honest about how it gets me: I have too much to get done, I’m always looking for ways to cut my own effort, and citations are the easiest thing in the world to skim past — they look right while you’re scrolling them. But it only takes one fabricated reference for someone to stop trusting everything else you wrote. So now I test every citation, every time.

It makes up numbers

Hand it a spreadsheet, ask for the summary, and where there’s a gap it will fill the gap — with a number that’s the right shape and the wrong value. It would rather give you a plausible figure than say “that cell is empty” or “I can’t compute this reliably.” Plausible-and-wrong is the single most expensive failure mode in business work, because it sails straight past the gut-check that catches obvious nonsense.

It writes confident, broken code

Code that reads beautifully and fails at runtime. Calls a method that doesn’t exist on that object. Handles the happy path and silently drops the edge case. Imports a package that was deprecated two versions ago.

For me, this is the least dangerous failure, because I have a safety net: I run it, the tests fail, I fix it. That safety net is the entire point — hold that thought.

It drifts over long documents

Ask for something long and page one will say one thing, page twelve will quietly contradict it, and nothing flagged the conflict. The model is optimizing to sound locally coherent — each paragraph consistent with its neighbors — not to be globally correct. Over a short answer you never notice. Over a forty-page document, the contradictions compound.

Why all of it happens

Every failure above is the same failure wearing different clothes.

Claude is, under the hood, predicting the next most plausible token given everything so far. It is extraordinarily good at plausible. It has no separate, internal sense of true that it checks against before answering. There’s no little fact-checker in there going “wait, does that paper exist?” There’s just: what’s the most likely continuation of this text? A fabricated-but-plausible citation is, to the model, a great continuation. So is a confident wrong number. So is code that looks exactly like working code.

Once you internalize that — it optimizes for plausibility, not truth — the failures stop being surprising and start being predictable. And predictable failures are the kind you can build around.

The gap nobody talks about

Here’s the part that matters most, and it’s the reason I worry more about business teams than engineering teams.

Engineers have a safety net for all of this. Tests. Code review. CI. A type checker yelling before anything ships. When Claude writes confident broken code on my machine, the broken-ness gets caught, automatically, in seconds. The verification is baked into how software already gets built.

Business teams have none of that. When Claude writes the customer email, the board summary, the contract redline, the campaign copy — there’s no test suite. There’s no CI. There’s just a person, often a busy one, often one who’s been told this tool is magic, hitting send. So when a wrong output reaches a customer, a partner, the public — it’s not an if. It’s a when. The only variable is whether someone checked first.

That’s the real divide in who’s getting value from AI right now. It’s not the teams who trust it the most. It’s the teams who know exactly where not to.

How I actually ship with it

So given all that — why do I use it every day, on real work, with my name on the output? Because the failures are predictable, and you can build a workflow that assumes them. Here’s mine, stripped down:

I treat every AI step as needing a check before it goes anywhere external. Not bureaucracy. Just a rule: nothing the model produces reaches a customer, a repo’s main branch, or a decision-maker without a human verifying the part that would actually hurt if it were wrong. The verification is targeted — I check the claims and the numbers and the edge cases, not the grammar.

I ask it to show its work, and I check the work, not the conclusion. “Give me the three sources” is a trap. “Give me the three sources and quote the exact sentence from each that supports the claim” makes the fabrication fall apart immediately, because there’s no sentence to quote.

I make the failure cheap to catch. For code, that’s tests — the model’s confident wrongness gets caught by a red test bar, not by me squinting. For documents, that’s asking a second pass: “find every place this contradicts itself.” The model that drifts is often good at spotting drift when you point it at the job directly.

I never let it be the last set of eyes on anything that matters. It’s a fast, tireless first drafter and a useful second opinion. It is not the final authority on anything with my name attached. My own rule is simple: I write a lot — engineering docs, marketing, sales, internal notes — and I never send any of it without reading it first. Sometimes the draft is just right and I change almost nothing; sometimes I rework it so my voice comes through. Either way, I’m checking that there’s no obvious error and nothing in there I wouldn’t actually say.

None of this slows me down in a way I notice. It’s the difference between a power tool with a guard and a power tool without one. Same speed. Many fewer trips to the ER.

The honest version

The tool is genuinely remarkable — I’m not hedging when I say I build things now that I couldn’t have, or couldn’t have afforded the time to, two years ago. But “remarkable” and “trustworthy” are different words, and the gap between them is exactly where the value is for someone who knows the terrain.

The companies winning with AI aren’t the believers. They’re the ones who learned, fast and a little painfully, precisely where it lies — and built the ten-second check that catches it before anyone else sees.

I’m Brian Fromme. I help mid-market teams get real results from AI — built and shipped, not slide-decked. More at atelier.purpleblossom.ai.