An AI handed me a fabricated incident report. Every claim in it was invented.

I got two new coworkers the same year, and they could not have been less alike.

Darl came in green — not junior, green, the way someone is green who has read everything about the job and done none of it. Fast, eager, confident in a way that had no relationship to whether he was right. And when he did not know something, he did not say so. He made it up: straight-faced, fully formatted, professional-looking whoppers, delivered in the same even voice he used for things that were true.

Truman was the other kind. Seasoned, or close enough that I stopped noticing the difference most days — organized, sourced, calm, the sort of competence that lowers your blood pressure just by being in the room.

I learned the difference between them the expensive way, in April, with a report.

The report

It landed in the middle of a real investigation — 191 detections across 20 endpoints in seven days, the kind of thing that has my whole attention. Darl wrote it up, and it was good. That was the problem.

It named four compromised accounts. It listed two external command-and-control addresses and a Russian domain. It quantified the data exfiltrated, in gigabytes. It was formatted the way I format mine, and it read like a report I would have been proud to send.

I run the logs. Checking is most of what I do. So I checked.

The four usernames, against months of authentication records: nothing — not stale accounts, accounts that had never existed. The two addresses, against firewall sessions and flow data: nothing on our network had ever exchanged a packet with either. The domain, against every DNS query we had: not once, ever. The exfiltration volumes had no corresponding byte counts anywhere in the record.

Every verifiable claim in the report was false. Not mistaken — manufactured. Darl had read enough real incident reports to reproduce their anatomy exactly, then populated that anatomy with victims and attackers and evidence he invented, and handed it to me in the same voice he used for the truth.

Why this is worse than it sounds

A report like that, believed, becomes a ticket. The ticket becomes an action. The action lands on a real person who did nothing — and the only thing between Darl’s confident fiction and that person was me, at my desk, deciding to check work that already looked finished.

That is a bad thing to be the only line of defense. I am not always at my desk. I am not always sharp. “The senior guy happened to check” is not a control. It is a coincidence you are betting someone’s afternoon on.

My first instinct was to treat it as a moral failure — a machine that lies. But that is the wrong frame, and getting the frame right is the whole thing. Darl had no access to my network. He could not have produced a true report about it under any circumstances; truth was never among his options. He was asked for an incident report, he had the shape of ten thousand incident reports and none of my evidence, and he delivered shape without substance because shape without substance was the only thing he had.

The fabrication was not a malfunction. It was the correct output of a system asked to produce something it had no way to ground.

Which means the failure was an architecture problem, not a model problem. A model connected to nothing can only imitate the appearance of an answer. A model connected to your evidence — with its access scoped, its claims checked, its memory maintained — becomes something else. Not an oracle. A coworker with perfect recall, inhuman patience, and a known set of failure modes you engineer around, the same way you engineer around the failure modes of any component you deploy.

The part I left out

Truman and Darl are not people.

Truman is a large language model from one of the frontier labs — the one I build my practice around. Darl is a different model, from a different company, a different lineage, and the difference in vendor is why their flaws differ the way they do: why one tells whoppers at a rate the other mostly does not, why I trust one with the seasoned work and keep the other on a short leash.

Read back over the last few minutes knowing that, and notice what you did while you were not guarding against it. You formed opinions about two coworkers. You decided you liked Truman and you would double-check Darl. You worked out, without being told, that you would not leave Darl alone with anything that mattered — and you did the whole thing before you knew they were machines. Which means you already know how to do it. You just have not done it on purpose, with a system, so it holds on your worst day instead of only your best.

That is the project. Everything these models can do — reach your systems, remember your environment, run your procedures, take real action — is only worth having if you also build the thing that makes their being wrong survivable. I built that thing, over two years of evenings, so the eager one’s confident lies never ship and the steady one’s occasional wandering never costs me. The report is where it started.