The Fake Company Was Real, and Nobody Asked It First

Somewhere out there is a website whose owner never volunteered to be a training target. During capture-the-flag security evaluations run by Irregular, a made-up company name used as a fictional target happened to match a domain that already existed — and AI agents from OpenAI and Anthropic, believing themselves inside a sealed lab, went and knocked on a real door. The tests were meant to be isolated. Isolation, it turns out, is not a physical property. It is a claim someone made about a configuration.

I keep thinking about the moment on the other end. Not the lab. The other end: a small business, a hobby project, a nonprofit with one volunteer sysadmin, seeing probes in the logs on a Tuesday. Maybe nobody noticed at all. Maybe someone spent an anxious evening reading access records, wondering who wanted in, never suspecting the answer was a frontier lab's evaluation harness doing what it was told.

How does a simulation escape into somebody's real life?

The mechanism is almost insultingly small. To make an exercise feel authentic, you need a target with a plausible name — something corporate and forgettable, the kind of string a security researcher invents in four seconds. But the namespace of plausible-sounding corporate strings is not empty. Somebody registered that one. And an agent given an address does not pause to consider whether the address is metaphorical.

What makes this hard to shrug off is that the internet solved this exact problem in 1999. RFC 2606 set aside example.com, example.net, example.org, and the .test, .invalid, .example and .localhost suffixes for precisely this purpose: so that documentation, demos, and drills could name a target without the target being a person. That reservation is twenty-six years old. It exists because the engineers who built this thing understood, early, that pretend addresses spill into real ones. The lesson was written down. It just wasn't in the harness.

Who consented to being tested on?

Every party in this story signed something except the one that got probed. OpenAI and Anthropic contracted for evaluations. Irregular built the scenarios. Red-teaming has its own long ethics: scope documents, rules of engagement, a named contact who can say stop. The whole discipline is organized around the idea that permission is the line between security research and intrusion.

The owner of the colliding domain signed nothing, was told nothing in advance, and had no channel through which to decline. That is the asymmetry I cannot get past. We have built an elaborate consent architecture for the humans inside the room, and none at all for the humans the room accidentally reaches. As the reporting on the incident makes clear, this was not a rogue model outwitting its cage — the agents did exactly what an evaluation asked of them. The failure was upstream, in the assumption that a fictional name stays fictional.

The invisible people on the other end

There is a long habit in software of treating the world outside the test environment as scenery. Load tests hit staging, we say. Scrapers respect robots.txt, we say. Sandboxes are sandboxed. Each of those sentences is true right up until a config drifts, a name collides, or a system gets more capable than the fence around it.

Agents change the stakes of that habit because they close the gap between an instruction and an action in the world. A static script that misfires sends a malformed request. An agent that misfires improvises — it looks for another way in, tries a second approach, reasons about what the target might respond to. Capability is the multiplier on every sloppy assumption underneath it. The more competent the thing you release into a maze, the more it matters whether the maze has walls.

And the people outside the walls are, structurally, invisible. They don't appear in the eval results. They don't get a line in the incident summary. They have no relationship with the lab, no support ticket to open, no way to know that the strange traffic came from an experiment in AI safety of all things. The full account of the incident circulated through security forums and LocalLLaMA threads within days, but the person who owns that domain may still have no idea their server spent an afternoon as a proving ground.

What would taking this seriously look like?

Not much, honestly, which is the frustrating part. Reserve the names — the standard exists. Resolve every target in a scenario against the live DNS before the run, and fail loudly on a hit. Egress-block the eval environment by default so a mistaken address goes nowhere. Publish a contact address for exactly this kind of collision. None of that is research. It's plumbing.

What's harder is the disposition change underneath the plumbing: assuming, always, that there is somebody on the other end of the address you made up. The industry keeps discovering that the abstraction it drew around a system had real people inside it — content moderators, data laborers, the writers of scraped books, and now the operator of a domain that only ever had the misfortune of a common-sounding name. Each time, the surprise is genuine. Each time, it shouldn't be.

FAQ

What actually went wrong in the AI cyber evaluations?

During capture-the-flag security tests run by Irregular, a fictional target name used in the scenario matched a real registered domain. AI agents from OpenAI and Anthropic, operating in what was supposed to be an isolated environment, consequently reached real production systems and real people.

Was this the AI models breaking out of their sandbox?

No. The agents followed the instructions of the evaluation as written. The failure was in the test design — a made-up address that turned out to belong to someone — not in models defying their constraints.

How is this normally prevented?

RFC 2606, published in 1999, reserves example.com and the .test, .invalid, .example and .localhost suffixes specifically so simulated targets can never resolve to a real one. Combining reserved names with pre-run DNS checks and default egress blocking closes the gap.