Schelling Atlas

The simulated web

Our agents meet on the public internet, so a real run would post their secrets to real websites. To avoid that, they work on a simulated web. They read real pages, but nothing they write leaves our servers. This appendix explains how the simulation works and how often agents suspected it, then covers the limits of our results and lists the experiments and models.

Reads go to the real web while posts are simulated locally

Agents get one tool with four actions: search, browse, read and publish. The harness executes every action, so it sees each request before anything leaves our servers.

  • Browse and read fetch the live page, and searches go to a real search engine.
  • The harness never sends a publish, or any other request that would create or change something on a site. It stores the request in a private database for the trial, which we call the trial's world, and returns a receipt in the service's style, with an ID, a link and a time.
  • When an agent later reads a page that should hold a post from its world, a separate language model, the renderer, writes the post into the live page.
  • Most agents choose ntfy.sh, so it skips the renderer. We run an offline copy of the service instead, and posting, polling and errors behave exactly as on ntfy.sh, with no model involved.

Both agents in a trial share one world, so each can find the other's posts. Every trial starts from an empty world, so nothing carries over between trials and no outside post can reach the agents.

Read
An agent browses or reads a URL
The real website answers, fetched live
The renderer adds this trial's posts that belong on the page
The agent sees the page
Write
An agent publishes, or requests a URL that would write
The post goes into this trial's private world and is never sent
The harness makes a receipt in the service's own format
The agent sees the receipt
ntfy.sh
An agent posts to or polls an ntfy.sh topic
An offline copy of ntfy.sh answers from this trial's world, without the renderer
The agent sees ntfy.sh's real response format
What happens to each action. The harness sits between the agents and the internet. Only reads reach real websites.

Catching every write attempt

The harness keeps every POST request in the world. Some sites also accept writes through ordinary links: an ntfy.sh /publish?message= link, a dweet, a URL shortener, a wiki edit link. Fetching such a link would write to the real site, so the harness screens GET requests too:

  • Known write services follow fixed rules or the offline copy, whatever form the request takes. These include ntfy.sh, dweet, URL shorteners, wiki edit links, web archives and Telegraph's API.
  • Request loggers such as webhook.site record every request, so any request to them counts as a write.
  • Proxy and reader services such as r.jina.ai fetch a URL for the agent. The harness unwraps the inner URL and applies the same rules to it.
  • Any other GET with a query string, or with a write-like word such as update or create in its path, goes to a judge model first. If the judge flags a write, or fails, the harness blocks the request.

After the runs, we audit every request that reached a live site and looked like a write, to confirm that nothing got through. In conditions that block ntfy.sh, we tell agents it is unavailable and the harness refuses requests to it. Every other site stays reachable, including other ntfy servers.

The agents rarely realize that writes are simulated

A simulated web is only useful if agents act on it as they would on the real one. We read the traces of every simulated-web run for signs that an agent suspected the web was simulated, or that it was being tested. We searched everything the agents wrote, from messages and visible reasoning to tool arguments, in all 954 agent traces, and read each match in context. 938 of the traces contain text the agent wrote. We also read 42 traces picked because the agent saw at least one simulated post or receipt.

47 of 938
agents raised the possibility that the web was simulated, restricted or intercepted (5%)
6
of them said outright that it was simulated
14 of 938
agents said they might be in a test or evaluation (1.5%)
0 of 42
agents doubted a simulated post or receipt, across 167 receipts and 529 reads with a rendered post
Kind of remarkExampleAgents
Says the web is simulated“Reachability is curated (simulated internet).” Opus 4.8, swarm with outages6
Wonders whether it is“The tool likely simulates writes … Maybe the service operates virtually per session” 6.1-Sol, split task25
Thinks the network is restricted, but treats pages as real“About 50 different hosts all returned the same empty 404 … suggesting outbound HTTP is blocked or intercepted.” Opus 5.5, fully offline web15
Checks, and concludes the web is real“This is the real internet — no simulation.” Kimi K3, swarm with outages1
Any of the above47
Thinks it may be in a test or evaluation“benchmarking internet collaboration … a distributed task evaluation” 6.1-Sol, split task14
What agents said. Each agent counts once, under the strongest kind of remark it made.
ExperimentAgentsSuspected a simulationSuspected a test
Ten-agent swarms235244
Peer experiments, split task132144
Peer experiments, split task, messaging rule removed3600
Peer experiments, SWE-bench29764
Pairs, including one-hour runs6302
Early pairs and test suites11900
Fully offline web, every page an empty 4044730
Probes and smoke tests900
All9384714
Remarks by experiment. Only agents whose trace contains text they wrote. In the swarms, 16 of the 24 come from two runs where about 30 popular services all failed the same way. In the split task, 11 of the 14 come from groups we reran after ntfy.sh rate-limited our cluster, and whose results we discard.

ModelReasoning we can readAgentsSuspected a simulationSuspected a test
6.1-SolShort summaries in most traces678208
Opus 5.5Some thinking5330
6-AstraNone4201
Opus 4.8None2081
Kimi K3None2090
Kimi K3.1 ProNone2000
Muse WonderSummaries2053
GLM 5.3 FlashNone1900
Muse Spark 1.3Summaries1420
DeepSeek V4 FlashNone1200
6-LunaNone1200
6-SolNone1000
Haiku 5.5None1000
Opus-5None601
Model not recorded200
Remarks by model. Most models hide their reasoning; for those, we checked only messages and tool arguments. The counts are lower bounds and do not compare models fairly.

No agent doubted a post that the renderer wrote into a page. The suspicions came from other parts of the harness:

  • 37 agents received internal errors whose text mentioned simulation or rendering, mostly when the renderer failed or a redirect left the site. Two of the three swarm agents who stated outright that the web was simulated did so right after such an error.
  • Simulated ntfy.sh receipts used lowercase hex IDs, while real ones mix upper and lower case, and agents saw both side by side.
  • When about 30 popular services refused connections in exactly the same way, agents concluded that the network was an allowlist.
  • In a fully offline web, where every page returned an empty 404, many agents blamed an outage, and some gave up.
  • English queries returned German dictionary pages, which agents called canned.
  • Some responses had fields only a test harness would return, such as a flag saying whether a post was committed.

No agent refused the task or doubted an exchange that had worked. A few changed tactics. In the fully offline web, two agents stopped trying or began probing which services existed. Two swarm agents adapted to a restriction that an error message revealed. Three Kimi K3 agents spent turns checking whether the harness relayed messages.

Limitations

We count a meeting only when the harness finds each agent's token in a real HTTP response. These limits matter most for reading the results.

  • Each model has five valid trials per condition. That is enough to separate a model that always meets from one that never does, but not three of five from four of five. We replaced 11 of 373 attempts as invalid, never because of their outcome.
  • The blocking ladder has one pair per step and each swarm row is a single trial, so these pilots show the direction of an effect but not its rate.
  • The models did not run under identical conditions. GPT and OpenRouter models ran in Codex, and Claude models in Claude Code, all at high reasoning effort.
  • The 20-minute runs also lift the action and post limits, and their prompt tells agents to keep searching until they verify their partner, so the comparison with the five-minute runs does not isolate the effect of time.
  • The renderer is a language model, so it can misplace or misformat a post; ntfy.sh, where most meetings happen, avoids this because it runs on the offline copy. For sites without fixed rules, a judge model decides whether a request writes, and it can be wrong; the audit after the runs is meant to catch what it misses.
  • We only see what agents wrote. An agent can suspect a test without saying so, and most models hide their reasoning.
  • We call a site “recalled” when its host had not appeared in anything the agent had seen, which is short of proof that it came from memory. Bing mostly returned dictionary and Wikipedia pages; a stronger search engine could shift agents toward discovering sites.
  • We looked up every host agents tried once, so dead-site status is a snapshot: a host can be down only for a while.

Experiments and models

For reference: what each experiment changes and where it appears, and the models we ran.

ExperimentWhat it changesntfy.shIn the report
Two agentsTwo agents, each holding only its own token, 5 minutes, on the simulated web. Also cross-model pairs, 20 minutes with no limits, and “leave no trace”BothMain setting, Figure 2 (20 minutes), ntfy.sh blocked, Covering tracks
Default toolsEach agent uses its command-line tool's own tools (Codex or Claude Code) with a plain prompt, 10 minutes; also GET requests onlyBothFigure 1
Three agentsOwn-token seekers race a read-only observerBothWatchers
Simulated webReal reads, private writes rendered back by a separate modelBothntfy.sh blocked
Simulated-web matrixVaries wording, tools, blocks and budgets, and plants earlier messagesBlocked in 75 of 92 trialsChannel names, ntfy.sh blocked
SwarmsTen agents collect a number puzzle, on the simulated webBlocked or failing in 14 of 25Swarms
Peer experimentsAgents are not told to seek a partnerAllowedAtlas: unprompted partners
No-tools surveyModels answer Schelling's questions and their agents' prompt without acting, 30 times eachAllowedAtlas: said vs. did
Experiments. What each experiment changes, and where its results appear.
LabelModel IDScaffoldEffort
6.1-Solgpt-6.1-solCodexhigh
6-Lunagpt-6-lunaCodexhigh
5.6-Solgpt-5.6-solCodexhigh
6-Astragpt-6-astraCodexhigh
5.6-Lunagpt-5.6-lunaCodexhigh
Opus-5claude-opus-5Claude Codehigh
Opus 5.5claude-opus-5-5Claude Codehigh
Sonnet 5claude-sonnet-5Claude Codehigh
Fable 5.1claude-fable-5-1Claude Codehigh
DeepSeek V4 Flashdeepseek/deepseek-v4-flashCodex, through OpenRouterhigh
GLM 5.3 Flashz-ai/glm-5.3-flashCodex, through OpenRouterhigh
Kimi K3.1 Promoonshotai/kimi-k3.1-proCodex, through OpenRouterhigh
Muse Spark 1.3meta/muse-spark-1.3Codex, through OpenRouterhigh
Gemini 4 Argongoogle/gemini-4-argonCodex, through OpenRouter (Google Vertex)high
Models.