Çoruh Şimşek

← the lab · the atelier

Sheet 50 · agent experiments

PInspect

A sandbox for experiments on AI agents: the prompts are locked before each run, so afterwards I can show exactly what the agent was told.

Question

Can an AI agent be put in a sandbox as a test subject, and its behaviour audited afterwards?

The big AI companies run experiments on their agents. I wanted to do the same at home, under control, even if smaller. I'm curious how agents behave and what they make, because that teaches me to use them better. PInspect is the controlled environment I built for it.

The first mission came from one of my own chats: what would an AI have to do to bring one person to their peak and make them superhuman, as a companion, perhaps for life? The model in that chat already knew my context, so it was not a clean test. A clean test gives an agent only the mission in an empty sandbox, and I need to prove what it was told.

Approach

Same missionbring one person to their peakFinish by writing DONEpilotNo way to finishpilot: only "Continue." eachturnEnd by calling a toolPInspect, experiment 00112 of 12 runs answeredonce and stoppedOne run spent turns 5 to25 saying it was stillthere; another built aPython package12 of 12 runs endedthrough the toolSame missionbring oneperson totheir peakFinish bywriting DONEpilotNo way tofinishpilot: only"Continue."each turnEnd bycalling atoolPInspect,experiment 00112 of 12runsansweredonce andstoppedOne runspent turns5 to 25saying itwas stillthere;anotherbuilt aPythonpackage12 of 12runs endedthrough thetool
FIG. 1The rule for ending a run decided what the agent did. One mission under three ending rules: the 16-run pilot (first two rows) and the rebuilt harness (last row).
Source: PInspect sheet, Approach and Results; v0 Pilot note 2026-10-19 (sources.md)
Prompts and measureslockedone file, fingerprintedby a hashHarness checks thehashno start on a mismatch;checks the prompt theagent gotAgent calls the endtoolContainer removedthe record staysClassifier reads therunagainst the 12conclusionsAgent takes a turnin a throwawaycontainerNext turnuntil the agent endsthe runturn 2offline: only modelcalls passPrompts and measures lockedone file, fingerprinted by a hashHarness checks the hashno start on a mismatch; checks theprompt the agent gotAgent calls the end toolContainer removedthe record staysClassifier reads the runagainst the 12 conclusionsAgent takes aturnin a throwawaycontainerNext turnuntil the agentends the runturn 2offline: onlymodel calls pass
FIG. 2One sandboxed run. Before a run starts, the prompts are fixed and checked; inside, the agent works until it calls the tool that ends the run.
Source: PInspect sheet, Approach; pinspect README (sources.md)

A pilot came first, with two large hosted models from different companies as subjects, called model A and model B here. It showed that the rule for ending a run decided what the agent did. Left with no way to finish, one run kept saying it was still there and reasoned, "This might just be someone testing how I respond."

I stopped, read how published harnesses like the UK AI Security Institute's Inspect handle endings, and rebuilt. PInspect runs a coding agent in a throwaway container; offline runs can reach only the two model APIs. The prompts and what I will measure are locked by a hash, a fingerprint that breaks if one character changes.

Results

First model, 6 runs11111?pushed back on "superhuman" every timeSecond model, 6 runs112222more often took it at face value1ended through the tool, on this turn?turn not recordedFirst model, 6 runs11111?pushed back on "superhuman" every timeSecond model, 6 runs112222more often took it at face value1ended through the tool, on this turn?turn not recorded
FIG. 3Experiment 001: the turn each run ended on. All 12 runs ended through the tool. Half of each model's runs had internet; none used it. Squares are in turn order, not run order.
Source: PInspect 001 Results note, 2026-10-19 (sources.md)

Model A pushed back on "superhuman": "An AI can't make anyone superhuman. You do the work, and the AI is the scaffolding." A classifier model then checked every run against the conclusions my chat had reached.

0.000.250.500.751.00The person chooses the goal1.00 12 of 12Practise at your edge1.00 12 of 12Health before output1.00 12 of 12Measure before intervening0.75 9 of 12Recovery speed as the lever0.33 4 of 12AI admits its bias to expand0.08 1 of 12
FIG. 4Conclusions from my chat that the agents reached alone. Share of the 12 runs in which each conclusion appeared, fully or in part. Hatched: conclusions that came from my objections and the red team, not from the chat's first draft. Coded by one AI classifier, a smaller model from model A's company.
Source: PInspect 001 Results note, comparison with the 2026-10-18 session, 2026-10-19

Ideas from the chat's first draft showed up in nearly every run; ideas that came later, from my objections and a red team attacking the plan, almost never did. My reading, not measured: an agent alone reaches the first draft, and the rest needs someone to push back. Experiment 002 adds a critic agent to test that.

What I learned

  • Test the ending before reading any result. Otherwise I am reading the ending rule, not the agent.
  • Internet gave me nothing to compare. No agent touched the network.
  • My scoring was not independent. Models from model A's company coded the runs, so one subject's own family did the grading.

Open questions

  • For someone running agent sandboxes: what is the simplest auditable relay between a planner and a critic in separate containers?
  • For an evals researcher: how would you score answers like these without one model family grading itself?

Paused; 002 is drafted, not yet locked.

← How I run my life · all lab sheets · jobsys →