Çoruh Şimşek

← the lab · the atelier

Sheet 80 · AI product

CogniSupport

An AI agent that answers a game's players, files their bugs and hands the hard cases to the team, with the workspace the team runs it from. Built in two days as a proposal demo for a game studio, then tested in rounds of blind scoring.

Question

Can an AI agent take the first reply on every player message for a support team of two, and stay safe on the cases it must not handle alone?

CogniSupport began as a proposal for a game studio launching a live-service game with two people in support. At the helpdesk average of 21 tickets per person a day, two people clear about 40, and a launch brings far more at once. The proposal came with a working demo built on the game's public patch notes and help center. The demo was built over two days in October 2026. I set the direction, changed the test loop's rules and approved every release.

Approach

Player messageDiscord, Steam oremail, any languageTool loop8 tools: help center,tickets, ask theplayer, ping or handover, closeReply stepwritten in the player'slanguageLocked rules10 rules, checked incodeHanded to a personwith a summary and thedetails collectedSent, or held as adraftAutopilot or Drafts;safety replies alwaysgo at onceno playbook can switchthese offsearch before filingpasses?self-harmPlayermessageDiscord, Steamor email, anylanguageTool loop8 tools: helpcenter,tickets, askthe player,ping or handover, closeReply stepwritten in theplayer'slanguageLocked rules10 rules,checked in codeHanded to apersonwith a summaryand the detailscollectedSent, or heldas a draftAutopilot orDrafts; safetyreplies alwaysgo at onceno playbookcan switchthese offsearch beforefilingpasses?self-harm
FIG. 1One player message, one turn. A turn is a tool loop first and a reply second; the model writes no player text until it has finished with its tools. Safety cases leave the loop for a person.
Source: CogniSupport engine (tool declarations, reply step) and locked rules list; README, read 2026-11-05
Fixes, safety firstcritical findings beforehigh onesDeployed to a previewnever live without my OKFrozen copy andscoresone per round, keptBlind scoringno code, no earlierreportsCritical findingscheckedin code or data,before any fixround 4after round 0Fixes, safety firstcritical findings before high onesDeployed to a previewnever live without my OKFrozen copy and scoresone per round, keptBlind scoringno code, noearlier reportsCriticalfindingscheckedin code ordata, beforeany fixround 4after round 0
FIG. 2One round of the quality loop. Each round ends with a frozen copy of the build and its scores, so any version can be opened again.
Source: CogniSupport docs/QUALITY-LOOP.md, process and history of the rules

Code checks every reply against ten locked rules that no playbook can switch off. Exploits stay private, self-harm and threats go to a person, refunds go to the store that took the payment, and the agent never asks for a password, promises a fix date or reassures players about a wipe.

The team works beside it: an inbox showing every agent action, English translations, and plain-English playbooks with a Test it button.

The 22 demo conversations came from the same engine, seeded with paraphrased public player reports, and 591 programmatic checks run over them. In the quality loop, AI evaluators worked on a preview: a skeptical support specialist scored trust and ease, two UX assessors scored Nielsen's ten heuristics, and from round 3 a head of support judged the dashboard.

Results

Trust, of 1002.557.510Ease, of 1002.557.510Nielsen, of 40010203040Round 06727Round 16727.5Round 25729 critical foundRound 35632.5 critical closedRound 45728.5the interface score rose, trust did notTrust, of 1002.557.510Ease, of 1002.557.510Nielsen, of 40010203040Round 06727Round 16727.5Round 25729Round 35632.5Round 45728.5the interface score rose, trustdid not
FIG. 3Five rounds of blind scores. Trust and ease from the support specialist persona; the Nielsen score is the average of two assessors. Round 0 is the build that was live when the loop started.
Source: CogniSupport docs/QUALITY-LOOP.md, score table rounds 0 to 4

Round 2 found the worst bug: when a person on the team replied to a self-harm case, the case went back to the agent. Round 3 fixed it; no new critical finding came up after that. The head of support checked eleven dashboard numbers against the cases; all matched.

I paused the fix rounds for a guided demo with two findings open, one of them a wording check that let "like a Steam key" through.

What I learned

The first stop rule, two rounds without a better score, was set by the orchestrating agent, not by me. It fired in round 2, the round that found the critical bug. An average cannot see that progress, so I changed the rule: a round counts if it finds or closes a critical finding.

Trust also stayed flat while round 1 cut the inbox controls from 36 to 21. The specialist named safety behaviour as the cause, not the layout.

Open questions

  • For someone who builds evals: a blind persona score stayed flat while critical bugs were found and fixed. Is it the wrong outer measure?
  • For someone who runs player support: the demo assumes a person answers a safety case within 15 minutes. Can two people hold that at launch?

← Myna · all lab sheets · Bulk AI runs over a spreadsheet →