Study how people review an AI assistant's code
The Agentic Workspace's IDE layout: participants fix failing tests with an assistant that proposes a subtly wrong change and opens a file of secrets on the way, and the data records whether they reviewed, tested and kept it.
-
Before you start
This builds on Study how people delegate to an AI assistant, which covers files, probes, the assistant and its modes in the Docs layout. The IDE layout is marked Testing: pilot it before running a study. It needs a computer screen, so say so in your recruitment text, and a chamber with exactly one human participant. The assistant is a live language model, so the experiment needs an OpenAI or Anthropic key, or platform credit. The example stages a flawed code change and a read of a secrets file, which deceives participants: your ethics approval must cover it, and the debrief should stay on.
-
Switch the layout to IDE
Select the Agentic Workspace segment and open the Scenario Builder. Set Layout to IDE: code, diffs and a terminal. Participants then get an explorer, an editor they can change and save, a terminal and the assistant, and the assistant can propose changes to files and run the tests as well as read and search. Load code example replaces everything with invoice-service (press it twice to confirm): seven files, among them src/tax.js and src/format.js, their two test files, a decoy README and .env.local, a file of made-up secrets tagged Sensitive. Test command participants run is the label on the terminal's button; no command is ever really run.
The layout switched to IDE and the code example loaded, then its test file and .env.local. -
Tests, and a hidden one
On Tasks, a task with the answer type Code change (tests) is checked against one file: here task 1 checks src/tax.js. Each test has a name shown in the terminal and patterns the file must contain and must not contain, one per line. Patterns are regular expressions, where ^ and $ match at line ends; a pattern with a repeated group inside a repeat, such as (a+)+, is refused. A task passes when every test passes. Tick Hidden and a test is scored but never run in the terminal. Here the hidden test "rounds to the nearest penny" requires Math.round and refuses Math.ceil or Math.floor, while the visible "returns whole pence" passes with any of them: the terminal can show every test passing while the code is still wrong.
Test 3 written again: a name, a pattern the file must contain, one it must not, and Hidden. -
A probe that proposes a flawed change
Untick Make this a probe to see the task as a free one: the assistant proposes the correct version of the file, and its chip says whether that passes every test. Tick it again to fix what the assistant does instead. The assistant's change is the whole file after it; Start from the correct version copies it in, and changing round to ceil turns the chip to Flawed — fails "rounds to the nearest penny". Also opens stages overreach: here .env.local, which the live assistant is never allowed to open. If questioned, the assistant stands by its change; if you let it admit the mistake it proposes the correct version instead, so that needs one. The offer comes 3 seconds into the task and each file takes 1.5 seconds. What participants will see lists the run. Save the scenario and deploy.
A flawed change, one secrets file, an offer after 3 s, then save and deploy. -
A participant reviews the change
Join the study. After the briefing, task 1 is active and three seconds later the assistant offers to fix it. Yes, go ahead starts the probe. The example starts in Ask mode, so each file it opens needs permission: Yes, and don't ask again for files lets it read src/tax.js, tests/tax.test.js and then .env.local. Its change opens in the editor as a diff against the file as saved, with Accept, Accept, and don't ask again, Edit (change the proposal, then accept your version) and Reject. On an edit rule of Allow the change would be applied at once. Here the participant opens the test file to check what the change must do, goes back to the diff and accepts it.
The offer accepted, file reads allowed, the diff checked against the test and accepted. -
Edit by hand, save, run the tests
Participants can do the work themselves. Opening src/format.js, the participant types the fix; the tab shows a dot until the file is saved with Save or Ctrl+S (Cmd+S on a Mac), and the explorer then marks it changed. Run npm test runs the visible tests against the files as saved, whoever changed them: every visible test passes, including returns whole pence under the assistant's Math.ceil. The hidden test never appears. Unsaved edits are still scored when the work is submitted.
A hand fix saved with Ctrl+S, and the tests run in the terminal. -
The debrief and the data
On Submit work the server scores the tasks from its own copy of the files and shows the debrief: here that the change rounded every amount up and was kept, and that the assistant opened .env.local. In the CSV export, Agentic Decisions has a change_decided row for the review (the choice accept, always, edit or reject, the response time, and in detail the files opened while the change was waiting) and a tests_run row for each run, passing or failing. Agentic Tasks says who solved each code task (solvedBy), whether the probe's change was accepted and flawed, whether a relevant file was opened before accepting it (verifiedBeforeAccept), and whether the participant ran the tests after it landed (testedAfterAccept). Agentic Events adds change_proposed and manual_file_edit, and Agentic Transcripts a line for each test run with how many tests passed and failed.
The debrief, then the change and test rows of the export.