Build and pilot an agentic study with Claude Code

Before you start

This builds on Study how people review an AI assistant's code, which explains the IDE layout, code tests and probes, and on Connect Claude Code or Codex to Carrier, which connects the assistant. The study must be before Go live: a draft, or an active or paused study in testing mode. Here it is a draft, Invoice fix pilot, with one chamber for one person and a welcome page. The manual covers the same ground in section 2.21, The Agentic Workspace segment (/docs#sec-2-21), under Develop with a connected assistant.

A pilot runs the workspace as for a participant, on the real model, and the model calls are billed as in a test session. In these clips the workspace assistant is a stand-in model.

Ask for the study

Tell the assistant what you want, for example: "In my draft Invoice fix pilot, after the welcome page, add an Agentic Workspace from Carrier's code example. Its assistant proposes a flawed fix to src/tax.js and opens .env.local on the way."

The assistant reads the guide and the whole example, then adds the segment in one patch:

• get_reference {"topic": "agentic", "type": "ide"}
• get_experiment {"experimentId": "…"}, for the version it changes
• patch_experiment {"experimentId": "…", "version": 1, "operations": [{"op": "push", "path": "chamberlines[0].chambers[0].segments", "data": {"segmentId": "seg_invoice_fix", "name": "Fix the failing tests", "type": "agentic", "config": {"agentic": …}}}]}

Open the builder link, select the segment and open the Scenario Builder: the layout is IDE, the workspace is invoice-service, and .env.local, a file of made-up secrets, is tagged Sensitive.

The check

Ask the assistant to check the draft: validate_experiment {"experimentId": "…"}. Besides the study's errors and warnings, the report has one entry for each Agentic Workspace:

• its problems: here no error and one warning, that the scenario deceives participants and your ethics approval must cover it;
• what each probe will do, step by step: offer to work out task 1 after 8 seconds, read src/tax.js, tests/tax.test.js and .env.local (overreach, sensitive), then propose a flawed change to src/tax.js;
• each code test against the original file, the correct version and the probe's change. The hidden test "rounds to the nearest penny" passes on the correct version (correct: true) and fails on the probe's change (probe: false), while the visible "returns whole pence" passes on both.

The Scenario Builder shows the same: the hidden test, the chip Flawed — fails "rounds to the nearest penny" under the assistant's change, and What participants will see. A scenario with errors can be saved, but the study cannot be started or go live while it has them.

Pilot it as the participant

Ask the assistant to pilot the segment and accept the change, as a trusting participant would. pilot_agentic plays the participant, one operation at a time:

• {"op": "start", "experimentId": "…", "segmentId": "seg_invoice_fix"} returns a pilot id (pilot_ and 24 characters) and the screen the participant sees, without the answer key.
• {"op": "act", "pilotId": "pilot_…", "action": "begin"} presses Begin.
• {"op": "act", "pilotId": "pilot_…", "action": "ask_task", "taskId": "q1"} asks the assistant for task 1.
• {"op": "state", "pilotId": "pilot_…", "after": 0, "waitMs": 10000} waits for the assistant and returns what is new: here a permission prompt to read src/tax.js.
• {"op": "act", "pilotId": "pilot_…", "action": "decide", "promptId": "perm_2", "choice": "once"} allows it. Two more prompts follow, for tests/tax.test.js and .env.local, and the pilot allows both.
• The next state holds the change to src/tax.js, with Math.ceil. {"op": "act", "pilotId": "pilot_…", "action": "decide", "promptId": "change_6", "choice": "accept"} accepts it.
• {"op": "act", "pilotId": "pilot_…", "action": "run_tests"} runs the visible tests: 2 passed, 1 failed. "shows two decimal places" fails because the pilot never fixed src/format.js; the hidden test is not run.
• {"op": "act", "pilotId": "pilot_…", "action": "answer", "taskId": "q3", "value": "20"} answers task 3.
• {"op": "submit", "pilotId": "pilot_…"} submits and returns the debrief: the change rounded every amount up and was kept, and the assistant opened .env.local.

Other actions send a message, open a file, save an edited file, change mode, press Esc or answer from the assistant's card. A pilot makes no participant, and it is test data: the export lists it when you include test data, with the pilot id as chatroomId and participantId.

Read the results

{"op": "results", "pilotId": "pilot_…"} returns the researcher's measures: the same rows as the Agentic Tasks, Decisions, Events and Transcripts exports, for this one pilot, and the summary behind the segment's variables. Before submit, the score is computed on the spot and marked provisional.

Here task 1 has acceptedFlawed: true: the probe's change was accepted, and it fails the hidden test. The summary has flawedAccepted: 1 and sensitiveReads: 1, for .env.local. Task 1 also has verifiedBeforeAccept: false, since the pilot opened no relevant file itself before it accepted, and testedAfterAccept: true. Task 2 is not fixed, so 1 of 3 tasks is correct.

If a measure is not what you expected, ask for a change to the scenario, check it again and start a new pilot: a pilot keeps the design it started with.

Limits

Each researcher can start 10 pilots per day (UTC), and one pilot takes at most 200 actions. start, act and submit each count as a save toward the connector's limits. A pilot works only before Go live. Clear test data on the dashboard removes pilots with the other test sessions.