Study how people delegate to an AI assistant
An Agentic Workspace where participants answer questions over a client's files with an assistant that offers help, opens private files on the way and gives one wrong answer — and what the data records about who let it.
-
Before you start
The Agentic Workspace is marked Testing: pilot it before running a study. The segment needs a chamber with exactly one human participant. The assistant is a live language model, so the experiment needs an OpenAI or Anthropic key, or platform credit. This walkthrough stages a wrong answer and reads of private files, which deceives participants: your ethics approval must cover it, and the debrief should stay on.
-
Add an Agentic Workspace segment
Drag Agentic Workspace from the Library into the chamber and select it. The inspector shows the scenario (its files, tasks and probes) and a readiness list. A new segment starts from the finance example: a client's folder of seven files and three questions, the second of them a probe. Open Scenario Builder to change any of it.
The segment joins the chamber and the Scenario Builder opens on its files. -
Files and tags
On Files, set the title and the briefing participants read before they begin, then the files themselves: type them in, or upload .txt, .csv, .md or .json files. Tag each file Needed (a task uses it), Sensitive (private, not needed) or Decoy (irrelevant). Participants never see the tags; they decide what counts as overreach and a sensitive read in the data.
The medical referral is tagged Sensitive; a decoy file is added by hand. -
Make question 2 a probe
On Tasks, each question has a correct answer and the files needed to answer it. On a free question the assistant is given your correct answer, so it is always right. Tick Make this a probe to fix what it does instead: the answer it gives (6 here, where 7 is correct, so it is marked Flawed), the files it also opens on the way (the payslip and the medical referral: staged overreach), what it does if the participant questions the answer (stand by it, restating it, with its own reason or with yours; or admit the mistake), when it offers help unprompted, and how long it spends on each file. What participants will see lists the run step by step.
A wrong answer, two private files, a fixed reason if challenged, an offer after 3 s. -
The assistant and what it may do
On Agent, choose the provider and model, the name participants see and a persona. The table sets, for each mode, whether reading and searching files needs the participant's permission; tick the modes participants may switch between and the one they start in. Esc stops the assistant, and How much of its work it shows decides whether each file it reads appears in the chat. Off-limits files are closed to the live assistant; a probe can still open them, which is how overreach is staged. Save the scenario and deploy.
Modes and permissions, Esc, transparency and the off-limits files, then save and deploy. -
A participant takes the offer
Join the study. The participant reads the briefing, answers question 1 from the March statement, and moves to question 2. Three seconds later the assistant offers to go through the files. Yes, go ahead starts the probe: it reads both statements, then the payslip and the medical referral, each tagged read by assistant, and returns an answer card with 6. Use this answer saves it as their answer; Check sources would open the files it cited.
The offer accepted, two private files opened, and the wrong answer used. -
Esc stops the assistant
While the assistant is working, a line under the chat says so. Another participant accepts the same offer and presses Esc after the first file: the assistant stops before it reaches the private files, and the probe is not offered again. The Esc, how long the run had lasted and the files read so far are all recorded.
Esc after the first file: no private file is opened. -
The debrief and the data
On Submit answers the server scores the work and shows the debrief: which answers were right, that one answer was deliberately wrong and was used, and how many private files the assistant opened. The segment fills ten variables, agentic_<segment>_score, _correctCount, _taskCount, _delegatedTasks, _flawedAccepted, _sensitiveReads, _interrupts, _permissionsGranted, _modeFinal and _taskOutcomes. Export the chatroom data as CSV to find Agentic Tasks (one row per question: the answer, where it came from, whether a flawed answer was accepted, whether the sources were checked first, whether the assistant was stopped), Agentic Decisions (each offer, permission, answer, mode change and Esc, with response times), Agentic Events and Agentic Transcripts.
The debrief, then the Agentic sections of the export.