Plate I · AgentLens · 2026
Bone-white pigment · a cool light round the infinity
A review workbench for agent-made changes that finds where a change drifts outside what was authorized, stops it before it runs, and hands a person one decision.
Agent drift / Approval / Explainability
Case summary
- Problem
- Agents proposed changes to support scripts, prices and refund limits. An automatic refund-template change went past policy and caused 2 incidents in which customers were promised too much. One "Approved" label left reviewers unable to tell whether a change had been authorized.
- Users
- Front-line reviewers — a support team lead and an operations configuration engineer.
- Key decision
- Build the list of changed fields from a JSON diff instead of a model summary, split one status into four records, and keep approval and execution as separate steps for price, refund-limit and permission changes.
- Result
- Correct reads of authorization rose from 21/40 to 34/40. Across 318 proposals in 6 weeks, there were 0 unintended executions.
Loading local demo…
The demo could not load. The image and case remain readable; you can try again.
One decision
Why the agent proposes, a person decides, and a receipt follows
AgentLens keeps what an agent wants, what a person decided and what actually ran apart. A change arrives as a proposal, a person records a decision, and only an approved decision can produce an execution receipt, followed by an evaluation result. Shown 20 screenshots and asked whether the agent may act, two front-line reviewers answered correctly 34/40 times (85%) with the four records, against 21/40 (53%) when the screen carried one “approved” label. Each answer took 8 seconds longer.
Engineering wanted fewer clicks: one status, and one “approve” that also ran the change. Before this, the agent had changed a refund template on its own, which led to 2 incidents of promising customers more than they were owed. Looking back, there were three ways to go: one label and one approve-and-run step for every change; separate records and separate steps for every change; or separate records for everything and separate steps where a mistake costs money. The first was fastest and kept the shape of those incidents. Two steps for every wording tweak risked teaching people to click straight through.
Two pieces of evidence settled it. The refund incidents showed what an approval that also executes lets through, so prices, refund limits and core permissions keep approval and execution apart. The comprehension test showed what one label hides: with it, reviewers read the agent’s authority right barely half the time. The four records had a weak spot too: 4 of the 6 wrong answers with them fell on “inconclusive”. The support lead asked whether that counted as success, and whether they could reply to the customer on it. We added a line beside the label: the change has not met its check, and nothing may be promised until the missing measures are in.
Low-risk wording changes now run on one approval and keep a one-step undo. Over 6 weeks and 318 proposals there were 0 unintended executions. The median time from request to execution for high-risk changes went from 4 to 9 minutes; low-risk changes took a median of 3. Reading the state cost 8 seconds a question. The base of incidents was small, so 0 means none happened in those 6 weeks and does not prove the risk is gone.
When an agent can act, write down what it proposes, what a person decides and what runs as separate records a person can read at a glance, and spend friction where a mistake costs money. The 8 seconds a question and the extra minutes on high-risk changes were both cheaper than another refund promise.
- Proposal
- Decision
- Low-risk wording: approval runs it, with a one-step undo
- Rejected or deferred: no receipt can exist
- Execution receipt
- Evaluation result
- Inconclusive: nothing may be promised until the missing measures are in
How the responsibilities connect
- 01ReviewProposal
- 02ReviewDecision
- 03ExecutionReceipt
- 04EvaluationResult
For price, refund-limit and permission changes, approval and execution stay separate steps. Records never act on external systems by themselves.
Brand card · Plate I
AgentLens
Four records a reviewer can read at a glance.
Colour
- Ink 1
#151413The record sheet - Bone
#EDE6D8A decision a person made - Gray 7
#B9B2A6Evidence and fields - Gray 5
#8E887FPending or inconclusive
Type
- GeistRecords and labels
- MonospaceIDs, fields and values
Motion
- Print
240 msThe records print in order: proposal, decision, receipt, result - Light
600 msOnly the state that allows action lights
Problem and context
The workbench reviews changes an agent proposes for customer-support and operations work: reply-script edits, price changes and refund-limit changes. The trouble with agents is often that they do more than they were asked to. Before this work, an automatic change to a refund template went past policy and caused 2 incidents in which customers were promised more than the policy allowed.
To catch that kind of drift, the screen has to answer three questions: what did the agent actually change, did a person agree, and did the change produce the intended result? The original screen showed one “Approved” label that folded all three into one.
User validation: can a reviewer tell when an agent overstepped?
I tested this with the two people who review these proposals every day, a support team lead and an operations configuration engineer. Each saw 20 screenshots of proposals at different lifecycle stages and answered the same question for every one: “From what is shown, is the agent authorized to perform this action?” That made 20 questions per person per round, 40 in total. We ran one round before the redesign and one after, 3 weeks apart, with the order shuffled.
| Version | Correct answers | Median time per answer |
|---|---|---|
| v1: one “Approved” label | 21/40 (53%) | 18 s |
| v2: Proposal, Decision, Execution Receipt and Evaluation Result as four records | 34/40 (85%) | 26 s |
Accuracy went up, and each answer took 8 seconds longer. I accepted that cost: a reviewer who cannot see that an agent overstepped can let a refund change run.
Four of the 6 wrong answers in v2 involved the same state, an evaluation marked “inconclusive”. The support lead asked: “It says the evidence is insufficient. So did it work or not? Can I reply to the customer based on this?” An inconclusive label gave a front-line person no instruction. We added a line next to it: “Not verified. Do not promise this change externally until X more checks pass.”
Evaluation: finding out what the agent actually changed
You cannot catch drift without knowing exactly what changed. Early on, a language model wrote the change summary for each proposal. On 50 proposals, 6 summaries left out a changed configuration field, and 1 invented a field that had not changed: the summary drifted too.
We moved the list of fields and value differences to a structured JSON diff, which covers every field by construction, and kept the model only for explaining the background and reason. After the change, field coverage reached 50/50. Reviewers marked 3/50 generated reasons as inconsistent with the diff, and the system now highlights that kind of summary automatically.
The lifecycle rules are covered by 62 deterministic tests in the internal workbench, all passing. One example: a rejected or deferred decision can never produce an execution receipt. The public AgentLens Review Kit contains 14 of those tests. They prove that the rules run as written. They say nothing about whether a person can read the screen, which is why the comprehension test above exists.
Team decision: split approval and execution where drift is costly
Engineering argued that clicking “approve” and then “execute” for every proposal was slow. The refund incidents argued the other way. We settled on two tiers:
- Display-only copy changes: a single approval runs the change, with one-step undo.
- Price, refund-limit and core permission changes: approval and execution are separate steps and cannot be merged.
Over 6 weeks and 318 proposals, there were 0 unintended executions. The median time from request to execution for high-risk changes went from 4 minutes to 9 minutes. Low-risk changes took a median of 3 minutes.
Design extension: requirement drift in coding agents
Coding agents drift the same way. The task says add a coupon; the agent also edits loyalty points and rewrites a done condition in its own favor. The demo at the top of this page carries the same approach into a Codex workflow, in two parts:
- Diagnose automatically. At the start, the task records a purpose anchor: the purpose, the paths it may write, the done conditions and a checksum. Every event the agent reports is compared against it. 4 deterministic rules check for writes outside scope, a real change to the goal or a condition (rewording alone is not flagged), evidence that predates later changes, and a rewritten anchor.
- Correct automatically. Once drift is found, the task stays at its last verified revision. In-scope progress is kept, out-of-scope writes are quarantined, a changed goal becomes a revision candidate, and the anchor is injected into the agent’s next turn. A person chooses once: continue the original task, update it, or split the extra work into a new task. A false alarm can be marked as not drift, and the same pattern stays quiet until new evidence arrives.
This part is at the design stage. The rules come from requirement documents, the demo is an interactive prototype built from them with fixed sample data, it has not shipped, and there is no user data for it yet.
Limits
The comprehension test had two participants. Past incidents were few, so 0 incidents in 6 weeks only says none happened in that window; it does not prove the risk is gone. The drift checks for coding agents have not been tested with users.
What I would do differently
I would test the “inconclusive” wording with reviewers before shipping the four-record screen, since that single state produced most of the remaining errors. I would also track how often reviewers open the diff before approving, in addition to whether they approve: the diff is where drift gets caught.
Watch the review flow
The 60-second film
Small enough to inspect
Follow a decision through.
Approved, executed, evaluated
- A proposal is recorded
An idea is pending review. Nothing has been approved or executed.
Inspect record
{ "kind": "review_proposal", "id": "proposal-approved", "at": "2026-09-18T09:00:00.000Z", "candidate_ref": "workflow:creative-review", "version": 1, "reason": "Try an explicit confirmation step before applying a suggested edit.", "status": "review_pending", "content_hash": "sha256:8a5600e85cbbf20e8eb9eff296d22b1b67745864b5a81317e28d8b31f01e4a08" } - A separate human decision
The reviewer permits a bounded trial, not a claim of success.
Inspect record
{ "kind": "review_decision", "id": "decision-approved", "at": "2026-09-18T09:01:00.000Z", "proposal_ref": "proposal-approved", "proposal_hash": "sha256:8a5600e85cbbf20e8eb9eff296d22b1b67745864b5a81317e28d8b31f01e4a08", "candidate_ref": "workflow:creative-review", "candidate_version": 1, "decided_by": "local_user", "status": "authorized_for_trial", "content_hash": "sha256:407589f684e2d6b36ab8902fd4f1ba374ee9ba302632138877515d2a4773c876" } - Execution has its own result
The execution receipt reports completion, without claiming product quality.
Inspect record
{ "kind": "execution_receipt", "id": "execution-approved", "at": "2026-09-18T09:02:00.000Z", "decision_ref": "decision-approved", "decision_hash": "sha256:407589f684e2d6b36ab8902fd4f1ba374ee9ba302632138877515d2a4773c876", "candidate_ref": "workflow:creative-review", "status": "executed", "change_ref": "change:synthetic-example", "content_hash": "sha256:82112bcb808f320146ebfbba53365228857e1dcaf6b122a9094198df63ca3398" } - Evaluate the evidence
All supplied example signals meet the confirmation rule.
Inspect record
{ "kind": "evaluation_result", "id": "evaluation-approved", "at": "2026-09-18T09:03:00.000Z", "execution_ref": "execution-approved", "execution_hash": "sha256:82112bcb808f320146ebfbba53365228857e1dcaf6b122a9094198df63ca3398", "candidate_ref": "workflow:creative-review", "status": "confirmed", "evidence_complete": true, "checks_passed": true, "regression_found": false, "content_hash": "sha256:3dca0e1c4d5a60b59cfd8957db0b9f7116de484cce6cc2dcc225ef7c87b99a2a" }
Rejected means no execution
- A proposal is recorded
An idea is pending review. Nothing has been approved or executed.
Inspect record
{ "kind": "review_proposal", "id": "proposal-rejected", "at": "2026-09-18T09:00:00.000Z", "candidate_ref": "workflow:creative-review", "version": 1, "reason": "Try an explicit confirmation step before applying a suggested edit.", "status": "review_pending", "content_hash": "sha256:54ff9008680878490a48a635b6e41faa28bd575e8310dae32475a79f8d7156a6" } - A separate human decision
The reviewer rejects this trial.
Inspect record
{ "kind": "review_decision", "id": "decision-rejected", "at": "2026-09-18T09:01:00.000Z", "proposal_ref": "proposal-rejected", "proposal_hash": "sha256:54ff9008680878490a48a635b6e41faa28bd575e8310dae32475a79f8d7156a6", "candidate_ref": "workflow:creative-review", "candidate_version": 1, "decided_by": "local_user", "status": "rejected", "content_hash": "sha256:233b373c1becf6683b02a2844cb66ec85996433758a6e3e8565b0715bd1498be" } - Execution is blocked
The module refuses to create an execution receipt from a rejected decision.
Inspect record
{ "kind": "blocked_attempt", "code": "TRIAL_NOT_AUTHORIZED" }
An execution can fail
- A proposal is recorded
An idea is pending review. Nothing has been approved or executed.
Inspect record
{ "kind": "review_proposal", "id": "proposal-failed", "at": "2026-09-18T09:00:00.000Z", "candidate_ref": "workflow:creative-review", "version": 1, "reason": "Try an explicit confirmation step before applying a suggested edit.", "status": "review_pending", "content_hash": "sha256:adfe434a3da6c2f4488c5d95bba4e257428dae647d493ba75cb0a392f8589652" } - A separate human decision
The reviewer permits a bounded trial, not a claim of success.
Inspect record
{ "kind": "review_decision", "id": "decision-failed", "at": "2026-09-18T09:01:00.000Z", "proposal_ref": "proposal-failed", "proposal_hash": "sha256:adfe434a3da6c2f4488c5d95bba4e257428dae647d493ba75cb0a392f8589652", "candidate_ref": "workflow:creative-review", "candidate_version": 1, "decided_by": "local_user", "status": "authorized_for_trial", "content_hash": "sha256:0c13d6cc93cb0c35efa0d5119863c1ebda2f60343018788473c5a33f7a581967" } - Execution has its own result
A failure remains a failure; the decision is not rewritten.
Inspect record
{ "kind": "execution_receipt", "id": "execution-failed", "at": "2026-09-18T09:02:00.000Z", "decision_ref": "decision-failed", "decision_hash": "sha256:0c13d6cc93cb0c35efa0d5119863c1ebda2f60343018788473c5a33f7a581967", "candidate_ref": "workflow:creative-review", "status": "execution_failed", "change_ref": "change:synthetic-example", "content_hash": "sha256:938dd4a03c4c16e257a65842ff16fb416fe7bb0acd00ee9be6f1e3ca8399d4d4" } - Evaluate the evidence
The evidence does not support confirmation. Keep the result inconclusive.
Inspect record
{ "kind": "evaluation_result", "id": "evaluation-failed", "at": "2026-09-18T09:03:00.000Z", "execution_ref": "execution-failed", "execution_hash": "sha256:938dd4a03c4c16e257a65842ff16fb416fe7bb0acd00ee9be6f1e3ca8399d4d4", "candidate_ref": "workflow:creative-review", "status": "inconclusive", "evidence_complete": false, "checks_passed": false, "regression_found": false, "content_hash": "sha256:aea45a6dc7e34ad691ee50c80ca498b0640115cf698cd92acaa0dacd1f246304" }
Done is not yet proven
- A proposal is recorded
An idea is pending review. Nothing has been approved or executed.
Inspect record
{ "kind": "review_proposal", "id": "proposal-uncertain", "at": "2026-09-18T09:00:00.000Z", "candidate_ref": "workflow:creative-review", "version": 1, "reason": "Try an explicit confirmation step before applying a suggested edit.", "status": "review_pending", "content_hash": "sha256:75498a88e2cbf7aaf63c75a94a7c851008af57bfd275df673b7716fb09f5d5f1" } - A separate human decision
The reviewer permits a bounded trial, not a claim of success.
Inspect record
{ "kind": "review_decision", "id": "decision-uncertain", "at": "2026-09-18T09:01:00.000Z", "proposal_ref": "proposal-uncertain", "proposal_hash": "sha256:75498a88e2cbf7aaf63c75a94a7c851008af57bfd275df673b7716fb09f5d5f1", "candidate_ref": "workflow:creative-review", "candidate_version": 1, "decided_by": "local_user", "status": "authorized_for_trial", "content_hash": "sha256:a8f062759c7be8a3528c0bffc04c193c094822440a1fd3ed7067eb5c70400476" } - Execution has its own result
The execution receipt reports completion, without claiming product quality.
Inspect record
{ "kind": "execution_receipt", "id": "execution-uncertain", "at": "2026-09-18T09:02:00.000Z", "decision_ref": "decision-uncertain", "decision_hash": "sha256:a8f062759c7be8a3528c0bffc04c193c094822440a1fd3ed7067eb5c70400476", "candidate_ref": "workflow:creative-review", "status": "executed", "change_ref": "change:synthetic-example", "content_hash": "sha256:6ef722d4e4726a926bf28b875dbaf78e79109ea0ca187813ec280a0a5c09bb78" } - Evaluate the evidence
The evidence does not support confirmation. Keep the result inconclusive.
Inspect record
{ "kind": "evaluation_result", "id": "evaluation-uncertain", "at": "2026-09-18T09:03:00.000Z", "execution_ref": "execution-uncertain", "execution_hash": "sha256:6ef722d4e4726a926bf28b875dbaf78e79109ea0ca187813ec280a0a5c09bb78", "candidate_ref": "workflow:creative-review", "status": "inconclusive", "evidence_complete": false, "checks_passed": false, "regression_found": false, "content_hash": "sha256:9b2f342354b9ff6922be283e176161e371e81726506eb5611ee443beecfee2e2" }