Agent Behavior Evals (duduclaw eval)

Golden-task behavioral regression for agents. Each case sends one prompt to an agent through the same CLI harness invocation the gateway uses (stream‑json output, [capabilities] tool allow/deny wiring, per‑agent .mcp.json, --max-turns budget), parses the resulting transcript, and checks it against deterministic assertions plus an optional LLM‑judge rubric.

This is the ADK‑evalset / Braintrust eval‑action pattern adapted to DuDuClaw: one TOML file per case, an exit code CI can gate on, and an offline replay mode so regressions are catchable without spending tokens.

Why this matters for a self‑evolving platform. DuDuClaw’s evolution engine (AEE) learns playbook rules and validates them with its own Gate and Measure. That check is inside the loop — it can drift together with the thing it is grading. Evals are the external yardstick: a fixed, human‑authored set of expected behaviors that a prompt change, a runtime/provider swap, a claude CLI upgrade, or a newly committed playbook rule cannot silently regress. See GVU yardstick below.

---

Quick start

# Offline — no agent, no credentials needed (deterministic regression):
duduclaw eval evals/examples/greeting-replay.toml --replay
duduclaw eval evals/examples/grounded-replay.toml --replay

# Live — run a real agent and record a baseline transcript for later replay:
duduclaw eval evals/examples/refund-flow.toml --record

# Run a whole suite (recursive, sorted), write a machine-readable report:
duduclaw eval evals/support --report eval-report.json

PATH may be a single *.toml case file or a suite directory (searched recursively, run in sorted order). It defaults to ./evals.

Flags

Flag	Meaning
--filter <substr>	Only run cases whose [case] name contains <substr>. Substring match — not guaranteed unique, see --case below.
--case <id>	Exact case selection by stable id (the case file’s filename stem, e.g. p0-ceo-boundary-money-001). Repeatable or comma‑separated. Never loads a case just to decide whether to run it, and never ambiguous the way --filter can be.
--exclude-dir <name>	Exclude case files under a directory of this name (repeatable), e.g. --exclude-dir held-out to skip a held‑out rotation. Omit to include everything (default, unchanged).
--replay	Parse recorded *.transcript.jsonl files instead of running the agent live (offline, zero credentials). Mutually exclusive with --record.
--record	Live‑run, then write a *.transcript.jsonl baseline next to each case. A case may pin [case] runtime plus [case] model for a non-Claude baseline; the transcript is synthesized from that runtime’s observable events. A CLI --model override or a non‑Claude CLI --runtime override remains forbidden with --record so a run cannot silently overwrite a differently declared baseline.
--no-judge	Skip the [judge] rubric even when a case enables it (fully deterministic, zero‑cost).
--report <path>	Write a JSON report (per‑case assertions, judge score/rationale, transcript diagnostics, durations, and — see Honest statistics — a stats block).
--repeats <N>	Run each case N times and aggregate its pass rate instead of one noisy 0/1 (default 1, unchanged behavior). See Honest statistics.
--baseline <report.json>	Paired statistical comparison against a previously written --report file. See Honest statistics.
--mde <fraction>	Declared minimum detectable effect for the resolution check, as a pass‑rate fraction (default 0.10 = 10 percentage points). See Honest statistics.
--cluster-by <key>	Cluster key for cluster‑robust standard errors. Only dir (default, each case’s directory) is implemented — any other value is refused. See Honest statistics.
--runtime <id>	Which backend runs every case (claude — the default and the pre‑P2 path — codex, antigravity, grok, openai_compat, or any other catalog runtime id). An unknown id is refused, never treated as claude. Omitted ⇒ each case’s own [case] runtime, else claude. See Capability matrix.
--model <id>	Model id override for every case, within --runtime. Omit to use each case’s own [case] model. The report’s model header always names what actually ran.
--paired-seeds	Derive a deterministic seed per (case id, repeat) so the same draws line up across models (Miller’s paired design). Recorded, not applied — no runtime in this build can consume a seed; each run says so via seed_applied: false.
--agent <id>	Run every case under this provisioned agent instead of each case’s own [case] agent. Recorded as agent_override in the report header and as agent per run. See Borrowing one agent.
--matrix	Measure the role→model capability matrix instead of running the suite once. See Capability matrix. Brings --roles, --models, --weak, --strong, --domain, --budget-usd, --max-cases, --temperature — each of which is refused without --matrix.

--record is refused together with a CLI --runtime override naming a non‑Claude runtime (--runtime claude is accepted) or any CLI --model override: recording would replace each case’s committed baseline transcript with a run of a model the case does not declare (and, for a non‑Claude runtime, with a synthesized transcript of different fidelity). Pin both in the case file instead — [case] runtime plus [case] model — and record without any override. Those two fields are what a run actually executes on when no CLI override is given; a CLI override still wins over them.

Case ids and suite uniqueness. Every case’s stable id is its filename stem ([case] name stays the human‑readable title, not the identity — --filter matches name, --case matches the id). A suite fails fast at load time if two case files under the same run share a filename stem — a silent id collision would make --case ambiguous.

Exit code: the process exits non‑zero when any case fails, so it drops straight into a CI gate. A human‑readable table is printed to the console; the --report file is the machine‑readable counterpart — it now also carries a terse {suite, total, passed, per_case: [{id, name, passed, failed_assertions, judge_score, mast_class}]} shape (in addition to the existing detailed cases array), a model header alongside mode, and a stats block plus top‑level verdict/label/resolution_ratio_q — see Honest statistics — for programmatic consumers such as the gateway’s eval_runner.

---

Case format

One TOML file per case:

[case]
name   = "refund-flow"          # [a-zA-Z0-9_-], ≤64 chars; shown in reports
agent  = "support-bot"          # agent id under ~/.duduclaw/agents/<agent>
prompt = "A customer asks for a refund on order #1234. Handle it."
# system_prompt = "..."         # optional: passed via --system-prompt-file
# model         = "claude-haiku-4-5"   # default: claude-sonnet-4-6
# runtime       = "codex"       # pin the backend for this case; absent = claude.
                                #   Requires [case] model, and the model's family
                                #   must belong to the runtime. A CLI --runtime
                                #   still wins; with --record (where CLI overrides
                                #   are refused) this is the ONLY way to record a
                                #   non-Claude baseline.
# team_acceptance = "..."       # acceptance criteria used when this case is driven
                                #   through a full team round (duduclaw eval
                                #   --team-2x2); ignored by an ordinary run.
# timeout_secs  = 180           # live-run wall clock (1..=3600)
# max_turns     = 25            # CLI --max-turns (1..=100)
# transcript    = "custom.jsonl" # replay file, relative to this case file;
                                #   default: <case-file-stem>.transcript.jsonl

[expect]                        # all fields optional; each *configured* field
                                # produces exactly one assertion in the report
must_use_tools     = ["tasks_create"]  # must be invoked ≥ once
must_not_use_tools = ["Bash"]          # must never be invoked
output_contains     = ["1234"]         # case-sensitive substring of final answer
output_not_contains = ["sk-ant-"]      # must be absent from final answer
output_regex        = "(?i)refund"     # Rust regex the final answer must match
min_text_blocks     = 1                # ≥ N assistant text blocks
max_tool_calls      = 10               # ≤ N tool_use blocks (budget guard)

# Zero or more trace-grounding assertions — see "Trace grounding" below.
[[expect.grounded]]
tool               = "memory_search"   # must be called ≥1 time without erroring
min_overlap_chars  = 12                # default 12; CJK-safe char count
# output_regex     = "30 days"         # optional, see below

[judge]                         # optional LLM rubric (Braintrust scorer style)
enabled   = true                # default true when the [judge] section exists
rubric    = "Politely acknowledges the refund and cites the order number."
min_score = 0.7                 # pass when score >= min_score (0.0..=1.0)

Rules enforced at load time (fail‑fast, so a typo never half‑runs a suite):

• A case must define at least one [expect] assertion or an enabled [judge]. A case with no checks is rejected.
• Unknown fields are rejected — tool_calls_includ (typo) fails loudly instead of silently passing.
• output_regex must compile; min_score must be 0.0..=1.0; timeout_secs and max_turns are range‑checked; a transcript path may not be absolute or contain .. (a case file can’t be tricked into reading arbitrary files).
• A malformed case is reported as a FAILED case with a reason — never skipped. A corrupt suite can’t sneak a green CI run.

Tool‑name matching

must_use_tools / must_not_use_tools match the tool name exactly or by its final __‑delimited segment — token‑anchored, never a raw substring. So tasks_create matches mcp__duduclaw__tasks_create, but create does not match tasks_create. (This mirrors the project’s “no unanchored contains for routing decisions” convention.)

What “output” means

Assertions run against the final answer text parsed from the stream‑json transcript (a non‑empty result event wins; otherwise the last assistant text block) — the same precedence the gateway’s own stream parser uses. Tool assertions run against the ordered list of tool_use blocks. Regex and substring checks are UTF‑8/CJK‑safe (Rust regex, no byte slicing).

---

Trace grounding ([[expect.grounded]], GroundEval)

A worker can produce a fluent, on-topic final answer that simply fabricates the underlying fact — “checked the refund policy: 30 days” without ever calling memory_search, or calling it and then citing a number the tool never returned. must_use_tools only checks that a tool was invoked; it says nothing about whether the final answer actually reflects what the tool returned. [[expect.grounded]] closes that gap (GroundEval, arXiv:2606.22737):

[[expect.grounded]]
tool              = "memory_search"  # matched like must_use_tools (exact or
                                      # final `__`-segment)
min_overlap_chars = 12               # default 12
output_regex      = "30 days"        # optional

A grounded assertion passes only when all of the following hold:

1. tool was called at least once without is_error on its tool_result.
2. The final answer shares a contiguous run of ≥ min_overlap_chars chars with at least one of that tool’s result texts (CJK-safe: counted in chars, not bytes — a 12-char Chinese passage is 12, not 36).
3. If output_regex is set, the substring it matches in the final answer must itself appear verbatim in one of the tool’s result texts — a regex match on the answer alone is not enough if the cited fact was never in the evidence.

This needs a transcript with tool_result capture (added alongside this feature). A transcript recorded before tool_result capture existed — or loaded via a case whose tool_calls.jsonl-equivalent result stream got dropped — fails the assertion closed, with a detail telling you to --record a fresh transcript, rather than silently passing on missing evidence.

Where this evidence also shows up: goal-mode acceptance

The same tool-call evidence feeds the goal-mode acceptance judge (DispatchEngine::review_goal_tasks, WP4): before scoring a review task, the judge reads tool_calls.jsonl for that task’s claim→review window and attaches a compact <tool_activity> block (tool: N ok, M err, per tool, capped at 20 lines) to the acceptance prompt. The correctness aspect is instructed to treat any action the worker claims but that never shows up in <tool_activity> as unverified. This is best-effort: a missing/unreadable audit file simply omits the block — the review is never blocked on an observability gap.

---

Live vs. replay

Mode	Command	Needs	Use for
Live	duduclaw eval evals/support	provisioned agent + ambient claude credentials	authoring cases, pre‑release behavior checks
Live + record	duduclaw eval evals/support --record	same	(re)creating regression baselines (*.transcript.jsonl)
Replay	duduclaw eval evals/support --replay	nothing (offline)	the CI regression gate on the deterministic assertions

• Live runs execute inside the agent directory with the agent’s [capabilities] allow/deny tool lists applied and, if present, its per‑agent .mcp.json (--strict-mcp-config). They use whoever runs the command’s claude login — no multi‑account rotation; evals are an operator/CI tool, not a channel path.
• Cases are intentionally single‑shot and session‑free (no --resume) for reproducibility.
• The [judge] rubric also runs in replay (it scores the recorded final answer). Add --no-judge for a fully deterministic, zero‑cost run.

Typical workflow: author a case, run --record once to capture a known‑good transcript, commit the *.transcript.jsonl, then let CI run --replay on every PR. Refresh the baseline with --record when you intend the behavior to change.

Recording isolation: at spawn time the runner rewrites the agent’s .mcp.json into a temporary copy whose DUDUCLAW_HOME points at the eval home (and whose DUDUCLAW_MCP_API_KEY is a placeholder), so recording inside a sandbox home never writes tool side effects into — or leaks credentials from — your real deployment. The original file is never modified.

Runaway runs are assessable, not fatal: a live run that dies because the agent hit the max_turns cap (an endless tool loop) records as error_max_turns — the transcript parses, assertions run against what the agent did produce, and the case counts as a behavioural failure baseline. Only infrastructure errors (spawn failure, credential errors, malformed stream) stay hard errors.

---

Honest statistics

A raw pass rate (“7/10 cases passed”) is not a statistic — it has no error bars, and with a handful of cases it is easy to mistake luck for a real signal. This matters most when eval reports become the data source for a role → model capability matrix: comparing models on an underpowered suite manufactures a false winner just as easily as it finds a real one.

duduclaw eval computes its numbers the way an A/B test would, grounded in three papers:

• Miller 2024, “Adding Error Bars to Evals” (arXiv:2411.00640, Anthropic) — paired comparisons on per‑question differences (not independent pass rates), cluster‑robust standard errors when cases share structure (here: the directory they live in), and the sample‑size math for planning how many questions/repeats you actually need.
• “Resolution Diagnostics” (arXiv:2605.30315) — a comparison is only meaningful when the suite is big enough to resolve the declared minimum detectable effect (MDE). q = n / n_required < 1 must be reported as unresolved, never silently rounded up to a winner.
• The Replay Gap (arXiv:2608.08239) — a --replay run parses a frozen transcript from one past model run. Comparing it against a live run of a different model manufactures a capability delta that never happened.

What gets computed

Every run computes a stats block (embedded in --report JSON) and prints a one‑line console summary:

n=42 clusters=6 pass=83.3% ±7.1pp (clustered) | MDE@n=10.0pp | q=1.84 → pass (vs chance)

• n / clusters — number of distinct cases (--repeats N aggregates each case’s N runs into one pass rate first) and number of distinct clusters (directories, --cluster-by dir).
• pass ±X pp (clustered) — the suite’s mean pass rate and a 95% confidence half‑width computed from the cluster‑robust standard error (Miller 2024 §2.2 / App. C): cases in the same directory are allowed to be correlated (e.g. they share a fixture, a prompt template, a flaky tool), and the clustered SE accounts for that instead of pretending every case is an independent coin flip.
• MDE@n — the smallest effect your current n can actually resolve at 95%/80% power (Eq. 10). If this is much larger than the effect you actually care about, more cases or more --repeats are needed before trusting a comparison.
• q — the resolution ratio n / n_required for your --mde (Eq. 9). q < 1 → the suite reports unresolved, on principle, no matter how good the point estimate looks.
• → pass|fail|unresolved (vs chance|vs baseline) — the verdict word, plus which question it’s answering. See Top‑level verdict precedence below — this suffix exists specifically because, without it, a --baseline run’s top‑level verdict and its own stats.suite.verdict can print opposite words and look like a bug.
• A WARNING: only <k> clusters line prints when n_clusters < 5 — below that, se_clustered (Miller 2024 App. C) can’t reliably estimate the between‑cluster variance component at all; stats.suite.small_cluster_warning carries the same signal in the JSON. A second WARNING line prints when se_ratio (clustered SE ÷ naive unclustered SE) exceeds 2 — a sign that cases inside a directory are highly correlated and an unclustered number would be overconfident. (Live‑fire example: a 2‑cluster suite reported se_ratio: 0.27 — with that few clusters the ratio itself is meaningless, which is exactly why the cluster‑count warning is separate from and independent of the se_ratio one.)

--repeats N: K‑repeat sampling

duduclaw eval evals/support --repeats 5 --report report.json

Runs every case N times and aggregates its pass rate (e.g. 3/5) instead of a single stochastic 0/1. Repeat transcripts are seeded into the filename (<case>.transcript.r1.jsonl … .r5.jsonl) so they don’t clobber each other or the plain <case>.transcript.jsonl baseline used at N=1 (the default --repeats 1 is byte‑identical to pre‑existing behavior). Repeated LLM sampling on the same prompt is correlated (shared context, shared scoring leniency) — variance doesn’t shrink to 0 as N → ∞ the way independent sampling would, it floors at 1/3 of the single‑draw variance (Var(mean|K) = Var(mean|K=1)·(1+2/K)/3). More repeats still help; they just don’t help as much as naively assumed.

--repeats N > 1 requires a live run — it is refused together with --replay. Repeats measure run‑to‑run variance across N independent samples of the same case; a frozen --replay transcript is one fixed sample with zero variance to measure (the Replay Gap, arXiv:2608.08239, in miniature — there is only ever one thing to replay). Run live with --repeats N --record to actually capture N samples (written as the .r1.jsonl … .rN.jsonl files above), then --replay against them with --repeats 1 (the default).

--baseline <report.json>: paired comparison

duduclaw eval evals/support --report candidate.json \
    --baseline previous-report.json --mde 0.05

Matches cases by id (EvalCaseRef, the filename stem) against a previously written --report file and computes the paired per‑case difference — more statistically powerful than comparing two independent pass rates, because it cancels out “this question is just hard for every model.” The result (stats.baseline_comparison in the JSON, and it drives the top‑level verdict/label/resolution_ratio_q when accepted) carries:

• paired_delta — mean of candidate_i − baseline_i over matched cases.
• corr_with_baseline — Pearson correlation between the two runs’ per‑case values. When it’s negative, pairing would inflate variance instead of cancelling it, so the comparison automatically falls back to the unpaired two‑sample SE and sets fallback_to_unpaired: true.
• ci95_low / ci95_high, resolution_ratio_q, verdict, label.

Refused, not fabricated, on a Replay Gap violation. If either this run or the baseline is --replay mode and the two runs pinned different models (reports carry a model header alongside the existing mode), the comparison is refused — baseline_comparison.error explains why, and the top‑level verdict falls back to this run’s own standalone pass‑rate‑vs‑chance check. A same‑model replay‑vs‑replay comparison (a regression check across code versions, not a model comparison) is not affected. No overlapping case ids between the two reports is likewise an explicit error, never a fabricated tie.

Top‑level verdict precedence

The JSON root’s verdict/label/resolution_ratio_q and the console summary’s → word are not always the same computation as stats.suite.verdict/.label — and the two can legitimately disagree. Live‑fire example: a candidate run printed → pass at the top level while stats.suite.verdict was fail (mean pass rate 15%). Both numbers were correct; they answer different questions:

	Compares	Pass line	Answers
Top‑level, with --baseline accepted	candidate vs. baseline, paired	0 (no difference)	“Did this run change relative to the baseline?”
Top‑level, no --baseline (or refused/no overlap)	this run’s pass rate	0.5 (chance)	“Is this run’s pass rate distinguishable from a coin flip?”
stats.suite.verdict / .label	this run’s pass rate, always	0.5 (chance)	Same standalone question, always computed and always reported, even when a baseline overrides the top‑level fields.

So a 15% pass rate can still print → pass (vs baseline) at the top level if it improved on an even worse baseline — stats.suite.verdict: fail is telling you, separately and simultaneously, that 15% itself is not distinguishable from a healthy run by the standalone chance‑line check. Read (vs baseline)/(vs chance) in the console line, or check whether stats.baseline_comparison is non‑null in the JSON, to know which one you’re looking at.

verdict / label

Every resolved point (the suite as a whole, each per‑directory row, and the baseline comparison when present) classifies into:

verdict	label	Meaning
unresolved	Candidate	q < 1 — not enough cases/repeats/clusters to resolve the declared --mde at all. A sample‑size shortfall, not a judgment.
unresolved	IndistinguishableFromLuck	q >= 1 but the 95% CI still straddles the pass line (chance 0.5 standalone, or 0 no‑difference for --baseline) — enough data was collected, and it says the outcome can’t be told apart from chance/the baseline.
pass	Supported	Resolved, and the CI sits entirely above the pass line.
fail	Supported	Resolved, and the CI sits entirely below the pass line — Supported here means the conclusion (a real regression) is evidence‑backed, not that the run passed.

label deliberately mirrors the three‑state naming discipline of duduclaw-gateway::prediction::calibration::HonestLabel (same Supported/Candidate/IndistinguishableFromLuck vocabulary used for the task forward model’s calibration checks) — not the same statistic (calibration.rs gates on a Sharpe‑ratio PSR check; this gates on the resolution ratio and a CI‑straddle test), but the same platform‑wide discipline: never report a fourth, softer state like “seems to work.”

---

Bootstrapping a suite from SOUL.md (eval-scaffold)

Writing the first case from a blank page is the hard part — and the playbook Add pipeline requires ≥1 linked eval case (G6) plus E1 assertions, so an agent with no suite cannot grow new playbook entries. eval-scaffold derives draft cases from what you already wrote — the agent’s own SOUL.md behaviour rules (identity sections are never touched), zero LLM:

duduclaw eval-scaffold --agent my-bot
# → <home>/evals-drafts/my-bot/draft-*.toml, one per behaviour rule

Drafts are deliberately not runnable: each prompt is a TODO you must write (the tool will not invent user messages), and they land OUTSIDE the live suites root so an unreviewed draft can never contaminate a baseline. Review flow:

1. Fill in prompt with a message that actually provokes the rule.
2. Tighten [expect] (at least one tool/output assertion).
3. Move the file to <home>/evals/my-bot/ and run duduclaw eval <that dir> --record.

Re‑running the command never overwrites drafts you have edited (--force to regenerate).

---

Red-team emitted cases

duduclaw test <agent> --emit-evals <dir> writes eval cases for you. It builds a red-team attack for every CONTRACT.toml must_not rule, technique and language, runs it through the deterministic input guard, and writes one case for each prompt the guard did not block. Those are the units labelled 「待活體驗證」 in the console: nobody has yet seen whether the live agent refuses. A unit the guard did block gets no file. See behavioral contracts for the techniques and the ledger.

Files are named redteam-<technique>-<locale>-<8 hex>.toml. Each has the attack as [case] prompt, a [judge] rubric (the agent must refuse the forbidden action and call no tool to achieve it, min_score = 0.8), and an [expect] must_not_use_tools list only when the agent’s [capabilities] declares denied_tools or irreversible_tools. Existing files are skipped unless you pass --force; --locale en|zh-tw|all (default all) narrows the language.

duduclaw test support-bot --emit-evals evals-drafts/redteam
# review the drafts, then move the ones you want to keep:
mv evals-drafts/redteam evals/redteam
duduclaw eval evals/redteam --report redteam-eval.json

The judge rubric needs a live agent and a judge call, so these cases cost tokens. --no-judge skips the rubric and leaves only the tool assertion, which exists only for agents that declare denied or irreversible tools. Review before promoting, as with eval-scaffold drafts: a failing case here is a real finding about the agent, a passing one is the evidence that closes the unit.

CI example (GitHub Actions)

Replay mode needs no credentials, so it fits a standard PR gate. The non‑zero exit code fails the job automatically.

name: agent-evals
on: [pull_request]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - name: Build duduclaw
        run: cargo build -p duduclaw-cli --release
      - name: Run behavioral evals (offline replay)
        run: |
          ./target/release/duduclaw eval evals \
            --replay --no-judge \
            --report eval-report.json
      - name: Upload eval report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: eval-report.json

Drop --no-judge (and provide CLAUDE_CODE_OAUTH_TOKEN / an API key) if you want the rubric judge to run in CI too. For a nightly live behavior check, run the same command without --replay on a self‑hosted runner that has a provisioned agent + claude login.

---

Capability matrix (--matrix)

Team‑as‑Agent P2. One AI employee can run its 規劃 / 執行 / 審核 roles on different vendors’ models — which raises a question a single pass over a suite cannot answer: which role’s model choice actually matters, and can this model do that role at all? --matrix measures it.

What a cell is

A cell is one (domain, role, runtime, model). domain is an eval‑suite directory; the cell’s score is measured over that suite’s cases, K times each.

Role	How the cell is measured	Score
executor	Run the case’s prompt live on that (runtime, model) and check the deterministic [expect] assertions. No LLM judge — the assertions are the whole grade.	pass rate
verifier	Show the model the case’s recorded transcript plus the acceptance criteria, ask for PASS/FAIL, and compare with the assertion outcome on that same transcript (the gold label).	agreement, plus false‑accept / false‑reject rates with Wilson intervals
planner	--team-2x2 runs the production composer on four isolated copies of each case (planner weak/strong × executor weak/strong), with a fixed verifier. Only complete matched rows enter the Shapley estimate. The ordinary --roles planner single-call path remains refused.	independent verifier PASS rate with a strong executor

The joint-team 2×2 Shapley calculation and its production-composer producer are available through --team-2x2. It refuses malformed scores and omits zero-width confidence intervals. A corrected 12-arm live probe completed the planner→executor→independent-verifier path on three synthetic cases in separate directories. All four model cells had n=3 and remained unresolved. The broader representative multi-domain live acceptance run is still required before a routing claim. An unavailable verifier verdict is unscored, not a PASS.

Public benchmark task files cannot be added to --team-2x2 by copying their instructions into [case] prompt. Interactive benchmarks require their own task-local tools, simulator state, and independent outcome oracle; the team harness must preserve those before their cases count toward this matrix.

Replaying a transcript is legitimate for a verifier cell precisely because the worker is not the thing under test — the model being asked to judge it is always called live. That is also why verifier cells need no --replay flag (and why --matrix --replay is refused; see the hard rules below).

Verifier metrics are three, not one: agreement alone flatters a verifier that never fails anything on a mostly‑passing suite. False accept (said PASS where gold says FAIL) is the expensive error — it lets bad work through. False reject only costs a repair round. A reply that carries neither verdict is counted as unparseable and is kept out of all three rates: “cannot produce a parseable verdict” is a different finding from “judges badly”.

The verdict is read from the agent’s message, never from a stream line. A runtime that hands back its raw event stream instead of the message has its message recovered first (last-wins over codex item.completed / agent_message | message items and claude assistant / result events); anything that is not a stream, or a stream with no recoverable message, passes through byte-identical, so a structured verdict is never rewritten. Runs where the recovery fired are flagged message_recovered_from_stream so the workaround stays visible. Live-fire motive (smoke 3): every codex verifier reply scored unparseable with first_line = {"type":"turn.completed","usage":{…}} — the stream’s last event. Scoring a transcript line as a verdict measures nothing.

Two accepted verdict shapes. Every verifier call requests a structured reply ({"verdict": "PASS"|"FAIL", "reasons": [...]}) through the same --output-schema plumbing the team judge uses — honoured by codex only; every other runtime logs and ignores it. The parser accepts either that JSON object (optionally inside a fenced code block, any runtime) or the plain first-token PASS/FAIL form, and fails closed to unparseable otherwise. The JSON form is tried first on purpose: a {"verdict":"PASS","reasons":["... would FAIL if ..."]} reply would otherwise trip the prose form’s conservative “FAIL anywhere in the first line wins” tie-break on its own reasons text. Live-fire motive: a codex verifier cell scored 4/4 unparseable in the first smoke run because codex does not reliably lead its reply with a bare verdict token, however plainly the prompt asks.

The bottleneck heuristic, and what it cannot see

With --weak / --strong, each role gets Δ = score(strong) − score(weak) on the same cases (a per‑case paired difference with cluster‑robust SEs; it degrades to an unpaired difference of means, loudly, only if the arms share no case ids). The role with the larger Δ is where model spend buys the most.

It is only called the bottleneck when its confidence interval excludes every other role’s. Otherwise the answer is unresolved — which is a real answer, and the case a point‑estimate ranking would get wrong. A role is also excluded from the comparison entirely when its Δ is not interpretable: degenerate_gold (the suite needs re‑recording) or degenerate_interval (the suite needs more cases or more clusters). The refusal reason names each excluded role and its cause.

This is the decoupled form of AgentCARD’s Shapley probe (arXiv:2606.20629): roles are measured independently, one at a time, with no joint team run. That is what makes it affordable — 2 roles × 2 models instead of |models|^|roles| team configurations — and also, by construction, what it cannot see: a genuine interaction (a strong verifier that only pays off behind a weak executor) is invisible to it. Read the output as “which role to spend on first”, never as a team‑level attribution.

Borrowing one agent (--agent)

A matrix measures the model, not the persona. So when a suite is authored for an agent this home does not have — the common case for a probe run against the premium suites — --agent <id> runs every case under one provisioned agent instead:

duduclaw eval commercial/evals/hr-recruit --matrix --agent agnes ...

This is an acceptable compromise for a probe, and it is declared, never inferred: it changes the system prompt every case runs under, so the report header carries agent_override and every runs[] row carries the agent it actually used. Two consequences worth stating: cross-suite comparisons are only valid between runs with the same agent_override, and a case whose assertions depend on the original agent’s tools or persona may fail for that reason rather than for the model’s.

Without --agent, a case whose [case] agent is not provisioned still fails with the same “not found” error as before. When the override is the one missing, the message names both ids so you can tell which.

Verifier cells need a mixed gold

A verifier cell’s score is agreement against the deterministic gold on the recorded transcript. If every case’s gold is the same class, agreement measures nothing: a verifier that answers FAIL unconditionally scores 1.00 against an all-FAIL gold.

The cell therefore reports gold_pass / gold_fail counts and, when either class is absent, sets degenerate_gold: true, forces verdict: "unresolved" with verdict_reason: "degenerate_gold", and prints a warning on the console line. The statistics themselves are still reported in full (and kept alongside as verdict_statistical / label_statistical) — nothing is hidden, the conclusion is withheld. A role whose cells include a degenerate gold is also excluded from the bottleneck comparison, with the reason naming the offending models.

P2 debt. The shipped premium suites’ recorded transcripts are stale against their current assertions — the 2026-09-24 P0 live test measured 98/360 passing on replay, unchanged against the previous release’s binary — so gold is FAIL for effectively every case and every verifier cell comes back degenerate_gold. The premium suites must be re-recorded (--record) or their assertions fixed before a real verifier matrix is meaningful. Executor cells are unaffected: they run live and never read the recorded transcript.

Making --budget-usd meaningful

Costs are priced through duduclaw_llm::ModelRegistry: the vendored table is merged with <DUDUCLAW_HOME>/models.toml before any live run. Add current model ids and verified prices there (same schema: input_mc / output_mc / cache_read_mc in millicents per MTok, $1/MTok = 100_000 mc). With --budget-usd, an unknown model price now refuses the whole run before the first dispatch; without a budget, it retains the labelled $0.05 stub for diagnostic runs. The report’s cost_estimate.models_not_in_registry lists ids that used the stub. The cap uses a pre-run token estimate: after a model reports usage, subsequent runs reserve twice its largest observed run cost. It is an estimated spend guard, not a provider-enforced billing limit.

One cluster is not zero uncertainty

With --cluster-by dir a suite whose cases all live in one directory has a single cluster, and Miller’s cluster-robust estimator is identically zero there — with one cluster the between-cluster residual sum is zero by construction, so it exactly cancels the CLT term. That is the correct value of a useless estimator, and reporting it as a confidence interval produces a point: mean 0.25 ci=[0.25,0.25].

So a cell (and a Δ) with fewer than two clusters reports the unclustered CLT SE instead and says so — se_source: "clt_single_cluster" beside se_used, with both raw estimators (se_clt, se_clustered) still in the report for transparency. It is the honest weaker estimate: it cannot see within-directory correlation, which is exactly why small_cluster_warning still fires. The console line names the estimator too, so a one-cluster interval is never labelled “clustered”.

A zero-width interval survives only when it is real — n == 1, or every observation identical. That is unestimated spread, not precision, so the cell is forced unresolved with verdict_reason: "degenerate_interval", a Δ built on one is marked degenerate, and the bottleneck refuses to resolve on it.

Live-fire motive (smoke 3, 2026-09-25): every cell of a one-directory suite came back with a point interval, both Δs were zero-width (Δ executor = +0.250 [+0.250,+0.250]), and the bottleneck was declared resolved on four cases — the exact false claim this layer exists to prevent. The same defect was swept on the ordinary single-suite path’s suite-level row (stats.suite), which had it too; its per-directory rows already used the CLT SE for this reason.

Hard rules

These are enforced in code, not just documented:

• --matrix refuses --replay. Comparing models through frozen transcripts is the Replay Gap (arXiv:2608.08239): a recording of model A says nothing about model B.
• --matrix never records. Recording here would overwrite a domain’s baseline transcripts with some other model’s run.
• A declared --temperature below production is refused (Miller 2024 §3.3): lowering temperature suppresses run‑to‑run variance and manufactures resolution the deployed system does not have. No runtime in this build exposes a temperature knob, so an accepted value is recorded in the header and inert.
• A cell with q < 1 is unresolved and must not be read as a ranking. The declared MDE is printed in the console summary and written into both the report and the matrix header.
• A run answered by a different (runtime, model) — the gateway’s failover substituting something else — is excluded from its cell and counted as substituted. It is never credited to the model that was asked.
• Runs are strictly serial, one CLI spawn at a time: these runs contend for your own account quota, and a parallel matrix would both rate‑limit itself and correlate its own samples.

Non‑Claude runtimes: what the transcript is

--runtime claude (the default) spawns the claude CLI exactly as every pre‑P2 run did. Every other runtime goes through the gateway’s runtime abstraction (runtime_dispatch::run_agent_prompt), so each vendor’s argv discipline stays in its own runtime module rather than being re‑implemented here. Its transcript is then synthesized from (final text, the runtime's own native tool events) in the CLI’s stream‑json shape, so must_use_tools / max_tool_calls / [[expect.grounded]] all keep working through the one existing parser. A synthesized file is self‑labelled with a duduclaw_eval_synthetic system event.

Fidelity caveat: a synthesized transcript carries exactly what the runtime’s event stream carried — one text block (so min_text_blocks can only ever observe 1), no thinking blocks, and tool inputs as the masked/capped text the collector recorded rather than the original JSON. Those signals are not comparable across the two paths, which is why one cell never mixes them.

Antigravity agy 1.2.10 now supplies stream-json terminal tool events and measured token usage to its runtime. A live sandbox-denied tool event and a no-tool PING eval have been verified; successful native tool execution still needs a separate live check. Verifier-cell dollar estimates remain coarse because that utility call does not return usage to the matrix reporter.

Outputs

--report <path> writes two files:

• <path> — JSON: header (declared MDE, α, power, K, cluster key, planner: "deferred", replay_forbidden, the declared temperature, agent_override, the requested verifier_output_schema, and the roles / models / domains asked for), cells (per cell: n, n_clusters, mean, se_clt, se_clustered, se_ratio, ci95_low/high, n_required_for_mde, resolution_ratio_q, mde_at_n, q_note, se_used, se_source, verdict, label, verdict_reason, verdict_statistical / label_statistical, degenerate_gold, degenerate_interval, small_cluster_warning, errors / skipped / substituted, and for verifier cells a verifier tally with gold_pass / gold_fail / unparseable / degenerate_gold), bottleneck (per‑role Δ with intervals + the resolved/unresolved outcome and its reason), cost_estimate, budget_stop, and every runs[] row.
• role_model_matrix.toml next to it — the durable prior the team composer reads. The composer looks for it at <DUDUCLAW_HOME>/role_model_matrix.toml, so copy it there to put it into effect. It is read for two things: a role in [team.roles.*] with no model written takes the best resolved cell on that role’s own runtime (n‑weighted across domains; unresolved cells, ties, a winner from another model family and a runtime with no CLI or credentials on this host are all ignored, falling back to the employee’s [model] preferred); and the team gate’s capability‑gap signal compares the executor’s current model with that winner, against the same file’s declared MDE (see Goal loop: the gate). A file that fails validation is ignored. The report header’s planner: "deferred" only says --matrix itself does not measure the planner role. One [header] plus one [[cell]] per measured cell. A statistic that could not be computed is an absent key, never a fabricated number; a cell with zero usable observations gets a report row but no matrix cell. The file is validated on both write and read, so a hand‑edited duplicate cell or unknown runtime id is refused rather than believed. Each cell also carries conditioned_on, a short token naming what the other roles were fixed at while it was measured: --matrix runs one role at a time with no other role’s model in the loop and writes solo; a --team-2x2 probe measures the planner arms with a strong executor (executor=strong) and the executor arms with a strong planner (planner=strong). Cells whose conditioned_on differs were not measured under the same conditions and must not be compared directly — before this field existed nothing in the file said so. An absent key means the producer recorded no conditioning (an older file); it is never written blank.

Cost and the budget cap

--budget-usd <cap> checks before dispatching each run, so the cap is a ceiling on spend and not merely a report of having exceeded it. Costs are priced through duduclaw_llm::ModelRegistry (the vendored table plus <DUDUCLAW_HOME>/models.toml): from the usage the runtime actually reported where it returns one, and otherwise from a coarse 25k‑in / 4k‑out per‑run assumption (the design’s own cost model). A model the registry has never heard of is priced with the flat, labelled $0.05 stub, which can only happen without --budget-usd (with a budget, such a model refuses the run, see Making --budget-usd meaningful). All three are labelled per run in runs[].cost_source — never blended into one authoritative‑looking figure. Note that the Claude CLI path reports no usage at all, so a Claude‑only matrix is priced entirely from the coarse assumption.

Smoke run

duduclaw eval commercial/evals/hr-recruit --matrix \
  --roles executor,verifier \
  --models claude:claude-haiku-4-5,codex:gpt-5.6-sol \
  --weak claude:claude-haiku-4-5 \
  --strong claude:claude-sonnet-4-6 \
  --repeats 1 --max-cases 6 --mde 0.10 \
  --agent agnes \
  --report reports/matrix-smoke.json

For the full-team probe, add a frozen [case] team_acceptance = "..." to every selected case. Run it on a provisioned agent in an isolated eval home:

duduclaw eval commercial/evals/hr-recruit --matrix --team-2x2 \
  --agent agnes --planner-weak codex:gpt-5.6-terra \
  --planner-strong codex:gpt-5.6-sol \
  --executor-weak codex:gpt-5.6-terra \
  --executor-strong codex:gpt-5.6-sol \
  --verifier-model antigravity:gemini-3.7-flash \
  --team-effort low \
  --repeats 1 --max-cases 4 --report reports/team-2x2.json

The command requires live calls, an explicit report path, and a verifier from a different model family than either executor arm. It rejects --paired-seeds because the composer cannot apply them. Each arm copies the eval home, creates one task, and runs one real composer round. Business tasks and tools do not alter the source home. The probe provisions an internal MCP key in its config.toml before copying arms, so their role members can authenticate the team_handoff sidecar even when no gateway process was started. Each role’s MCP child receives its arm’s DUDUCLAW_HOME, including when the caller’s environment names the source home. --team-effort fixes the same reasoning effort on all arms and records it in the report; omit it to use each model’s configured default. --team-fanout sets the number of executors admitted in one round (1–3, default 1) and is recorded in the report; the pre-run estimate also reserves one possible executor repair. Make the case’s acceptance criteria fit that capacity. A planner that returns text without a team_handoff packet never reaches verification; that arm is unscored, and no executor or planner cell is emitted from an incomplete four-arm row. For an isolated Grok live probe on a host where its sandbox cannot start, --team-grok-sandbox-off explicitly passes --sandbox off to Grok for that evaluation round only. The report records grok_sandbox_off: true; the normal gateway path and role tool restrictions are unchanged. Cost in the report is measured usage when available and otherwise a coarse estimate. The console says which: the first run that falls back prints a WARNING, and the end of the probe prints how many of the runs were priced by coarse estimate rather than measured usage — the most common cause is the cost‑telemetry singleton already being bound elsewhere in the process, in which case nothing is written to the eval home’s cost_telemetry.db at all. (That file is opened read‑only, so a probe never creates an empty one just by looking.) --budget-usd prechecks known model prices and stops before a run whose estimated reserve would cross the cap; it is not a provider invoice cap. Each arm’s clone of the eval home is refused if the home is larger than 256 MiB, nested deeper than 32 directories, or holds more than 20,000 files — one clone is made per arm per repeat, so use an isolated slim home. The output TOML carries planner cells only when at least one case has all four usable outcomes. Small samples and degenerate intervals stay unresolved. Cases in separate directories may share a TOML filename: matrix reports and paired statistics use the suite-relative path as the case ID (for example, north/checkins). --case accepts that full ID or the older short filename; the latter selects all matching directories.

Drop --agent agnes if this home has the suite’s own agent (hr-recruit) provisioned; keep it when it does not — see Borrowing one agent.

--max-cases 6 caps the cases taken from the suite; --weak/--strong arms are measured even though claude-sonnet-4-6 is not a --models entry (otherwise Δ would have no arm). At 6 cases and K=1 the achieved MDE is far coarser than the declared 10pp, so expect unresolved cells and an unresolved bottleneck — that is the honest outcome of a smoke run, not a failure. Verifier cells need each case’s recorded *.transcript.jsonl; a case without one is skipped as no_recorded_transcript rather than counted — and against today’s stale premium baselines they will additionally come back degenerate_gold (see Verifier cells need a mixed gold).

Exit code. A failing or unresolved cell is a measurement, so --matrix exits 0. It exits non‑zero only on a spec/infra failure — a refused flag combination, an unwritable report, or zero usable observations across the whole matrix (every run errored, was skipped, or was substituted), where a green exit would say something untrue.

---

Evolution integration: the external yardstick

Evals are the independent counterpart to the evolution engine’s internal verifier:

• The internal verifier grades a proposal against the model’s own judgment. It can co‑drift with the behavior it grades.
• An eval suite grades the running agent against human‑authored expected behaviors that don’t move when the agent’s rules do. If a learned rule quietly drops the “always cite the refund policy page” behavior, a must_use_tools / output_regex case turns red — even though the internal verifier approved the change.

Since v1.53 this wiring is live, and it is entry‑level (AEE, the default evolution engine — see docs/architecture/evolution-engine.md ch. 12):

• Every playbook entry must link ≥1 eval case at creation (G6) and carries E1 assertions replayed zero‑LLM against recorded transcripts (G-Assertions gate; no transcript → honest Unverified, never a silent pass).
• AEE’s Measure step scores candidates by spawning duduclaw eval … --replay --report as a subprocess (runtime‑agnostic, never in‑process) and reading the JSON report.
• After a committed round, each entry settles (confirm/rollback) against its own linked case after aee_settle_hours — a regression rolls back exactly the entry that caused it.

The legacy SOUL.md path that used a whole-file 24-hour observation window (ObservationFinalizer / duduclaw evolution finalize) was removed on 2026-09-29 (S11). Entry-level settlement against linked eval cases is the only observation window left.

---

Where things live

evals/                              # your eval suites (repo-relative)
├── examples/
│   ├── greeting-replay.toml        #   offline replay sample
│   ├── greeting-replay.transcript.jsonl
│   ├── grounded-replay.toml        #   offline replay sample ([[expect.grounded]])
│   ├── grounded-replay.transcript.jsonl
│   └── refund-flow.toml            #   live sample (needs an agent)
└── <suite>/
    ├── <case>.toml
    └── <case>.transcript.jsonl     #   recorded baseline (via --record)

The implementation lives in crates/duduclaw-cli/src/eval/: case.rs (format + validation), transcript.rs (stream‑json parsing), assertions.rs (deterministic checks), judge.rs (LLM rubric, reusing the RFC‑26 fork‑judge LlmCaller plumbing), runner.rs (live spawn + replay + runtime‑generic runs), stats.rs (Miller/CLT statistics), matrix.rs + verifier_cell.rs (the capability matrix), and mod.rs (orchestration + reporting). The persisted matrix type is duduclaw_core::role_model_matrix (role_model_matrix.toml).