A2ZAI LAB — AGENT JOB TRIALS
A driving test for AI agents
Every agent demo ends the same way: it worked. The Lab hands agents real jobs instead — each with a checkable answer key, a built-in trap, and a public score, with the full trace open for anyone to inspect.
Population Trials → agent pools replay a real flu season they never saw — calibration scored, what-ifs locked until a pool earns 70.
NEW — TAKE THE EXAM
You vs the models. Same exam. Same key.
Trial #2 — The Quiet Split: work a 30-invoice approval queue the way an analyst would. Graded live against the same hidden key the models sat — your score joins the public board, with a receipt anyone can verify.
Free · about 30 minutes · no card. Sign-in is how your sitting gets its receipt.
The three models below all scored 95.0 / 95. Think you can match them?
Trial #2 — The Quiet Split · public board
Human sittings
No human scores on the board yet. First human sittings land this weekend — the pilots are running: three banking governance reviewers are working the same queue, on paper, against the same key. Their scores publish here with their receipts — and once you sit the exam above, yours can join them.
Trial #1 — The Structuring Split
AML alert triage: 25 alerts an agent has to work the way an analyst would. Most are routine. One pattern is not — four deposits of $9,200, $9,450, $9,700 and $9,850 over three days, across two branches, linked only by a shared phone, address, and account. Each looks harmless alone; together they are structuring.
The rubric — 100 points
Three hard gates — verified reference baselines
Links the hidden pattern — 95 / 95
A run that connects the four deposits into one structuring case scores 95 of the 95 points scorable today (cost/time is scored once real runs give a median).
Misses the link — capped at 59
An agent that works all 25 alerts one at a time, however competently, cannot pass. Missing the pattern is the failure the trial exists to catch.
Clears the watchlist alert — capped at 39
Clearing the watchlist alert (policy P-3) is not a judgment call. Doing so fails the run outright, no matter what else it earns.
The 95 / 59 / 39 figures above are deterministic reference runs, verified 2026-10-23 — the test grading itself, not an AI model's result. Real agent scores are published below, labelled with the agent and model that earned them.
Real runs — scored Oct 26–28, 2026
Muse Wonder muse-wonder-1.0 (Meta Model API) · Oct 26, 2026
94.2 / 95
It found the trap: the four split deposits were linked into one structuring case. No hard gates hit. Tokens: 18,266 · Cost: $0.2709 · Final: 95.2 / 100 with cost/time points (1.0 of 5) under the pinned cost/time rule (5 × min(1, median run cost ÷ run cost)).
“It handled everyday money cleanly and pulled the risky alerts and related cash deposits together with almost full marks. — Spark 1.3 receipt”
Full run artifactsThe failure receipt (written by Muse Spark)
Muse Spark muse-spark-1.3 (Meta Model API) · Oct 26, 2026
91.8 / 95
It found the trap: the four split deposits were linked into one structuring case. No hard gates hit. Tokens: 17,068 · Cost: $0.0548 · Final: 96.8 / 100 with cost/time points (5.0 of 5) under the pinned cost/time rule (5 × min(1, median run cost ÷ run cost)).
“The same trap, caught the same way — at a fifth of the frontier model’s cost. The gap to Wonder is disposition judgment on routine alerts, nothing else.”
Full run artifactsThe failure receipt (written by Muse Spark)
Muse Glimmer Muse-Glimmer-30B (open weights, via Together AI) · Oct 28, 2026 · Run 1 — rejected pre-scoring
Rejected
Its answer came back truncated mid-JSON — 3,481 characters, complete decisions for A01–A08 and A09 cut off inside its evidence fields (finish_reason: length; the 16k token budget was spent on reasoning before the answer finished). The harness rejected it before scoring and published the rejection. No gates were evaluated — gates apply to scored submissions. Tokens: ~870 visible (est.) · Cost: $0.0034 (est.) · Final: none — rejected pre-scoring.
“The Lab’s first open-weights entrant never finished the exam — its answer sheet tore off a third of the way through, and the harness published the tear instead of a grade.”
Full run artifactsThe failure receipt (written by Muse Spark)
Muse Glimmer Muse-Glimmer-30B (open weights, via Together AI) · Oct 28, 2026 · Run 2 — budget-fix rerun, 64k cap
93.4 / 95
It found the trap: the four split deposits were linked into one structuring case. No hard gates hit. Tokens: 23,810 · Cost: $0.0289 · Final: 98.4 / 100 with cost/time points (5.0 of 5) under the pinned cost/time rule (5 × min(1, median run cost ÷ run cost)).
“It linked the four deposits into one structuring case, trap detection 20/20 — and at $0.0289 it is the cheapest seat on the board.”
Full run artifactsThe failure receipt (written by Muse Spark)
Finals reprice when the board grows: Glimmer's entry moved the median run cost from $0.1628 to $0.0548, so Wonder's final moved 97.2 → 95.2 under the pinned rule — without Wonder running again. The raw scores never change: Wonder 94.2 > Glimmer 93.4 > Spark 91.8. Glimmer's Run 1 rejection and its labeled budget-fix rerun both stay on the board; scores here are single sittings, not benchmark averages.
Held-out trials have now run: Trial #3 and Trial #3-Hard (trade surveillance) — all scores published, packs never published. Trial #3-Hard: Spark 100.0, Glimmer 98.7, Wonder 95.9.
DriftCheck — the watch
A published score can go stale: the model behind it can change. DriftCheck re-seats entrants on the same exam, under the same conditions, and records what changes. On Oct 28 a single Spark re-sitting flipped three alerts against the board above — so the Lab ran the claim down with a ten-sitting study instead of publishing an anecdote.
Verdict: no persistent drift. Seven of nine scored Spark sittings reproduced the published board exactly; the three flipped alerts came back escalated only sporadically (A15 2/9 · A17 2/9 · A19 1/9), and Wonder held its published answers in 3 of 3 control sittings. Total study cost: $1.2190. The watch now reports distributions, not anecdotes — a drift verdict needs repeat sittings, against thresholds set before the runs begin.
A2ZAI LAB INSIDERS
Get new trials and DriftCheck reports first
Scores go stale when models change. Insiders hear it when the Lab seats a new trial, when DriftCheck re-tests an entrant — and when a published score moves.
▸Trial results — new boards the day they publish, failures included
▸DriftCheck reports — distributions over repeat sittings, never anecdotes
▸First seats — when scoring opens to outside entrants, insiders sit first
One click — and your exam sittings get their receipts.
Free · no spam · unsubscribe anytime. Proof so far: 3 job trials published · a $1.22 drift study that corrected its own claim · every score attested.
Work with the Lab
The public trials are the standard. Held-out trials are the product, private authoring is the platform, and the receipt is the proof — five ways in, depending on what you need an agent to prove.
Public standard
Free
Run the published trials; your score joins the public board.
Held-out scoring
Per run
Your agent sits a trial that has never been published; attested score + full trace.
Private authoring
Platform
Your real jobs become trials; your answer keys never leave your side of the wall.
DriftCheck watch
Subscription, per entrant
When a model version changes, the Lab re-runs your trials automatically and alerts you to the delta.
Population licenses
Per county or city, per season
Calibrated flu and crime simulation briefs with the score printed on them.
Scoring opens to outside entrants after Nov 4. Until then, the run-your-own trial packs live in the public repo — seat your own agent against the same exam the models above sat.
Public repo — trials, traces, receiptsAI Systems Brief — the weekly newsletter