Band: a recorded session on a scripted test performer, 139 questions, 2026-08-27.
Your score moved. What moved it?
AIM measures the judge, locks the questions and puts an error bar on each number. A move bigger than its error bar points at the model.
Judge measured. Questions locked.
Four things can move a score: the model, the judge, the questions or the prompt. Here the judge is measured and the questions and their keys are locked. Chance shows up as the error bar. What is left points at the model.
- Model: free to move.
- Judge: measured.
- Questions and keys: locked, with a fingerprint.
- Grading prompt: held constant. Its record is pending.
Your own task mix in production is outside this ruler.
In a simulation on our own judge telemetry, a run set to a standard error of 0.20 came in at 1.16 with judge harshness left in. With it corrected, 0.21.1
Simulation.
Two questions to ask any ruler.
Has the ruler moved?
The questions cannot change.
The question set is frozen, fingerprinted and dated. Every run checks the fingerprint first. Whether the ruler holds steady over months is the next exhibit, and it is not done yet.
Re-measurement run: not yet published.2
Is the ruler circular?
Defined before anything is scored.
Each level is fixed before any answer is graded. New questions stay sealed, so no model could have learned them. For new questions, the writer, the attacker and the judges come from separate companies. Each judge is checked against the others.
The judges for levels 7 to 9 are a certified set of 12. Their credentialing run is in progress, with no results yet.
A check against outside human experts: still ahead.
Measured, not averaged.
Each level carries its own error bar. No average sits on top. A failure at one level stays visible.
Pick any score. It resolves to one question in a locked question set, with its answer key and a fingerprint.
Measured on the reasoning ruler. Gate passed 2026-08-22. Each level's error bar is on the Evidence page, with its unit.
If your domain is not on the list, we say so before the run.
Reasoning ruler, error bar at each level
| Level | Standard error | Judgments |
|---|---|---|
| Level 8 (Concrete) | ± 0.070 | 342 |
| Level 9 (Abstract) | ± 0.056 | 230 |
| Level 10 (Pre-Formal / Abstract-Formal transition) | ± 0.056 | 230 |
| Level 11 (Formal) | ± 0.032 | 232 |
| Level 12 (Systematic) | ± 0.019 | 225 |
No total row. Gate passed 2026-08-22. Unit: one level of the ruler.
Trace
Each row resolves to its question IDs, their answer keys and the bank fingerprint, in the dated report of the certifying run.
Know the cheapest judge that can.
Two judges have published ceilings. Today you pick the judge and see how far up the ruler it grades reliably. The run stops when the error bar is tight enough, or at the question limit you set. Hand-off to a stronger judge is in build.
Cheapest judge first. A stronger judge on stand-by, in build. Stops you set: the widest error bar, the most questions, the most spend.3
In build.
Open to reviewers. Closed to training.
The engine is open to reviewers under Apache-2.0, so reviewers can check the method. The banks stay closed. That keeps the questions out of public training data.
If you run an eval loop, AIM is one step in it. It replaces nothing you built.
Engine
Apache-2.0
open to reviewers
Question banks
closed
sha256 fingerprint
One outside view.
“I've always stressed the importance of ethics in persuasion, and Dr. Matt Barney's AI assessment tool brings unprecedented scientific rigor to this domain. I am optimistic that his method holds immense promise in proactively preventing the misuse of persuasion techniques, both by people and emerging technologies, and augmenting their long-term use correctly”
Every measure, with its date.
Every measure on this page carries its error bar and its date. When a number moves, you can see which parts were held still.
Tell us which number you do not trust.
You will get a short note from Matt when a measurement slot opens. Nothing else, no drip sequence.
- Gate passed 2026-08-22. Criterion met: a standard error at or below 0.07 of a level, with at least 30 judgments, at every level from 8 to 12.
- Ceiling rule met. Criterion: a judge's ceiling is the highest level it passes before a break; passes above a break do not count. Certifying runs: the reasoning-ceiling probe runs of 2026-08-19 and 2026-08-22, audited by our quality gate.
- The simulation used 15,000 judge decisions, dated 2026-09-09. Method and results are in the whitepaper. Whitepaper
- Fifteen artifacts sit under sha256 fingerprints, checked at the start of every run.
- Two ceilings are published. Both sit at level 11, with a stated condition. Ceiling rule met
- Method: Chapter 3 of a book on models and metrology. Edited by Fisher and Pendrill. Published by De Gruyter in 2024.