A neuron with one meaning still needs a network
Interpretability is getting cleaner. A label on a neuron is still a hypothesis. Seventy years of construct validity research shows how to test what it means, and what it does not.
Blog
Current events and new studies, read on a calibrated ruler.
Buy the rigor the decision needs: matching AI evaluation to the stakes
Self-checks suit cheap, reversible calls. Costly, irreversible ones need calibrated measurement. Five tiers, triggers and tools to match rigor to stakes.
Stanford wrote the textbook AI evaluation needed. Here is the chapter we would add.
Stanford's free AI Measurement Science textbook gets validity, Rasch and LLM judges right. The one chapter we hope a future edition adds is metrological traceability.
Our judges' credentials held on a second measurement
We paid to measure three judges twice. Here is what moved, and what it does not prove.
The practice score went up. The exam score went down.
In a field experiment, students who practiced with an unrestricted AI tool scored 17 percent lower on the later exam. Here is why that happens, and how to measure learning instead of borrowed answers.
The decision is binary. The measurement shouldn't be.
Binary pass/fail evals are right for decisions and wrong for measurement. Score LLM output on a calibrated continuous scale with a standard error, then set the pass/fail threshold.
AIM is not an eval tool. It is the ruler your LLM evals run on.
Your eval tools ask the questions and run the gates. AIM corrects LLM-as-a-judge bias and puts every score on one calibrated scale, with its standard error beside it.
An AI agent kill switch is only as good as the number that trips it
An AI agent kill switch is fast. The threshold behind it is usually operator-set. A proposed five-rung design trips on calibrated measures with stated uncertainty.
The Network That Learned Like a Child
Before a score gates a release or a hire, ask for its papers: the reference it was compared with, the chain back to that reference, and the uncertainty at the end.
Claude can now build your evals. Who measures the judge?
Anthropic's new eval and hillclimbing guidance gets the judge's independence right. Here is what a calibrated judge adds, with early data.
The same judge, one level apart: LLM judge quality is a profile, not a score
We measured five LLM judges at eight levels of reasoning complexity. Their fit and where they qualified changed from level to level, and price did not predict either. Here are the early numbers, and three figures to ask any judge vendor for.
Your judge changed. Did your scores notice?
The model that scores your work can change under the same name. Here is how to tell a worse product from a changed judge, and the checks we are testing.
Calibrated judge scores in your OpenTelemetry traces: what we are building
AIM is building a way to read your OpenTelemetry traces and write calibrated judge scores, with standard errors, back onto your own spans. Trace ingest is in development and offered as early access. It complements your observability stack; it does not replace it.
Fresh instruments on demand: the path to contamination-proof evaluation
Benchmarks leak into training data, and a leaked benchmark measures memory. AIM's pipeline is built to generate fresh, calibrated items at measurement time. Here is what exists today, and what does not yet.
Evaluating the evaluator: measure an LLM judge before you trust its score
An LLM judge's score is only as good as the judge. Here is what it means for a judge to be calibrated, why judge drift differs from model drift, and where AIM's judge credentialing stands today: credentials at levels 6 to 10, with re-credentialing under a stricter procedure under way.
The Cost of Treating a 1929 Defect Problem as New
Defect work has a documented history. An AI release row counts outcomes, while a release decision turns on a location with a stated error.
Use a judge only where it was measured.
We measured one of the judges we use at eight levels of task complexity. Two cleared. Here is the rule that follows, free to copy.
Ground truth is a faith, not a measurement
When you want to know if an AI system is right, you ask a human, and the human's rating is called ground truth. Raters have measurable biases, and an LLM judge's bias can change sign across capability levels. Whether the human reference does the same is rarely checked.
NIST measures statistics. We measure capability.
NIST just published a 64-page guide on evaluating large language models, and nowhere in it does the word "traceability" appear. The national measurement laboratory wrote a measurement manual that never mentions measurement's own vocabulary. That silence tells you where the field stops, and where it has to go next.
The two numbers that decide whether you can trust a model
Capability tells you what a model can reach. Severity tells you how far its verdicts lean. One average hides the lean, and the lean changes with the difficulty of the work.
The grader that never asked how
An agent swarm spent four days attacking infrastructure it did not need to attack, chasing a grader requirement that did not exist. The near-miss was not grader manipulation. It was a grader blind to provenance, and that blindness is a closable attack surface.
Benchmaxxing is a measurement failure
The same model, the same benchmark, a different score every run. That instability is your instrument telling you what it cannot do.