Stanford wrote the textbook AI evaluation needed. Here is the chapter we would add.

For years, people who measure AI systems have borrowed tools from psychometrics one paper at a time. Now there is a textbook. Sang Truong and Sanmi Koyejo of AI Measurement Science (AIMS), a center within Stanford HAI, have published AI Measurement Science, a free living textbook on the foundations and practice of AI evaluation: aimslab.stanford.edu/textbook. Section numbers below refer to the version of September 7, 2026.

Read it. Then ask the one question the book leaves open: when an AI evaluation reports a score, traceable to what?

What the book gets right

It starts with validity, not leaderboards. Chapter 1 sets out the kinds of validity evidence and a catalogue of threats, and later chapters say where each threat is diagnosed. The book says plainly that a number is not yet a measurement (chap. 3). It asks what a score means before it reports one.

Chapter 3 treats measurement models with real care. It explains specific objectivity: under the Rasch model, the comparison of two performers does not depend on which items you used. It derives the sufficiency result, ties Rasch to additive conjoint measurement, and calls Rasch a probabilistic analogue of those axioms, not a deduction from them. It also shows that under the two-parameter model the ability scale is interval by stipulation, with its unit fixed by convention (sec. 3.6).

Chapter 5 treats LLM judges as raters, which is the right frame. It decomposes judge effects into variance components (sec. 5.2), covers agreement coefficients, and uses threshold models to keep a judge's scale use out of ability (sec. 5.9). It also makes a point every evaluation team should post on the wall: a perfectly reliable judge can still be invalid, and two judges can share the same bias (sec. 5.9).

Two more tools we plan to borrow, with credit. The cost-aware D-study tells you how many judges and items buy a target reliability (sec. 5.6). Prediction-powered inference gives an honest interval when a small human-labelled set corrects a cheap judge (sec. 12.5.2).

Their model choice, stated fairly

The book teaches Rasch as one rung of a ladder, with two- and three-parameter models beside it as legitimate trade-offs. It states both camps fairly: only Rasch earns an interval scale from the data's structure, while a fitted two-parameter scale is interval enough for ranking and keeps information about which items discriminate (sec. 3.6). If the goal is to predict benchmark outcomes, that is a sound choice.

We weigh it differently because our goal is a ruler that carries from one system to another and from one year to the next. Three properties are lost under a two-parameter model:

• item order changes with ability, because item curves cross;
• the raw score is no longer sufficient, so the leaderboard number is no longer the measurement;
• person and item parameters no longer separate, so specific objectivity is lost.

Extra slope parameters can also absorb misfit that a Rasch fit test would flag. The book names this disagreement itself (sec. 3.6).

The chapter we would add: metrological traceability

The book does not cover metrological traceability. We searched all thirteen chapters for "traceability", "metrology" and the VIM (the International Vocabulary of Metrology) and found no hits. "Calibration" in the book means IRT item calibration or ML probability calibration.

Metrological traceability asks a different question: can this number be related to a stated reference through a documented chain of calibrations, each with a stated uncertainty? In physical measurement that chain is routine: a torque wrench on a factory floor is checked against a working standard, which is checked against a national reference, with the uncertainty stated at every link (VIM 2.41). In AI evaluation the chain is mostly missing. A chapter on it would cover three things.

• A calibration hierarchy. Name the reference standard, the working standards and the instruments in use, and show how each links to the one above it.
• Reference standards for meaning. For a developmental scale, a construct map says what each level means, so "this response is at level 11" means the same thing across systems.
• An uncertainty budget. The book reports standard errors, generalizability coefficients and PPI intervals. A budget itemizes each source: estimation error, judge-severity error, linking error between calibrations, and the gap between the engine used to calibrate and the engine used at runtime.

A regulator who asks "traceable to what?" is asking for that chapter. Our own frame for answering it is one measurement model for human, AI and robot performers, with judges treated as performers who earn credentials too.

A narrower point about judges

The book gives each judge a leniency term on the log-odds scale and summarizes judges through their variance (sec. 5.2, 5.9). It names the many-facet Rasch model, applied to treatment conditions (chap. 2). The step we would add is to report each judge's severity as a fixed location in logits, with a standard error, and remove it from each performer's score.

That location becomes a credential. It supports fit statistics for each judge, linking judges across calibration occasions, and monitoring each judge's severity for drift. It also allows a credential per level, because a judge can be sound at one level of complexity and unreliable at a higher one. A single agreement coefficient hides that.

The book's own remedy for shared bias is calibration against a human gold set (sec. 5.9). A many-facet model puts that remedy on a scale. Severity is estimated relative to an anchor, so a harshness that every judge shares becomes visible only when a human reference anchors the scale.

Constant severity is what the model removes. Position and length preferences depend on the response, so we test for them separately, as the book advises (sec. 5.9).

Where we stand

Our own calibration chain is designed and partly built, not finished. Our judge credentials are anchored today to a reference judge and synthetic fixtures; the human-anchored reference waits on data we are still collecting. Read this post as a proposal for a chapter, not a claim that we have written it.

The goal is worth the work: a score that means the same thing in every lab and every year. We would rather build it together. If you teach from this book, build on it, or wrote it, tell us what a metrological traceability chapter would need to earn a place next to chapter 5: leave a note here. Our one-hour class works through a calibration chain on real judge data: see the class.