NIST measures statistics. We measure capability.

NIST AI 800-3, "Expanding the AI Evaluation Toolbox with Statistical Models" (Keller et al., 2026), frames generalized linear mixed models as "one promising approach which could form a foundation for more principled AI evaluation statistics." It addresses statistical validity, and across its 64 pages it never uses the word "traceability," which is where metrological validity begins.

Here is a puzzle. The U.S. National Institute of Standards and Technology, the body that defines the meter and the second, just published a 64-page guide on evaluating large language models. It is rigorous, careful, and openly statistical. Yet nowhere in those 64 pages does the word "traceability" appear. Nowhere does "metrological." Nowhere "severity," "leniency," or "calibration."

How can the nation's measurement laboratory publish a measurement manual that never mentions measurement's own vocabulary?

The answer tells you where the field stops, and where it has to go next.

What NIST actually said

NIST AI 800-3, "Expanding the AI Evaluation Toolbox with Statistical Models," is a good paper. It says so itself, in its own framing: "This paper addresses a critical piece of these growing measurement validity concerns: statistical validity" (Keller et al., 2026, Sec. 1, p. 1 / PDF p. 9). Not metrological validity. Not traceability. Statistical validity, defined as "the extent to which reported statistical conclusions about evaluation results, including quantifications of uncertainty, are supported by assumptions and available data."

The model at the paper's core is a generalized linear mixed model, deliberately close to the Rasch model. NIST acknowledges the lineage: "Our GLMM modeling approach is very similar to the Rasch model [25] or the one-parameter Item Response Theory (IRT) model" (Sec. 2.2, p. 5 / PDF p. 13). It calls itself "an intermediate step that incorporates concepts from psychometrics while relying on a simpler statistical model than multi-parameter IRT models."

And here is the anchor. NIST concedes that latent-scale intervals have meaning, and raw benchmark scores do not: "the intervals between LLM capabilities theta on the latent scale have meaning independent of the difficulties of the specific tasks selected into the benchmark. Raw benchmark scores do not have this property" (Sec. 3, p. 6 / PDF p. 14). A 10-point gap between 80 and 90 percent may not equal the gap between 70 and 80 percent, even with no change in underlying capability.

That is the right observation. It is also where NIST stops.

The three tiers of evaluation

Evaluation quality climbs a ladder. Each rung answers a question the rung below cannot.

Tier 1: Statistical alignment. This is NIST 800-3's home. It asks: are your reported conclusions supported by the data and the model's assumptions? It gives you standard errors on benchmark differences, and it shows that latent scales fix the interval problem. It is necessary. It is not sufficient.

Tier 2: Rasch measurement. Rasch measurement goes further. Rasch (1960) gave measurement a probabilistic model with specific objectivity: when the data fit, comparisons between persons do not depend on which items were used, and comparisons between items do not depend on which persons were sampled. NIST cites Rasch as reference [25] (p. 36 / PDF p. 44), yet stays with a simpler statistical family.

Tier 3: Metrological traceability. This is the rung NIST never names. Pendrill (2019) shows that Rasch parameter separation enables traceable measurement standards comparable to physical metrology. Pendrill and Fisher (2015) formally compare psychometric and metrological measurement. Mari and Wilson (2014) place the bridge in one shared concept system. Traceability means every score carries an unbroken chain of comparisons to a reference standard, with an uncertainty budget at each link. NIST cites the GUM, the international guide to measurement uncertainty, as reference [34] (p. 36 / PDF p. 44), yet never operationalizes it.

NIST 800-3 is a Tier 1 document with Tier 2 leanings. XLNC/AIM is built for Tier 3, and our results there are early evidence, not yet certified.

The tiers are cumulative, not competitive. A Tier 3 instrument still runs Tier 1 statistics and Tier 2 item calibrations; it refuses to stop there. Every value XLNC/AIM publishes carries a stated standard error and a calibration date, and every judge that contributes to it is modeled as a rater with its own severity, because a measure without those is an opinion wearing a number.

What the silence costs

The document-wide negative finding is stark. Across 64 pages, zero occurrences of "metrological," "traceability," "severity," "leniency," "judge," "specific objectivity," "GUM" in the body text, or "calibration" as a measurement process. The one appearance of "metrolog" is the contact email caisi-metrology@nist.gov (PDF p. 3).

Where raters appear at all, NIST treats them as an afterthought: one sentence noting that "GLMMs can also incorporate annotator (i.e., grader or rater) characteristics and multiple annotators per item [21, 22]," listed under "additional GLMM applications and extensions not illustrated in this work" (Sec. 6.1, p. 28 / PDF p. 36). No severity parameter. No leniency parameter. No rater facet.

The assumption structure confirms the ceiling. NIST's GLMM, "shared by the Rasch model, implies that all items are equally discriminating of LLM performance between latent capability levels" (Sec. 6.2, p. 31 / PDF p. 39). That is a strong simplification, and NIST is honest about it. But honesty about a simplification is not the same as escaping it.

Aligned, and beyond

We are not opposing NIST. The paper's own self-description is statistical validity, and it achieves that. Its GLMM advances "the statistical validity of LLM evaluations," while it defers construct and external validity to others (Sec. 2, p. 4 / PDF p. 12). NIST's Rasch adjacency makes the document a stepping stone, not an opponent.

But statistical alignment is not measurement. A score you cannot trace to a standard, with a rater facet you do not model, and an uncertainty you do not budget, is a number, not a measure.

XLNC/AIM's MASEMS architecture (Thornton, Barney, & Fisher, in preparation) extends the metrological canon across social, educational, and medical sciences. The Hexagon Measurement Framework (Wilson & Mari, 2026) sets out a blueprint for measurement in the human sciences. Our own prior work applied many-facet Rasch measurement with LLMs as raters (Barney, Wind, & Krishna, 2026) and set out how metrology and psychometrics combine for LLM-based measurement (Barney & Barney, 2024).

The difference is not attitude. It is architecture. NIST gives you a better benchmark. We give you a ruler.

What this means for a buyer

If you are evaluating an LLM for a high-stakes decision, ask one question: can this score be traced? Not "is it statistically significant," but "where does the interval come from, what rater biases are estimated, and what is the uncertainty budget?"

NIST 800-3 shows the field has learned to ask the question of statistical alignment. The traceability and uncertainty questions are still open. That is the gap XLNC/AIM is built to close. Our calibration chain is early evidence, not yet certified, and every link in it is published with its error and its date.

If you want to put those questions to your own numbers, start with the Design-Partner Readout. Send a redacted slice of your eval results or traces, and in about 30 days you get a written early read of how much of a score change came from the model, the judge, the dataset, the task, and the prompt. Ask for one at xlnc.co/solutions.

Different teams need different things, from a hosted service to a self-hosted harness that keeps your data on your side. Tell us which fits yours at xlnc.co/solutions.

References

• Barney, M., & Barney, F. (2024). Transdisciplinary measurement through AI: Hybrid metrology and psychometrics powered by large language models. In W. P. Fisher, Jr. & L. Pendrill (Eds.), Models, measurement, and metrology extending the SI (pp. 103-132). De Gruyter. https://doi.org/10.1515/9783111036496-003
• Barney, M., Wind, S. A., & Krishna, V. (2026). Using large language models to evaluate ethical persuasion text: A measurement modeling approach. International Journal of Assessment Tools in Education, 13(1), 224-247. https://doi.org/10.21449/ijate.1788563
• Joint Committee for Guides in Metrology. (2008). Evaluation of measurement data: Guide to the expression of uncertainty in measurement (JCGM 100:2008). https://doi.org/10.59161/JCGM100-2008E
• Keller, A. J., Kwegyir-Aggrey, K., Steed, R., Rao, A. K., Sharp, J. L., & Bergman, A. S. (2026). Expanding the AI evaluation toolbox with statistical models (NIST AI 800-3). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.800-3
• Mari, L., & Wilson, M. (2014). An introduction to the Rasch measurement approach for metrologists. Measurement, 51, 315-327. https://doi.org/10.1016/j.measurement.2014.02.014
• Pendrill, L. R. (2019). Quality assured measurement: Unification across social and physical sciences. Springer. https://doi.org/10.1007/978-3-030-28695-8
• Pendrill, L., & Fisher, W. P., Jr. (2015). Counting and quantification: Comparing psychometric and metrological perspectives on visual perceptions of number. Measurement, 71, 46-55. https://doi.org/10.1016/j.measurement.2015.04.010
• Rasch, G. (1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.
• Thornton, A. M. A., Barney, M., & Fisher, W. P., Jr. (in preparation). Metrological Architecture for the Social, Educational, and Medical Sciences (MASEMS): From measurement chaos to traceability.
• Wilson, M., & Mari, L. (2026). Mapping out the Hexagon Measurement Framework as a blueprint underlying measurement in the human sciences. Journal of Educational Measurement, 63(1), e70036. https://doi.org/10.1111/jedm.70036