Ground truth is a faith, not a measurement

Human ground truth is not a fixed reference. Raters carry measurable severity, leniency, halo and central-tendency effects (Myford and Wolfe, 2003, 2004), and in our early probe one LLM judge's bias changed sign across capability levels: 2.60 raw points lenient on weak higher-level work, 0.46 points severe on strong work. Whether human raters' bias flips the same way is rarely checked.

Here is a belief so common it goes unquestioned: when you want to know if an AI system is right, you ask a human. The human's rating is the "ground truth." The machine is judged against it. Case closed.

But what if the ground truth moves?

Not philosophically. Measurably. What if the human reference carries systematic bias that changes with the capability of the work being judged, and what if almost nobody checks?

The weakness of human ground truth

Psychometrics did not invent the many-facet Rasch model because human ratings were reliable. It invented the model because they were not.

Myford and Wolfe (2003, 2004) catalogued the taxonomy: severity and leniency, halo, central tendency, restriction of range. These are not noise. They are "rater effects," systematic and estimable, which is why Linacre (1989) built the MFRM to parameterize them. Engelhard and Wind (2018) devote an entire monograph to "rater-mediated assessment," a subfield that exists precisely because raters are fallible instruments requiring calibration. Eckes (2015) teaches the same taxonomy as standard curriculum.

Human raters drift. They vary with expertise. They need training, monitoring, and recalibration (Linacre et al., 1994; Myford & Wolfe, 2009; Noh & Matore, 2020; Sampson et al., 2024; Yan & Chuang, 2023, as cited in Thornton, Barney, & Fisher, in preparation). And humans cannot be re-run deterministically. The same rater, the same response, a different day, a different mood, a different lunch: the observation changes.

This is not a scandal. It is a known property of human judgment, and psychometrics built machinery to handle it. The scandal is that the AI evaluation field largely ignores the machinery while still treating the human as the fixed point.

The estimable machine

An LLM judge differs from a human rater in one practical way: you can ask it again.

You can fix the prompt and the settings and send the same response through the same judge many times. Hosted models are not perfectly deterministic, so you get a distribution rather than one fixed output, and that distribution can be characterized. A judge you can re-run is a judge whose bias can be estimated, and a bias you can estimate is a bias you can correct. The precision of that estimate is limited by how many distinct responses and levels you sample, not only by how many times you repeat one of them.

The economics are stark. In our early cost ledger, certifying a judge costs $0.10. Skipping certification risks a $7.74 to $15 wave re-run plus bank contamination. The Phase 1 probe ran 480 cells for $0.1224 total. Deterministic re-runs at negligible marginal cost are not a hope. They are telemetry.

The sign flip nobody was looking for

Here is what happens when you actually measure an LLM judge across capability levels.

In an early XLNC Phase 1 probe, Nemotron Nano 30B, a frozen W2R2 bank, 480 cells: on low-capability weak responses, the judge is lenient by +2.29 raw points (n=12). On high-capability weak responses, lenient by +2.60 (n=15). On high-capability strong responses, severe by -0.46 (n=24).

The samples are small and the reference is other models, not people. The direction is still clear enough to matter: lenient on weak work, slightly severe on strong work.

The bias changes sign. Lenient on the weak, severe on the strong. A single-number correction, the kind you would apply if you treated the judge as a fixed reference, would average the +2.6 and the -0.46 into a small residual and call it calibrated. The error would remain exactly where it matters most: on the boundary between weak and strong, where band assignment is decided.

A larger reading showed the same shape in a different model. grok-4-6 rated work above the reference at levels 8 through 10 (by 1.43, 1.57, and 1.07 stages) and below it at levels 11 and 12 (by 0.38 and 1.03 stages), with at least 225 judgments per level. We published it, with per-level errors, in an earlier post.

The honest asymmetry

Here is what we have not done. We have not measured capability-level-varying bias in human raters under our design.

Our early probe measures LLM judges against the consensus of other LLM judges. There is no human rater arm, and some of those reference judges returned unusable output on part of the items, which weakens the consensus itself. Our own manuscript in preparation describes a planned comparison, "a subset of approximately 5,000 responses that receive both human and LLM scores," but states these are "predicted outcomes ... not completed empirical findings."

The asymmetry is the point. The field treats humans as the reference. Our early data show that a judge's bias can change sign across the very levels you care about, and the rater-effects literature above shows humans carry level-dependent biases of their own. Whether human bias changes sign with capability, under a design like ours, is still an open question. We have not tested it, and it is rarely tested anywhere.

That is a limitation of our work, stated plainly. It is also the reason the question deserves a measurement instead of an assumption.

Falsifiability beats faith

Rasch measurement with specific objectivity (Rasch, 1960) gives you a falsifiable claim: once each judge's severity is estimated and removed, the same response yields the same measure from different judges, within a stated standard error. If it does not, the model is wrong, and you can see where. The claim can fail. That is what makes it measurement.

"Ground truth" cannot fail. It is defined as true. You cannot recalibrate it because there is no calibration target above it. You cannot estimate its bias because it is the baseline against which bias is defined. It is not a measurement. It is a faith.

The MFRM canon (Myford & Wolfe, 2003, 2004; Linacre, 1989; Engelhard & Wind, 2018; Eckes, 2015) exists because human raters are measurable instruments with estimable biases. The metrological canon (Pendrill, 2019; Pendrill & Fisher, 2015; Mari & Wilson, 2014) exists because measurement requires traceability and uncertainty budgets. NIST AI 800-3 stays statistical and never crosses into metrology.

XLNC/AIM is built on both canons, and our own results are early evidence and not yet certified. The MASEMS architecture (Thornton, Barney, & Fisher, in preparation) extends metrological traceability to social and educational measurement. Our prior work used many-facet Rasch measurement with LLMs as raters (Barney, Wind, & Krishna, 2026) and set out how metrology and psychometrics combine for LLM-based measurement (Barney & Barney, 2024).

What to do instead

Stop asking "is it aligned with ground truth?" Start asking three questions.

What is the judge's bias at each capability level, with what standard error? Is the bias stable across replications, or does it drift? And what is the uncertainty budget from raw response to reported score?

A human rater cannot be re-run to answer them. An LLM judge can. That is no reason to trust the machine blindly. It is a reason to measure the machine, correct what you find, and measure the human reference too, instead of assuming it.

If your team scores against a human reference and wants that reference measured too, that is custom measurement work: a proof engagement built around one decision you face, scoped with your team, with your LLM judges and your human raters on one ruler. Ask for one at xlnc.co/solutions.

Different teams need different things, from a Readout on one score change to a self-hosted harness that keeps your data on your side. Tell us which fits yours at xlnc.co/solutions.

References

• Barney, M., & Barney, F. (2024). Transdisciplinary measurement through AI: Hybrid metrology and psychometrics powered by large language models. In W. P. Fisher, Jr. & L. Pendrill (Eds.), Models, measurement, and metrology extending the SI (pp. 103-132). De Gruyter. https://doi.org/10.1515/9783111036496-003
• Barney, M., Wind, S. A., & Krishna, V. (2026). Using large language models to evaluate ethical persuasion text: A measurement modeling approach. International Journal of Assessment Tools in Education, 13(1), 224-247. https://doi.org/10.21449/ijate.1788563
• Eckes, T. (2015). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd revised and updated ed.). Peter Lang.
• Engelhard, G., Jr., & Wind, S. A. (2018). Invariant measurement with raters and rating scales: Rasch models for rater-mediated assessments. Routledge.
• Keller, A. J., Kwegyir-Aggrey, K., Steed, R., Rao, A. K., Sharp, J. L., & Bergman, A. S. (2026). Expanding the AI evaluation toolbox with statistical models (NIST AI 800-3). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.800-3
• Linacre, J. M. (1989). Many-facet Rasch measurement. MESA Press.
• Mari, L., & Wilson, M. (2014). An introduction to the Rasch measurement approach for metrologists. Measurement, 51, 315-327. https://doi.org/10.1016/j.measurement.2014.02.014
• Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386-422.
• Myford, C. M., & Wolfe, E. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189-227.
• Pendrill, L. R. (2019). Quality assured measurement: Unification across social and physical sciences. Springer. https://doi.org/10.1007/978-3-030-28695-8
• Pendrill, L., & Fisher, W. P., Jr. (2015). Counting and quantification: Comparing psychometric and metrological perspectives on visual perceptions of number. Measurement, 71, 46-55. https://doi.org/10.1016/j.measurement.2015.04.010
• Rasch, G. (1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.
• Thornton, A. M. A., Barney, M., & Fisher, W. P., Jr. (in preparation). Metrological Architecture for the Social, Educational, and Medical Sciences (MASEMS): From measurement chaos to traceability.