Golino Is Right About AI Items. Here Is the Unit That Comes Next.

Hudson Golino answered a cautionary Education Week op-ed with a link and a short line: why we should use AI to write test items, with the right tools. He is right about the tools he named. The open question is what those tools leave for the next step.

The debate in one screen

On Nov 2, 2026, Cipriano, Huggins-Manley and Lu published an Education Week opinion, "Five reasons for caution when building, scoring, or interpreting tests". Three of those cautions land hard on anyone who generates or scores with AI. Growth measurement "requires a stable metric across time points, comparable administration conditions, and an explicit model of change". "AI-generated interpretation of assessment data introduces a new, largely invisible source of measurement error". And for AI scoring, they advise: "have a human score them independently of the AI's scores". The op-ed notes the Standards are "currently under revision specifically to address AI-integrated assessment." (A Joint Committee is revising the 2014 edition.)

On Nov 5, 2026, at 05:48 Manila, @GolinoHudson replied. One post said "Why we actually should, with the right tools:" and another said "Why we should, with the right methods:" Both linked the open-access AI-GENIE paper. Both sides want evidence. They disagree on whether AI belongs in the workflow. Golino's answer is better methods, not abstinence. That framing is sound.

Where Golino is right

Russell-Lasalandra, Christensen and Golino (2026) introduce AI-GENIE in Behavior Research Methods, open access, published 01 July 2026. AI-GENIE applies UVA, EGA and bootEGA to LLM item embeddings. Across five nationally representative U.S. samples recruited from Prolific (N = 4,964; final Ns 989 to 997), the authors report structural validity, meaning internal structure (dimensionality and item stability), comparable to traditional expert-developed measures. Structure is what they checked.

They also scope their own work with care. The authors scope "validation" to the "structural phase"; it "does not provide evidence for the remaining four sources (test content, response processes, relations to other variables, or consequences of testing)". That honesty is a strength of the paper, not a weakness.

Golino has also announced a forthcoming paper, "Network Methods Outperform PCA in Recovering Dimensional Structure from Large Language Model Item Embeddings" (announced on X, not yet peer reviewed). Its abstract says EGA "recovered the correct number of dimensions in nearly every replication". That supports his structural point. Call what AI-GENIE does a structural check, not a finished instrument.

Structure is not a unit

EGA estimates dimensional structure, and bootEGA and EGAnet's invariance tools check its stability across resamples and groups. What it does not set out to do is place items and people on a common interval unit with a standard error for each measure, the specific objectivity Rasch models are built for. That is scope, not a charge.

The paper says so itself. "AI-GENIE's structural validation does not substitute for IRT-based examination of item difficulty and discrimination, which remains an important complementary step." The paper suggests post-selection IRT such as 2PL. It does not propose Rasch.

A related preprint leans the same way. In Cummins et al. (2026, preprint), students answered the LLM-written items more accurately, and a 2PL model found them less discriminating; the 2PL difficulty shift did not meet the authors' own decision rule. Face-valid items can still be psychometrically weaker. Structure alone does not settle difficulty, discrimination or a shared unit.

In our reading, caution 2 (subgroups) asks for invariance, caution 3 asks for a "stable metric," and caution 4 asks that interpretation error be made visible, which needs a stated uncertainty. The op-ed never says "unit" or "Rasch". The mapping is ours. Structure is necessary. It is not sufficient.

Fitting the data versus measuring

AI-GENIE's proposed next step is 2PL IRT. Cummins uses IRT difficulty and discrimination. Both sit in a long argument about what a measurement model is for.

Andrich (2004) frames the controversy as "Kuhn's concept of legitimate, incompatible paradigms": some "saw Rasch models as simply special cases of IRT models and those who saw them as essentially different"; Rasch proponents "emphasize the models' compatibility with fundamental measurement found in physical science". Andrich (2004) describes two paradigms. In one, which he calls defensible and rooted in statistics, when the Rasch model does not account for the data, a more complex model such as the 2PL is tried. In the other, the Rasch model is treated as a statement of what measurement requires, so misfitting data are studied and items revised rather than the model changed.

The fair counterpoint is old and still live. Letters reprinted in Rasch Measurement Transactions record Lord's Nov 26, 1965 line: "test items can have the same difficulty level and still differ very much in discriminating power"; and Wright's Nov 23, 1965 reply that Rasch "allows for separating parameters of objects and agents". Real items do differ in discrimination. That is why 2PL exists.

Why do metrologists working on human-based measurement lean the other way? Melin et al. (2023) write: "The Rasch model is the only item response theory model to have an additive latent model, and therefore parameter separation (compared with two parameters logistic (2PL) or three parameters logistic (3PL) item response theory models)." And: "conditional inference results in sample-free item estimates, providing invariant measurement and specific objectivity to ensure traceability". These authors reject 2PL and 3PL for this purpose. They do not reject IRT as a family; they call Rasch an IRT model with the additive property they need.

The same paper: "Traditional psychometric methods (i.e., based on classical test theory) do not account for the ordinal nature of data generated by human responses, and lack the ability to separate person ability and item difficulty". Pendrill (2014), quoted there, calls Rasch a "specifically metrological approach to human-based measurement". Melin et al. (2023) treat the Rasch standard error as an estimate of random error used to express measurement uncertainty, and state that the contribution of systematic errors "ha[s] not, as yet," been assessed. That candor matches the limit we state for our own work.

Metrologists working on human-based measurement, such as the EMPIR NeuroMET team (Melin et al., 2023), Pendrill (2019), and Fisher and Stenner (2016), build on Rasch models. Pendrill (2019) discusses Rasch units for traceability. Wright (1997) asks whether item response models meet the requirements of fundamental measurement and argues that only models with sufficient statistics, such as Rasch's, can.

A substantial literature, much of it from Rasch and metrology authors, argues that measurement needs models whose person and item parameters separate (Wright, 1997; Andrich, 2004; Mari and Wilson, 2014; Melin et al., 2023). It is not a consensus: IRT authors from Lord onward answer that real items differ in discrimination, and Michell questions whether any of these models tests that the attribute is quantitative. Metrologists working on human-based measurement, such as Pendrill and the NeuroMET team, have predominantly used Rasch models. Mari and Wilson (2014) introduced the Rasch approach to metrologists as a member of the IRT family, and Mari, Wilson and Maul (2023) use it as their worked example within a general measurement framework.

Wright pressed the same fork from the MESA laboratory. Wright (1997) put the contrast bluntly: for the 3PL, "the model must fit, else find another model"; for Rasch, "the data must fit, else find better data" (quoted from the MESA Memo 62 version). Wright (1997) argued that because the 2PL gives each item its own slope, the slope does not cancel when two people are compared, so person and item parameters cannot be estimated separately; Rasch's raw-score sufficiency lets them separate (quoted from the MESA Memo 62 version). Wright (1997) argued that when item curves cross, item order changes with ability, and he set the goal as "one ruler for everyone, everywhere, every time" (quoted from the MESA Memo 62 version). That line is Wright's goal for what data must be built to yield, not a finding that Rasch scales achieve it.

On discrimination and guessing, those checks are diagnostics. Wright wrote that "Variation in discrimination, a sure symptom of item bias and multi-dimensionality, is also rejected (Masters, 1988)"; "sure symptom" is Wright's phrase, not Masters'. Masters (1993) put it more evenly: "Both unusually low and unusually high discriminations merit further investigation". Wright (1997): Rasch "recognizes guessing, not as an item asset but, as an unreliable person liability"; "When it does occur, it is not items that do the guessing. The place to look for guessing is among guessers." Linacre (2004): "Upper and lower asymptotes are notoriously difficult to estimate"; Rasch-based estimates "can be useful in diagnosing whether the behavior they reflect could be distorting the Rasch measures". Reviewing two IRT texts in 1984, Wright argued that their own reports showed discrimination and guessing estimates were unstable and sample dependent. Estimation methods have moved on since 1984. That 1984 review is not a current census of 3PL practice.

One working example of a commercial metrological network in reading is Stenner and Fisher (2013): "The Lexile Framework for Reading constitutes a commercial metrological traceability network linking books, test results, instructional materials, and students in elementary and secondary English and Spanish language reading education in the U.S., Canada, Mexico, and Australia". Stenner is listed as CEO of MetaMetrics, which owns the Lexile Framework, and the paper itself calls the network commercial. Even Rasch advocates warn that the model alone is not enough. Stenner, Fisher, Stone and Burdick (2013): "Rasch analysis, absent construct theory and an associated specification equation, is a black box in which understanding may be more illusory than not".

The difference is not that one approach uses statistics and the other does not. Both are statistical models. The difference is the direction of fit. In what Andrich (2004) calls the Rasch paradigm, the model states what measurement requires, and when items misfit, the items are studied and improved. In the traditional paradigm, which Andrich calls defensible and rooted in statistics, a more flexible model such as the 2PL or 3PL is chosen to fit the data. Wright (1997) put it bluntly: for the 3PL, "the model must fit, else find another model"; for Rasch, "the data must fit, else find better data." Metrologists working on human-based measurement, such as Pendrill and the NeuroMET team, have predominantly used Rasch models because parameter separation supports traceability (Melin et al., 2023). Even Rasch advocates warn that fitting the model is not enough without a construct theory (Stenner et al., 2013).

Short form: model-led versus data-led. IRT and classical test theory are not "wrong". They answer a different question.

Calibrate, then pick judges level by level

Rasch-family models give a common unit, a standard error for each estimate, and testable invariance (DIF), not guaranteed invariance. That is the layer the structural check does not claim to supply.

For AI scoring, keep the op-ed's advice in view: "have a human score them independently of the AI's scores". Beside that, XLNC picks AI judges per scale level using MCpk, XLNC's own holistic judge capability index across infit, outfit, point-measure correlation and multidimensionality (closer to 1.0 is better). One overall agreement number can hide level-specific judge misfit; a per-level check is built to surface it. (MCpk is XLNC's own method; its process excellence recheck is pending.)

If the ruler is called "traceable", rest that word on Barney and Barney (2024), a chapter in Fisher and Pendrill (Eds.). List MASEMS only as: Thornton, A., Barney, M., and Fisher, W. P., Jr. (in preparation). MASEMS: Metrological Architecture for the Social, Educational, and Medical Sciences.

What AIM adds

EGA finds an instrument's structure, and EGAnet checks that structure's stability across resamples and groups. By its own scope, it does not set out to give Rasch's specific objectivity or a measure in a stated unit. AIM builds on Rasch-family models and reports each measure with its standard error; full metrological traceability is work in progress (Barney and Barney, 2024; MASEMS, in preparation).

AIM reads results against the buyer's quality, cost, quantity and cycle time targets (Barney, 2013). Today buyers set the error bar they need; other settings are on the roadmap.

In short, EGA, as AI-GENIE uses it, finds an instrument's structure and item stability. By its own scope, it does not set out to give Rasch's specific objectivity or a measure in a stated unit. AIM builds on Rasch-family models, reports each measure with its standard error, and reads results against the buyer's quality, cost, quantity and cycle time targets (Barney, 2013). Today buyers set the error bar they need; other settings are on the roadmap, and full metrological traceability is work in progress (MASEMS, in preparation).

AIM builds on the same Rasch-family models these authors use. Nothing here implies that Wright, Andrich, Pendrill, Mari, Wilson or any cited author endorses AIM.

Close

Structural checking plus calibration is a fuller pipeline than either alone. Golino is right that the answer to the op-ed is better methods. AI-GENIE checks structure. The next layer is a common unit with a standard error for each measure, and judges checked where they fit. The op-ed notes the Standards are "currently under revision specifically to address AI-integrated assessment." (A Joint Committee is revising the 2014 edition.) In our reading, a revision that takes those cautions seriously will need both layers.

Going deeper on calibrating AI-written items and AI judges: join the Jan 7, 2027 Maven lesson (9:00 AM ET). https://maven.com/p/3f03a8/advanced-ai-evaluation-beyond-pass-rates?utm_source=xlnc_blog&utm_medium=blog&utm_campaign=golino_metrology_2026-11&utm_content=a

References

• Andrich, D. (2004). Controversy and the Rasch model: A characteristic of incompatible paradigms? Medical Care, 42(1 Suppl.), I7 to I16. doi:10.1097/01.mlr.0000103528.48582.7c
• Barney, M. F. (2013). Leading Value Creation. Palgrave Macmillan. doi:10.1057/9781137361509
• Barney, M., and Barney, C. (2024). Chapter in Fisher, W. P., Jr., and Pendrill, L. (Eds.). doi:10.1515/9783111036496-003
• Bond, T., Yan, Z., and Heene, M. (2020). Applying the Rasch Model (4th ed.).
• Cipriano, C., Huggins-Manley, A. C., and Lu, Y. (2026, Nov 2). Education Week opinion.
• Cummins, J., Grinschgl, S., Elson, M., Borter, N., and Schulte-Mecklenbeck, M. (2026). Looks good on paper: LLM-generated exams are face-valid but psychometrically weaker than human assessments. PsyArXiv preprint. doi:10.31234/osf.io/vaf2y_v1
• Linacre, J. M. (2004). Discrimination, guessing and carelessness asymptotes: Estimating IRT parameters with Rasch. Rasch Measurement Transactions, 18(1), 959 to 960.
• Lord, F. M., and Wright, B. D. (2010). Fred Lord and Ben Wright discuss Rasch and IRT models. Rasch Measurement Transactions, 24(3), 1289 to 1290.
• Mari, L., and Wilson, M. (2014). An introduction to the Rasch measurement approach for metrologists. Measurement, 51, 315 to 327. doi:10.1016/j.measurement.2014.02.014
• Mari, L., Wilson, M., and Maul, A. (2023). Measurement across the sciences (2nd ed.). Springer. doi:10.1007/978-3-031-22448-5
• Masters, G. N. (1988). Item discrimination: When more is worse. Journal of Educational Measurement, 25(1), 15 to 29. doi:10.1111/j.1745-3984.1988.tb00288.x
• Masters, G. N. (1993). Undesirable item discrimination. Rasch Measurement Transactions, 7(2), 289.
• Melin, J., Cano, S. J., Gillman, A., Marquis, S., Floel, A., Goschel, L., and Pendrill, L. R. (2023). Traceability and comparability through crosswalks with the NeuroMET Memory Metric. Scientific Reports, 13, 5179. doi:10.1038/s41598-023-32208-0
• Pendrill, L. (2014). Man as a measurement instrument. NCSLI Measure, 9(4), 24 to 35.
• Pendrill, L. (2019). Quality assured measurement. Springer. doi:10.1007/978-3-030-28695-8
• Rasch, G. (1960/1980). Probabilistic Models for Some Intelligence and Attainment Tests.
• Russell-Lasalandra, L. L., Christensen, A. P., and Golino, H. (2026). Behavior Research Methods, 58(8), 217. Open access, published 01 July 2026. doi:10.3758/s13428-026-03082-1
• Stenner, A. J., and Fisher, W. P., Jr. (2013). Metrological traceability in the social sciences: A model from reading measurement. Journal of Physics: Conference Series, 459, 012025. doi:10.1088/1742-6596/459/1/012025
• Stenner, A. J., Fisher, W. P., Jr., Stone, M. H., and Burdick, D. S. (2013). Causal Rasch models. Frontiers in Psychology, 4, 536. doi:10.3389/fpsyg.2013.00536
• Thornton, A., Barney, M., and Fisher, W. P., Jr. (in preparation). MASEMS: Metrological Architecture for the Social, Educational, and Medical Sciences.
• Wright, B. D. (1984). Contemporary Education Review, 3(1), 281 to 288. rasch.org/memo41.htm
• Wright, B. D. (1997). A history of social science measurement. Educational Measurement: Issues and Practice, 16(4), 33 to 45. doi:10.1111/j.1745-3992.1997.tb00606.x; quoted from the MESA Memo 62 version, rasch.org/memo62.htm
• Wright, B. D., and Masters, G. N. (1982). Rating Scale Analysis. MESA Press.