XLNCXLNC Book a 20-minute scoping call
Menu

Blog

Which question is your AI evaluation paying to answer?

Stopping when the decision is settled, what it costs, and what it gets wrong

By · · Measurement practice

Most AI evaluations pay to answer one question: where exactly does this text score? Many decisions only need a smaller one: is it above our standard or below it? This post shows what the smaller question costs, how often it gets the answer wrong, and when the bigger one is worth paying for.

In short

  • Asking only which side of your standard a text is on cost about half as much per text in a replay of real reads ($0.0090 against $0.0184), with no difference in wrong calls when every text is forced to a decision.
  • In a simulation of insurer letters, the more a wrong pass costs, the higher the confidence level worth paying for.
  • If a wrong pass costs little and is easy to undo, you don't need AIM.
  • This way of stopping is in build. Today AIM stops on the precision you set.

Why the question matters: denial letters

In the opening week of June 2019, 15 of the largest Medicare Advantage organizations issued 12,273 prior-authorization denials of service requests. The HHS Office of Inspector General later estimated that 13% of those denials met Medicare coverage rules (HHS Office of Inspector General, 2022).

Now picture an insurer that has an AI model draft the reasoning in letters like these and wants each letter checked before it goes out. That insurer is our stand-in, not a real case.

Its evaluation team can ask for a precise score on every letter: where exactly does this reasoning sit on the scale? Or it can ask something smaller: is this letter above our standard or below it? They sound alike, but they ask different things and go wrong in different ways. Which one the decision needs depends partly on what a wrong pass costs.

Two questions, two stopping rules

A few terms to start. A judge is an AI model that scores a text. A read is one score from one judge. The cut is your pass line on the scale, and a logit is the scale's unit.

Maximum precision asks where a text sits, and keeps reading until the score is as tight as the setting you chose. This is how AIM stops today. Precision here means the smallest difference the measurement can reliably detect, about 3.96 times the standard error (95% confidence, 80% power). A setting of SE 0.30, for example, detects a difference of about 1.19 logits.

Hypothesis testing answers a narrower question: which side of the buyer's cut a text is on. It does not give a score at a stated precision. In this mode, AIM would add judges one at a time and stop once it is 95% sure of the side.

  • A text far from the cut settles after a few reads.
  • A text near it gets more.
  • A text that stays close is handed to a person rather than forced into a verdict.

Hypothesis-test stopping is in build; you cannot buy it from us today.

Nursing licensure already works this way. The NCLEX computer "seeks to determine with 95% certainty whether the candidate's true ability is above or below the passing standard", and scores are never reported, only pass or fail (NCSBN, 2026, p. 7). In 2025, for US-educated NCLEX-RN candidates testing for the initial time, its decision consistency was 0.90; 59.1% stopped at the minimum length and 20.5% ran to the maximum (NCSBN, 2026, pp. 7, 10).

What the narrower question costs, on real reads

We replayed stored reads from a six-judge research panel on 3,119 reasoning texts, with no new model calls. Every figure in this section is real replay, box refit, not of record: an internal reanalysis of real reads, not a certified result. The texts are reasoning texts, not insurer letters.

The cut is a buyer's, and here it is illustrative: level 10 or above, an assumption. At that cut, stopping at 95% settled about two in three texts with at most one read from each judge. The rest would need repeat reads or a person. The cost was about half that of having every judge read every text, with no difference in wrong calls between the two (exact McNemar test; undecided texts force-decided).

Stop at 95%Stop at 95%, one judge may settle a textEvery judge reads every text
Texts settled65.9%70.0%All read
Model cost per text$0.0090$0.0074$0.0184
Reads per text, average4.19 (4.14 to 4.24)3.33 (3.26 to 3.41)All six judges
Median time per text34.3 s (33.5 to 35.5)21.0 s (19.3 to 23.1)56.1 s
90th percentile time69.4 s66.5 s84.7 s

Real replay, box refit, not of record; cut illustrative; model time only, per-call latencies resampled from the real ledger (Oct 13 to 15, 2026 UTC), reads run one after another; excludes the person's time on the texts not settled (34.1% / 30.0%).

How often was it wrong? With every text forced to a decision, wrong passes were 3.0% of texts and wrong fails 1.5%. Those rates are scored against an 8-judge reference that includes the six panel judges' own reads. On the subset an independent reference was sure of, wrong passes were 3.9%.

Against the same 8-judge reference, the 95% rule made fewer wrong calls than maximum precision set at SE 0.45 or 0.50, which detect differences of about 1.78 and 1.98 logits (McNemar, Holm-adjusted; undecided texts force-decided).

The question itself changes the answer. Asked which two adjacent levels a text sits in, the panel settled 97.6% of texts (97.0 to 98.1) at $0.0049 per text (0.0047 to 0.0051), with a median of 23.2 seconds and a 90th percentile of 52.9 seconds. Among decided texts, its verdicts agreed 99.3% (98.9 to 99.5) with the 8-judge reference that includes the panel's own reads. That rule does not place a text against a buyer's cut, so its 97.6% is not a cut-score result.

The letter example, simulated

Every figure in this section is simulation, with reviewers assumed correct. The stand-in insurer grades the reasoning in 12,273 letters a week (the OIG batch size), with 13% of letters below the standard (exact), mapped from the OIG share by assumption.

Stopping at 95% settled 85.7% of letters and handed 14.3% to a person. Wrong passes among settled letters came to 1.26% of all letters. Model cost was $0.0206 per letter, against $0.0737 for maximum precision at SE 0.30 (a detectable difference of about 1.19 logits).

One input is uncertain: how alike two reads from the same judge are (their correlation, ρ). The results above use ρ 0.288, a pooled cross-prompt lower bound (0.241 to 0.337); no same-prompt repeat reads exist for these judges at this level. The planning value is 0.5. The read model is about 3 percentage units optimistic on the share settled.

At the planning value ρ 0.5, stopping at 95% settled 83.0%, handed 17.0% to a person and cost $0.0221 per letter, and its settled wrong passes came to 1.23% of all letters. Maximum precision at SE 0.30 became unreachable: repeat reads from the same judges stop adding enough information, and every letter used all 30 reads at $0.0921.

Two-panel line chart, labeled as a simulation study of a six-judge quick panel (research), 20,000 simulated texts per location. The horizontal axis in both panels is the true location minus the buyer-set cut, from -1.5 to +1.5 logits on the PCM scale, with the prior centered on the cut. Thick lines use ρ 0.288, a pooled cross-prompt lower bound (0.241 to 0.337), because no same-prompt repeat reads exist for these judges; thin lines use the planning value ρ 0.5. The read model is about 3 percentage units optimistic on the share settled. Left panel, model cost in US dollars per 1,000 texts: - Max precision with an SE target of 0.30 (detects about 1.19 logits) is flat at about $74 (ρ 0.5: about $92, because the target is never reached and all 30 reads are used). - Max precision with an SE target of 0.20 (detects about 0.79 logits) is flat at about $92 at both ρ values, because it never reaches its target and uses all 30 reads. - One judge, one read is flat at about $0.40 (one read, so ρ does not apply). - Hypothesis testing at 95% is a peak shape: about $8 at ±1.5 logits, $22 at ±1.0 (ρ 0.5: $25), $57 at ±0.5 (ρ 0.5: $61), rising to about $80 at the cut (ρ 0.5: $81). Right panel, wrong calls settled by the panel as a percent of texts: - One judge, one read is highest: about 49% at the cut, 34% at ±0.5 and 12% at ±1.5. - Both max-precision lines peak at about 43% to 44% at the cut (ρ 0.5: about 44%) and drop to about 5% to 6% at ±0.5 (ρ 0.5: about 8%). - Hypothesis testing at 95% peaks at about 7% to 8% at the cut (ρ 0.5: about 6% to 7%), is about 3% at ±0.2, and is under 1% from ±0.5 outward at both ρ values. - A dash-dot line shows the share hypothesis testing sends to a person, the complement of the share settled; these are not wrong calls. It is about 80% at the cut (ρ 0.5: 83%), 42% at ±0.5 (ρ 0.5: 54%) and 4% at ±1.0 (ρ 0.5: 11%).
Figure 1. Simulation, 20,000 simulated texts per location. Left: model cost per 1,000 texts. Right: wrong calls as a percent of texts. Hypothesis testing at 95% costs the most near the cut and stays under 1% wrong from 0.5 logits outward; one judge, one read is wrong about 49% of the time at the cut.

Are you spending too much or too little on AI evaluation?

The two mistakes point in opposite directions.

Too much looks like reading every text with every judge. In the real replay, that cost $0.0184 per text against $0.0090 for the 95% rule, with no difference in wrong calls when undecided texts are force-decided (real replay, box refit, not of record).

Too little looks like one judge, one read, scored as if it were exact. Near the cut, within 0.5 logits by the 8-judge estimate, a single read from any one of the six judges was wrong 25.8% to 28.3% of the time (real replay, box refit, not of record).

Metrology has names for both habits. Accepting a result at the limit with no guard band is "simple acceptance", with up to a 50% chance of a false accept at the limit. The ISO 14253-1:2017 guard band of 0.83U holds the false-accept probability under 5% (ILAC, 2019, pp. 8, 12; see also Joint Committee for Guides in Metrology, 2012b, p. 20). A 95% stopping rule applies the same idea to AI evaluation.

The price of not deciding

Sending letters unchecked also has a cost. Premier reports that providers spend an average of $43.84 to fight a denial, and that the average denial is tied to charges of $14,000 and up (Premier Inc., 2024). Those are provider-side figures, used here as analogs, not as measured insurer losses.

The table shows weekly batch totals in the simulation: wrong-pass losses plus reviews plus model cost. Each column is a different loss per wrong pass. Figures in parentheses use the planning value ρ 0.5.

Mode$43.84 per wrong pass (cost to fight a denial)$500 per wrong pass (assumption)$14,000 per wrong pass (value at stake)
Send every letter unchecked (no ρ dependence)$69,900$798,000$22.3 M
Maximum precision, SE 0.30 (detects about 1.19 logits)$47,000 ($48,900; target never reached)$242,000 ($274,000)$6.02 M ($6.93 M)
Maximum precision, SE 0.20 (detects about 0.79 logits; never reached, all 30 reads)$46,600 ($48,900)$233,000 ($274,000)$5.75 M ($6.94 M)
Hypothesis testing 95%, every letter forced to a verdict$46,100 ($48,200)$244,000 ($278,000)$6.09 M ($7.09 M)
Hypothesis testing 99%, every letter forced to a verdict$45,900 ($48,100)$232,000 ($272,000)$5.75 M ($6.91 M)
Hypothesis testing 90%, close letters to a person$50,100 ($52,400)$183,000 ($183,000)$4.12 M ($4.05 M)
Hypothesis testing 95%, close letters to a person$53,900 ($57,700)$124,000 ($126,000)$2.21 M ($2.16 M)
Hypothesis testing 99%, close letters to a person$69,000 ($76,800)$83,500 ($91,400)$0.51 M ($0.53 M)

Simulation, stand-in insurer scenario. Batch of 12,273 letters (source: HHS OIG, 2022). 13% of letters below the standard (exact; mapped from the OIG share by assumption). Loss per wrong pass: $43.84 and $14,000 (source: Premier Inc., 2024; provider-side analogs, not insurer losses), and $500 (assumption). Review cost $20 per letter sent to a person (assumption), charged for hand-offs, wrong fails and correct fails. Reviewers assumed correct. Model cost from real ledger per-call costs. Repeat-read correlation ρ 0.288, a pooled cross-prompt lower bound (0.241 to 0.337), with the planning value ρ 0.5 in parentheses. The read model is about 3 percentage units optimistic on the share settled. Box refit, not of record.

Simulation precision: about ±$100 in the $43.84 column, ±$1,000 in the $500 column and ±$0.02 M in the $14,000 column; differences under $1,000 at $43.84 are ties.

The right confidence level rises with the stakes

At a $43.84 loss, every rule that forces a verdict costs about the same, about $46k to $47k a week (ρ 0.5: about $48k to $49k).

At $500 or more per wrong pass, stopping at 99% and handing close letters to a person costs the least: $83,500 a week at $500 (ρ 0.5: $91,400) against $232,000 to $244,000 for the rules that force a verdict, and $0.51 M at $14,000 (ρ 0.5: $0.53 M) against $5.75 M to $6.09 M (simulation, labeled as in the table above).

So in the simulation, the right confidence level rises as the cost of a wrong pass rises. A higher-stakes letter justifies more reads and more hand-offs. That matches metrology's cost-based approach, which sets measurement effort so that the cost of measuring plus the cost of wrong decisions is as small as possible (Pendrill, 2014; simulation, ρ 0.288 lower bound, planning value 0.5, reviewers assumed correct).

To read the table for your own letters, start with the column nearest your loss per wrong pass. Every row assumes $20 for each letter sent to a person.

What makes a 95% answer worth trusting

A 95% answer depends on the scale underneath it. Three parts of AIM's design carry that weight.

The scale comes with metrological traceability. AIM is designed so each result has an unbroken, documented chain of calibrations back to a stated reference, each step contributing to the stated measurement uncertainty (Joint Committee for Guides in Metrology, 2012a). The chain runs to waypoints on the calibrated scale: content-defined levels that mean the same thing for every buyer. Your cut score sits outside that chain. It is your decision rule, placed on the scale, and it can move when your costs change while the waypoints stay put.

The panel spans model families and adjusts for harsh or lenient judges. The panel is designed to span several model families and to adjust for each judge's severity before combining reads, a method known as many-facet Rasch measurement (Linacre, 1989). The research panel here spans six families. Dropping any one family still settled 61% to 64% of texts, with 4.7% to 6.1% wrong (real replay, box refit, not of record). We are still building out the panel's family and country coverage; AIM does not yet meet that coverage at every level.

The settings are yours. You choose the cut, the confidence level and what happens to close cases, and you can match them to the stakes. A draft check and a regulator-facing letter do not need the same evidence. Custom settings are scoped in an enterprise engagement.

Where this stands

Hypothesis-test stopping is in build. Today AIM stops on the precision you set. Real-replay figures are a box refit, not of record. Simulated figures assume reviewers are correct and carry the repeat-read correlation caveat above. The insurer is a stand-in, and the cut is illustrative.

Before your next evaluation run, write down the decision each score is for. If the decision is a side of a cut, ask for the side. If it needs a location, for a trend or a second cut, ask for precision. If a wrong call is cheap and easy to undo, you don't need AIM for it.

If you'd like to work through that with others, I'm teaching a free Maven Lightning Lesson, Beyond Pass Rates: AI Evaluation That Moves the Business, on Thursday, January 7, 2027, 9:00 AM ET: maven.com/p/3f03a8.

Quick answers

Is hypothesis-testing mode available today?

No. It is in build. AIM stops on the precision you set today.

Does hypothesis testing give me a score?

No. It answers which side of your cut a text is on, at the confidence you chose. When you need a location with a stated precision, maximum precision is still the right tool.

What happens to texts it cannot settle?

They go to a person, or to more reads, under the rule you set. In the simulated letter example (reviewers assumed correct; repeat-read correlation at its lower bound) that was 14.3% of letters (17.0% at the planning correlation).

Who sets the cut score?

You do. AIM keeps the calibrated scale and its waypoints; you place your decision rule on it.

References

Tell us which number you do not trust.

You will get a short note from Matt when a measurement slot opens. Nothing else, no drip sequence.

We store what you send with the few providers our Privacy Policy names, and use it to reply to you. We do not sell it. Email matt@xlnc.co and we delete it. Read our Privacy Policy.