The Ruler That Changes Length: Reading AI Index Scores Across Versions

Imagine your team chose a model in June because it led an AI index. In October you check again. Its score has dropped and two rivals sit above it.

Did your model get worse? The number cannot tell you. Between those two readings, the index itself was rebuilt, with new tests, new weights and new graders.

Nobody hid anything. The changes are listed in a public changelog. But you are now holding two numbers from two different rulers.

What changed, by the record

The Artificial Analysis Intelligence Index combines nine evaluations into one number. In its current version, v5.0.1, they sit in four categories: Knowledge Work, Coding, Science and General, each worth 25% (methodology).

Its public version history is unusually candid. It lists nine releases dated 2026, from v4.0 in January to v5.0.1 in November. A few of the changes:

• v4.0 (January) set four categories at 25% each: Agents, Coding, General and Scientific Reasoning.
• v4.1 (June) moved them to Agents 34%, Coding 24%, Scientific Reasoning 24% and General 18%, "to further emphasize agentic tasks."
• In August and September, v4.1.1 changed the grader model for three evaluations, and v4.3.1 refreshed the judge panels that score pairwise comparisons.
• v5.0 (October) returned to 25% each, with two categories renamed and GDP.pdf moved between them. It cut GDP.pdf from 10% to 5%, removed four evaluations (GDPval-AA, SciCode, AA-LCR and Humanity's Last Exam), added FrontierCode-AA 2.0 (15%), Terminal-Bench-Science (15%) and HLE-Diamond (10%), and swapped CritPt for a harder CritPt v2 subset at the same 10%.
• v5.0.1 (November), a patch release, regraded all models on CritPt v2.

In November 2026, a practitioner charted the weights version by version. His summary: "They vary quite a bit!"

The fair reply

One reply to that chart made a fair argument. When you add or remove benchmarks, you have to change the weights, and that makes sense.

It does. A frontier index should retire tests that no longer separate the leaders, and add harder ones. Artificial Analysis says plainly what follows: "Each version of the Intelligence Index is therefore measured on a different set of evaluations, and scores across versions are not intended for long-run comparison" (Progress Index methodology).

So the risk sits with the reader. It shows up when someone puts a June score next to an October score and calls the gap progress.

A ruler that changes length

Here is the principle. A change in a number means something only if the ruler held still between the two readings.

When the tasks, the weights and the graders all move, a change in score blends the model with the test and with the people or models doing the grading. One number cannot pull those apart. A model can rise because it improved, or because the index now weighs the work it was already good at.

The graders deserve more attention than they get, because that is where we spend our days. A word on scope before the numbers: none of what follows shows that any index's judge refresh moved its scores. We have not tested that. It shows that a change of graders can move results, so it has to be measured rather than assumed away.

In our own calibrations, LLM judges ran harsh at one capability level and lenient at the next. Agreement among AI judges ranged from 0.46 to 0.84 against the rest of the panel. Human rater pools differ too. In the DICES-990 safety ratings, a harshness shift between rater pools changed about 135 of 990 verdicts (95% interval 60 to 222). That is a difference between rater pools, not a country effect.

What a stable scale looks like

Artificial Analysis has already moved in this direction. Alongside v5 it launched the Progress Index, which uses item response theory to put model capability and task difficulty on one shared scale. By its own description, models link through the tasks they have in common, so a model tested on older evaluations can be compared with one tested on newer ones. The scale is anchored: o3-mini (high) sits at 220 and GPT-5.5 (xhigh) at 320 (methodology). That is the right instinct.

Stated uncertainty is the other half. When Terminal-Bench moved to 4.1, the 4.0 leaderboard trials were re-graded with the new verifiers. About 1% of rewards flipped, and the announcement said leaderboard scores stayed within the confidence intervals reported before (announcement). Change the instrument, then show how much moved against a band you stated in advance. That is how you keep a ruler honest.

This is how we handle the judge problem above. For judge-based evaluation we use Rasch measurement with separate facets, meaning the difficulty of each item and the harshness of each judge are estimated separately and adjusted for. The scale is anchored, so a new version can be checked against the old one. Scores are reported with an uncertainty band and a trail back to the reads behind them. In one fit, 21 judges carried the information of about 19 to 20 independent ones, so we count shared mistakes before we count votes. In our replay, all 21,833 judge reads link to their source records.

Before you compare two scores

1. Check the version. If two scores come from different index versions, patch releases included, do not subtract them.

2. Look underneath. Find the component that matches your use case and read that instead of the composite. As the same practitioner put it: "look underneath" and don't "fly towards the index score blindly."

3. Ask for the band. A gap smaller than the stated uncertainty cannot be told apart from a tie. Leaving the uncertainty off a benchmark chart is, in his words, a chart crime.

4. Ask about the graders. If judges score the work, ask how their harshness is measured and whether it held still between versions.

5. Re-grade before you switch. When you change a test, a weight or a judge in your own evaluation, re-score a fixed set of past runs on both versions and report what moved against your stated band. Then decide whether to adopt the change.

The takeaway

Composite indexes are good at one job: showing where the frontier stands today. Measuring how far anything moved is a different job, and it needs a ruler that holds still.

A moving ruler can tell you who is tallest today. Only a fixed one can tell you who grew.

---

If you want to go deeper on telling a real difference from noise, Matt Barney is teaching a 60-minute Maven Lightning Lesson, Beyond Pass Rates: AI Evaluation That Moves the Business, on January 7, 2027, at 9 AM ET.