One index at the top; click any row to go down: area, kind of action, skill class, benchmark. At the bottom of
Code optimisation are the techniques we saw models use in our own runs. Every number links to the report it was read from.
Aziz Latipov · data as of · first version
SourcesShow models measured on at least of the parts
How the numbers are made
Benchmark: per model, the best score among the rows the chosen sources allow (labs usually report their best setting; the setting is in the evidence table). Scores in % or 0–1 are used; Elo and other scales are shown but not rolled up.
Everything above a benchmark: the mean of its parts, weight 1 each, over the parts measured for that model. "k/n" = parts measured of the parts that have any data. A model is ranked only if it has at least the chosen share of parts.
Compare: the table under the ranking compares the models you pick (click a bar to add or remove). "Only parts all of them have" recomputes every number on the same parts for all compared models, which is the fair comparison.
First version: all weights are 1 for now, and where a benchmark sits in the tree is a first placement; both will change. Numbers are as the labs and leaderboards published them, read from their pages and checked by script against saved copies of those pages; a random sample of 60 was re-checked by a second model for the right model column (no errors found).
Code optimisation: our Quern runs (Claude models only): a technique counts 100 if the model used it in at least one run, else 0.
Sources: "labs about own models" = launch posts, model and system cards of the model's developer. Numbers a lab published about a competitor and third-party leaderboards are added by the switch. Data from Artificial Analysis' own files is not used.