What this benchmark cannot show, in decreasing order of severity, and what is being done about each. Source of record: METHODOLOGY.md §7 and research/reports/2026-09.md.
As of this writing no real-provider calibration batch has been run, so every leaderboard row is a scripted baseline or a pipeline test. The live banner on every ranking page says so and will change on its own once real matches land. Nothing on this site should be read as evidence about any language model until it does.
| Mitigation | Runner built and verified end to end (tools/run_calibration_batch.py). Blocked on provider keys on the backend. |
Visitors are not a calibrated panel and each match is voted on once, so there is no inter-rater agreement (κ). Vote noise is measurable through the bootstrap intervals; selection bias is not.
| Mitigation | Expert and casual tiers are fitted separately, never pooled. A multi-vote sample track is planned; not built. |
A model that answers slowly falls back or times out and is penalised, independent of its tactical quality. Fast providers are therefore favoured.
| Mitigation | Latency, fallback and invalid-action rates are published beside every rating and per turn in the actions table. Strict policy excludes any fallen-back match from ranking. The effect is measured, not corrected. |
Only physics and scripted brains replay bit-for-bit. A published match is re-simulated from its stored action log, which reproduces the fight but not the model call.
| Mitigation | Provenance (model used, provider, tokens, latency) is stored per turn so the call is at least auditable. |
The 2026-09-08 bump added movement hints for ranged play. It was additive, but v1 and v2 bow ratings measure slightly different questions.
| Mitigation | Every row carries prompt_version; fits are per version. The fingerprint changed with it (029281ed627a → 09de66effd02) and CI now asserts it per version. |
The public roster leans on free OpenRouter and Groq tiers, which rotate and throttle. Coverage of frontier paid models is thin unless a visitor brings their own key.
| Mitigation | BYOK is supported per match; keys are never stored. The calibration design targets 4–6 models across ~3 providers, not the whole roster. |
There is no prompt-injection or jailbreak evaluation. Out of scope for a competitive-play rating; stated so it is not assumed.
| Mitigation | None planned. |
Damage per turn, hit rate and the like are physics-derived and vote-independent, but they measure aggression as readily as skill. They are published beside ratings and never used to rank.
| Mitigation | Kept as a separate board with that caveat attached. |
- No leaderboard rank is shown without its sample size and interval.
- No pair of models is called different unless their 95 % intervals do not overlap; otherwise they share a tie letter and are “not separable”.
- Scripted baselines are labelled as such on every row, every export and every report, and are never presented as model results.
- Every report names its dataset version, benchmark version, prompt version and spec fingerprint (
tools/benchmark_report.py§0). - “Infrastructure validated” and “model conclusions validated” are separate claims; today only the first is true.