STICKBLADE ARENA is a physics-grounded, pairwise-preference benchmark for language models: two models control ragdoll fencers in a deterministic 2-D simulation, humans watch blind and vote, and ratings with confidence intervals are fitted from those votes. This page answers the seven questions a reviewer asks first. Everything here is backed by a file in the public repository.

Embodied tactical decision-making under a fixed physical ruleset, judged by human preference. Each turn a model receives a structured state (positions, distances, HP, weapon zones, last events) and returns one action plus footwork (macro mode) or raw joint targets (joint mode). The physics engine resolves the turn; the model never sees or edits the outcome directly. The ranked signal is the tactical vote (“who fought smarter?”); execution and entertainment votes are collected separately and are not ranked.

Because a physical ruleset cannot be memorised or negotiated with. Text benchmarks are contaminated by training data and graded by other models; here the grader is a rigid-body simulator and the rules are frozen in a versioned spec (docs/BENCHMARK_SPEC.md) with a fingerprint stored on every match. Consequences of a decision are computed, not asserted: a “thrust” from 400 px away misses because the geometry says so. The same physics also makes every non-LLM part of a match reproducible from a seed, which is what turns a leaderboard into an auditable claim (see /reproducibility).

• Matches are blind: fighters are “A” and “B”; the model identities are revealed only after the vote. Canvas sides are randomised per match (flip) so colour/side bias cancels; research batches pin sides to a balanced design instead.
• Ratings are segmented per cell (weapon × arena × control mode × sharp zone × blindfolded) so a model good with a bow on ice is not averaged with itself using a sword on grass.
• Elo (K = 32) is shown as a live scoreboard. The scientific rating is Bradley–Terry with Davidson ties, refitted over all votes in a cell with ridge shrinkage and bootstrap intervals (stickblade/ratings.py).
• Every row also carries a data-quality label — scripted baseline / mixed-provider / real-provider — and a status (ranking-eligible / exploratory only / reference baseline), from stickblade/data_quality.py.
• Scripted baselines (bot:*, mock:*) exist to calibrate the pipeline and give a floor. They are labelled as such everywhere and are never presented as model results.

Two models are called different only when their 95 % bootstrap intervals do not overlap (the lower bound of one above the upper bound of the other). Models whose intervals overlap share a tie letter on the leaderboard and are described as not separable. No rank is shown without its sample size and interval. In addition, a model’s ranking is only eligible once it has at least 10 real-provider, ranking-eligible matches in the cell, and a board is only treated as model evidence once it has 30 — below that every ranking is labelled exploratory and a banner says so.

The current honest statement, from the latest generated report (research/reports/2026-09.md): 0 of 1 pairs separable, on scripted data. That is the pipeline working, not a model finding.

The live strip at the top of this page is the answer, straight from the database. Every finished match stores: benchmark, physics and prompt versions, spec fingerprint, seed, match length, fallback policy, both requested models, both models that actually answered, provider per side, per-turn latency, billed tokens and API calls, fallback and invalid-action counts, engine result, per-turn action log, and the replay. Votes store the four axes, confidence, and a self-declared voter tier — no IPs, no user ids. The dataset page documents every column: /data.

As of this writing no real-provider calibration batch has been run; the runner exists and is verified end to end on scripted fighters (tools/run_calibration_batch.py, research/calibration/README.md). When it runs, the strip above changes on its own.

• Self-selected, uncalibrated voter pool; one vote per match, so no inter-rater agreement yet.
• Provider latency is part of the outcome: a slow strong model is penalised by timeouts. Latency, fallback and invalid-action rates are published so the effect is visible, not normalised away.
• LLM decisions are not reproducible from a seed; only physics and scripted brains are. Reproducibility is audited by replaying the stored action log.
• Prompt version 2 is not comparable with version 1 bow cells; filter by prompt_version.
• Everything above rests, today, on scripted-baseline data.

Full list with severity ordering: /limitations.

• Clone, install, run the 250-test suite offline (pytest tests); it includes seeded-replay determinism and the rating fitter’s interval behaviour.
• Download a versioned release (or build one from a local database with tools/export_dataset.py build), then tools/export_dataset.py verify re-hashes every file and refits the ranking from the files alone.
• Regenerate the monthly report from that release with tools/benchmark_report.py --export …/matches/matches.jsonl; §0 of the report names the dataset version and evidence level.
• Re-run any seeded scripted match bit-for-bit with tools/run_match.py; re-run a whole calibration design with tools/run_calibration_batch.py (backend and offline runners produce identical rows).

Step-by-step commands: /reproducibility.