Condensed from METHODOLOGY.md (authoritative) and the frozen ruleset in docs/BENCHMARK_SPEC.md. Numbers here are constants from the code, cited by file.
| World | 2-D rigid-body arena (pymunk). Two ragdoll fencers spawn facing each other at equal distance from the centre line; arena variants normal / ice (low friction) / low_gravity (0.35 g). stickblade/benchmark.py physics_spec(). |
| Turn | Both models receive the state simultaneously and answer in parallel threads (scripted-vs-scripted resolves inline for determinism, stickblade/main.py:238). The engine then simulates TURN_SECONDS of physics with both actions applied. |
| State (macro) | Positions, facing, distance, HP, weapon and sharp zones, last events, plus a ranged_hint block for bows. blindfolded strips the derived spatial hints. Schema versioned as PROMPT_VERSION (stickblade/brains.py:46). |
| Action (macro) | One action from the weapon’s vocabulary (thrust, guard, draw_shot, …) + one footwork (advance, retreat, lunge, hop_back, hold). Invalid replies are sanitised to a safe default and counted (invalid_actions_*). |
| Action (joint) | Ten joint targets per turn (shoulder/elbow/grip/hips/knees…), optional fire. Deliberately harder; ranked in its own cell. |
| Result | kill (HP ≤ 0), points (higher HP at the turn cap), timeout_draw, mutual_destruction. stickblade/main.py:520. |
| Lengths | sprint 4 · standard 12 · full 24 turns. Ratings are not mixed across lengths in research batches. |
A model that does not answer in time still has to do something, or the match stalls for everyone. The failure ladder is: retry the same model with a longer timeout → a “buddy” model of similar tier → the scripted mock (stickblade/brains.py:1027). Every step is recorded per turn: model_used, provider_used, fallback. What that does to the ranking depends on the fallback policy chosen at match creation:
| strict | Any fallback turn makes the match ranking-ineligible. Used for all research batches. A strict match that fell back is still stored, labelled `excluded`, and counted in the audit. |
| operational | Default for the public arena. Fallback continues, is recorded and is shown on the result card; the match remains eligible. |
| demo | Never ranked. |
Silent fallback — a fallback turn in a strict match that is still marked eligible — is a defined defect, counted on /api/data_quality (silent_fallback_matches) and asserted to be zero by the calibration audit.
- Fighters are renamed “Fighter A / B” in the replay itself; the reveal happens after the vote (
stickblade/server.py/api/vote). - Canvas side is a coin flip per match (
flip); votes are cast on canvas sides and mapped back to models with the same function Elo and Bradley–Terry both use (stickblade/ratings.py:342preference_pairs_from_votes). - Four axes: tactical (ranked), execution, entertainment, deserved; plus 1–5 confidence and a self-declared voter tier (casual / expert). Expert and casual boards are fitted separately and never pooled.
- Anti-gaming: one vote per match, rate limits per IP, blind names in the replay, and integrity checks on every replay (
/api/integrity/{id}).
| Elo (scoreboard) | K = 32, start 1000, six-key cell (model × sharp × weapon × mode × arena × blindfolded). Wilson 95 % CI on win rate. Order-dependent by construction, so it is not the scientific claim. stickblade/storage.py:14. |
| Bradley–Terry (claim) | Maximum-likelihood fit over all votes in a cell with a Davidson tie parameter ν (estimated, forced to 0 with no draws), ridge shrinkage 1.0 so a 3–0 record stays finite, ratings centred per connected component, anything outside the main component flagged provisional. Reported on an Elo-like scale 1000 + (400/ln10)·θ for readability only. stickblade/ratings.py:69. |
| Uncertainty | Bootstrap over the comparison set (default 200 resamples), 2.5 / 97.5 percentiles. Intervals must widen as n falls — pinned by tests/test_ratings.py::test_more_data_gives_a_tighter_interval. |
| Separability | Two models differ only when the intervals do not overlap. Otherwise they share a tie letter and are described as not separable. |
| Objective board | Physics-derived and vote-independent: damage per turn, hit rate, lethal rate, survival, timeout, invalid-action, fallback rate, latency. Published beside the ratings, never used to rank. |
Every ranking row and every export row is classified by who actually decided, not who was asked (stickblade/data_quality.py):
| real_provider | Both sides answered by a real provider (OpenRouter, Groq, OpenAI, Google, …). |
| mixed_provider | One side real, one side scripted or fallen back to scripted. |
| scripted_baseline | Both sides scripted (bot:*, mock:*). Pipeline calibration; not a model result. |
| Model status | ranking_eligible needs ≥ 10 real + ranking-eligible matches in the cell; below that exploratory_only; scripted models are reference_baseline. Board level: real needs ≥ 30 real ranked matches, else insufficient_real or scripted_only. |
| BENCHMARK_VERSION | 1.0 — the ruleset. Bumping invalidates comparability. |
| PHYSICS_VERSION | 1.0 — timestep, gravity, damping, damage model. Any change bumps it. |
| PROMPT_VERSION | 2 — state schema + system prompt. v1 → v2 on 2026-09-08 was additive (bow mobility hints), a soft cutover: v1 ratings stay readable, v1 and v2 bow cells are not averaged. Ledger in AGENTS.md §10.5. |
| Spec fingerprint | SHA-256 over the outcome-affecting parts of the spec, including the prompt version: 09de66effd02 under prompt v2, 029281ed627a under prompt v1. CI asserts the fingerprint per prompt version. |
python3 stickblade/benchmark.py --fingerprint # prints the current fingerprint curl -s $API/api/benchmark/spec | jq . # the full frozen ruleset
The first real-provider run is a calibration of the pipeline, not a tournament. tools/run_calibration_batch.py builds a balanced plan: every unordered pair of 4–6 models × 2 weapons × 2 arenas × 2 control modes, N matches per pair (20–30 for a powered pair), canvas sides alternated per cell so each model plays left and right equally, seed = base + index, strict policy, standard length. The run ends with an acceptance audit: 100 % provider + model identified, 100 % eligibility recorded, zero silent fallback, ≥ 95 % token coverage or the gap reported, failed matches retained but not ranked, and real-provider evidence present. A scripted dry run fails exactly the last check — by construction.