Reproducibility

What can be re-run, what cannot, and the commands for each. Nothing below needs an API key, a database or a deployed server except where it says so.

ResearchMethodologyDataReproducibilityLimitationsStatus
What is and is not reproducible
Physics + scripted brainsBit-for-bit from a seed. Pinned by tests/test_replay_determinism.py, including the case where the server asks for pre-fight quips first (a 2026-09-09 fix — quips used to advance the decision RNG).
LLM decisionsNot reproducible from a seed (providers are non-deterministic). Every decision is stored in the action log, so a published match is re-simulated from the log, not by re-calling the provider.
Ratingstools/export_dataset.py verify refits Bradley–Terry from a release’s matches + votes and compares point estimates with the manifest. Tampering with one vote fails it (tested).
SpecThe fingerprint is recomputed from code on every CI run and asserted per prompt version (.github/workflows/ci.yml).
0. Set up (offline, ~1 minute)
git clone https://github.com/Cometbuster4969/STICKBLADE-ARENA
cd STICKBLADE-ARENA
pip install -r stickblade/requirements.txt pytest
export SDL_VIDEODRIVER=dummy          # headless pygame
python3 -m pytest tests -q            # expect: 25x passed
1. Re-run a seeded match bit-for-bit
./tools/run_match.py simulate --a bot:pro --b mock:duelist --seed 20260909 \
    --weapon bow --arena ice --length sprint --fallback-policy strict --out /tmp/m.json
./tools/run_match.py verify /tmp/m.json        # replay integrity audit
python3 stickblade/benchmark.py --fingerprint  # 09de66effd02 under prompt v2

Run it twice; the frames and action log are identical.

2. Re-run the calibration design
# offline dry run (scripted fighters, ~10 s) — the committed one is research/calibration/dry-run/
python3 tools/run_calibration_batch.py run --local \
    --models bot:pro,mock:duelist,bot:greedy,bot:distance --n 16 --length sprint \
    --out-dir /tmp/cal --run-id cal-dryrun-2026-09-09
# expect audit: 5 ✅, 1 ❌ (real_provider_evidence_present) → NOT ACCEPTED

# same plan through a running backend lands identical rows (verified 24/24)
python3 tools/run_calibration_batch.py run --backend http://localhost:8000 \
    --plan /tmp/cal/plan.json --out-dir /tmp/cal_backend
3. Rebuild and verify a dataset release
python3 tools/export_dataset.py build --backend https://pioneer37-stickman-arena.hf.space \
    --out-dir research/exports --version v2026.09.09 --replays 500
python3 tools/export_dataset.py verify research/exports/v2026.09.09
# expect: hashes OK · row counts match · ratings refit point estimates match · RESULT: VERIFIED
4. Regenerate the report
python3 tools/benchmark_report.py --export research/exports/v2026.09.09/matches/matches.jsonl \
    --out /tmp/report.md
# §0 of the report names dataset version, benchmark/prompt versions, fingerprint and evidence level
5. Recompute the leaderboard from the API
curl -s "$API/api/leaderboard/bradley_terry?weapon=sword&mode=macro&bootstraps=500" | jq '.rows[] | {model, rating, ci_low, ci_high, matches, data_quality}'
curl -s "$API/api/data_quality" | jq .summary

Every row carries matches, ci_low/ci_high and a data_quality block. No rank is shown without them.

Where the constants live