Results
Each row is one search configuration, graded on the same 224 questions. A hit means the right rule, in the version in force on the question's date, is in the top 5. Brackets are 95% Wilson intervals; comparisons are exact McNemar tests on the same questions. Every row had a written prediction before it ran.
| Configuration | Expert rulings (160) | Card rulings (113) | Version changes (56) | Against the row it was tested on | Kept? |
|---|---|---|---|---|---|
| Naive vector search, nomic | 44% [36 to 51%] | 35% [26 to 44%] | 39% [27 to 52%] | baseline | baseline |
| + date filter | 58% [50 to 65%] | 52% [43 to 61%] | 59% [46 to 71%] | version changes 11 gained / 0 lost, p = 0.0010 | kept |
| + Qwen3 embedder | 63% [55 to 70%] | 55% [46 to 64%] | 80% [68 to 89%] | version changes 12 gained / 1 lost, p = 0.0034 | current best |
| Keyword search only | 53% [45 to 61%] | 49% [40 to 58%] | 35% [24 to 49%] | version changes 0 gained / 24 lost, p < 0.001 | reference |
| Hybrid (vector + keyword, RRF) | 66% [59 to 73%] | 60% [51 to 69%] | 65% [51 to 76%] | version changes 0 gained / 8 lost, p = 0.0078 | dropped |
| Reranker (bge-reranker-base) | 66% [58 to 73%] | 58% [49 to 67%] | 89% [78 to 95%] | version changes 6 gained / 1 lost, p = 0.13, 10.5 s per question | dropped |
BaselineKeptCurrent bestNo edge: tested and dropped
Why the current best misses
Every question the current best misses at top 5, in exactly one category, by a fixed rule with no model. “Names a card” means the source files the ruling under cards; it marks the questions whose card text the corpus lacks, and does not prove that caused the miss. Version-change questions have no source category, so the ones not found stay unclassified rather than guessed.
| Questions | Found, ranked 6 to 20 | Not found, names a card | Not found, rules only | Unclassified | Misses |
|---|---|---|---|---|---|
| Expert rulings | 37 | 21 | 2 | 0 | 60 |
| FAQ rulings | 3 | 2 | 0 | 0 | 5 |
| Version changes | 7 | 0 | 0 | 4 | 11 |
Predictions against results
Written before each run. 21 right, 3 close, 26 wrong.
| Slice | Prediction | Result | Verdict |
|---|---|---|---|
| Naive baseline (5) | Expert rulings recall@5 about 25% | 44% | wrong |
| Naive baseline (5) | Version changes recall@5 about 55% | 39% | wrong |
| Naive baseline (5) | Outdated copy in top 5 on version changes, at least 30% | 45% | right |
| Naive baseline (5) | Version-blind recall at least 5 points above recall@5 | 13 points | right |
| Date filter (6) | No question loses its hit | 0 lost | right |
| Date filter (6) | Wrong-version copies in top k drop to 0 | 0 | right |
| Date filter (6) | Version changes about 52% | 59% | right |
| Date filter (6) | Expert rulings about 47% | 58% | wrong |
| Qwen3 embedder (7) | Expert rulings about 62% | 62% | right |
| Qwen3 embedder (7) | Card questions gain the most | +3 points, the least | wrong |
| Qwen3 embedder (7) | Version changes flat, about 60% | 79% | wrong |
| Qwen3 embedder (7) | No group significant | version changes p = 0.0034 | wrong |
| Hybrid search (8) | Expert rulings about 65% | 66% | right |
| Hybrid search (8) | Version changes about 78%, no gain | 66%, 8 lost | wrong |
| Hybrid search (8) | Keyword-only card questions about 30% | 49% | wrong |
| Reranker (9) | Version changes about 82% | 88% | right |
| Reranker (9) | Significant gain on expert rulings | p = 0.54 | wrong |
| Reranker (9) | About 2 s per question on CPU | 10.5 s | wrong |
| Failure taxonomy (12) | About 80% of misses outside the top 20 name a card | 92% (23 of 25) | close |
| Failure taxonomy (12) | About 5 rules-only misses outside the top 20 | 2 | close |
| Failure taxonomy (12) | Exactly 4 unclassified | 4 | right |
| Failure taxonomy (12) | Card share higher outside the top 20 than below the cutoff | 92% against 80% | right |
| MCP server (15) | At least 222 of 224 searches identical to the snapshot | 224 of 224 | right |
| MCP server (15) | Recall@5 through the server equal to the snapshot | identical | right |
| MCP server (15) | Warm search under 0.3 s over stdio | 0.036 s | right |
| MCP server (15) | About 9 s to parse a rules PDF on first use | 2.4 to 3.1 s | wrong |
| New rules version (18) | About 80 rules changed from v1.4 to v1.5 | 81 | right |
| New rules version (18) | About 60 rules added (25 to 150) | 159 | wrong |
| New rules version (18) | About 15 rules removed (0 to 40) | 44 | wrong |
| New rules version (18) | About 10% of ruling questions cite a rule v1.5 changed | 49% | wrong |
| Answers + judge (11) | Rulings about 55% correct given a hit | 54% | right |
| Answers + judge (11) | Version changes about 70% correct given a hit | 64% | right |
| Answers + judge (11) | Judge agreement with hand labels: kappa about 0.5 | 0.77 | wrong |
| Answers + judge (11) | Judge 10+ points more lenient than hand labels | 2 points | wrong |
| Answers + judge (11) | About 4 s per answer on GPU | 1.9 s | wrong |
| Answers + judge (11) | Under 5% of answers truncated | 4.9% | right |
| Confidence (13) | Calibrated, narrowly (ECE about 0.08) | ECE 0.068 | right |
| Confidence (13) | Brier skill about 0.05 | +0.23 | wrong |
| Confidence (13) | Top fifth about 20 points above bottom | +64 points | wrong |
| Prompt injection (16) | Production (framed) prompt about 20% injected | 36% | wrong |
| Prompt injection (16) | Framed significantly beats naive | 69% to 36%, p = 2e-10 | right |
| Prompt injection (16) | Datamarking about 8%, better than framed | 40%, worse | wrong |
| Prompt injection (16) | Fake end-of-block strongest, about 35% on framed | 100% | wrong |
| Agent (14) | Router about 85% accurate | 80% | close |
| Agent (14) | Re-search fires on about 30% | 47% | wrong |
| Agent (14) | Loop adds about 3 points, not significant | +5 points, p = 0.08 | right |
| Agent (14) | About 35% more generation time | +123% | wrong |
| Prompt injection (16) | Escaping the delimiter stops the fake system message (about 15%) | 95% | wrong |
| New rules version (18) | 224 of 224 searches unchanged after adding v1.5 | 224 of 224 | right |
| Determinism | CPU generation fully repeatable | 2 of 10; cause is Ollama's prompt cache | wrong |
Generated at build time from site/data/results.json, which make site-data computes from the committed runs in evals/runs/.