vaxxbt
$vaxxbt scoreboardlive
NetMHCpan baseline
25.7%
recall@34 · micro · 96 / 373 epitopes caught
14 / 14 cohorts complete
vaxxbt
—
not yet run  0 / 14 cohorts scored
results post to @vaxxbt as they land

vaxxbt has not run on this dataset yet

An earlier run scored 9.5% against the baseline's 16.7% on one cohort — a clear loss. That result has been withdrawn, because a code review found the dataset it was scored on had mislabeled ground truth.

It is not deleted, it is void. Scoring against a broken answer key tells you nothing in either direction. The run happens again on the corrected data, and whatever it says goes up here.

Correction log — 19 Aug 2026

An independent review found six integrity bugs before launch. All are fixed, each with a regression test. The three that touched the numbers:

  • Correct answers were scored as wrong. IEDB grades responses Positive-High / Intermediate / Low; only the plain Positive label was being read. 24 validated epitopes were missing, and ~21 of them sat in the decoy pool labeled non-immunogenic.
  • The scorer could be cheated in one line. Submitting the same correct peptide 34 times scored 3400% recall. Duplicates are now dropped before the slot cap, and each one costs a slot.
  • The head-to-head could be decided by network luck. Each selector fetched its own binding scores; an IEDB hiccup during one run would hand the other better data. Scores are now computed once, frozen into the dataset, and covered by the dataset hash.

Baseline moved from 27.2% to 25.7% on the corrected data. Published because a benchmark that quietly revises its own numbers is worth nothing.

$vaxxbt build-datasetcomplete

Ground truth is pulled live from IEDB — free, public, no key. Melanoma-associated human MHC-I T-cell assays.

raw positives
2,348 plain Positive + 144 graded Positive-High/Int/Low
raw negatives
12,395 peptides assayed and found inert
after cleaning
373 positives · 2,222 decoys · 14 cohorts
dropped
151 peptides reported both positive and negative across studies
binding coverage
98.8% — scored once, frozen; both selectors read these exact values
dataset digest
74dad43c18698c34c7ec96f39debd600

The decoys are the point. They are peptides a researcher thought worth testing that came back inert — hard negatives. A benchmark built on random peptides is trivially winnable and proves nothing.

$vaxxbt run --selector baseline14 cohorts

Rank every candidate by NetMHCpan predicted binding, take the top 34 — the same slot count mRNA-4157 uses per patient. A strong opponent, not a strawman.

HLA allelepooltrue+caught recall@34vs random 
micro recall
25.7%  — all positives pooled; big cohorts dominate
macro recall
35.7%  — per-cohort mean; every cohort weighs equally
median
28.3%
random picking
27.4%

Read this before celebrating the baseline

On micro recall the predictor scores 25.7% against random's 27.4% — it is at or slightly below chance. The macro figure looks far better only because small cohorts, where 34 slots cover most of the pool, are easy. Both numbers are shown because reporting only the flattering one is how benchmarks lie.

$cat SOURCES.mdcitations

Moderna's published selection recall: of the neoantigens patient T-cells actually recognized, their algorithm included 41%. In 39% of patients, not one confirmed-reactive target made the cut.

Source: AACR 2020 Abstract 6539, Cancer Research 80(16_Suppl):6539 — doi:10.1158/1538-7445.AM2020-6539

41% is a reference line, not our opponent

  • It is colorectal, not melanoma. The Phase 3 that just read out is melanoma.
  • It is a conference abstract, not a peer-reviewed paper.
  • It is not this benchmark's task — Moderna evaluated on patient exomes; we evaluate on curated IEDB assays. Different problems, different difficulty.

The only comparison this project calls a result is vaxxbt vs. NetMHCpan on identical cohorts — same candidates, same 34 slots, same ground truth. Anything implying we are racing Moderna head-to-head is a lie this project does not support.

$vaxxbt verify --run-id <run>reproducible

There is no blockchain here and no trusted third party. Every run produces a digest that pins the exact candidates, the ground truth, the binding scores, the picks and their order, and the selector code that produced them.

dataset digest
74dad43c18698c34c7ec96f39debd600
baseline run
ee2a083128cae92d96771ab5…
ground truth
IEDB — public, free, no key required
predictor
NetMHCpan-4.1 via IEDB's hosted tools

You don't have to trust us. Every input is public. Rebuild the dataset, re-run the selector, and check you land on the same digest. If you don't, we're wrong — and you can show it.

That is a stronger claim than a timestamp. A timestamp says we didn't edit this afterwards. Reproducibility says you can check the whole thing yourself, which is the only guarantee that survives us disappearing.

$cat EXPERIMENT.md | grep -A4 falsificationpre-registered

Falsification condition

If vaxxbt's recall@34 does not beat the baseline across ≥30 cohorts with a sign test p < 0.05, vaxxbt adds nothing — and we publish that.

A directional edge that fails the sign test is reported as "no significant difference", never as a win. Melanoma alone yields 14 cohorts, so reaching 30 means widening the disease set — never loosening thresholds, which would make the benchmark easier rather than larger.

Tuning the prompt until vaxxbt wins and publishing only that run is the specific failure the pre-registration exists to prevent. Prompt revisions are new registered runs, not replacements for old ones.

Every result — including the ones where it loses — posts to @vaxxbt.

$_try it
vaxxbt 0.1.0 — type `help` for commands.
$