Benchmark methodology
How the head-to-head against NetMHCpan-4.2c was constructed, and how we decontaminated the test data.
Primary benchmark: multi-study mono-allelic HLA-I EL
Union of three post-cutoff, publicly-available PRIDE proteomics studies. 601,565 unique 8–14mer positives across 13 HLA-A and HLA-B alleles. Scored subset: 496,240 candidates (cap 500 positives per sample; 19:1 source-protein-matched decoys).
| PRIDE accession | cell line(s) | alleles | positives |
|---|---|---|---|
| PXD082505 | 721.221 mono-allelic (Kim et al., FDA CBERS) | 3 HLA-B | 2,051 |
| PXD070374 | C1R-B*57:01 (ExCID FragPipe) | 1 HLA-B | 9,118 |
| PXD073207 | 10 mono-allelic FDA01 lines (timsTOF-SCP DDA-PASEF, Nilsson lab) | 10 HLA-A/B | 590,396 |
Benchmark protocol
1. Temporal gate
Every benchmark row has first-public date ≥ 2025-08-08 — the day after NetMHCpan-4.2's release. This guarantees that both models' training data was frozen strictly before the test observations.
2. Overlap audit vs the reference model's public training data
Every benchmark's positives are audited against a 17.6 M-row union catalogue assembled from the reference predictor's public training partitions, keyed on the exact (peptide, HLA) pair and any contiguous 8-mer sub-string overlap (via an MD5-hashed 8-mer index of 12 M unique hashes). Two locked analysis sets result: exact-disjoint and strict-sequence-disjoint.
3. Multi-study mono-allelic EL (primary result)
Three post-cutoff PRIDE studies (see table above). 601,565 unique positives across 13 alleles; 86.06 % exact-disjoint from the training corpus (517,679 novel peptide–HLA pairs).
4. Source-protein-matched decoys
For every positive, decoys are drawn from k-mers of the same source proteins observed in that sample's positive set. This removes the "protein-was-expressed" trivial signal that length-matched random-proteome decoys leave in — a stricter test than most published benchmarks.
5. Reference predictor
Version-pinned NetMHCpan-4.2c executable, unmodified from the DTU
distribution, invoked with -BA for IC50 output. HLA-B*07:01 is
unsupported by NetMHCpan-4.2 and is excluded from paired comparisons.
6. Statistics
Bootstrap CIs: 1,000 iterations at seed 42, study-clustered where possible. Paired-delta CIs on the intersection of covered rows.
Decontamination audit — all 4 benchmarks
Percent of positives that are novel relative to the reference model's public training data.
| Benchmark | n | exact pep-HLA leak | exact-disjoint | strict-seq-disjoint |
|---|---|---|---|---|
| BA temporal (IEDB post-2025-08-08) | 592 | 0.00% | 100.00% | 98.82% |
| IEDB auto_bench pooled (post-2025-08-08) | 594 | 2.53% | 97.47% | 42.42% |
| Mono-allelic EL v1 (superseded) | 2,051 | 73.67% | 26.33% | 0.49% |
| Mono-allelic EL v2 (primary) | 601,565 | 13.94% | 86.06% | 8.01% |
Head-to-head result — mono-allelic EL v2
| Predictor | Pooled AUROC | 95% CI | Per-allele macro AUROC | Alleles won |
|---|---|---|---|---|
| DeepNeo-CL v4.2 (EL) | 0.8100 | [0.8068, 0.8132] | 0.8436 | 12 / 12 |
| NetMHCpan-4.2c | 0.7762 | [0.7728, 0.7801] | 0.8157 | — |
Paired Δ AUROC (BA head, common-support subset) = +0.0113 [+0.0094, +0.0134], bootstrap p ≈ 0. Full per-allele + interactive Plotly charts on the Benchmarks page.
Live service
/api/health— service liveness/api/alleles— supported HLA-I allele list/api/docs— interactive OpenAPI docs for the predict endpoint