msmarco-genqa

System

BM25 → dense (MiniLM L6) → cross-encoder rerank → T5-small generation

Dataset

MS MARCO passage, 8.8 M docs, 6 980 paired queries (dev/small)

Main result

Reranking lifts Token-F1 from 0.197 to 0.368 (Δ +0.171, 95 % CI [+0.163, +0.178], paired bootstrap)

External retrieval

TREC-DL 2019/2020 full-corpus BM25 + cross-encoder benchmark, independently cross-checked

Reproducibility

Schema-v2 manifest contract; make reproduce-baseline from clean clone

Status

Active — CPU-only, single-machine, CI-backed test suite

Problem

Surface generation metrics (Token-F1, ROUGE-L) on retrieve → rerank → generate pipelines do not, by themselves, tell you whether the generator’s output is actually supported by the retrieved evidence. Reranking the retrieval stage can simultaneously raise surface metrics and reduce grounding on the same fixed generator. Quantifying this requires a pipeline where every component is exchangeable, every run is reproducible to a bit-level manifest, and statistical comparisons are paired and confidence-bounded.

What I built

Four entry-point scripts under experiments/ — one per pipeline stage — sharing a single source-of-truth config and writing a schema-versioned manifest.json next to every metrics.json. The contract is enforced at write time: a run that fails to capture the six required reproducibility fields cannot land an artifact.

Headline result

Generation × retrieval source, full MS MARCO dev/small, 6 980 paired queries, paired bootstrap (N = 10 000).

Retrieval-only, BM25 on the full 8.8 M-passage corpus (6 980 queries):

Evidence status

Validated result

On MS MARCO dev/small, reranked dense top-3 improves T5-small surface metrics over BM25 top-3 on 6 980 paired queries. The dense and reranking stages use a documented 50 000-passage qrels-anchored pool, not a full-corpus dense first stage.

Validated external retrieval

On all 43 TREC-DL 2019 and 54 TREC-DL 2020 judged topics, cross-encoder reranking improves MRR@10 and graded nDCG@10 over full-corpus BM25. The protocol and checked artifact are in the repository's TREC-DL report.

Implemented, pending evaluation

The T5-base generator-capacity sweep and configurable alternative-generator paths exist, but no T5-base or FLAN-T5 headline result is claimed without a complete versioned run.

Not established

The current artifacts do not show that TREC-DL retrieval gains transfer to generation, that dense retrieval beats BM25 under a fair full-corpus condition, or that the findings generalize beyond MS MARCO passages.

Technical components

Sparse retrieval

BM25, k₁ = 1.5, b = 0.75, top-k = 1000

Dense retrieval

sentence-transformers/all-MiniLM-L6-v2 + FAISS flat IP, qrels-anchored 50 000-passage sample

Reranking

cross-encoder/ms-marco-MiniLM-L-6-v2 over dense top-100

Generation

t5-small, frozen, max_new_tokens = 64

Retrieval metrics

MRR@k, Recall@k, nDCG@k

Generation metrics

Token-F1, Exact-Match, ROUGE-L, BLEU (best-of-N references)

Grounding

Lexical content-token, 3-gram, NLI entailment via cross-encoder/nli-deberta-v3-small

Statistical core

Paired bootstrap, N = 10 000 resamples, seed 42

Manifest contract

Required at write time: git.commit, git.dirty, extra.seed, extra.resolved_config_hash, extra.data_fingerprint, extra.env_fingerprint

CI

GitHub Actions, Python 3.10, CPU-only torch — pytest -q + ruff check on every push

Reproduce

Clean clone, CPU-only laptop:

make install
make reproduce-baseline

make reproduce-baseline runs the BM25 stage under a clean-tree checkpoint and then validates the produced manifest against committed fingerprints via scripts/verify_reproduction.py.

Current status

Active. The paired MS MARCO generation comparison and the TREC-DL 2019/2020 BM25-plus-reranker benchmark are completed, versioned evidence. The T5-base capacity study and broader generator comparisons remain implemented or planned work; they are not reported as empirical findings until their artifacts land.

Repo

github.com/GioiaZheng/msmarco-genqa