Pyserini Reproductions: MS MARCO V2 Passage

The two-click^* reproduction matrix below provides commands for reproducing experimental results reported in a number of papers, denoted by the references in square brackets. These runs take advantage of prebuilt indexes in Pyserini, so you don't need access to the raw corpus. Instructions for programmatic execution are shown at the bottom of this page.

		TREC 2021			TREC 2022			TREC 2023			dev		dev2
		AP	nDCG@10	R@1K	AP	nDCG@10	R@1K	AP	nDCG@10	R@1K	RR@100	R@1K	RR@100	R@1K
[1] (1a)	BM25 original passage	0.1357	0.4458	0.6149	0.0325	0.2692	0.3321	0.0793	0.2627	0.4346	0.0719	0.5733	0.0802	0.5839
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-default.dl21.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-default.dl22.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-default.dl23.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-default.dev.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-default.dev2.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-default.dev2.txt`
[1] (1b)	BM25 augmented passage	0.0977	0.3977	0.5835	0.0346	0.2742	0.3666	0.0751	0.2061	0.4514	0.0872	0.6925	0.0917	0.6933
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-augmented-default.dl21.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-augmented-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-augmented-default.dl22.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-augmented-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-augmented-default.dl23.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-augmented-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-augmented-default.dev.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-augmented-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-augmented-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-augmented-default.dev2.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-augmented-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-augmented-default.dev2.txt`
[1] (1c)	BM25+RM3 original passage	0.1666	0.4455	0.6616	0.0310	0.2686	0.3559	0.0806	0.2602	0.4748	0.0630	0.5947	0.0659	0.6062
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-rm3-default.dl21.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-rm3-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-rm3-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-rm3-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-rm3-default.dl22.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-rm3-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-rm3-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-rm3-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-rm3-default.dl23.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-rm3-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-rm3-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-rm3-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-rm3-default.dev.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-rm3-default.dev2.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-default.dev2.txt`
[1] (1d)	BM25+RM3 augmented passage	0.1050	0.3869	0.6298	0.0318	0.2571	0.3441	0.0646	0.1798	0.4583	0.0667	0.6857	0.0700	0.6826
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-rm3-augmented-default.dl21.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-rm3-augmented-default.dl22.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-rm3-augmented-default.dl23.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-rm3-augmented-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-rm3-augmented-default.dev.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-augmented-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-augmented-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-rm3-augmented-default.dev2.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-augmented-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-augmented-default.dev2.txt`

[1] (2a)	BM25 w/ doc2query-T5 original passage	0.1874	0.4816	0.7078	0.0748	0.3599	0.4632	0.1085	0.3156	0.5608	0.1072	0.7083	0.1123	0.7151
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5 \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-d2q-t5-default.dl21.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5 \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-d2q-t5-default.dl22.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5 \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-d2q-t5-default.dl23.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-d2q-t5-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5 \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-d2q-t5-default.dev.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-d2q-t5-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-d2q-t5-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5 \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-d2q-t5-default.dev2.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-d2q-t5-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-d2q-t5-default.dev2.txt`
[1] (2b)	BM25 w/ doc2query-T5 augmented passage	0.1649	0.4702	0.6962	0.0735	0.3609	0.4739	0.1078	0.2926	0.5689	0.1172	0.7647	0.1170	0.7659
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5 \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl21.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5 \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl22.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5 \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl23.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5 \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dev.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5 \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dev2.txt \ --bm25` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-d2q-t5-augmented-default.dev2.txt`
[1] (2c)	BM25+RM3 w/ doc2query-T5 original passage	0.2285	0.5098	0.7537	0.0872	0.3721	0.4964	0.1135	0.3132	0.5978	0.0947	0.7181	0.0984	0.7222
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5-docvectors \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl21.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5-docvectors \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl22.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5-docvectors \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl23.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5-docvectors \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dev.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.d2q-t5-docvectors \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dev2.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-d2q-t5-default.dev2.txt`
[1] (2d)	BM25+RM3 w/ doc2query-T5 augmented passage	0.1930	0.4812	0.7672	0.0821	0.3749	0.4914	0.1059	0.2719	0.5623	0.0883	0.7607	0.0904	0.7649
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5-docvectors \ --topics dl21 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl21.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5-docvectors \ --topics dl22 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl22.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5-docvectors \ --topics dl23 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl23.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5-docvectors \ --topics msmarco-v2-passage-dev \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dev.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage-augmented.d2q-t5-docvectors \ --topics msmarco-v2-passage-dev2 \ --output run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dev2.txt \ --bm25 --rm3` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.bm25-rm3-d2q-t5-augmented-default.dev2.txt`

[1] (3a)	uniCOIL noexp: PyTorch	0.2193	0.5756	0.6897	0.0754	0.4077	0.4423	0.1112	0.3262	0.5070	0.1342	0.7010	0.1385	0.7114
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-noexp-0shot \ --topics dl21 \ --encoder castorini/unicoil-noexp-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-noexp-pytorch.dl21.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-noexp-0shot \ --topics dl22 \ --encoder castorini/unicoil-noexp-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-noexp-pytorch.dl22.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-noexp-0shot \ --topics dl23 \ --encoder castorini/unicoil-noexp-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-noexp-pytorch.dl23.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.unicoil-noexp-pytorch.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-noexp-0shot \ --topics msmarco-v2-passage-dev \ --encoder castorini/unicoil-noexp-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-noexp-pytorch.dev.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.unicoil-noexp-pytorch.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.unicoil-noexp-pytorch.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-noexp-0shot \ --topics msmarco-v2-passage-dev2 \ --encoder castorini/unicoil-noexp-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-noexp-pytorch.dev2.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.unicoil-noexp-pytorch.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.unicoil-noexp-pytorch.dev2.txt`
[1] (3b)	uniCOIL w/ doc2query-T5: PyTorch	0.2538	0.6159	0.7551	0.1050	0.4614	0.5253	0.1437	0.3855	0.5541	0.1499	0.7616	0.1577	0.7671
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-0shot \ --topics dl21 \ --encoder castorini/unicoil-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-pytorch.dl21.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.unicoil-pytorch.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.unicoil-pytorch.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.unicoil-pytorch.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-0shot \ --topics dl22 \ --encoder castorini/unicoil-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-pytorch.dl22.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.unicoil-pytorch.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.unicoil-pytorch.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.unicoil-pytorch.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-0shot \ --topics dl23 \ --encoder castorini/unicoil-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-pytorch.dl23.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.unicoil-pytorch.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.unicoil-pytorch.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.unicoil-pytorch.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-0shot \ --topics msmarco-v2-passage-dev \ --encoder castorini/unicoil-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-pytorch.dev.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.unicoil-pytorch.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.unicoil-pytorch.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.unicoil-0shot \ --topics msmarco-v2-passage-dev2 \ --encoder castorini/unicoil-msmarco-passage \ --output run.msmarco-v2-passage.unicoil-pytorch.dev2.txt \ --hits 1000 --impact` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.unicoil-pytorch.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.unicoil-pytorch.dev2.txt`

[2]	SLIM++ (`norefine`, `tau`=0.5, `min_idf`=1)	0.2820	0.6337	0.8389	0.1616	0.5017	0.6159	0.1772	0.4746	0.6412	0.1915	0.8710	0.1901	0.8681
TREC 2021 TREC 2022 TREC 2023 dev dev2 Command to generate run on TREC 2021 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.slimr-pp \ --topics dl21 \ --encoder castorini/slimr-pp-msmarco-passage \ --output run.msmarco-v2-passage.slimr-pp.dl21.txt \ --hits 1000 --impact --min-idf 1` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl21-passage run.msmarco-v2-passage.slimr-pp.dl21.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl21-passage run.msmarco-v2-passage.slimr-pp.dl21.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl21-passage run.msmarco-v2-passage.slimr-pp.dl21.txt` Command to generate run on TREC 2022 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.slimr-pp \ --topics dl22 \ --encoder castorini/slimr-pp-msmarco-passage \ --output run.msmarco-v2-passage.slimr-pp.dl22.txt \ --hits 1000 --impact --min-idf 1` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl22-passage run.msmarco-v2-passage.slimr-pp.dl22.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl22-passage run.msmarco-v2-passage.slimr-pp.dl22.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl22-passage run.msmarco-v2-passage.slimr-pp.dl22.txt` Command to generate run on TREC 2023 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.slimr-pp \ --topics dl23 \ --encoder castorini/slimr-pp-msmarco-passage \ --output run.msmarco-v2-passage.slimr-pp.dl23.txt \ --hits 1000 --impact --min-idf 1` Evaluation commands: `python -m pyserini.eval.trec_eval -c -l 2 -M 100 -m map dl23-passage run.msmarco-v2-passage.slimr-pp.dl23.txt python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl23-passage run.msmarco-v2-passage.slimr-pp.dl23.txt python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl23-passage run.msmarco-v2-passage.slimr-pp.dl23.txt` Command to generate run on dev queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.slimr-pp \ --topics msmarco-v2-passage-dev \ --encoder castorini/slimr-pp-msmarco-passage \ --output run.msmarco-v2-passage.slimr-pp.dev.txt \ --hits 1000 --impact --min-idf 1` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev run.msmarco-v2-passage.slimr-pp.dev.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev run.msmarco-v2-passage.slimr-pp.dev.txt` Command to generate run on dev2 queries: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v2-passage.slimr-pp \ --topics msmarco-v2-passage-dev2 \ --encoder castorini/slimr-pp-msmarco-passage \ --output run.msmarco-v2-passage.slimr-pp.dev2.txt \ --hits 1000 --impact --min-idf 1` Evaluation commands: `python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-v2-passage-dev2 run.msmarco-v2-passage.slimr-pp.dev2.txt python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-v2-passage-dev2 run.msmarco-v2-passage.slimr-pp.dev2.txt`

References

[1] Xueguang Ma, Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. Document Expansions and Learned Sparse Lexical Representations for MS MARCO V1 and V2. Proceedings of the 45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022), July 2022. (Numbers in parentheses correspond to rows in Table 1 of the paper.)
[2] Minghan Li, Sheng-Chieh Lin, Xueguang Ma, and Jimmy Lin. SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes. Proceedings of the 46th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023), pages 1954–1959, July 2023, Taipei, Taiwan.

Programmatic Execution

All experimental runs shown in the above table can be programmatically executed based on the instructions below. To list all the experimental conditions:

python -m pyserini.2cr.msmarco --collection v2-passage --list-conditions

These conditions correspond to the table rows above.

For all conditions, just show the commands in a "dry run":

python -m pyserini.2cr.msmarco --collection v2-passage --all --display-commands --dry-run

To actually run all the experimental conditions:

python -m pyserini.2cr.msmarco --collection v2-passage --all --display-commands

With the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

To show the commands for a specific condition:

python -m pyserini.2cr.msmarco --collection v2-passage --condition bm25-default --display-commands --dry-run

This will generate exactly the commands for a specific condition above (corresponding to a row in the table).

To actually run a specific condition:

python -m pyserini.2cr.msmarco --collection v2-passage --condition bm25-default --display-commands

Again, with the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

Finally, to generate this page:

python -m pyserini.2cr.msmarco --collection v2-passage --generate-report --output msmarco-v2-passage.html

The output file msmarco-v2-passage.html should be identical to this page.

MS MARCO V2 Passage Reproductions

References

Programmatic Execution