Pyserini: MS MARCO V1 Doc

The two-click* reproduction matrix below provides commands for reproducing experimental results reported in a number of papers, denoted by the references in square brackets. These runs take advantage of prebuilt indexes in Pyserini, so you don't need access to the raw corpus. Instructions for programmatic execution are shown at the bottom of this page.

The experimental conditions and reference scores are defined in the YAML configuration. Not all experimental conditions are shown in the table below.

TREC 2019TREC 2020MS MARCO dev
AP@100nDCG@10R@1KAP@100nDCG@10R@1KRR@100R@1K

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-doc-default.dl19-doc.txt \
  --bm25 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-doc-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-doc-default.dl20.txt \
  --bm25 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-doc-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-doc-default.msmarco-doc-dev.txt \
  --bm25 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-doc-segmented-default.dl19-doc.txt \
  --bm25 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-doc-segmented-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-doc-segmented-default.dl20.txt \
  --bm25 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-doc-segmented-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-doc-segmented-default.msmarco-doc-dev.txt \
  --bm25 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-segmented-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-segmented-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-doc-default.dl19-doc.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-doc-default.dl20.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-doc-default.msmarco-doc-dev.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl19-doc.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl20.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.msmarco-doc-dev.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-segmented-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-default.dl19-doc.txt \
  --bm25 --rocchio --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-default.dl20.txt \
  --bm25 --rocchio --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-default.msmarco-doc-dev.txt \
  --bm25 --rocchio --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl19-doc.txt \
  --bm25 --rocchio --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl20.txt \
  --bm25 --rocchio --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.msmarco-doc-dev.txt \
  --bm25 --rocchio --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-segmented-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-doc-tuned.dl19-doc.txt \
  --bm25

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-doc-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-doc-tuned.dl20.txt \
  --bm25

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-doc-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-doc-tuned.msmarco-doc-dev.txt \
  --bm25

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl19-doc.txt \
  --bm25 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl20.txt \
  --bm25 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-doc-segmented-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-doc-segmented-tuned.msmarco-doc-dev.txt \
  --bm25 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-segmented-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-doc-segmented-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl19-doc.txt \
  --bm25 --rm3

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl20.txt \
  --bm25 --rm3

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-doc-tuned.msmarco-doc-dev.txt \
  --bm25 --rm3

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl19-doc.txt \
  --bm25 --rm3 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl20.txt \
  --bm25 --rm3 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.msmarco-doc-dev.txt \
  --bm25 --rm3 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-doc-segmented-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl19-doc.txt \
  --bm25 --rocchio

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl20.txt \
  --bm25 --rocchio

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-tuned.msmarco-doc-dev.txt \
  --bm25 --rocchio

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl19-doc.txt \
  --bm25 --rocchio --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl20.txt \
  --bm25 --rocchio --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.msmarco-doc-dev.txt \
  --bm25 --rocchio --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rocchio-doc-segmented-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5 \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl19-doc.txt \
  --bm25 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5 \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl20.txt \
  --bm25 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5 \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-default.msmarco-doc-dev.txt \
  --bm25 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5 \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl19-doc.txt \
  --bm25 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5 \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl20.txt \
  --bm25 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5 \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.msmarco-doc-dev.txt \
  --bm25 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5-docvectors \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl19-doc.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5-docvectors \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl20.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5-docvectors \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.msmarco-doc-dev.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5-docvectors \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl19-doc.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5-docvectors \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl20.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5-docvectors \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.msmarco-doc-dev.txt \
  --bm25 --rm3 --k1 0.9 --b 0.4 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-default.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5 \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl19-doc.txt \
  --bm25

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5 \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl20.txt \
  --bm25

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5 \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.msmarco-doc-dev.txt \
  --bm25

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5 \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl19-doc.txt \
  --bm25 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5 \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl20.txt \
  --bm25 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5 \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.msmarco-doc-dev.txt \
  --bm25 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-d2q-t5-doc-segmented-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5-docvectors \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl19-doc.txt \
  --bm25 --rm3

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5-docvectors \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl20.txt \
  --bm25 --rm3

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc.d2q-t5-docvectors \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.msmarco-doc-dev.txt \
  --bm25 --rm3

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5-docvectors \
  --topics dl19-doc \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl19-doc.txt \
  --bm25 --rm3 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5-docvectors \
  --topics dl20 \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl20.txt \
  --bm25 --rm3 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.d2q-t5-docvectors \
  --topics msmarco-doc-dev \
  --output run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.msmarco-doc-dev.txt \
  --bm25 --rm3 --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.bm25-rm3-d2q-t5-doc-segmented-tuned.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.unicoil-noexp \
  --topics dl19-doc \
  --encoder castorini/unicoil-noexp-msmarco-passage \
  --output run.msmarco-v1-doc.unicoil-noexp-pytorch.dl19-doc.txt \
  --impact --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.unicoil-noexp-pytorch.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.unicoil-noexp-pytorch.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.unicoil-noexp-pytorch.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.unicoil-noexp \
  --topics dl20 \
  --encoder castorini/unicoil-noexp-msmarco-passage \
  --output run.msmarco-v1-doc.unicoil-noexp-pytorch.dl20.txt \
  --impact --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.unicoil-noexp-pytorch.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.unicoil-noexp-pytorch.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.unicoil-noexp-pytorch.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.unicoil-noexp \
  --topics msmarco-doc-dev \
  --encoder castorini/unicoil-noexp-msmarco-passage \
  --output run.msmarco-v1-doc.unicoil-noexp-pytorch.msmarco-doc-dev.txt \
  --impact --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.unicoil-noexp-pytorch.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.unicoil-noexp-pytorch.msmarco-doc-dev.txt

topics: dl19-doc / qrels: dl19-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.unicoil \
  --topics dl19-doc \
  --encoder castorini/unicoil-msmarco-passage \
  --output run.msmarco-v1-doc.unicoil-pytorch.dl19-doc.txt \
  --impact --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl19-doc run.msmarco-v1-doc.unicoil-pytorch.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-doc run.msmarco-v1-doc.unicoil-pytorch.dl19-doc.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl19-doc run.msmarco-v1-doc.unicoil-pytorch.dl19-doc.txt

topics: dl20 / qrels: dl20-doc

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.unicoil \
  --topics dl20 \
  --encoder castorini/unicoil-msmarco-passage \
  --output run.msmarco-v1-doc.unicoil-pytorch.dl20.txt \
  --impact --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m map dl20-doc run.msmarco-v1-doc.unicoil-pytorch.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-doc run.msmarco-v1-doc.unicoil-pytorch.dl20.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 dl20-doc run.msmarco-v1-doc.unicoil-pytorch.dl20.txt

topics: msmarco-doc-dev / qrels: msmarco-doc-dev

Retrieval command

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index msmarco-v1-doc-segmented.unicoil \
  --topics msmarco-doc-dev \
  --encoder castorini/unicoil-msmarco-passage \
  --output run.msmarco-v1-doc.unicoil-pytorch.msmarco-doc-dev.txt \
  --impact --hits 10000 --max-passage-hits 1000 --max-passage

Evaluation commands

python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank msmarco-doc-dev run.msmarco-v1-doc.unicoil-pytorch.msmarco-doc-dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-doc-dev run.msmarco-v1-doc.unicoil-pytorch.msmarco-doc-dev.txt

References

Programmatic Execution

All experimental runs shown in the above table can be programmatically executed based on the instructions below. To list all the experimental conditions:

python -m pyserini.2cr.msmarco --collection v1-doc --list-conditions

These conditions correspond to the table rows above.

For all conditions, just show the commands in a "dry run":

python -m pyserini.2cr.msmarco --collection v1-doc --all --display-commands --dry-run

To actually run all the experimental conditions:

python -m pyserini.2cr.msmarco --collection v1-doc --all --display-commands

With the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

To show the commands for a specific condition:

python -m pyserini.2cr.msmarco --collection v1-doc --condition bm25-doc-default --display-commands --dry-run

This will generate exactly the commands for a specific condition above (corresponding to a row in the table).

To actually run a specific condition:

python -m pyserini.2cr.msmarco --collection v1-doc --condition bm25-doc-default --display-commands

Again, with the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

Finally, to generate this page:

python -m pyserini.2cr.msmarco --collection v1-doc --generate-report --output msmarco-v1-doc.html

The output file msmarco-v1-doc.html should be identical to this page.