Pyserini Reproductions

The two-click^* reproductions below provides commands for reproducing experimental results on BEIR. Instructions for programmatic execution are shown at the bottom of this page (scroll down).

Main Results

The main results table provides commands for reproducing runs using the following models:

BM25 Flat: BM25 "flat" baseline (bm25-flat) [1]
BM25 MF: BM25 "multifield" baseline (bm25-multifield) [1]
SPLADE: SPLADE++ CoCondenser-EnsembleDistil (splade-pp-ed) [2]
Contriever: Contriever FT MS MARCO (contriever-msmarco) [3]
BGE: BGE-base-en-v1.5 (Faiss flat index) (bge-base-en-v1.5.faiss) [4]
Cohere: Cohere embed-english-v3.0 (Faiss flat index) (cohere-embed-english-v3.0) [5]

The symbol shown in parentheses (e.g., bm25-flat) refers the parameter used in the --condition option for programmatic execution (see below). In addition, we provide the following models whose runs can be reproduced, but are not shown in the results table:

BGE-base-en-v1.5 (Lucene flat index) (bge-base-en-v1.5.lucene-flat) [4]
BGE-base-en-v1.5 (Lucene HNSW index) (bge-base-en-v1.5.lucene-hnsw) [4]

	BM25 Flat		BM25 MF		SPLADE		Contriever		BGE		Cohere
	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100
trec-covid	0.595	0.109	0.656	0.114	0.727	0.128	0.596	0.091	0.781	0.141	0.818	0.159
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics beir-v1.0.0-trec-covid-test \ --output run.beir.bm25-flat.trec-covid.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-covid-test \ run.beir.bm25-flat.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-covid-test \ run.beir.bm25-flat.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-covid-test \ run.beir.bm25-flat.trec-covid.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.multifield \ --topics beir-v1.0.0-trec-covid-test \ --output run.beir.bm25-multifield.trec-covid.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-covid-test \ run.beir.bm25-multifield.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-covid-test \ run.beir.bm25-multifield.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-covid-test \ run.beir.bm25-multifield.trec-covid.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics beir-v1.0.0-trec-covid.test.splade-pp-ed \ --output run.beir.splade-pp-ed.trec-covid.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-covid-test \ run.beir.splade-pp-ed.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-covid-test \ run.beir.splade-pp-ed.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-covid-test \ run.beir.splade-pp-ed.trec-covid.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-trec-covid.contriever-msmarco \ --topics beir-v1.0.0-trec-covid-test \ --output run.beir.contriever-msmarco.trec-covid.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-covid-test \ run.beir.contriever-msmarco.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-covid-test \ run.beir.contriever-msmarco.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-covid-test \ run.beir.contriever-msmarco.trec-covid.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics beir-v1.0.0-trec-covid-test \ --output run.beir.bge-base-en-v1.5.faiss.trec-covid.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-covid-test \ run.beir.bge-base-en-v1.5.faiss.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-covid-test \ run.beir.bge-base-en-v1.5.faiss.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-covid-test \ run.beir.bge-base-en-v1.5.faiss.trec-covid.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-trec-covid.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-trec-covid-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-trec-covid-test \ --output run.beir.cohere-embed-english-v3.0.trec-covid.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-covid-test \ run.beir.cohere-embed-english-v3.0.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-covid-test \ run.beir.cohere-embed-english-v3.0.trec-covid.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-covid-test \ run.beir.cohere-embed-english-v3.0.trec-covid.txt`
bioasq	0.522	0.769	0.465	0.715	0.498	0.739	0.383	0.607	0.415	0.632	0.457	0.679
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-bioasq.flat \ --topics beir-v1.0.0-bioasq-test \ --output run.beir.bm25-flat.bioasq.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-bioasq-test \ run.beir.bm25-flat.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-bioasq-test \ run.beir.bm25-flat.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-bioasq-test \ run.beir.bm25-flat.bioasq.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-bioasq.multifield \ --topics beir-v1.0.0-bioasq-test \ --output run.beir.bm25-multifield.bioasq.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-bioasq-test \ run.beir.bm25-multifield.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-bioasq-test \ run.beir.bm25-multifield.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-bioasq-test \ run.beir.bm25-multifield.bioasq.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-bioasq.splade-pp-ed \ --topics beir-v1.0.0-bioasq.test.splade-pp-ed \ --output run.beir.splade-pp-ed.bioasq.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-bioasq-test \ run.beir.splade-pp-ed.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-bioasq-test \ run.beir.splade-pp-ed.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-bioasq-test \ run.beir.splade-pp-ed.bioasq.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-bioasq.contriever-msmarco \ --topics beir-v1.0.0-bioasq-test \ --output run.beir.contriever-msmarco.bioasq.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-bioasq-test \ run.beir.contriever-msmarco.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-bioasq-test \ run.beir.contriever-msmarco.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-bioasq-test \ run.beir.contriever-msmarco.bioasq.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-bioasq.bge-base-en-v1.5 \ --topics beir-v1.0.0-bioasq-test \ --output run.beir.bge-base-en-v1.5.faiss.bioasq.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-bioasq-test \ run.beir.bge-base-en-v1.5.faiss.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-bioasq-test \ run.beir.bge-base-en-v1.5.faiss.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-bioasq-test \ run.beir.bge-base-en-v1.5.faiss.bioasq.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-bioasq.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-bioasq-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-bioasq-test \ --output run.beir.cohere-embed-english-v3.0.bioasq.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-bioasq-test \ run.beir.cohere-embed-english-v3.0.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-bioasq-test \ run.beir.cohere-embed-english-v3.0.bioasq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-bioasq-test \ run.beir.cohere-embed-english-v3.0.bioasq.txt`
nfcorpus	0.322	0.246	0.325	0.250	0.347	0.284	0.328	0.301	0.373	0.337	0.386	0.351
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-nfcorpus.flat \ --topics beir-v1.0.0-nfcorpus-test \ --output run.beir.bm25-flat.nfcorpus.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nfcorpus-test \ run.beir.bm25-flat.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nfcorpus-test \ run.beir.bm25-flat.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nfcorpus-test \ run.beir.bm25-flat.nfcorpus.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-nfcorpus.multifield \ --topics beir-v1.0.0-nfcorpus-test \ --output run.beir.bm25-multifield.nfcorpus.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nfcorpus-test \ run.beir.bm25-multifield.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nfcorpus-test \ run.beir.bm25-multifield.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nfcorpus-test \ run.beir.bm25-multifield.nfcorpus.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-nfcorpus.splade-pp-ed \ --topics beir-v1.0.0-nfcorpus.test.splade-pp-ed \ --output run.beir.splade-pp-ed.nfcorpus.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nfcorpus-test \ run.beir.splade-pp-ed.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nfcorpus-test \ run.beir.splade-pp-ed.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nfcorpus-test \ run.beir.splade-pp-ed.nfcorpus.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-nfcorpus.contriever-msmarco \ --topics beir-v1.0.0-nfcorpus-test \ --output run.beir.contriever-msmarco.nfcorpus.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nfcorpus-test \ run.beir.contriever-msmarco.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nfcorpus-test \ run.beir.contriever-msmarco.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nfcorpus-test \ run.beir.contriever-msmarco.nfcorpus.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-nfcorpus.bge-base-en-v1.5 \ --topics beir-v1.0.0-nfcorpus-test \ --output run.beir.bge-base-en-v1.5.faiss.nfcorpus.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nfcorpus-test \ run.beir.bge-base-en-v1.5.faiss.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nfcorpus-test \ run.beir.bge-base-en-v1.5.faiss.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nfcorpus-test \ run.beir.bge-base-en-v1.5.faiss.nfcorpus.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-nfcorpus.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-nfcorpus-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-nfcorpus-test \ --output run.beir.cohere-embed-english-v3.0.nfcorpus.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nfcorpus-test \ run.beir.cohere-embed-english-v3.0.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nfcorpus-test \ run.beir.cohere-embed-english-v3.0.nfcorpus.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nfcorpus-test \ run.beir.cohere-embed-english-v3.0.nfcorpus.txt`
nq	0.305	0.751	0.329	0.760	0.538	0.930	0.498	0.925	0.541	0.942	0.616	0.956
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-nq.flat \ --topics beir-v1.0.0-nq-test \ --output run.beir.bm25-flat.nq.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nq-test \ run.beir.bm25-flat.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nq-test \ run.beir.bm25-flat.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nq-test \ run.beir.bm25-flat.nq.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-nq.multifield \ --topics beir-v1.0.0-nq-test \ --output run.beir.bm25-multifield.nq.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nq-test \ run.beir.bm25-multifield.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nq-test \ run.beir.bm25-multifield.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nq-test \ run.beir.bm25-multifield.nq.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-nq.splade-pp-ed \ --topics beir-v1.0.0-nq.test.splade-pp-ed \ --output run.beir.splade-pp-ed.nq.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nq-test \ run.beir.splade-pp-ed.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nq-test \ run.beir.splade-pp-ed.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nq-test \ run.beir.splade-pp-ed.nq.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-nq.contriever-msmarco \ --topics beir-v1.0.0-nq-test \ --output run.beir.contriever-msmarco.nq.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nq-test \ run.beir.contriever-msmarco.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nq-test \ run.beir.contriever-msmarco.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nq-test \ run.beir.contriever-msmarco.nq.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-nq.bge-base-en-v1.5 \ --topics beir-v1.0.0-nq-test \ --output run.beir.bge-base-en-v1.5.faiss.nq.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nq-test \ run.beir.bge-base-en-v1.5.faiss.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nq-test \ run.beir.bge-base-en-v1.5.faiss.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nq-test \ run.beir.bge-base-en-v1.5.faiss.nq.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-nq.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-nq-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-nq-test \ --output run.beir.cohere-embed-english-v3.0.nq.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-nq-test \ run.beir.cohere-embed-english-v3.0.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-nq-test \ run.beir.cohere-embed-english-v3.0.nq.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-nq-test \ run.beir.cohere-embed-english-v3.0.nq.txt`
hotpotqa	0.633	0.796	0.603	0.740	0.687	0.818	0.638	0.777	0.726	0.873	0.707	0.823
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-hotpotqa.flat \ --topics beir-v1.0.0-hotpotqa-test \ --output run.beir.bm25-flat.hotpotqa.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-hotpotqa-test \ run.beir.bm25-flat.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-hotpotqa-test \ run.beir.bm25-flat.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-hotpotqa-test \ run.beir.bm25-flat.hotpotqa.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-hotpotqa.multifield \ --topics beir-v1.0.0-hotpotqa-test \ --output run.beir.bm25-multifield.hotpotqa.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-hotpotqa-test \ run.beir.bm25-multifield.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-hotpotqa-test \ run.beir.bm25-multifield.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-hotpotqa-test \ run.beir.bm25-multifield.hotpotqa.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-hotpotqa.splade-pp-ed \ --topics beir-v1.0.0-hotpotqa.test.splade-pp-ed \ --output run.beir.splade-pp-ed.hotpotqa.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-hotpotqa-test \ run.beir.splade-pp-ed.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-hotpotqa-test \ run.beir.splade-pp-ed.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-hotpotqa-test \ run.beir.splade-pp-ed.hotpotqa.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-hotpotqa.contriever-msmarco \ --topics beir-v1.0.0-hotpotqa-test \ --output run.beir.contriever-msmarco.hotpotqa.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-hotpotqa-test \ run.beir.contriever-msmarco.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-hotpotqa-test \ run.beir.contriever-msmarco.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-hotpotqa-test \ run.beir.contriever-msmarco.hotpotqa.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-hotpotqa.bge-base-en-v1.5 \ --topics beir-v1.0.0-hotpotqa-test \ --output run.beir.bge-base-en-v1.5.faiss.hotpotqa.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-hotpotqa-test \ run.beir.bge-base-en-v1.5.faiss.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-hotpotqa-test \ run.beir.bge-base-en-v1.5.faiss.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-hotpotqa-test \ run.beir.bge-base-en-v1.5.faiss.hotpotqa.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-hotpotqa.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-hotpotqa-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-hotpotqa-test \ --output run.beir.cohere-embed-english-v3.0.hotpotqa.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-hotpotqa-test \ run.beir.cohere-embed-english-v3.0.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-hotpotqa-test \ run.beir.cohere-embed-english-v3.0.hotpotqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-hotpotqa-test \ run.beir.cohere-embed-english-v3.0.hotpotqa.txt`
fiqa	0.236	0.539	0.236	0.539	0.347	0.631	0.329	0.656	0.406	0.742	0.421	0.736
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics beir-v1.0.0-fiqa-test \ --output run.beir.bm25-flat.fiqa.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fiqa-test \ run.beir.bm25-flat.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fiqa-test \ run.beir.bm25-flat.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fiqa-test \ run.beir.bm25-flat.fiqa.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.multifield \ --topics beir-v1.0.0-fiqa-test \ --output run.beir.bm25-multifield.fiqa.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fiqa-test \ run.beir.bm25-multifield.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fiqa-test \ run.beir.bm25-multifield.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fiqa-test \ run.beir.bm25-multifield.fiqa.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics beir-v1.0.0-fiqa.test.splade-pp-ed \ --output run.beir.splade-pp-ed.fiqa.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fiqa-test \ run.beir.splade-pp-ed.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fiqa-test \ run.beir.splade-pp-ed.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fiqa-test \ run.beir.splade-pp-ed.fiqa.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-fiqa.contriever-msmarco \ --topics beir-v1.0.0-fiqa-test \ --output run.beir.contriever-msmarco.fiqa.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fiqa-test \ run.beir.contriever-msmarco.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fiqa-test \ run.beir.contriever-msmarco.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fiqa-test \ run.beir.contriever-msmarco.fiqa.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics beir-v1.0.0-fiqa-test \ --output run.beir.bge-base-en-v1.5.faiss.fiqa.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fiqa-test \ run.beir.bge-base-en-v1.5.faiss.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fiqa-test \ run.beir.bge-base-en-v1.5.faiss.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fiqa-test \ run.beir.bge-base-en-v1.5.faiss.fiqa.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-fiqa.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-fiqa-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-fiqa-test \ --output run.beir.cohere-embed-english-v3.0.fiqa.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fiqa-test \ run.beir.cohere-embed-english-v3.0.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fiqa-test \ run.beir.cohere-embed-english-v3.0.fiqa.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fiqa-test \ run.beir.cohere-embed-english-v3.0.fiqa.txt`
signal1m	0.330	0.370	0.330	0.370	0.301	0.340	0.278	0.322	0.289	0.311	0.263	0.283
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-signal1m.flat \ --topics beir-v1.0.0-signal1m-test \ --output run.beir.bm25-flat.signal1m.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-signal1m-test \ run.beir.bm25-flat.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-signal1m-test \ run.beir.bm25-flat.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-signal1m-test \ run.beir.bm25-flat.signal1m.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-signal1m.multifield \ --topics beir-v1.0.0-signal1m-test \ --output run.beir.bm25-multifield.signal1m.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-signal1m-test \ run.beir.bm25-multifield.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-signal1m-test \ run.beir.bm25-multifield.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-signal1m-test \ run.beir.bm25-multifield.signal1m.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-signal1m.splade-pp-ed \ --topics beir-v1.0.0-signal1m.test.splade-pp-ed \ --output run.beir.splade-pp-ed.signal1m.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-signal1m-test \ run.beir.splade-pp-ed.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-signal1m-test \ run.beir.splade-pp-ed.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-signal1m-test \ run.beir.splade-pp-ed.signal1m.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-signal1m.contriever-msmarco \ --topics beir-v1.0.0-signal1m-test \ --output run.beir.contriever-msmarco.signal1m.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-signal1m-test \ run.beir.contriever-msmarco.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-signal1m-test \ run.beir.contriever-msmarco.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-signal1m-test \ run.beir.contriever-msmarco.signal1m.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-signal1m.bge-base-en-v1.5 \ --topics beir-v1.0.0-signal1m-test \ --output run.beir.bge-base-en-v1.5.faiss.signal1m.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-signal1m-test \ run.beir.bge-base-en-v1.5.faiss.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-signal1m-test \ run.beir.bge-base-en-v1.5.faiss.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-signal1m-test \ run.beir.bge-base-en-v1.5.faiss.signal1m.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-signal1m.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-signal1m-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-signal1m-test \ --output run.beir.cohere-embed-english-v3.0.signal1m.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-signal1m-test \ run.beir.cohere-embed-english-v3.0.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-signal1m-test \ run.beir.cohere-embed-english-v3.0.signal1m.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-signal1m-test \ run.beir.cohere-embed-english-v3.0.signal1m.txt`
trec-news	0.395	0.447	0.398	0.422	0.415	0.441	0.428	0.492	0.442	0.499	0.504	0.543
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics beir-v1.0.0-trec-news-test \ --output run.beir.bm25-flat.trec-news.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-news-test \ run.beir.bm25-flat.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-news-test \ run.beir.bm25-flat.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-news-test \ run.beir.bm25-flat.trec-news.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.multifield \ --topics beir-v1.0.0-trec-news-test \ --output run.beir.bm25-multifield.trec-news.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-news-test \ run.beir.bm25-multifield.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-news-test \ run.beir.bm25-multifield.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-news-test \ run.beir.bm25-multifield.trec-news.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics beir-v1.0.0-trec-news.test.splade-pp-ed \ --output run.beir.splade-pp-ed.trec-news.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-news-test \ run.beir.splade-pp-ed.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-news-test \ run.beir.splade-pp-ed.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-news-test \ run.beir.splade-pp-ed.trec-news.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-trec-news.contriever-msmarco \ --topics beir-v1.0.0-trec-news-test \ --output run.beir.contriever-msmarco.trec-news.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-news-test \ run.beir.contriever-msmarco.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-news-test \ run.beir.contriever-msmarco.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-news-test \ run.beir.contriever-msmarco.trec-news.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics beir-v1.0.0-trec-news-test \ --output run.beir.bge-base-en-v1.5.faiss.trec-news.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-news-test \ run.beir.bge-base-en-v1.5.faiss.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-news-test \ run.beir.bge-base-en-v1.5.faiss.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-news-test \ run.beir.bge-base-en-v1.5.faiss.trec-news.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-trec-news.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-trec-news-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-trec-news-test \ --output run.beir.cohere-embed-english-v3.0.trec-news.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-trec-news-test \ run.beir.cohere-embed-english-v3.0.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-trec-news-test \ run.beir.cohere-embed-english-v3.0.trec-news.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-trec-news-test \ run.beir.cohere-embed-english-v3.0.trec-news.txt`
robust04	0.407	0.375	0.407	0.375	0.468	0.385	0.473	0.392	0.444	0.351	0.541	0.417
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-robust04.flat \ --topics beir-v1.0.0-robust04-test \ --output run.beir.bm25-flat.robust04.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-robust04-test \ run.beir.bm25-flat.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-robust04-test \ run.beir.bm25-flat.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-robust04-test \ run.beir.bm25-flat.robust04.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-robust04.multifield \ --topics beir-v1.0.0-robust04-test \ --output run.beir.bm25-multifield.robust04.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-robust04-test \ run.beir.bm25-multifield.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-robust04-test \ run.beir.bm25-multifield.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-robust04-test \ run.beir.bm25-multifield.robust04.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-robust04.splade-pp-ed \ --topics beir-v1.0.0-robust04.test.splade-pp-ed \ --output run.beir.splade-pp-ed.robust04.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-robust04-test \ run.beir.splade-pp-ed.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-robust04-test \ run.beir.splade-pp-ed.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-robust04-test \ run.beir.splade-pp-ed.robust04.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-robust04.contriever-msmarco \ --topics beir-v1.0.0-robust04-test \ --output run.beir.contriever-msmarco.robust04.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-robust04-test \ run.beir.contriever-msmarco.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-robust04-test \ run.beir.contriever-msmarco.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-robust04-test \ run.beir.contriever-msmarco.robust04.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-robust04.bge-base-en-v1.5 \ --topics beir-v1.0.0-robust04-test \ --output run.beir.bge-base-en-v1.5.faiss.robust04.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-robust04-test \ run.beir.bge-base-en-v1.5.faiss.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-robust04-test \ run.beir.bge-base-en-v1.5.faiss.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-robust04-test \ run.beir.bge-base-en-v1.5.faiss.robust04.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-robust04.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-robust04-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-robust04-test \ --output run.beir.cohere-embed-english-v3.0.robust04.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-robust04-test \ run.beir.cohere-embed-english-v3.0.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-robust04-test \ run.beir.cohere-embed-english-v3.0.robust04.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-robust04-test \ run.beir.cohere-embed-english-v3.0.robust04.txt`
arguana	0.397	0.932	0.414	0.943	0.520	0.974	0.446	0.977	0.636	0.992	0.540	0.982
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics beir-v1.0.0-arguana-test \ --output run.beir.bm25-flat.arguana.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-arguana-test \ run.beir.bm25-flat.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-arguana-test \ run.beir.bm25-flat.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-arguana-test \ run.beir.bm25-flat.arguana.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.multifield \ --topics beir-v1.0.0-arguana-test \ --output run.beir.bm25-multifield.arguana.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-arguana-test \ run.beir.bm25-multifield.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-arguana-test \ run.beir.bm25-multifield.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-arguana-test \ run.beir.bm25-multifield.arguana.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics beir-v1.0.0-arguana.test.splade-pp-ed \ --output run.beir.splade-pp-ed.arguana.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-arguana-test \ run.beir.splade-pp-ed.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-arguana-test \ run.beir.splade-pp-ed.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-arguana-test \ run.beir.splade-pp-ed.arguana.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-arguana.contriever-msmarco \ --topics beir-v1.0.0-arguana-test \ --output run.beir.contriever-msmarco.arguana.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-arguana-test \ run.beir.contriever-msmarco.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-arguana-test \ run.beir.contriever-msmarco.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-arguana-test \ run.beir.contriever-msmarco.arguana.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "" \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics beir-v1.0.0-arguana-test \ --output run.beir.bge-base-en-v1.5.faiss.arguana.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-arguana-test \ run.beir.bge-base-en-v1.5.faiss.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-arguana-test \ run.beir.bge-base-en-v1.5.faiss.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-arguana-test \ run.beir.bge-base-en-v1.5.faiss.arguana.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-arguana.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-arguana-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-arguana-test \ --output run.beir.cohere-embed-english-v3.0.arguana.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-arguana-test \ run.beir.cohere-embed-english-v3.0.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-arguana-test \ run.beir.cohere-embed-english-v3.0.arguana.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-arguana-test \ run.beir.cohere-embed-english-v3.0.arguana.txt`
webis-touche2020	0.442	0.582	0.367	0.538	0.247	0.471	0.204	0.442	0.257	0.487	0.326	0.516
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-webis-touche2020.flat \ --topics beir-v1.0.0-webis-touche2020-test \ --output run.beir.bm25-flat.webis-touche2020.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-webis-touche2020-test \ run.beir.bm25-flat.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-webis-touche2020-test \ run.beir.bm25-flat.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-webis-touche2020-test \ run.beir.bm25-flat.webis-touche2020.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-webis-touche2020.multifield \ --topics beir-v1.0.0-webis-touche2020-test \ --output run.beir.bm25-multifield.webis-touche2020.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-webis-touche2020-test \ run.beir.bm25-multifield.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-webis-touche2020-test \ run.beir.bm25-multifield.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-webis-touche2020-test \ run.beir.bm25-multifield.webis-touche2020.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-webis-touche2020.splade-pp-ed \ --topics beir-v1.0.0-webis-touche2020.test.splade-pp-ed \ --output run.beir.splade-pp-ed.webis-touche2020.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-webis-touche2020-test \ run.beir.splade-pp-ed.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-webis-touche2020-test \ run.beir.splade-pp-ed.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-webis-touche2020-test \ run.beir.splade-pp-ed.webis-touche2020.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-webis-touche2020.contriever-msmarco \ --topics beir-v1.0.0-webis-touche2020-test \ --output run.beir.contriever-msmarco.webis-touche2020.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-webis-touche2020-test \ run.beir.contriever-msmarco.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-webis-touche2020-test \ run.beir.contriever-msmarco.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-webis-touche2020-test \ run.beir.contriever-msmarco.webis-touche2020.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-webis-touche2020.bge-base-en-v1.5 \ --topics beir-v1.0.0-webis-touche2020-test \ --output run.beir.bge-base-en-v1.5.faiss.webis-touche2020.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-webis-touche2020-test \ run.beir.bge-base-en-v1.5.faiss.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-webis-touche2020-test \ run.beir.bge-base-en-v1.5.faiss.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-webis-touche2020-test \ run.beir.bge-base-en-v1.5.faiss.webis-touche2020.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-webis-touche2020.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-webis-touche2020-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-webis-touche2020-test \ --output run.beir.cohere-embed-english-v3.0.webis-touche2020.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-webis-touche2020-test \ run.beir.cohere-embed-english-v3.0.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-webis-touche2020-test \ run.beir.cohere-embed-english-v3.0.webis-touche2020.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-webis-touche2020-test \ run.beir.cohere-embed-english-v3.0.webis-touche2020.txt`
cqadupstack (average)	0.302	0.580	0.299	0.606	0.334	0.650	0.345	0.663	0.424	0.762	0.415	0.745
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-android.flat \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.bm25-flat.cqadupstack-android.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-english.flat \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.bm25-flat.cqadupstack-english.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gaming.flat \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.bm25-flat.cqadupstack-gaming.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gis.flat \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.bm25-flat.cqadupstack-gis.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-mathematica.flat \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.bm25-flat.cqadupstack-mathematica.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-physics.flat \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.bm25-flat.cqadupstack-physics.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-programmers.flat \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.bm25-flat.cqadupstack-programmers.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-stats.flat \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.bm25-flat.cqadupstack-stats.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-tex.flat \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.bm25-flat.cqadupstack-tex.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-unix.flat \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.bm25-flat.cqadupstack-unix.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-webmasters.flat \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.bm25-flat.cqadupstack-webmasters.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-wordpress.flat \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.bm25-flat.cqadupstack-wordpress.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt Command to generate run: python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-android.multifield \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.bm25-multifield.cqadupstack-android.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-english.multifield \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.bm25-multifield.cqadupstack-english.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gaming.multifield \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.bm25-multifield.cqadupstack-gaming.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gis.multifield \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.bm25-multifield.cqadupstack-gis.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-mathematica.multifield \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.bm25-multifield.cqadupstack-mathematica.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-physics.multifield \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.bm25-multifield.cqadupstack-physics.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-programmers.multifield \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.bm25-multifield.cqadupstack-programmers.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-stats.multifield \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.bm25-multifield.cqadupstack-stats.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-tex.multifield \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.bm25-multifield.cqadupstack-tex.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-unix.multifield \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.bm25-multifield.cqadupstack-unix.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-webmasters.multifield \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.bm25-multifield.cqadupstack-webmasters.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-wordpress.multifield \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.bm25-multifield.cqadupstack-wordpress.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0 Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt Command to generate run: python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-android.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-android.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-android.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-english.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-english.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-english.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gaming.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-gaming.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-gaming.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gis.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-gis.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-gis.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-mathematica.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-mathematica.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-mathematica.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-physics.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-physics.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-physics.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-programmers.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-programmers.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-programmers.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-stats.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-stats.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-stats.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-tex.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-tex.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-tex.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-unix.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-unix.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-unix.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-webmasters.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-webmasters.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-webmasters.txt \ --output-format trec \ --hits 1000 --impact --remove-query python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-wordpress.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-wordpress.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-wordpress.txt \ --output-format trec \ --hits 1000 --impact --remove-query Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt Command to generate run: python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-android.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.contriever-msmarco.cqadupstack-android.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-english.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.contriever-msmarco.cqadupstack-english.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-gaming.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.contriever-msmarco.cqadupstack-gaming.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-gis.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.contriever-msmarco.cqadupstack-gis.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-mathematica.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.contriever-msmarco.cqadupstack-mathematica.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-physics.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.contriever-msmarco.cqadupstack-physics.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-programmers.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.contriever-msmarco.cqadupstack-programmers.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-stats.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.contriever-msmarco.cqadupstack-stats.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-tex.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.contriever-msmarco.cqadupstack-tex.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-unix.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.contriever-msmarco.cqadupstack-unix.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-webmasters.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.contriever-msmarco.cqadupstack-webmasters.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-wordpress.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.contriever-msmarco.cqadupstack-wordpress.txt \ --hits 1000 --remove-query Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt Command to generate run: python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-android.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-english.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-gaming.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-gis.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-mathematica.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-physics.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-programmers.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-stats.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-tex.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-unix.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-webmasters.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-wordpress.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt \ --hits 1000 --remove-query Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt Command to generate run: python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-android.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-android-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-android-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-english.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-english-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-english-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-gaming.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-gis.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-gis-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-mathematica.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-physics.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-physics-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-programmers.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-stats.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-stats-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-tex.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-tex-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-unix.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-unix-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-webmasters.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt \ --hits 1000 --remove-query python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-wordpress.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt \ --hits 1000 --remove-query Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt
quora	0.789	0.973	0.789	0.973	0.834	0.986	0.865	0.994	0.889	0.997	0.887	0.996
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-quora.flat \ --topics beir-v1.0.0-quora-test \ --output run.beir.bm25-flat.quora.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-quora-test \ run.beir.bm25-flat.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-quora-test \ run.beir.bm25-flat.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-quora-test \ run.beir.bm25-flat.quora.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-quora.multifield \ --topics beir-v1.0.0-quora-test \ --output run.beir.bm25-multifield.quora.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-quora-test \ run.beir.bm25-multifield.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-quora-test \ run.beir.bm25-multifield.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-quora-test \ run.beir.bm25-multifield.quora.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-quora.splade-pp-ed \ --topics beir-v1.0.0-quora.test.splade-pp-ed \ --output run.beir.splade-pp-ed.quora.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-quora-test \ run.beir.splade-pp-ed.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-quora-test \ run.beir.splade-pp-ed.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-quora-test \ run.beir.splade-pp-ed.quora.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-quora.contriever-msmarco \ --topics beir-v1.0.0-quora-test \ --output run.beir.contriever-msmarco.quora.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-quora-test \ run.beir.contriever-msmarco.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-quora-test \ run.beir.contriever-msmarco.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-quora-test \ run.beir.contriever-msmarco.quora.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "" \ --index beir-v1.0.0-quora.bge-base-en-v1.5 \ --topics beir-v1.0.0-quora-test \ --output run.beir.bge-base-en-v1.5.faiss.quora.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-quora-test \ run.beir.bge-base-en-v1.5.faiss.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-quora-test \ run.beir.bge-base-en-v1.5.faiss.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-quora-test \ run.beir.bge-base-en-v1.5.faiss.quora.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-quora.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-quora-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-quora-test \ --output run.beir.cohere-embed-english-v3.0.quora.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-quora-test \ run.beir.cohere-embed-english-v3.0.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-quora-test \ run.beir.cohere-embed-english-v3.0.quora.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-quora-test \ run.beir.cohere-embed-english-v3.0.quora.txt`
dbpedia-entity	0.318	0.468	0.313	0.398	0.437	0.562	0.413	0.541	0.407	0.530	0.434	0.536
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics beir-v1.0.0-dbpedia-entity-test \ --output run.beir.bm25-flat.dbpedia-entity.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-dbpedia-entity-test \ run.beir.bm25-flat.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-dbpedia-entity-test \ run.beir.bm25-flat.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-dbpedia-entity-test \ run.beir.bm25-flat.dbpedia-entity.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.multifield \ --topics beir-v1.0.0-dbpedia-entity-test \ --output run.beir.bm25-multifield.dbpedia-entity.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-dbpedia-entity-test \ run.beir.bm25-multifield.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-dbpedia-entity-test \ run.beir.bm25-multifield.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-dbpedia-entity-test \ run.beir.bm25-multifield.dbpedia-entity.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics beir-v1.0.0-dbpedia-entity.test.splade-pp-ed \ --output run.beir.splade-pp-ed.dbpedia-entity.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-dbpedia-entity-test \ run.beir.splade-pp-ed.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-dbpedia-entity-test \ run.beir.splade-pp-ed.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-dbpedia-entity-test \ run.beir.splade-pp-ed.dbpedia-entity.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-dbpedia-entity.contriever-msmarco \ --topics beir-v1.0.0-dbpedia-entity-test \ --output run.beir.contriever-msmarco.dbpedia-entity.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-dbpedia-entity-test \ run.beir.contriever-msmarco.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-dbpedia-entity-test \ run.beir.contriever-msmarco.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-dbpedia-entity-test \ run.beir.contriever-msmarco.dbpedia-entity.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics beir-v1.0.0-dbpedia-entity-test \ --output run.beir.bge-base-en-v1.5.faiss.dbpedia-entity.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-dbpedia-entity-test \ run.beir.bge-base-en-v1.5.faiss.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-dbpedia-entity-test \ run.beir.bge-base-en-v1.5.faiss.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-dbpedia-entity-test \ run.beir.bge-base-en-v1.5.faiss.dbpedia-entity.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-dbpedia-entity.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-dbpedia-entity-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-dbpedia-entity-test \ --output run.beir.cohere-embed-english-v3.0.dbpedia-entity.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-dbpedia-entity-test \ run.beir.cohere-embed-english-v3.0.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-dbpedia-entity-test \ run.beir.cohere-embed-english-v3.0.dbpedia-entity.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-dbpedia-entity-test \ run.beir.cohere-embed-english-v3.0.dbpedia-entity.txt`
scidocs	0.149	0.348	0.158	0.356	0.159	0.373	0.165	0.378	0.217	0.496	0.203	0.451
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scidocs.flat \ --topics beir-v1.0.0-scidocs-test \ --output run.beir.bm25-flat.scidocs.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scidocs-test \ run.beir.bm25-flat.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scidocs-test \ run.beir.bm25-flat.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scidocs-test \ run.beir.bm25-flat.scidocs.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scidocs.multifield \ --topics beir-v1.0.0-scidocs-test \ --output run.beir.bm25-multifield.scidocs.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scidocs-test \ run.beir.bm25-multifield.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scidocs-test \ run.beir.bm25-multifield.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scidocs-test \ run.beir.bm25-multifield.scidocs.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scidocs.splade-pp-ed \ --topics beir-v1.0.0-scidocs.test.splade-pp-ed \ --output run.beir.splade-pp-ed.scidocs.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scidocs-test \ run.beir.splade-pp-ed.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scidocs-test \ run.beir.splade-pp-ed.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scidocs-test \ run.beir.splade-pp-ed.scidocs.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-scidocs.contriever-msmarco \ --topics beir-v1.0.0-scidocs-test \ --output run.beir.contriever-msmarco.scidocs.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scidocs-test \ run.beir.contriever-msmarco.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scidocs-test \ run.beir.contriever-msmarco.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scidocs-test \ run.beir.contriever-msmarco.scidocs.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-scidocs.bge-base-en-v1.5 \ --topics beir-v1.0.0-scidocs-test \ --output run.beir.bge-base-en-v1.5.faiss.scidocs.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scidocs-test \ run.beir.bge-base-en-v1.5.faiss.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scidocs-test \ run.beir.bge-base-en-v1.5.faiss.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scidocs-test \ run.beir.bge-base-en-v1.5.faiss.scidocs.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-scidocs.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-scidocs-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-scidocs-test \ --output run.beir.cohere-embed-english-v3.0.scidocs.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scidocs-test \ run.beir.cohere-embed-english-v3.0.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scidocs-test \ run.beir.cohere-embed-english-v3.0.scidocs.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scidocs-test \ run.beir.cohere-embed-english-v3.0.scidocs.txt`
fever	0.651	0.918	0.753	0.931	0.788	0.946	0.758	0.949	0.863	0.972	0.890	0.965
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fever.flat \ --topics beir-v1.0.0-fever-test \ --output run.beir.bm25-flat.fever.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fever-test \ run.beir.bm25-flat.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fever-test \ run.beir.bm25-flat.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fever-test \ run.beir.bm25-flat.fever.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fever.multifield \ --topics beir-v1.0.0-fever-test \ --output run.beir.bm25-multifield.fever.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fever-test \ run.beir.bm25-multifield.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fever-test \ run.beir.bm25-multifield.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fever-test \ run.beir.bm25-multifield.fever.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fever.splade-pp-ed \ --topics beir-v1.0.0-fever.test.splade-pp-ed \ --output run.beir.splade-pp-ed.fever.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fever-test \ run.beir.splade-pp-ed.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fever-test \ run.beir.splade-pp-ed.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fever-test \ run.beir.splade-pp-ed.fever.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-fever.contriever-msmarco \ --topics beir-v1.0.0-fever-test \ --output run.beir.contriever-msmarco.fever.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fever-test \ run.beir.contriever-msmarco.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fever-test \ run.beir.contriever-msmarco.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fever-test \ run.beir.contriever-msmarco.fever.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-fever.bge-base-en-v1.5 \ --topics beir-v1.0.0-fever-test \ --output run.beir.bge-base-en-v1.5.faiss.fever.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fever-test \ run.beir.bge-base-en-v1.5.faiss.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fever-test \ run.beir.bge-base-en-v1.5.faiss.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fever-test \ run.beir.bge-base-en-v1.5.faiss.fever.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-fever.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-fever-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-fever-test \ --output run.beir.cohere-embed-english-v3.0.fever.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-fever-test \ run.beir.cohere-embed-english-v3.0.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-fever-test \ run.beir.cohere-embed-english-v3.0.fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-fever-test \ run.beir.cohere-embed-english-v3.0.fever.txt`
climate-fever	0.165	0.425	0.213	0.436	0.230	0.521	0.237	0.575	0.312	0.636	0.259	0.581
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-climate-fever.flat \ --topics beir-v1.0.0-climate-fever-test \ --output run.beir.bm25-flat.climate-fever.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-climate-fever-test \ run.beir.bm25-flat.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-climate-fever-test \ run.beir.bm25-flat.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-climate-fever-test \ run.beir.bm25-flat.climate-fever.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-climate-fever.multifield \ --topics beir-v1.0.0-climate-fever-test \ --output run.beir.bm25-multifield.climate-fever.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-climate-fever-test \ run.beir.bm25-multifield.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-climate-fever-test \ run.beir.bm25-multifield.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-climate-fever-test \ run.beir.bm25-multifield.climate-fever.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-climate-fever.splade-pp-ed \ --topics beir-v1.0.0-climate-fever.test.splade-pp-ed \ --output run.beir.splade-pp-ed.climate-fever.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-climate-fever-test \ run.beir.splade-pp-ed.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-climate-fever-test \ run.beir.splade-pp-ed.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-climate-fever-test \ run.beir.splade-pp-ed.climate-fever.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-climate-fever.contriever-msmarco \ --topics beir-v1.0.0-climate-fever-test \ --output run.beir.contriever-msmarco.climate-fever.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-climate-fever-test \ run.beir.contriever-msmarco.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-climate-fever-test \ run.beir.contriever-msmarco.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-climate-fever-test \ run.beir.contriever-msmarco.climate-fever.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-climate-fever.bge-base-en-v1.5 \ --topics beir-v1.0.0-climate-fever-test \ --output run.beir.bge-base-en-v1.5.faiss.climate-fever.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-climate-fever-test \ run.beir.bge-base-en-v1.5.faiss.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-climate-fever-test \ run.beir.bge-base-en-v1.5.faiss.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-climate-fever-test \ run.beir.bge-base-en-v1.5.faiss.climate-fever.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-climate-fever.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-climate-fever-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-climate-fever-test \ --output run.beir.cohere-embed-english-v3.0.climate-fever.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-climate-fever-test \ run.beir.cohere-embed-english-v3.0.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-climate-fever-test \ run.beir.cohere-embed-english-v3.0.climate-fever.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-climate-fever-test \ run.beir.cohere-embed-english-v3.0.climate-fever.txt`
scifact	0.679	0.925	0.665	0.908	0.704	0.935	0.677	0.947	0.741	0.967	0.718	0.963
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics beir-v1.0.0-scifact-test \ --output run.beir.bm25-flat.scifact.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scifact-test \ run.beir.bm25-flat.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scifact-test \ run.beir.bm25-flat.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scifact-test \ run.beir.bm25-flat.scifact.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.multifield \ --topics beir-v1.0.0-scifact-test \ --output run.beir.bm25-multifield.scifact.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scifact-test \ run.beir.bm25-multifield.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scifact-test \ run.beir.bm25-multifield.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scifact-test \ run.beir.bm25-multifield.scifact.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics beir-v1.0.0-scifact.test.splade-pp-ed \ --output run.beir.splade-pp-ed.scifact.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scifact-test \ run.beir.splade-pp-ed.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scifact-test \ run.beir.splade-pp-ed.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scifact-test \ run.beir.splade-pp-ed.scifact.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-scifact.contriever-msmarco \ --topics beir-v1.0.0-scifact-test \ --output run.beir.contriever-msmarco.scifact.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scifact-test \ run.beir.contriever-msmarco.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scifact-test \ run.beir.contriever-msmarco.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scifact-test \ run.beir.contriever-msmarco.scifact.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics beir-v1.0.0-scifact-test \ --output run.beir.bge-base-en-v1.5.faiss.scifact.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scifact-test \ run.beir.bge-base-en-v1.5.faiss.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scifact-test \ run.beir.bge-base-en-v1.5.faiss.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scifact-test \ run.beir.bge-base-en-v1.5.faiss.scifact.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-scifact.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-scifact-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-scifact-test \ --output run.beir.cohere-embed-english-v3.0.scifact.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-scifact-test \ run.beir.cohere-embed-english-v3.0.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-scifact-test \ run.beir.cohere-embed-english-v3.0.scifact.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-scifact-test \ run.beir.cohere-embed-english-v3.0.scifact.txt`

CQADupStack Results

The "cqadupstack (average)" row above represents the average across the individual CQADupStack datasets, shown below.

	BM25 Flat		BM25 MF		SPLADE		Contriever		BGE		Cohere
	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100
android	0.380	0.683	0.371	0.689	0.390	0.740	0.425	0.750	0.508	0.845	0.500	0.832
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-android.flat \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.bm25-flat.cqadupstack-android.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-flat.cqadupstack-android.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-android.multifield \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.bm25-multifield.cqadupstack-android.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.bm25-multifield.cqadupstack-android.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-android.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-android.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-android.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.splade-pp-ed.cqadupstack-android.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-android.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.contriever-msmarco.cqadupstack-android.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.contriever-msmarco.cqadupstack-android.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-android.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-android-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-android.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-android.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-android-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-android-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-android-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-android.txt`
english	0.345	0.576	0.332	0.584	0.408	0.695	0.433	0.694	0.486	0.759	0.491	0.757
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-english.flat \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.bm25-flat.cqadupstack-english.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-flat.cqadupstack-english.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-english.multifield \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.bm25-multifield.cqadupstack-english.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.bm25-multifield.cqadupstack-english.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-english.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-english.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-english.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.splade-pp-ed.cqadupstack-english.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-english.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.contriever-msmarco.cqadupstack-english.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.contriever-msmarco.cqadupstack-english.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-english.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-english-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-english.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-english.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-english-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-english-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-english-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-english.txt`
gaming	0.482	0.765	0.442	0.757	0.496	0.813	0.528	0.848	0.597	0.904	0.605	0.900
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gaming.flat \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.bm25-flat.cqadupstack-gaming.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-flat.cqadupstack-gaming.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gaming.multifield \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.bm25-multifield.cqadupstack-gaming.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bm25-multifield.cqadupstack-gaming.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gaming.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-gaming.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-gaming.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.splade-pp-ed.cqadupstack-gaming.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-gaming.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.contriever-msmarco.cqadupstack-gaming.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.contriever-msmarco.cqadupstack-gaming.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-gaming.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gaming.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-gaming.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-gaming-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-gaming-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gaming-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gaming.txt`
gis	0.290	0.612	0.290	0.646	0.315	0.632	0.302	0.627	0.413	0.768	0.392	0.744
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gis.flat \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.bm25-flat.cqadupstack-gis.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-flat.cqadupstack-gis.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gis.multifield \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.bm25-multifield.cqadupstack-gis.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bm25-multifield.cqadupstack-gis.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-gis.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-gis.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-gis.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.splade-pp-ed.cqadupstack-gis.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-gis.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.contriever-msmarco.cqadupstack-gis.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.contriever-msmarco.cqadupstack-gis.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-gis.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-gis.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-gis.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-gis-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-gis-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-gis-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-gis.txt`
mathematica	0.202	0.488	0.205	0.521	0.238	0.580	0.235	0.573	0.316	0.692	0.304	0.667
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-mathematica.flat \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.bm25-flat.cqadupstack-mathematica.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-flat.cqadupstack-mathematica.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-mathematica.multifield \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.bm25-multifield.cqadupstack-mathematica.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bm25-multifield.cqadupstack-mathematica.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-mathematica.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-mathematica.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-mathematica.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.splade-pp-ed.cqadupstack-mathematica.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-mathematica.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.contriever-msmarco.cqadupstack-mathematica.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.contriever-msmarco.cqadupstack-mathematica.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-mathematica.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-mathematica.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-mathematica.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-mathematica-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-mathematica-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-mathematica-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-mathematica.txt`
physics	0.321	0.633	0.325	0.649	0.360	0.720	0.416	0.762	0.472	0.808	0.438	0.784
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-physics.flat \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.bm25-flat.cqadupstack-physics.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-flat.cqadupstack-physics.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-physics.multifield \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.bm25-multifield.cqadupstack-physics.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bm25-multifield.cqadupstack-physics.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-physics.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-physics.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-physics.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.splade-pp-ed.cqadupstack-physics.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-physics.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.contriever-msmarco.cqadupstack-physics.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.contriever-msmarco.cqadupstack-physics.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-physics.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-physics.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-physics.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-physics-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-physics-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-physics-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-physics.txt`
programmers	0.280	0.559	0.296	0.619	0.340	0.658	0.357	0.719	0.424	0.786	0.437	0.789
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-programmers.flat \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.bm25-flat.cqadupstack-programmers.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-flat.cqadupstack-programmers.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-programmers.multifield \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.bm25-multifield.cqadupstack-programmers.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bm25-multifield.cqadupstack-programmers.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-programmers.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-programmers.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-programmers.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.splade-pp-ed.cqadupstack-programmers.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-programmers.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.contriever-msmarco.cqadupstack-programmers.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.contriever-msmarco.cqadupstack-programmers.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-programmers.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-programmers.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-programmers.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-programmers-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-programmers-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-programmers-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-programmers.txt`
stats	0.271	0.534	0.279	0.572	0.299	0.589	0.309	0.586	0.373	0.673	0.352	0.643
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-stats.flat \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.bm25-flat.cqadupstack-stats.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-flat.cqadupstack-stats.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-stats.multifield \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.bm25-multifield.cqadupstack-stats.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bm25-multifield.cqadupstack-stats.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-stats.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-stats.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-stats.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.splade-pp-ed.cqadupstack-stats.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-stats.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.contriever-msmarco.cqadupstack-stats.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.contriever-msmarco.cqadupstack-stats.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-stats.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-stats.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-stats.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-stats-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-stats-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-stats-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-stats.txt`
tex	0.224	0.469	0.209	0.495	0.253	0.516	0.221	0.498	0.311	0.649	0.308	0.624
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-tex.flat \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.bm25-flat.cqadupstack-tex.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-flat.cqadupstack-tex.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-tex.multifield \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.bm25-multifield.cqadupstack-tex.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bm25-multifield.cqadupstack-tex.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-tex.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-tex.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-tex.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.splade-pp-ed.cqadupstack-tex.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-tex.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.contriever-msmarco.cqadupstack-tex.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.contriever-msmarco.cqadupstack-tex.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-tex.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-tex.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-tex.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-tex-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-tex-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-tex-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-tex.txt`
unix	0.275	0.542	0.279	0.572	0.317	0.621	0.326	0.616	0.422	0.780	0.406	0.754
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-unix.flat \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.bm25-flat.cqadupstack-unix.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-flat.cqadupstack-unix.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-unix.multifield \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.bm25-multifield.cqadupstack-unix.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bm25-multifield.cqadupstack-unix.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-unix.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-unix.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-unix.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.splade-pp-ed.cqadupstack-unix.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-unix.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.contriever-msmarco.cqadupstack-unix.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.contriever-msmarco.cqadupstack-unix.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-unix.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-unix.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-unix.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-unix-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-unix-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-unix-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-unix.txt`
webmasters	0.306	0.582	0.301	0.610	0.317	0.636	0.339	0.703	0.407	0.777	0.407	0.749
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-webmasters.flat \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.bm25-flat.cqadupstack-webmasters.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-flat.cqadupstack-webmasters.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-webmasters.multifield \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.bm25-multifield.cqadupstack-webmasters.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bm25-multifield.cqadupstack-webmasters.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-webmasters.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-webmasters.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-webmasters.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.splade-pp-ed.cqadupstack-webmasters.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-webmasters.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.contriever-msmarco.cqadupstack-webmasters.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.contriever-msmarco.cqadupstack-webmasters.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-webmasters.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-webmasters.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-webmasters.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-webmasters-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-webmasters-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-webmasters-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-webmasters.txt`
wordpress	0.248	0.515	0.256	0.553	0.273	0.595	0.253	0.577	0.355	0.705	0.343	0.694
BM25 Flat BM25 MF SPLADE Contriever BGE Cohere Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-wordpress.flat \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.bm25-flat.cqadupstack-wordpress.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-flat.cqadupstack-wordpress.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-wordpress.multifield \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.bm25-multifield.cqadupstack-wordpress.txt \ --output-format trec \ --hits 1000 --bm25 --remove-query --fields contents=1.0 title=1.0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bm25-multifield.cqadupstack-wordpress.txt` Command to generate run: `python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-cqadupstack-wordpress.splade-pp-ed \ --topics beir-v1.0.0-cqadupstack-wordpress.test.splade-pp-ed \ --output run.beir.splade-pp-ed.cqadupstack-wordpress.txt \ --output-format trec \ --hits 1000 --impact --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.splade-pp-ed.cqadupstack-wordpress.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class contriever --encoder facebook/contriever-msmarco \ --index beir-v1.0.0-cqadupstack-wordpress.contriever-msmarco \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.contriever-msmarco.cqadupstack-wordpress.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.contriever-msmarco.cqadupstack-wordpress.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class auto --encoder BAAI/bge-base-en-v1.5 --l2-norm \ --query-prefix "Represent this sentence for searching relevant passages:" \ --index beir-v1.0.0-cqadupstack-wordpress.bge-base-en-v1.5 \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.bge-base-en-v1.5.faiss.cqadupstack-wordpress.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --index beir-v1.0.0-cqadupstack-wordpress.cohere-embed-english-v3.0 \ --topics beir-v1.0.0-cqadupstack-wordpress-test \ --encoded-queries cohere-embed-english-v3.0-beir-v1.0.0-cqadupstack-wordpress-test \ --output run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt \ --hits 1000 --remove-query` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.10 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.100 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt python -m pyserini.eval.trec_eval \ -c -m recall.1000 beir-v1.0.0-cqadupstack-wordpress-test \ run.beir.cohere-embed-english-v3.0.cqadupstack-wordpress.txt`

References

[1] Ehsan Kamalloo, Nandan Thakur, Carlos Lassance, Xueguang Ma, Jheng-Hong Yang, and Jimmy Lin. Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard. arXiv:2306.07471, June 2023.
[2] Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2353–2359.
[3] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Towards Unsupervised Dense Information Retrieval with Contrastive Learning. arXiv:2112.09118, December 2021.
[4] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597, December 2023.

Programmatic Execution

All experimental runs shown in the above table can be programmatically executed based on the instructions below. To list all the experimental conditions:

python -m pyserini.2cr.beir --list-conditions

These conditions correspond to the table rows above.

For all conditions, just show the commands in a "dry run":

python -m pyserini.2cr.beir --all --display-commands --dry-run

To actually run all the experimental conditions:

python -m pyserini.2cr.beir --all --display-commands

With the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

To show the commands for a specific condition:

python -m pyserini.2cr.beir --condition bm25-flat --display-commands --dry-run

This will generate exactly the commands for a specific condition above (corresponding to a row in the table).

To actually run a specific condition:

python -m pyserini.2cr.beir --condition bm25-flat --display-commands

Again, with the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

Finally, to generate this page:

python -m pyserini.2cr.beir --generate-report --output beir.html

The output file beir.html should be identical to this page.

Pyserini BEIR Regressions

Main Results

CQADupStack Results

References

Programmatic Execution