Pyserini Reproductions

	nDCG@20, Test Set B	Hausa	Somali	Swahili	Yoruba	Avg
	BM25 Human QT	0.2121	0.1725	0.1727	0.3459	0.2258
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.lucene \ --language ha \ --topics ciral-v1.0-ha-test-b-native \ --index ciral-v1.0-ha \ --output run.ciral.bm25-qt.ha.test-b.txt --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-ha-test-b \ run.ciral.bm25-qt.ha.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --language so \ --topics ciral-v1.0-so-test-b-native \ --index ciral-v1.0-so \ --output run.ciral.bm25-qt.so.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-so-test-b \ run.ciral.bm25-qt.so.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --language sw \ --topics ciral-v1.0-sw-test-b-native \ --index ciral-v1.0-sw \ --output run.ciral.bm25-qt.sw.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-sw-test-b \ run.ciral.bm25-qt.sw.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --language yo \ --topics ciral-v1.0-yo-test-b-native \ --index ciral-v1.0-yo \ --output run.ciral.bm25-qt.yo.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-yo-test-b \ run.ciral.bm25-qt.yo.test-b.txt`
	BM25 Machine DT	0.2124	0.2186	0.2582	0.3700	0.2648
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-ha-test-b \ --index ciral-v1.0-ha-en \ --output run.ciral.bm25-dt.ha.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-ha-test-b \ run.ciral.bm25-dt.ha.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-so-test-b \ --index ciral-v1.0-so-en \ --output run.ciral.bm25-dt.so.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-so-test-b \ run.ciral.bm25-dt.so.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-sw-test-b \ --index ciral-v1.0-sw-en \ --output run.ciral.bm25-dt.sw.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-sw-test-b \ run.ciral.bm25-dt.sw.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-yo-test-b \ --index ciral-v1.0-yo-en \ --output run.ciral.bm25-dt.yo.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-yo-test-b \ run.ciral.bm25-dt.yo.test-b.txt`
	mDPR (tied encoders), pre-FT w/ MS MARCO	0.0397	0.0635	0.1227	0.1458	0.0929
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-ha-test-b \ --index ciral-v1.0-ha-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.ha.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-ha-test-b \ run.ciral.mdpr-tied-pft-msmarco.ha.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-so-test-b \ --index ciral-v1.0-so-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.so.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-so-test-b \ run.ciral.mdpr-tied-pft-msmarco.so.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-sw-test-b \ --index ciral-v1.0-sw-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.sw.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-sw-test-b \ run.ciral.mdpr-tied-pft-msmarco.sw.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-yo-test-b \ --index ciral-v1.0-yo-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.yo.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-yo-test-b \ run.ciral.mdpr-tied-pft-msmarco.yo.test-b.txt`
	AfriBERTa, pre-FT w/ MS MARCO FT w/ latin Mr. TyDi	0.2028	0.1682	0.2166	0.1157	0.1758
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-ha-test-b \ --index ciral-v1.0-ha-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.ha.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-ha-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.ha.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-so-test-b \ --index ciral-v1.0-so-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.so.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-so-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.so.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-sw-test-b \ --index ciral-v1.0-sw-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.sw.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-sw-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.sw.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-yo-test-b \ --index ciral-v1.0-yo-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.yo.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-yo-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.yo.test-b.txt`
	RRF Fusion of BM25 Machine DT and AfriBERTa-DPR	0.2935	0.2878	0.3187	0.3435	0.3109
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.ha.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.ha.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.ha.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-ha-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.ha.test-b.txt` Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.so.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.so.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.so.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-so-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.so.test-b.txt` Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.sw.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.sw.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.sw.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-sw-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.sw.test-b.txt` Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.yo.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.yo.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.yo.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m ndcg_cut.20 ciral-v1.0-yo-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.yo.test-b.txt`

	Recall@100, Test Set B	Hausa	Somali	Swahili	Yoruba	Avg
	BM25 Human QT	0.3800	0.3479	0.4166	0.6434	0.4470
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.lucene \ --language ha \ --topics ciral-v1.0-ha-test-b-native \ --index ciral-v1.0-ha \ --output run.ciral.bm25-qt.ha.test-b.txt --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-ha-test-b \ run.ciral.bm25-qt.ha.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --language so \ --topics ciral-v1.0-so-test-b-native \ --index ciral-v1.0-so \ --output run.ciral.bm25-qt.so.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-so-test-b \ run.ciral.bm25-qt.so.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --language sw \ --topics ciral-v1.0-sw-test-b-native \ --index ciral-v1.0-sw \ --output run.ciral.bm25-qt.sw.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-sw-test-b \ run.ciral.bm25-qt.sw.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --language yo \ --topics ciral-v1.0-yo-test-b-native \ --index ciral-v1.0-yo \ --output run.ciral.bm25-qt.yo.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-yo-test-b \ run.ciral.bm25-qt.yo.test-b.txt`
	BM25 Machine DT	0.4394	0.4637	0.4918	0.7348	0.5324
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-ha-test-b \ --index ciral-v1.0-ha-en \ --output run.ciral.bm25-dt.ha.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-ha-test-b \ run.ciral.bm25-dt.ha.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-so-test-b \ --index ciral-v1.0-so-en \ --output run.ciral.bm25-dt.so.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-so-test-b \ run.ciral.bm25-dt.so.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-sw-test-b \ --index ciral-v1.0-sw-en \ --output run.ciral.bm25-dt.sw.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-sw-test-b \ run.ciral.bm25-dt.sw.test-b.txt` Command to generate run: `python -m pyserini.search.lucene \ --topics ciral-v1.0-yo-test-b \ --index ciral-v1.0-yo-en \ --output run.ciral.bm25-dt.yo.test-b.txt \ --batch 128 --threads 16 --bm25 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-yo-test-b \ run.ciral.bm25-dt.yo.test-b.txt`
	mDPR (tied encoders), pre-FT w/ MS MARCO	0.1027	0.1345	0.3019	0.3249	0.2160
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-ha-test-b \ --index ciral-v1.0-ha-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.ha.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-ha-test-b \ run.ciral.mdpr-tied-pft-msmarco.ha.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-so-test-b \ --index ciral-v1.0-so-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.so.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-so-test-b \ run.ciral.mdpr-tied-pft-msmarco.so.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-sw-test-b \ --index ciral-v1.0-sw-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.sw.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-sw-test-b \ run.ciral.mdpr-tied-pft-msmarco.sw.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/mdpr-tied-pft-msmarco \ --topics ciral-v1.0-yo-test-b \ --index ciral-v1.0-yo-mdpr-tied-pft-msmarco \ --output run.ciral.mdpr-tied-pft-msmarco.yo.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-yo-test-b \ run.ciral.mdpr-tied-pft-msmarco.yo.test-b.txt`
	AfriBERTa, pre-FT w/ MS MARCO FT w/ latin Mr. TyDi	0.3900	0.3558	0.4608	0.2907	0.3743
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-ha-test-b \ --index ciral-v1.0-ha-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.ha.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-ha-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.ha.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-so-test-b \ --index ciral-v1.0-so-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.so.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-so-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.so.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-sw-test-b \ --index ciral-v1.0-sw-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.sw.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-sw-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.sw.test-b.txt` Command to generate run: `python -m pyserini.search.faiss \ --encoder-class auto \ --encoder castorini/afriberta-dpr-pft-msmarco-ft-latin-mrtydi \ --topics ciral-v1.0-yo-test-b \ --index ciral-v1.0-yo-afriberta-dpr-ptf-msmarco-ft-latin-mrtydi \ --output run.ciral.afriberta-pft-msmarco-ft-mrtydi.yo.test-b.txt \ --batch 128 --threads 16 --hits 1000` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-yo-test-b \ run.ciral.afriberta-pft-msmarco-ft-mrtydi.yo.test-b.txt`
	RRF Fusion of BM25 Machine DT and AfriBERTa-DPR	0.6007	0.5618	0.7007	0.7525	0.6539
Hausa Somali Swahili Yoruba Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.ha.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.ha.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.ha.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-ha-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.ha.test-b.txt` Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.so.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.so.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.so.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-so-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.so.test-b.txt` Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.sw.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.sw.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.sw.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-sw-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.sw.test-b.txt` Command to generate run: `python -m pyserini.fusion \ --runs run.ciral.bm25-dt.yo.test-b.txt run.ciral.afriberta-pft-msmarco-ft-mrtydi.yo.test-b.txt \ --runtag rrf-afridpr-bmdt --method rrf --rrf.k 60 \ --output run.ciral.bm25-dt-afriberta-dpr-fusion.yo.test-b.txt` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.100 ciral-v1.0-yo-test-b \ run.ciral.bm25-dt-afriberta-dpr-fusion.yo.test-b.txt`

Programmatic Execution

All experimental runs shown in the above table can be programmatically executed based on the instructions below. To list all the experimental conditions:

python -m pyserini.2cr.ciral --list-conditions

Run all languages for a specific condition and show commands:

python -m pyserini.2cr.ciral --condition bm25-qt --display-commands

Run a particular language for a specific condition and show commands:

python -m pyserini.2cr.ciral --condition bm25-qt --language somali --display-commands

Run all languages for all conditions and show commands:

python -m pyserini.2cr.ciral --all --display-commands

With the above commands, run files will be placed in the current directory. Use the option --directory runs to place the runs in a sub-directory.

For a specific condition, just show the commands and do not run:

python -m pyserini.2cr.ciral --condition bm25-qt --display-commands --dry-run

This will generate exactly the commands for a specific condition above (corresponding to a row in the table).

For a specific condition and language, just show the commands and do not run:

python -m pyserini.2cr.ciral --condition bm25-qt --language somali --display-commands --dry-run

For all conditions, just show the commands and do not run and skip evaluation:

python -m pyserini.2cr.ciral --all --display-commands --dry-run --skip-eval

Finally, to generate this page:

python -m pyserini.2cr.ciral --generate-report --output docs/2cr/ciral.html --display-split test-b

The output file ciral.html should be identical to this page.

CIRAL

Programmatic Execution