Pyserini: M-BEIR

The two-click^* reproduction matrix below provides commands for reproducing experimental results reported in a number of papers, denoted by the references in square brackets. Instructions for programmatic execution are shown at the bottom of this page (scroll down).

		CLIP-SF-Large [1]				BLIP-FF-Large [1]
	R@5	R@10	S@5	S@10	R@5	R@10	S@5	S@10
union_cirr_task7	0.4378	0.5637	0.4384	0.5640	0.4824	0.5883	0.4827	0.5887
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-cirr_task7-test \ --output-format trec \ --output run.m-beir-cirr_task7.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-cirr_task7-test \ --output-format trec \ --output run.m-beir-cirr_task7.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.union.blip-ff-large.txt`
union_edis_task2	0.3209	0.4536	0.5251	0.6387	0.2969	0.4113	0.4968	0.6023
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-edis_task2-test \ --output-format trec \ --output run.m-beir-edis_task2.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-edis-task2 \ run.m-beir-edis_task2.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-edis-task2 \ run.m-beir-edis_task2.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-edis-task2 \ run.m-beir-edis_task2.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-edis-task2 \ run.m-beir-edis_task2.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-edis_task2-test \ --output-format trec \ --output run.m-beir-edis_task2.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-edis-task2 \ run.m-beir-edis_task2.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-edis-task2 \ run.m-beir-edis_task2.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-edis-task2 \ run.m-beir-edis_task2.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-edis-task2 \ run.m-beir-edis_task2.union.blip-ff-large.txt`
union_fashion200k_task0	0.0731	0.1128	0.1239	0.1786	0.1194	0.1758	0.1914	0.2600
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task0-test \ --output-format trec \ --output run.m-beir-fashion200k_task0.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task0-test \ --output-format trec \ --output run.m-beir-fashion200k_task0.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.union.blip-ff-large.txt
union_fashion200k_task3	0.1264	0.1806	0.1264	0.1806	0.1921	0.2727	0.1921	0.2727
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task3-test \ --output-format trec \ --output run.m-beir-fashion200k_task3.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task3-test \ --output-format trec \ --output run.m-beir-fashion200k_task3.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.union.blip-ff-large.txt
union_fashioniq_task7	0.1755	0.2425	0.1756	0.2425	0.2192	0.2901	0.2194	0.2902
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-fashioniq_task7-test \ --output-format trec \ --output run.m-beir-fashioniq_task7.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-fashioniq_task7-test \ --output-format trec \ --output run.m-beir-fashioniq_task7.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.union.blip-ff-large.txt
union_infoseek_task6	0.1336	0.2045	0.2222	0.2921	0.1051	0.1582	0.1999	0.2677
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-infoseek_task6-test \ --output-format trec \ --output run.m-beir-infoseek_task6.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-infoseek_task6-test \ --output-format trec \ --output run.m-beir-infoseek_task6.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.union.blip-ff-large.txt
union_infoseek_task8	0.2668	0.3873	0.4435	0.5336	0.1271	0.1961	0.2623	0.3371
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-infoseek_task8-test \ --output-format trec \ --output run.m-beir-infoseek_task8.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-infoseek_task8-test \ --output-format trec \ --output run.m-beir-infoseek_task8.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.union.blip-ff-large.txt
union_mscoco_task0	0.7162	0.8009	0.7173	0.8022	0.6684	0.7619	0.6695	0.7632
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-mscoco_task0-test \ --output-format trec \ --output run.m-beir-mscoco_task0.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-mscoco_task0-test \ --output-format trec \ --output run.m-beir-mscoco_task0.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.union.blip-ff-large.txt`
union_mscoco_task3	0.5227	0.6717	0.9160	0.9580	0.4805	0.6465	0.8972	0.9498
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-mscoco_task3-test \ --output-format trec \ --output run.m-beir-mscoco_task3.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-mscoco_task3-test \ --output-format trec \ --output run.m-beir-mscoco_task3.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.union.blip-ff-large.txt`
union_nights_task4	0.3170	0.5231	0.3170	0.5231	0.3189	0.5472	0.3189	0.5472
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-nights_task4-test \ --output-format trec \ --output run.m-beir-nights_task4.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-nights-task4 \ run.m-beir-nights_task4.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-nights-task4 \ run.m-beir-nights_task4.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-nights-task4 \ run.m-beir-nights_task4.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-nights-task4 \ run.m-beir-nights_task4.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-nights_task4-test \ --output-format trec \ --output run.m-beir-nights_task4.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-nights-task4 \ run.m-beir-nights_task4.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-nights-task4 \ run.m-beir-nights_task4.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-nights-task4 \ run.m-beir-nights_task4.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-nights-task4 \ run.m-beir-nights_task4.union.blip-ff-large.txt`
union_oven_task6	0.2627	0.3489	0.3960	0.4559	0.2339	0.3093	0.3949	0.4542
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-oven_task6-test \ --output-format trec \ --output run.m-beir-oven_task6.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task6 \ run.m-beir-oven_task6.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task6 \ run.m-beir-oven_task6.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task6 \ run.m-beir-oven_task6.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task6 \ run.m-beir-oven_task6.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-oven_task6-test \ --output-format trec \ --output run.m-beir-oven_task6.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task6 \ run.m-beir-oven_task6.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task6 \ run.m-beir-oven_task6.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task6 \ run.m-beir-oven_task6.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task6 \ run.m-beir-oven_task6.union.blip-ff-large.txt`
union_oven_task8	0.2945	0.4436	0.6019	0.6587	0.2386	0.3513	0.5603	0.6116
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-oven_task8-test \ --output-format trec \ --output run.m-beir-oven_task8.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task8 \ run.m-beir-oven_task8.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task8 \ run.m-beir-oven_task8.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task8 \ run.m-beir-oven_task8.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task8 \ run.m-beir-oven_task8.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-oven_task8-test \ --output-format trec \ --output run.m-beir-oven_task8.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task8 \ run.m-beir-oven_task8.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task8 \ run.m-beir-oven_task8.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task8 \ run.m-beir-oven_task8.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task8 \ run.m-beir-oven_task8.union.blip-ff-large.txt`
union_visualnews_task0	0.4254	0.5159	0.4254	0.5159	0.2265	0.2960	0.2265	0.2960
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-visualnews_task0-test \ --output-format trec \ --output run.m-beir-visualnews_task0.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-visualnews_task0-test \ --output-format trec \ --output run.m-beir-visualnews_task0.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.union.blip-ff-large.txt
union_visualnews_task3	0.3867	0.4916	0.3867	0.4916	0.2095	0.2882	0.2095	0.2882
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-visualnews_task3-test \ --output-format trec \ --output run.m-beir-visualnews_task3.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-visualnews_task3-test \ --output-format trec \ --output run.m-beir-visualnews_task3.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.union.blip-ff-large.txt
union_webqa_task1	0.5818	0.6801	0.8342	0.8969	0.5147	0.6035	0.7796	0.8415
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-webqa_task1-test \ --output-format trec \ --output run.m-beir-webqa_task1.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-webqa_task1-test \ --output-format trec \ --output run.m-beir-webqa_task1.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.union.blip-ff-large.txt`
union_webqa_task2	0.6806	0.7744	0.7762	0.8534	0.6810	0.7752	0.7718	0.8542
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-union.clip-sf-large \ --topics-format mbeir \ --topics m-beir-webqa_task2-test \ --output-format trec \ --output run.m-beir-webqa_task2.union.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-union.blip-ff-large \ --topics-format mbeir \ --topics m-beir-webqa_task2-test \ --output-format trec \ --output run.m-beir-webqa_task2.union.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.union.blip-ff-large.txt`
local_cirr_task7	0.4526	0.5817	0.4532	0.5820	0.5030	0.6133	0.5034	0.6137
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-cirr_task7.clip-sf-large \ --topics-format mbeir \ --topics m-beir-cirr_task7-test \ --output-format trec \ --output run.m-beir-cirr_task7.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-cirr_task7.blip-ff-large \ --topics-format mbeir \ --topics m-beir-cirr_task7-test \ --output-format trec \ --output run.m-beir-cirr_task7.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-cirr-task7 \ run.m-beir-cirr_task7.local.blip-ff-large.txt`
local_edis_task2	0.3291	0.4623	0.5319	0.6455	0.3055	0.4163	0.5051	0.6066
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-edis_task2.clip-sf-large \ --topics-format mbeir \ --topics m-beir-edis_task2-test \ --output-format trec \ --output run.m-beir-edis_task2.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-edis-task2 \ run.m-beir-edis_task2.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-edis-task2 \ run.m-beir-edis_task2.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-edis-task2 \ run.m-beir-edis_task2.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-edis-task2 \ run.m-beir-edis_task2.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-edis_task2.blip-ff-large \ --topics-format mbeir \ --topics m-beir-edis_task2-test \ --output-format trec \ --output run.m-beir-edis_task2.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-edis-task2 \ run.m-beir-edis_task2.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-edis-task2 \ run.m-beir-edis_task2.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-edis-task2 \ run.m-beir-edis_task2.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-edis-task2 \ run.m-beir-edis_task2.local.blip-ff-large.txt`
local_fashion200k_task0	0.0734	0.1144	0.1245	0.1803	0.1211	0.1770	0.1931	0.2612
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-fashion200k_task0.clip-sf-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task0-test \ --output-format trec \ --output run.m-beir-fashion200k_task0.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-fashion200k_task0.blip-ff-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task0-test \ --output-format trec \ --output run.m-beir-fashion200k_task0.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task0 \ run.m-beir-fashion200k_task0.local.blip-ff-large.txt
local_fashion200k_task3	0.1262	0.1810	0.1262	0.1810	0.2082	0.2849	0.2082	0.2849
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-fashion200k_task3.clip-sf-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task3-test \ --output-format trec \ --output run.m-beir-fashion200k_task3.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-fashion200k_task3.blip-ff-large \ --topics-format mbeir \ --topics m-beir-fashion200k_task3-test \ --output-format trec \ --output run.m-beir-fashion200k_task3.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashion200k-task3 \ run.m-beir-fashion200k_task3.local.blip-ff-large.txt
local_fashioniq_task7	0.1772	0.2455	0.1772	0.2455	0.2222	0.2943	0.2224	0.2944
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-fashioniq_task7.clip-sf-large \ --topics-format mbeir \ --topics m-beir-fashioniq_task7-test \ --output-format trec \ --output run.m-beir-fashioniq_task7.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-fashioniq_task7.blip-ff-large \ --topics-format mbeir \ --topics m-beir-fashioniq_task7-test \ --output-format trec \ --output run.m-beir-fashioniq_task7.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-fashioniq-task7 \ run.m-beir-fashioniq_task7.local.blip-ff-large.txt
local_infoseek_task6	0.1312	0.1991	0.2723	0.3485	0.1028	0.1478	0.2354	0.3026
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-infoseek_task6.clip-sf-large \ --topics-format mbeir \ --topics m-beir-infoseek_task6-test \ --output-format trec \ --output run.m-beir-infoseek_task6.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-infoseek_task6.blip-ff-large \ --topics-format mbeir \ --topics m-beir-infoseek_task6-test \ --output-format trec \ --output run.m-beir-infoseek_task6.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task6 \ run.m-beir-infoseek_task6.local.blip-ff-large.txt
local_infoseek_task8	0.2414	0.3446	0.4906	0.5839	0.1255	0.1914	0.3057	0.3892
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-infoseek_task8.clip-sf-large \ --topics-format mbeir \ --topics m-beir-infoseek_task8-test \ --output-format trec \ --output run.m-beir-infoseek_task8.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-infoseek_task8.blip-ff-large \ --topics-format mbeir \ --topics m-beir-infoseek_task8-test \ --output-format trec \ --output run.m-beir-infoseek_task8.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-infoseek-task8 \ run.m-beir-infoseek_task8.local.blip-ff-large.txt
local_mscoco_task0	0.8074	0.8833	0.8084	0.8846	0.7942	0.8725	0.7955	0.8737
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-mscoco_task0.clip-sf-large \ --topics-format mbeir \ --topics m-beir-mscoco_task0-test \ --output-format trec \ --output run.m-beir-mscoco_task0.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-mscoco_task0.blip-ff-large \ --topics-format mbeir \ --topics m-beir-mscoco_task0-test \ --output-format trec \ --output run.m-beir-mscoco_task0.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task0 \ run.m-beir-mscoco_task0.local.blip-ff-large.txt`
local_mscoco_task3	0.5383	0.6852	0.9202	0.9596	0.5209	0.6631	0.9086	0.9546
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-mscoco_task3.clip-sf-large \ --topics-format mbeir \ --topics m-beir-mscoco_task3-test \ --output-format trec \ --output run.m-beir-mscoco_task3.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-mscoco_task3.blip-ff-large \ --topics-format mbeir \ --topics m-beir-mscoco_task3-test \ --output-format trec \ --output run.m-beir-mscoco_task3.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-mscoco-task3 \ run.m-beir-mscoco_task3.local.blip-ff-large.txt`
local_nights_task4	0.3175	0.5222	0.3175	0.5222	0.3189	0.5472	0.3189	0.5472
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-nights_task4.clip-sf-large \ --topics-format mbeir \ --topics m-beir-nights_task4-test \ --output-format trec \ --output run.m-beir-nights_task4.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-nights-task4 \ run.m-beir-nights_task4.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-nights-task4 \ run.m-beir-nights_task4.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-nights-task4 \ run.m-beir-nights_task4.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-nights-task4 \ run.m-beir-nights_task4.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-nights_task4.blip-ff-large \ --topics-format mbeir \ --topics m-beir-nights_task4-test \ --output-format trec \ --output run.m-beir-nights_task4.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-nights-task4 \ run.m-beir-nights_task4.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-nights-task4 \ run.m-beir-nights_task4.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-nights-task4 \ run.m-beir-nights_task4.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-nights-task4 \ run.m-beir-nights_task4.local.blip-ff-large.txt`
local_oven_task6	0.2017	0.2333	0.4600	0.5348	0.1955	0.2267	0.4261	0.5040
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-oven_task6.clip-sf-large \ --topics-format mbeir \ --topics m-beir-oven_task6-test \ --output-format trec \ --output run.m-beir-oven_task6.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task6 \ run.m-beir-oven_task6.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task6 \ run.m-beir-oven_task6.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task6 \ run.m-beir-oven_task6.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task6 \ run.m-beir-oven_task6.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-oven_task6.blip-ff-large \ --topics-format mbeir \ --topics m-beir-oven_task6-test \ --output-format trec \ --output run.m-beir-oven_task6.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task6 \ run.m-beir-oven_task6.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task6 \ run.m-beir-oven_task6.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task6 \ run.m-beir-oven_task6.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task6 \ run.m-beir-oven_task6.local.blip-ff-large.txt`
local_oven_task8	0.1408	0.1537	0.6888	0.7471	0.1256	0.1382	0.5625	0.6322
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-oven_task8.clip-sf-large \ --topics-format mbeir \ --topics m-beir-oven_task8-test \ --output-format trec \ --output run.m-beir-oven_task8.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task8 \ run.m-beir-oven_task8.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task8 \ run.m-beir-oven_task8.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task8 \ run.m-beir-oven_task8.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task8 \ run.m-beir-oven_task8.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-oven_task8.blip-ff-large \ --topics-format mbeir \ --topics m-beir-oven_task8-test \ --output-format trec \ --output run.m-beir-oven_task8.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-oven-task8 \ run.m-beir-oven_task8.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-oven-task8 \ run.m-beir-oven_task8.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-oven-task8 \ run.m-beir-oven_task8.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-oven-task8 \ run.m-beir-oven_task8.local.blip-ff-large.txt`
local_visualnews_task0	0.4284	0.5190	0.4284	0.5190	0.2305	0.2996	0.2305	0.2996
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-visualnews_task0.clip-sf-large \ --topics-format mbeir \ --topics m-beir-visualnews_task0-test \ --output-format trec \ --output run.m-beir-visualnews_task0.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-visualnews_task0.blip-ff-large \ --topics-format mbeir \ --topics m-beir-visualnews_task0-test \ --output-format trec \ --output run.m-beir-visualnews_task0.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task0 \ run.m-beir-visualnews_task0.local.blip-ff-large.txt
local_visualnews_task3	0.4261	0.5188	0.4261	0.5188	0.2291	0.3016	0.2291	0.3016
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-visualnews_task3.clip-sf-large \ --topics-format mbeir \ --topics m-beir-visualnews_task3-test \ --output-format trec \ --output run.m-beir-visualnews_task3.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.clip-sf-large.txt Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-visualnews_task3.blip-ff-large \ --topics-format mbeir \ --topics m-beir-visualnews_task3-test \ --output-format trec \ --output run.m-beir-visualnews_task3.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-visualnews-task3 \ run.m-beir-visualnews_task3.local.blip-ff-large.txt
local_webqa_task1	0.5893	0.6870	0.8387	0.9014	0.5228	0.6166	0.7866	0.8485
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-webqa_task1.clip-sf-large \ --topics-format mbeir \ --topics m-beir-webqa_task1-test \ --output-format trec \ --output run.m-beir-webqa_task1.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-webqa_task1.blip-ff-large \ --topics-format mbeir \ --topics m-beir-webqa_task1-test \ --output-format trec \ --output run.m-beir-webqa_task1.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task1 \ run.m-beir-webqa_task1.local.blip-ff-large.txt`
local_webqa_task2	0.6886	0.7822	0.7853	0.8606	0.6945	0.7905	0.7845	0.8650
CLIP-SF-Large BLIP-FF-Large Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder clip_sf_large \ --index m-beir-webqa_task2.clip-sf-large \ --topics-format mbeir \ --topics m-beir-webqa_task2-test \ --output-format trec \ --output run.m-beir-webqa_task2.local.clip-sf-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.clip-sf-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.clip-sf-large.txt` Command to generate run: `python -m pyserini.search.faiss \ --threads 16 --batch-size 512 \ --encoder-class uniir \ --encoder blip_ff_large \ --index m-beir-webqa_task2.blip-ff-large \ --topics-format mbeir \ --topics m-beir-webqa_task2-test \ --output-format trec \ --output run.m-beir-webqa_task2.local.blip-ff-large.txt \ --hits 1000 --fp16 --device cuda:0 --faiss-device cuda:0` Evaluation commands: `python -m pyserini.eval.trec_eval \ -c -m recall.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m recall.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.5 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.blip-ff-large.txt python -m pyserini.eval.trec_eval \ -c -m success.10 m-beir-webqa-task2 \ run.m-beir-webqa_task2.local.blip-ff-large.txt`

Pyserini: M-BEIR

References

Programmatic Execution