Pyserini: DSE

The two-click* reproductions below provides commands for reproducing DSE experimental results. Instructions for programmatic execution are shown at the bottom of this page (scroll down).

Wiki-SS-NQ Results

DSE-Phi3-vision [1]
Accuracy@1 Accuracy@5 Accuracy@10 Accuracy@20
Command to generate run: 
python -m pyserini.search.faiss \
  --encoder-class dse \
  --encoder Tevatron/dse-phi3-v1.0 --pooling last \
  --index wiki-ss.dse \
  --topics wiki-ss-nq \
  --output run.dse.dse.wiki-ss-nq.txt \
  --hits 20 --device cuda:0 --batch-size 128
Evaluation commands: 
python scripts/dse/evaluate_wiki_ss_run.py --run_file run.dse.dse.wiki-ss-nq.txt --k 1
python scripts/dse/evaluate_wiki_ss_run.py --run_file run.dse.dse.wiki-ss-nq.txt --k 5
python scripts/dse/evaluate_wiki_ss_run.py --run_file run.dse.dse.wiki-ss-nq.txt --k 10
python scripts/dse/evaluate_wiki_ss_run.py --run_file run.dse.dse.wiki-ss-nq.txt --k 20

SlideVQA Results

DSE-Phi3-vision [1]
nDCG@10 Recall@10
Command to generate run: 
python -m pyserini.search.faiss \
  --encoder-class dse \
  --encoder Tevatron/dse-phi3-v1.0 --pooling last \
  --index slidevqa.dse \
  --topics slidevqa \
  --output run.dse.dse.slidevqa.txt \
  --hits 20 --device cuda:0 --batch-size 128
Evaluation commands: 
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 slidevqa run.dse.dse.slidevqa.txt
python -m pyserini.eval.trec_eval -c -m recall.10 slidevqa run.dse.dse.slidevqa.txt

References

Programmatic Execution

All experimental runs shown in the above table can be programmatically executed based on the instructions below. To list all the experimental conditions:

python -m pyserini.2cr.dse --list-conditions

These conditions correspond to the table rows above.

For all conditions, just show the commands in a "dry run":

python -m pyserini.2cr.dse --all --display-commands --dry-run

To actually run all the experimental conditions:

python -m pyserini.2cr.dse --all --display-commands

With the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

To show the commands for a specific condition:

python -m pyserini.2cr.dse --condition dse --display-commands --dry-run

This will generate exactly the commands for a specific condition above (corresponding to a row in the table).

To actually run a specific condition:

python -m pyserini.2cr.dse --condition dse --display-commands

Again, with the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

Finally, to generate this page:

python -m pyserini.2cr.dse --generate-report --output dse.html

The output file dse.html should be identical to this page.