← home
Github: allenai/ir_datasets

trec_eval & ir_datasets

trec_eval is a popular tool for calculating IR evaluation metrics, and is usually the official measure implementation for TREC shared tasks.

To get started with trec_eval, see this guide.

Basic Usage

ir_datasets export produces qrels in a trec_eval-compatible format by default. To use with trec_eval, first save save the output of the ir_datasets export command as a file, and then use that file as input to trec_eval.

ir_datasets export trec-robust04 qrels > trec-robust04.qrels
trec_eval trec-robust04.qrels path/to/run
runid all run num_q all 249 num_ret all 60639 num_rel all 17412 num_rel_ret all 5293 map all 0.1619 gm_map all 0.0667 Rprec all 0.2199 ...

Note that trec_eval does not accept named pipes as source files, so using process substitution will result in an error.