← home
Github: datasets/msmarco_passage.py

ir_datasets: MSMARCO (passage)

Index
  1. msmarco-passage
  2. msmarco-passage/dev
  3. msmarco-passage/dev/judged
  4. msmarco-passage/dev/small
  5. msmarco-passage/eval
  6. msmarco-passage/eval/small
  7. msmarco-passage/train
  8. msmarco-passage/train/judged
  9. msmarco-passage/train/medical
  10. msmarco-passage/train/split200-train
  11. msmarco-passage/train/split200-valid
  12. msmarco-passage/trec-dl-2019
  13. msmarco-passage/trec-dl-2019/judged
  14. msmarco-passage/trec-dl-2020
  15. msmarco-passage/trec-dl-2020/judged

"msmarco-passage"

A passage ranking benchmark with a collection of 8.8 million passages and question queries. Most relevance judgments are shallow (typically at most 1-2 per query), but the TREC Deep Learning track adds deep judgments. Evaluation typically conducted using MRR@10.

Note that the original document source files for this collection contain a double-encoding error that cause strange sequences like "å¬" and "ðºð". These are automatically corrrected (properly converting previous examples to "公" and "🇺🇸").

docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
Citation
bibtex: @inproceedings{Bajaj2016MSMA, title={MS MARCO: A Human Generated MAchine Reading COmprehension Dataset}, author={Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, Tong Wang}, booktitle={InCoCo@NIPS}, year={2016} }

"msmarco-passage/dev"

Official dev set.

scoreddocs are the top 1000 results from BM25. These are used for the "re-ranking" setting. Note that these are sub-sampled to about 1/8 of the total avaialable dev queries by the MSMARCO authors for faster evaluation. The BM25 scores from scoreddocs are not available (all have a score of 0).

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>

"msmarco-passage/dev/judged"

Subset of msmarco-passage/dev that only includes queries that have at least one qrel.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev/judged')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev/judged')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev/judged')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev/judged')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>

"msmarco-passage/dev/small"

Official "small" version of the dev set, consisting of 6,980 queries (6.9% of the full dev set).

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev/small')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev/small')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/dev/small')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

"msmarco-passage/eval"

Official eval set for submission to MS MARCO leaderboard. Relevance judgments are hidden.

scoreddocs are the top 1000 results from BM25. These are used for the "re-ranking" setting. Note that these are sub-sampled to about 1/8 of the total avaialable eval queries by the MSMARCO authors for faster evaluation. The BM25 scores from scoreddocs are not available (all have a score of 0).

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/eval')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/eval')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/eval')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>

"msmarco-passage/eval/small"

Official "small" version of the eval set, consisting of 6,837 queries (6.8% of the full eval set).

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/eval/small')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/eval/small')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>

"msmarco-passage/train"

Official train set.

Not all queries have relevance judgments. Use msmarco-passage/train/judged for a filtered list that only includes documents that have at least one qrel.

scoreddocs are the top 1000 results from BM25. These are used for the "re-ranking" setting. Note that these are sub-sampled to about 1/8 of the total avaialable train queries by the MSMARCO authors for faster evaluation. The BM25 scores from scoreddocs are not available (all have a score of 0).

docpairs provides access to the "official" sequence for pairwise training.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>
docpairs
Document Pair type:
GenericDocPair: (namedtuple)
  1. query_id: str
  2. doc_id_a: str
  3. doc_id_b: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train')
for docpair in dataset.docpairs_iter():
    docpair # namedtuple<query_id, doc_id_a, doc_id_b>

"msmarco-passage/train/judged"

Subset of msmarco-passage/train that only includes queries that have at least one qrel.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/judged')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/judged')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/judged')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/judged')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>
docpairs
Document Pair type:
GenericDocPair: (namedtuple)
  1. query_id: str
  2. doc_id_a: str
  3. doc_id_b: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/judged')
for docpair in dataset.docpairs_iter():
    docpair # namedtuple<query_id, doc_id_a, doc_id_b>

"msmarco-passage/train/medical"

Subset of msmarco-passage/train that only includes queries that have a layman or expert medical term. Note that this includes about 20% false matches due to terms with multiple senses.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/medical')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/medical')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/medical')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/medical')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>
docpairs
Document Pair type:
GenericDocPair: (namedtuple)
  1. query_id: str
  2. doc_id_a: str
  3. doc_id_b: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/medical')
for docpair in dataset.docpairs_iter():
    docpair # namedtuple<query_id, doc_id_a, doc_id_b>
Citation
bibtex: @inproceedings{macavaney:emnlp2020-sledge, author = {MacAvaney, Sean and Cohan, Arman and Goharian, Nazli}, title = {SLEDGE-Zero: A Zero-Shot Baseline for COVID-19 Literature Search}, booktitle = {EMNLP}, year = {2020} }

"msmarco-passage/train/split200-train"

Subset of msmarco-passage/train without 200 queries that are meant to be used as a small validation set. From various works.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-train')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-train')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-train')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-train')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>
docpairs
Document Pair type:
GenericDocPair: (namedtuple)
  1. query_id: str
  2. doc_id_a: str
  3. doc_id_b: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-train')
for docpair in dataset.docpairs_iter():
    docpair # namedtuple<query_id, doc_id_a, doc_id_b>

"msmarco-passage/train/split200-valid"

Subset of msmarco-passage/train with only 200 queries that are meant to be used as a small validation set. From various works.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-valid')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-valid')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Labeled by crowd worker as relevant

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-valid')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-valid')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>
docpairs
Document Pair type:
GenericDocPair: (namedtuple)
  1. query_id: str
  2. doc_id_a: str
  3. doc_id_b: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train/split200-valid')
for docpair in dataset.docpairs_iter():
    docpair # namedtuple<query_id, doc_id_a, doc_id_b>

"msmarco-passage/trec-dl-2019"

Queries from the TREC Deep Learning (DL) 2019 shared task, which were sampled from msmarco-passage/eval. A subset of these queries were judged by NIST assessors, (filtered list available in msmarco-passage/trec-dl-2019/judged).

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
0Irrelevant: The passage has nothing to do with the query.
1Related: The passage seems related to the query but does not answer it.
2Highly relevant: The passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information.
3Perfectly relevant: The passage is dedicated to the query and contains the exact answer.

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>
Citation
bibtex: @inproceedings{Craswell2020OverviewOT, title={Overview of the TREC 2019 deep learning track}, author={Nick Craswell and Bhaskar Mitra and Emine Yilmaz and Daniel Campos and Ellen Voorhees}, booktitle={TREC 2019}, year={2019} }

"msmarco-passage/trec-dl-2019/judged"

Subset of msmarco-passage/trec-dl-2019, only including queries with qrels.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019/judged')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019/judged')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
0Irrelevant: The passage has nothing to do with the query.
1Related: The passage seems related to the query but does not answer it.
2Highly relevant: The passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information.
3Perfectly relevant: The passage is dedicated to the query and contains the exact answer.

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019/judged')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019/judged')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>

"msmarco-passage/trec-dl-2020"

Queries from the TREC Deep Learning (DL) 2020 shared task, which were sampled from msmarco-passage/eval. A subset of these queries were judged by NIST assessors, (filtered list available in msmarco-passage/trec-dl-2020/judged).

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
0Irrelevant: The passage has nothing to do with the query.
1Related: The passage seems related to the query but does not answer it.
2Highly relevant: The passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information.
3Perfectly relevant: The passage is dedicated to the query and contains the exact answer.

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>
Citation
bibtex: @inproceedings{Craswell2021OverviewOT, title={Overview of the TREC 2020 deep learning track}, author={Nick Craswell and Bhaskar Mitra and Emine Yilmaz and Daniel Campos}, booktitle={TREC}, year={2020} }

"msmarco-passage/trec-dl-2020/judged"

Subset of msmarco-passage/trec-dl-2020, only including queries with qrels.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020/judged')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020/judged')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
0Irrelevant: The passage has nothing to do with the query.
1Related: The passage seems related to the query but does not answer it.
2Highly relevant: The passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information.
3Perfectly relevant: The passage is dedicated to the query and contains the exact answer.

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020/judged')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>
scoreddocs
Scored Document type:
GenericScoredDoc: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. score: float

Example

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2020/judged')
for scoreddoc in dataset.scoreddocs_iter():
    scoreddoc # namedtuple<query_id, doc_id, score>