`ir_datasets`: MSMARCO (passage, version 2)

Index

msmarco-passage-v2
msmarco-passage-v2/dev1
msmarco-passage-v2/dev2
msmarco-passage-v2/train
msmarco-passage-v2/trec-dl-2021
msmarco-passage-v2/trec-dl-2021/judged

`"msmarco-passage-v2"`

Version 2 of the MS MARCO passage ranking dataset. The corpus contains 138M passages, which can be linked up with documents in msmarco-document-v2.

Version 1 of dataset: msmarco-passage
Documents: Text extracted from web pages
Queries: Natural language questions (from query log)
Dataset Paper

Change Log

On July 21, 2021, the task organizers updated the train, dev1, and dev2 qrels to remove duplicate entries from the files. This should not have change results from evaluation tools, but may result in non-repeatable results if these files were used in another process (e.g., model training). The original qrels file for msmarco-passage-v2/train can be found here to aid in result repeatability.

docs Citation Metadata

138M docs

Language: en

Document type:

MsMarcoV2Passage: (namedtuple)

doc_id: str
text: str
spans: Tuple[Tuple[int,int], ...]
msmarco_document_id: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-passage-v2")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text, spans, msmarco_document_id>

You can find more details about the Python API here.

ir_datasets export msmarco-passage-v2 docs



[doc_id]    [text]    [spans]    [msmarco_document_id]
...

You can find more details about the CLI here.

{
  "docs": {
    "count": 138364198,
    "fields": {
      "doc_id": {
        "max_len": 28,
        "common_prefix": "msmarco_passage_"
      }
    }
  }
}

`"msmarco-passage-v2/dev1"`

Official dev1 set with 3,903 queries.

Note that that qrels in this dataset are not directly human-assessed; labels from msmarco-passage are mapped to documents via URL, these documents are re-passaged, and then the best approximate match is identified.

Official evaluation measures: RR@10

queries docs qrels scoreddocs Citation Metadata

3.9K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-passage-v2/dev1")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-passage-v2/dev1 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Based on mapping from v1 of MS MARCO	`4.0K`	100.0%

{
  "docs": {
    "count": 138364198,
    "fields": {
      "doc_id": {
        "max_len": 28,
        "common_prefix": "msmarco_passage_"
      }
    }
  },
  "queries": {
    "count": 3903
  },
  "qrels": {
    "count": 4009,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 4009
        }
      }
    }
  },
  "scoreddocs": {
    "count": 390300
  }
}

`"msmarco-passage-v2/dev2"`

Official dev2 set with 4,281 queries.

Official evaluation measures: RR@10

queries docs qrels scoreddocs Citation Metadata

4.3K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-passage-v2/dev2")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-passage-v2/dev2 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Based on mapping from v1 of MS MARCO	`4.4K`	100.0%

{
  "docs": {
    "count": 138364198,
    "fields": {
      "doc_id": {
        "max_len": 28,
        "common_prefix": "msmarco_passage_"
      }
    }
  },
  "queries": {
    "count": 4281
  },
  "qrels": {
    "count": 4411,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 4411
        }
      }
    }
  },
  "scoreddocs": {
    "count": 428100
  }
}

`"msmarco-passage-v2/train"`

Official train set with 277,144 queries.

Official evaluation measures: RR@10

queries docs qrels scoreddocs Citation Metadata

277K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-passage-v2/train")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-passage-v2/train queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Based on mapping from v1 of MS MARCO	`284K`	100.0%

{
  "docs": {
    "count": 138364198,
    "fields": {
      "doc_id": {
        "max_len": 28,
        "common_prefix": "msmarco_passage_"
      }
    }
  },
  "queries": {
    "count": 277144
  },
  "qrels": {
    "count": 284212,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 284212
        }
      }
    }
  },
  "scoreddocs": {
    "count": 27713673
  }
}

`"msmarco-passage-v2/trec-dl-2021"`

Official topics for the TREC Deep Learning (DL) 2021 shared task.

Note that at this time, qrels are only available to those with TREC active participant login credentials.

Official evaluation measures: AP@100, nDCG@10, P(rel=2)@10, RR(rel=2)

queries docs qrels scoreddocs Metadata

477 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-passage-v2/trec-dl-2021")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-passage-v2/trec-dl-2021 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: The passage has nothing to do with the query.	`4.3K`	40.1%
1	Related: The passage seems related to the query but does not answer it.	`3.1K`	28.3%
2	Highly relevant: The passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information.	`2.3K`	21.6%
3	Perfectly relevant: The passage is dedicated to the query and contains the exact answer.	`1.1K`	10.0%

{
  "docs": {
    "count": 138364198,
    "fields": {
      "doc_id": {
        "max_len": 28,
        "common_prefix": "msmarco_passage_"
      }
    }
  },
  "queries": {
    "count": 477
  },
  "qrels": {
    "count": 10828,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 4338,
          "3": 1086,
          "1": 3063,
          "2": 2341
        }
      }
    }
  },
  "scoreddocs": {
    "count": 47700
  }
}

`"msmarco-passage-v2/trec-dl-2021/judged"`

msmarco-passage-v2/trec-dl-2021, but filtered down to the 53 queries with qrels.

Note that at this time, this is only available to those with TREC active participant login credentials.

Official evaluation measures: AP@100, nDCG@10, P(rel=2)@10, RR(rel=2)

queries docs qrels scoreddocs Metadata

53 queries

Language: multiple/other/unknown

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-passage-v2/trec-dl-2021/judged")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-passage-v2/trec-dl-2021/judged queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: The passage has nothing to do with the query.	`4.3K`	40.1%
1	Related: The passage seems related to the query but does not answer it.	`3.1K`	28.3%
2	Highly relevant: The passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information.	`2.3K`	21.6%
3	Perfectly relevant: The passage is dedicated to the query and contains the exact answer.	`1.1K`	10.0%

{
  "docs": {
    "count": 138364198,
    "fields": {
      "doc_id": {
        "max_len": 28,
        "common_prefix": "msmarco_passage_"
      }
    }
  },
  "queries": {
    "count": 53
  },
  "qrels": {
    "count": 10828,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 4338,
          "3": 1086,
          "1": 3063,
          "2": 2341
        }
      }
    }
  },
  "scoreddocs": {
    "count": 5300
  }
}

ir_datasets: MSMARCO (passage, version 2)

"msmarco-passage-v2"

"msmarco-passage-v2/dev1"

"msmarco-passage-v2/dev2"

"msmarco-passage-v2/train"

"msmarco-passage-v2/trec-dl-2021"

"msmarco-passage-v2/trec-dl-2021/judged"

`ir_datasets`: MSMARCO (passage, version 2)

`"msmarco-passage-v2"`

`"msmarco-passage-v2/dev1"`

`"msmarco-passage-v2/dev2"`

`"msmarco-passage-v2/train"`

`"msmarco-passage-v2/trec-dl-2021"`

`"msmarco-passage-v2/trec-dl-2021/judged"`