MSMARCO (document, version 2)

`"msmarco-document-v2"`

Version 2 of the MS MARCO document ranking dataset. The corpus contains 12M documents (roughly 3x as many as version 1).

Version 1 of dataset: msmarco-document
Documents: Text extracted from web pages
Queries: Natural language questions (from query log)
Dataset Paper

docs Citation Metadata

12M docs

Language: en

Document type:

MsMarcoV2Document: (namedtuple)

doc_id: str
url: str
title: str
headings: str
body: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, url, title, headings, body>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2 docs



[doc_id]    [url]    [title]    [headings]    [body]
...

You can find more details about the CLI here.

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  }
}

`"msmarco-document-v2/anchor-text"`

For version 2 of MS MARCO, the anchor text collection enriches 4,821,244 documents with anchor text extracted from six Common Crawl snapshots. To keep the collection size reasonable, we sampled 1,000 anchor texts for documents with more than 1,000 anchor texts (this sampling yields that all anchor text is included for 97% of the documents). The text field contains the anchor texts concatenated and the anchors field contains the anchor texts as list. The raw dataset with additional information (roughly 100GB) is available online.

docs Citation Metadata

4.8M docs

Language: en

Document type:

MsMarcoV2AnchorTextDocument: (namedtuple)

doc_id: str
text: str
anchors: List[str]

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/anchor-text")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text, anchors>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/anchor-text docs



[doc_id]    [text]    [anchors]
...

You can find more details about the CLI here.

{
  "docs": {
    "count": 4821244,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  }
}

`"msmarco-document-v2/dev1"`

Official dev1 set with 4,552 queries.

queries docs qrels scoreddocs Citation Metadata

4.6K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/dev1")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/dev1 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Document contains a passage labeled as relevant in msmarco-passage	`4.7K`	100.0%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 4552
  },
  "qrels": {
    "count": 4702,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 4702
        }
      }
    }
  },
  "scoreddocs": {
    "count": 455200
  }
}

`"msmarco-document-v2/dev2"`

Official dev2 set with 5,000 queries.

queries docs qrels scoreddocs Citation Metadata

5.0K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/dev2")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/dev2 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Document contains a passage labeled as relevant in msmarco-passage	`5.2K`	100.0%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 5000
  },
  "qrels": {
    "count": 5178,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 5178
        }
      }
    }
  },
  "scoreddocs": {
    "count": 500000
  }
}

`"msmarco-document-v2/train"`

Official train set with 322,196 queries.

queries docs qrels scoreddocs Citation Metadata

322K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/train")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/train queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Document contains a passage labeled as relevant in msmarco-passage	`332K`	100.0%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 322196
  },
  "qrels": {
    "count": 331956,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 331956
        }
      }
    }
  },
  "scoreddocs": {
    "count": 32218809
  }
}

`"msmarco-document-v2/trec-dl-2019"`

Queries from the TREC Deep Learning (DL) 2019 shared task, which were sampled from msmarco-document/eval. A subset of these queries were judged by NIST assessors, (filtered list available in msmarco-document-v2/trec-dl-2019/judged).

Shared Task Paper

queries docs qrels Citation Metadata

200 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/trec-dl-2019")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/trec-dl-2019 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: Document does not provide any useful information about the query	`8.2K`	59.0%
1	Relevant: Document provides some information relevant to the query, which may be minimal.	`4.0K`	28.4%
2	Highly relevant: The content of this document provides substantial information on the query.	`1.0K`	7.2%
3	Perfectly relevant: Document is dedicated to the query, it is worthy of being a top result in a search engine.	`745`	5.3%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 200
  },
  "qrels": {
    "count": 13940,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 8229,
          "1": 3957,
          "2": 1009,
          "3": 745
        }
      }
    }
  }
}

`"msmarco-document-v2/trec-dl-2019/judged"`

Subset of msmarco-document-v2/trec-dl-2019, only including queries with qrels.

queries docs qrels Citation Metadata

43 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/trec-dl-2019/judged")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/trec-dl-2019/judged queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: Document does not provide any useful information about the query	`8.2K`	59.0%
1	Relevant: Document provides some information relevant to the query, which may be minimal.	`4.0K`	28.4%
2	Highly relevant: The content of this document provides substantial information on the query.	`1.0K`	7.2%
3	Perfectly relevant: Document is dedicated to the query, it is worthy of being a top result in a search engine.	`745`	5.3%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 43
  },
  "qrels": {
    "count": 13940,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 8229,
          "1": 3957,
          "2": 1009,
          "3": 745
        }
      }
    }
  }
}

`"msmarco-document-v2/trec-dl-2020"`

Queries from the TREC Deep Learning (DL) 2020 shared task, which were sampled from msmarco-document/eval. A subset of these queries were judged by NIST assessors, (filtered list available in msmarco-document-v2/trec-dl-2020/judged).

Shared Task Paper

queries docs qrels Citation Metadata

200 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/trec-dl-2020")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/trec-dl-2020 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: Document does not provide any useful information about the query	`6.4K`	80.2%
1	Relevant: Document provides some information relevant to the query, which may be minimal.	`1.1K`	13.3%
2	Highly relevant: The content of this document provides substantial information on the query.	`279`	3.5%
3	Perfectly relevant: Document is dedicated to the query, it is worthy of being a top result in a search engine.	`233`	2.9%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 200
  },
  "qrels": {
    "count": 7942,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 6371,
          "3": 233,
          "1": 1059,
          "2": 279
        }
      }
    }
  }
}

`"msmarco-document-v2/trec-dl-2020/judged"`

Subset of msmarco-document-v2/trec-dl-2020, only including queries with qrels.

queries docs qrels Citation Metadata

45 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/trec-dl-2020/judged")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/trec-dl-2020/judged queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: Document does not provide any useful information about the query	`6.4K`	80.2%
1	Relevant: Document provides some information relevant to the query, which may be minimal.	`1.1K`	13.3%
2	Highly relevant: The content of this document provides substantial information on the query.	`279`	3.5%
3	Perfectly relevant: Document is dedicated to the query, it is worthy of being a top result in a search engine.	`233`	2.9%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 45
  },
  "qrels": {
    "count": 7942,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 6371,
          "3": 233,
          "1": 1059,
          "2": 279
        }
      }
    }
  }
}

`"msmarco-document-v2/trec-dl-2021"`

Official topics for the TREC Deep Learning (DL) 2021 shared task.

Note that at this time, qrels are only available to those with TREC active participant login credentials.

Official evaluation measures: AP@100, nDCG@10, P@10, RR(rel=2)

queries docs qrels scoreddocs Metadata

477 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/trec-dl-2021")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/trec-dl-2021 queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: Document does not provide any useful information about the query	`4.9K`	37.2%
1	Relevant: Document provides some information relevant to the query, which may be minimal.	`4.2K`	32.0%
2	Highly relevant: The content of this document provides substantial information on the query.	`2.8K`	21.2%
3	Perfectly relevant: Document is dedicated to the query, it is worthy of being a top result in a search engine.	`1.3K`	9.6%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 477
  },
  "qrels": {
    "count": 13058,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "2": 2769,
          "0": 4855,
          "3": 1256,
          "1": 4178
        }
      }
    }
  },
  "scoreddocs": {
    "count": 47700
  }
}

`"msmarco-document-v2/trec-dl-2021/judged"`

msmarco-document-v2/trec-dl-2021, but filtered down to the 57 queries with qrels.

Note that at this time, this is only available to those with TREC active participant login credentials.

Official evaluation measures: AP@100, nDCG@10, P@10, RR(rel=2)

queries docs qrels scoreddocs Metadata

57 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("msmarco-document-v2/trec-dl-2021/judged")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export msmarco-document-v2/trec-dl-2021/judged queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant: Document does not provide any useful information about the query	`4.9K`	37.2%
1	Relevant: Document provides some information relevant to the query, which may be minimal.	`4.2K`	32.0%
2	Highly relevant: The content of this document provides substantial information on the query.	`2.8K`	21.2%
3	Perfectly relevant: Document is dedicated to the query, it is worthy of being a top result in a search engine.	`1.3K`	9.6%

{
  "docs": {
    "count": 11959635,
    "fields": {
      "doc_id": {
        "max_len": 25,
        "common_prefix": "msmarco_doc_"
      }
    }
  },
  "queries": {
    "count": 57
  },
  "qrels": {
    "count": 13058,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "2": 2769,
          "0": 4855,
          "3": 1256,
          "1": 4178
        }
      }
    }
  },
  "scoreddocs": {
    "count": 5700
  }
}

ir_datasets: MSMARCO (document, version 2)

"msmarco-document-v2"

"msmarco-document-v2/anchor-text"

"msmarco-document-v2/dev1"

"msmarco-document-v2/dev2"

"msmarco-document-v2/train"

"msmarco-document-v2/trec-dl-2019"

"msmarco-document-v2/trec-dl-2019/judged"

"msmarco-document-v2/trec-dl-2020"

"msmarco-document-v2/trec-dl-2020/judged"

"msmarco-document-v2/trec-dl-2021"

"msmarco-document-v2/trec-dl-2021/judged"

`ir_datasets`: MSMARCO (document, version 2)

`"msmarco-document-v2"`

`"msmarco-document-v2/anchor-text"`

`"msmarco-document-v2/dev1"`

`"msmarco-document-v2/dev2"`

`"msmarco-document-v2/train"`

`"msmarco-document-v2/trec-dl-2019"`

`"msmarco-document-v2/trec-dl-2019/judged"`

`"msmarco-document-v2/trec-dl-2020"`

`"msmarco-document-v2/trec-dl-2020/judged"`

`"msmarco-document-v2/trec-dl-2021"`

`"msmarco-document-v2/trec-dl-2021/judged"`