ANTIQUE - ir_datasets

`"antique"`

"ANTIQUE is a non-factoid quesiton answering dataset based on the questions and answers of Yahoo! Webscope L6."

Documents: Short answer passages (from Yahoo Answers)
Queries: Natural language questions (from Yahoo Answers)
Dataset Paper

docs Citation Metadata

404K docs

Language: en

Document type:

GenericDoc: (namedtuple)

doc_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("antique")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>

You can find more details about the Python API here.

ir_datasets export antique docs



[doc_id]    [text]
...

You can find more details about the CLI here.

{
  "docs": {
    "count": 403666,
    "fields": {
      "doc_id": {
        "max_len": 10,
        "common_prefix": ""
      }
    }
  }
}

`"antique/test"`

Official test set of the ANTIQUE dataset.

queries docs qrels Citation Metadata

200 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("antique/test")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export antique/test queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	It is completely out of context or does not make any sense.	`1.6K`	24.9%
2	It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.	`2.4K`	36.7%
3	It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.	`1.2K`	18.2%
4	It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".	`1.3K`	20.2%

{
  "docs": {
    "count": 403666,
    "fields": {
      "doc_id": {
        "max_len": 10,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 200
  },
  "qrels": {
    "count": 6589,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "4": 1334,
          "1": 1642,
          "2": 2417,
          "3": 1196
        }
      }
    }
  }
}

`"antique/test/non-offensive"`

antique/test without a set of queries deemed by the authors of ANTIQUE to be "offensive (and noisy)."

queries docs qrels Citation Metadata

176 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("antique/test/non-offensive")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export antique/test/non-offensive queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	It is completely out of context or does not make any sense.	`1.4K`	24.5%
2	It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.	`2.1K`	36.5%
3	It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.	`1.0K`	18.2%
4	It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".	`1.2K`	20.8%

{
  "docs": {
    "count": 403666,
    "fields": {
      "doc_id": {
        "max_len": 10,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 176
  },
  "qrels": {
    "count": 5752,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "4": 1195,
          "1": 1407,
          "2": 2101,
          "3": 1049
        }
      }
    }
  }
}

`"antique/train"`

Official train set of the ANTIQUE dataset.

queries docs qrels Citation Metadata

2.4K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("antique/train")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export antique/train queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	It is completely out of context or does not make any sense.	`1.3K`	4.6%
2	It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.	`6.3K`	23.1%
3	It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.	`8.1K`	29.5%
4	It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".	`12K`	42.8%

{
  "docs": {
    "count": 403666,
    "fields": {
      "doc_id": {
        "max_len": 10,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 2426
  },
  "qrels": {
    "count": 27422,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "4": 11733,
          "3": 8080,
          "2": 6337,
          "1": 1272
        }
      }
    }
  }
}

`"antique/train/split200-train"`

antique/train without the 200 queries used by antique/train/split200-valid.

queries docs qrels Citation Metadata

2.2K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("antique/train/split200-train")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export antique/train/split200-train queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	It is completely out of context or does not make any sense.	`1.2K`	4.6%
2	It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.	`5.8K`	23.1%
3	It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.	`7.4K`	29.5%
4	It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".	`11K`	42.7%

{
  "docs": {
    "count": 403666,
    "fields": {
      "doc_id": {
        "max_len": 10,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 2226
  },
  "qrels": {
    "count": 25229,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "4": 10782,
          "3": 7447,
          "2": 5829,
          "1": 1171
        }
      }
    }
  }
}

`"antique/train/split200-valid"`

A held-out subset of 200 queries from antique/train. Use in conjunction with antique/train/split200-train.

queries docs qrels Citation Metadata

200 queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("antique/train/split200-valid")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export antique/train/split200-valid queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	It is completely out of context or does not make any sense.	`101`	4.6%
2	It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.	`508`	23.2%
3	It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.	`633`	28.9%
4	It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".	`951`	43.4%

{
  "docs": {
    "count": 403666,
    "fields": {
      "doc_id": {
        "max_len": 10,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 200
  },
  "qrels": {
    "count": 2193,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "4": 951,
          "2": 508,
          "3": 633,
          "1": 101
        }
      }
    }
  }
}

ir_datasets: ANTIQUE

"antique"

"antique/test"

"antique/test/non-offensive"

"antique/train"

"antique/train/split200-train"

"antique/train/split200-valid"

`ir_datasets`: ANTIQUE

`"antique"`

`"antique/test"`

`"antique/test/non-offensive"`

`"antique/train"`

`"antique/train/split200-train"`

`"antique/train/split200-valid"`