← home
Github: datasets/antique.py

ir_datasets: ANTIQUE

Index
  1. antique
  2. antique/test
  3. antique/test/non-offensive
  4. antique/train
  5. antique/train/split200-train
  6. antique/train/split200-valid

"antique"

"ANTIQUE is a non-factoid quesiton answering dataset based on the questions and answers of Yahoo! Webscope L6."

  • Documents: Short answer passages (from Yahoo Answers)
  • Queries: Natural language questions (from Yahoo Answers)
  • Dataset Paper
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
Citation
bibtex: @inproceedings{Hashemi2020Antique, title={ANTIQUE: A Non-Factoid Question Answering Benchmark}, author={Helia Hashemi and Mohammad Aliannejadi and Hamed Zamani and Bruce Croft}, booktitle={ECIR}, year={2020} }

"antique/test"

Official test set of the ANTIQUE dataset.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/test')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/test')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1It is completely out of context or does not make any sense.
2It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.
3It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.
4It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".

Example

import ir_datasets
dataset = ir_datasets.load('antique/test')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

"antique/test/non-offensive"

antique/test without a set of queries deemed by the authors of ANTIQUE to be "offensive (and noisy)."

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/test/non-offensive')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/test/non-offensive')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1It is completely out of context or does not make any sense.
2It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.
3It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.
4It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".

Example

import ir_datasets
dataset = ir_datasets.load('antique/test/non-offensive')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

"antique/train"

Official train set of the ANTIQUE dataset.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/train')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/train')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1It is completely out of context or does not make any sense.
2It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.
3It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.
4It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".

Example

import ir_datasets
dataset = ir_datasets.load('antique/train')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

"antique/train/split200-train"

antique/train without the 200 queries used by antique/train/split200-valid.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/train/split200-train')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/train/split200-train')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1It is completely out of context or does not make any sense.
2It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.
3It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.
4It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".

Example

import ir_datasets
dataset = ir_datasets.load('antique/train/split200-train')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

"antique/train/split200-valid"

A held-out subset of 200 queries from antique/train. Use in conjunction with antique/train/split200-train.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/train/split200-valid')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: en

Document type:
GenericDoc: (namedtuple)
  1. doc_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('antique/train/split200-valid')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, text>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1It is completely out of context or does not make any sense.
2It does not answer the question or if it does, it provides anunreasonable answer, however, it is not out of context. Therefore, you cannot accept it as an answer to the question.
3It can be an answer to the question, however, it is notsufficiently convincing. There should be an answer with much better quality for the question.
4It looks reasonable and convincing. Its quality is on parwith or better than the "Possibly Correct Answer". Note that it does not have to provide the same answer as the "PossiblyCorrect Answer".

Example

import ir_datasets
dataset = ir_datasets.load('antique/train/split200-valid')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>