Github: datasets/codec.py

`ir_datasets`: CODEC

Index

codec
codec/economics
codec/history
codec/politics

Data Access Information

To use this dataset, you need a copy the document corpus from here.

The process involves emailing a dataset author, who will provide instructions for downloading the dataset.

ir_datasets expects the source file to be copied/linked under ~/.ir_datasets/codec/v1/comets_documents.jsonl.

`"codec"`

CODEC Document Ranking sub-task.

Documents: curated web articles
Queries: challenging, entity-focused queries
Task Repository
See also: kilt/codec, the entity ranking subtask

queries docs qrels Citation Metadata

42 queries

Language: en

Query type:

CodecQuery: (namedtuple)

query_id: str
query: str
domain: str
guidelines: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("codec")
for query in dataset.queries_iter():
    query # namedtuple<query_id, query, domain, guidelines>

You can find more details about the Python API here.

ir_datasets export codec queries



[query_id]    [query]    [domain]    [guidelines]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Not Relevant. Not useful or on topic.	`2.4K`	38.0%
1	Not Valuable. Consists of definitions or background.	`2.2K`	35.7%
2	Somewhat Valuable. Includes valuable topic-specific arguments, evidence, or knowledge.	`1.2K`	19.5%
3	Very Valuable. Includes central topic-specific arguments, evidence, or knowledge. This does not include general definitions or background.	`416`	6.7%

{
  "docs": {
    "count": 729824,
    "fields": {
      "doc_id": {
        "max_len": 32,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 42
  },
  "qrels": {
    "count": 6186,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "2": 1207,
          "0": 2353,
          "1": 2210,
          "3": 416
        }
      }
    }
  }
}

`"codec/economics"`

Subset of codec that only contains topics about economics.

queries docs qrels Citation Metadata

14 queries

Language: en

Query type:

CodecQuery: (namedtuple)

query_id: str
query: str
domain: str
guidelines: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("codec/economics")
for query in dataset.queries_iter():
    query # namedtuple<query_id, query, domain, guidelines>

You can find more details about the Python API here.

ir_datasets export codec/economics queries



[query_id]    [query]    [domain]    [guidelines]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Not Relevant. Not useful or on topic.	`660`	33.5%
1	Not Valuable. Consists of definitions or background.	`693`	35.2%
2	Somewhat Valuable. Includes valuable topic-specific arguments, evidence, or knowledge.	`458`	23.2%
3	Very Valuable. Includes central topic-specific arguments, evidence, or knowledge. This does not include general definitions or background.	`159`	8.1%

{
  "docs": {
    "count": 729824,
    "fields": {
      "doc_id": {
        "max_len": 32,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 14
  },
  "qrels": {
    "count": 1970,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "2": 458,
          "0": 660,
          "1": 693,
          "3": 159
        }
      }
    }
  }
}

`"codec/history"`

Subset of codec that only contains topics about history.

queries docs qrels Citation Metadata

14 queries

Language: en

Query type:

CodecQuery: (namedtuple)

query_id: str
query: str
domain: str
guidelines: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("codec/history")
for query in dataset.queries_iter():
    query # namedtuple<query_id, query, domain, guidelines>

You can find more details about the Python API here.

ir_datasets export codec/history queries



[query_id]    [query]    [domain]    [guidelines]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Not Relevant. Not useful or on topic.	`998`	49.3%
1	Not Valuable. Consists of definitions or background.	`618`	30.5%
2	Somewhat Valuable. Includes valuable topic-specific arguments, evidence, or knowledge.	`292`	14.4%
3	Very Valuable. Includes central topic-specific arguments, evidence, or knowledge. This does not include general definitions or background.	`116`	5.7%

{
  "docs": {
    "count": 729824,
    "fields": {
      "doc_id": {
        "max_len": 32,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 14
  },
  "qrels": {
    "count": 2024,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 998,
          "1": 618,
          "2": 292,
          "3": 116
        }
      }
    }
  }
}

`"codec/politics"`

Subset of codec that only contains topics about politics.

queries docs qrels Citation Metadata

14 queries

Language: en

Query type:

CodecQuery: (namedtuple)

query_id: str
query: str
domain: str
guidelines: str

Examples:

Python API CLI PyTerrier

import ir_datasets
dataset = ir_datasets.load("codec/politics")
for query in dataset.queries_iter():
    query # namedtuple<query_id, query, domain, guidelines>

You can find more details about the Python API here.

ir_datasets export codec/politics queries



[query_id]    [query]    [domain]    [guidelines]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Not Relevant. Not useful or on topic.	`695`	31.7%
1	Not Valuable. Consists of definitions or background.	`899`	41.0%
2	Somewhat Valuable. Includes valuable topic-specific arguments, evidence, or knowledge.	`457`	20.8%
3	Very Valuable. Includes central topic-specific arguments, evidence, or knowledge. This does not include general definitions or background.	`141`	6.4%

{
  "docs": {
    "count": 729824,
    "fields": {
      "doc_id": {
        "max_len": 32,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 14
  },
  "qrels": {
    "count": 2192,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "3": 141,
          "2": 457,
          "1": 899,
          "0": 695
        }
      }
    }
  }
}

ir_datasets: CODEC

Data Access Information

"codec"

"codec/economics"

"codec/history"

"codec/politics"

`ir_datasets`: CODEC

`"codec"`

`"codec/economics"`

`"codec/history"`

`"codec/politics"`