← home
Github: datasets/trec_tot.py

ir_datasets: TREC Tip-of-the-Tongue

Index
  1. trec-tot
  2. trec-tot/2023
  3. trec-tot/2023/dev
  4. trec-tot/2023/train
  5. trec-tot/2024
  6. trec-tot/2024/test
  7. trec-tot/2025
  8. trec-tot/2025/dev1
  9. trec-tot/2025/dev2
  10. trec-tot/2025/dev3
  11. trec-tot/2025/test
  12. trec-tot/2025/train

"trec-tot"

Tip of the tongue: The phenomenon of failing to retrieve something from memory, combined with partial recall and the feeling that retrieval is imminent. More details are available on the official page for the TREC Tip-of-the-Tongue (ToT) Track.

"trec-tot/2023"

Corpus for the TREC 2023 tip-of-the-tongue search track.

docs
232K docs

Language: en

Document type:
TipOfTheTongueDoc: (namedtuple)
  1. doc_id: str
  2. page_title: str
  3. wikidata_id: str
  4. wikidata_classes: List[str]
  5. text: str
  6. sections: Dict[str,str]
  7. infoboxes: List[Dict[str,str]]

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2023")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, page_title, wikidata_id, wikidata_classes, text, sections, infoboxes>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2023 docs
[doc_id]    [page_title]    [wikidata_id]    [wikidata_classes]    [text]    [sections]    [infoboxes]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2023')
# Index trec-tot/2023
indexer = pt.IterDictIndexer('./indices/trec-tot_2023')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['page_title', 'wikidata_id', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2023')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

Metadata 
{
  "docs": {
    "count": 231852,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  }
}

"trec-tot/2023/dev"

Dev query set for TREC 2023 tip-of-the-tongue search track.

queries
150 queries

Language: en

Query type:
TipOfTheTongueQuery: (namedtuple)
  1. query_id: str
  2. url: str
  3. domain: str
  4. title: str
  5. text: str
  6. sentence_annotations: List[Dict[str,str]]

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2023/dev")
for query in dataset.queries_iter():
    query # namedtuple<query_id, url, domain, title, text, sentence_annotations>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2023/dev queries
[query_id]    [url]    [domain]    [title]    [text]    [sentence_annotations]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2023/dev')
index_ref = pt.IndexRef.of('./indices/trec-tot_2023') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics('url'))

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2023.dev.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
232K docs

Inherits docs from trec-tot/2023

Language: en

Document type:
TipOfTheTongueDoc: (namedtuple)
  1. doc_id: str
  2. page_title: str
  3. wikidata_id: str
  4. wikidata_classes: List[str]
  5. text: str
  6. sections: Dict[str,str]
  7. infoboxes: List[Dict[str,str]]

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2023/dev")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, page_title, wikidata_id, wikidata_classes, text, sections, infoboxes>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2023/dev docs
[doc_id]    [page_title]    [wikidata_id]    [wikidata_classes]    [text]    [sections]    [infoboxes]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2023/dev')
# Index trec-tot/2023
indexer = pt.IterDictIndexer('./indices/trec-tot_2023')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['page_title', 'wikidata_id', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2023.dev')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

qrels
150 qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.DefinitionCount%
0Not Relevant0 0.0%
1Relevant150 100.0%

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2023/dev")
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2023/dev qrels --format tsv
[query_id]    [doc_id]    [relevance]    [iteration]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
from pyterrier.measures import *
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2023/dev')
index_ref = pt.IndexRef.of('./indices/trec-tot_2023') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pt.Experiment(
    [pipeline],
    dataset.get_topics('url'),
    dataset.get_qrels(),
    [MAP, nDCG@20]
)

You can find more details about PyTerrier experiments here.

XPM-IR 
from datamaestro import prepare_dataset
qrels = prepare_dataset('irds.trec-tot.2023.dev.qrels')  # AdhocAssessments
for topic_qrels in qrels.iter():
    print(topic_qrels)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocAssessments.

Metadata 
{
  "docs": {
    "count": 231852,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 150
  },
  "qrels": {
    "count": 150,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 150
        }
      }
    }
  }
}

"trec-tot/2023/train"

Train query set for TREC 2023 tip-of-the-tongue search track.

queries
150 queries

Language: en

Query type:
TipOfTheTongueQuery: (namedtuple)
  1. query_id: str
  2. url: str
  3. domain: str
  4. title: str
  5. text: str
  6. sentence_annotations: List[Dict[str,str]]

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2023/train")
for query in dataset.queries_iter():
    query # namedtuple<query_id, url, domain, title, text, sentence_annotations>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2023/train queries
[query_id]    [url]    [domain]    [title]    [text]    [sentence_annotations]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2023/train')
index_ref = pt.IndexRef.of('./indices/trec-tot_2023') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics('url'))

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2023.train.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
232K docs

Inherits docs from trec-tot/2023

Language: en

Document type:
TipOfTheTongueDoc: (namedtuple)
  1. doc_id: str
  2. page_title: str
  3. wikidata_id: str
  4. wikidata_classes: List[str]
  5. text: str
  6. sections: Dict[str,str]
  7. infoboxes: List[Dict[str,str]]

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2023/train")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, page_title, wikidata_id, wikidata_classes, text, sections, infoboxes>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2023/train docs
[doc_id]    [page_title]    [wikidata_id]    [wikidata_classes]    [text]    [sections]    [infoboxes]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2023/train')
# Index trec-tot/2023
indexer = pt.IterDictIndexer('./indices/trec-tot_2023')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['page_title', 'wikidata_id', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2023.train')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

qrels
150 qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.DefinitionCount%
0Not Relevant0 0.0%
1Relevant150 100.0%

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2023/train")
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2023/train qrels --format tsv
[query_id]    [doc_id]    [relevance]    [iteration]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
from pyterrier.measures import *
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2023/train')
index_ref = pt.IndexRef.of('./indices/trec-tot_2023') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pt.Experiment(
    [pipeline],
    dataset.get_topics('url'),
    dataset.get_qrels(),
    [MAP, nDCG@20]
)

You can find more details about PyTerrier experiments here.

XPM-IR 
from datamaestro import prepare_dataset
qrels = prepare_dataset('irds.trec-tot.2023.train.qrels')  # AdhocAssessments
for topic_qrels in qrels.iter():
    print(topic_qrels)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocAssessments.

Metadata 
{
  "docs": {
    "count": 231852,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 150
  },
  "qrels": {
    "count": 150,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 150
        }
      }
    }
  }
}

"trec-tot/2024"

Corpus for the TREC 2024 tip-of-the-tongue search track.

docs
3.2M docs

Language: en

Document type:
TipOfTheTongueDoc2024: (namedtuple)
  1. doc_id: str
  2. title: str
  3. wikidata_id: str
  4. text: str
  5. sections: Dict[str,str]

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2024")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, wikidata_id, text, sections>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2024 docs
[doc_id]    [title]    [wikidata_id]    [text]    [sections]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2024')
# Index trec-tot/2024
indexer = pt.IterDictIndexer('./indices/trec-tot_2024')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'wikidata_id', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2024')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

Metadata 
{
  "docs": {
    "count": 3185450,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  }
}

"trec-tot/2024/test"

Test query set for TREC 2024 tip-of-the-tongue search track.

queries
600 queries

Language: en

Query type:
TipOfTheTongueQuery2024: (namedtuple)
  1. query_id: str
  2. query: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2024/test")
for query in dataset.queries_iter():
    query # namedtuple<query_id, query>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2024/test queries
[query_id]    [query]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2024/test')
index_ref = pt.IndexRef.of('./indices/trec-tot_2024') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics())

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2024.test.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
3.2M docs

Inherits docs from trec-tot/2024

Language: en

Document type:
TipOfTheTongueDoc2024: (namedtuple)
  1. doc_id: str
  2. title: str
  3. wikidata_id: str
  4. text: str
  5. sections: Dict[str,str]

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2024/test")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, wikidata_id, text, sections>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2024/test docs
[doc_id]    [title]    [wikidata_id]    [text]    [sections]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2024/test')
# Index trec-tot/2024
indexer = pt.IterDictIndexer('./indices/trec-tot_2024')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'wikidata_id', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2024.test')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

Metadata 
{
  "docs": {
    "count": 3185450,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 600
  }
}

"trec-tot/2025"

(no description provided)

docs
6.4M docs

Language: en

Document type:
TrecToT2025Doc: (namedtuple)
  1. doc_id: str
  2. title: str
  3. url: str
  4. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, url, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025 docs
[doc_id]    [title]    [url]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025')
# Index trec-tot/2025
indexer = pt.IterDictIndexer('./indices/trec-tot_2025')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'url', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2025')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

Metadata 
{
  "docs": {
    "count": 6407814,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  }
}

"trec-tot/2025/dev1"

(no description provided)

queries
142 queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev1")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev1 queries
[query_id]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev1')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_dev1') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics())

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2025.dev1.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
6.4M docs

Language: en

Document type:
TrecToT2025Doc: (namedtuple)
  1. doc_id: str
  2. title: str
  3. url: str
  4. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev1")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, url, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev1 docs
[doc_id]    [title]    [url]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev1')
# Index trec-tot/2025/dev1
indexer = pt.IterDictIndexer('./indices/trec-tot_2025_dev1')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'url', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2025.dev1')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

qrels
142 qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.DefinitionCount%
0Not Relevant0 0.0%
1Relevant142 100.0%

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev1")
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev1 qrels --format tsv
[query_id]    [doc_id]    [relevance]    [iteration]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
from pyterrier.measures import *
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev1')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_dev1') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pt.Experiment(
    [pipeline],
    dataset.get_topics(),
    dataset.get_qrels(),
    [MAP, nDCG@20]
)

You can find more details about PyTerrier experiments here.

XPM-IR 
from datamaestro import prepare_dataset
qrels = prepare_dataset('irds.trec-tot.2025.dev1.qrels')  # AdhocAssessments
for topic_qrels in qrels.iter():
    print(topic_qrels)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocAssessments.

Metadata 
{
  "docs": {
    "count": 6407814,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 142
  },
  "qrels": {
    "count": 142,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 142
        }
      }
    }
  }
}

"trec-tot/2025/dev2"

(no description provided)

queries
143 queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev2")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev2 queries
[query_id]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev2')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_dev2') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics())

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2025.dev2.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
6.4M docs

Language: en

Document type:
TrecToT2025Doc: (namedtuple)
  1. doc_id: str
  2. title: str
  3. url: str
  4. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev2")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, url, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev2 docs
[doc_id]    [title]    [url]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev2')
# Index trec-tot/2025/dev2
indexer = pt.IterDictIndexer('./indices/trec-tot_2025_dev2')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'url', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2025.dev2')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

qrels
143 qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.DefinitionCount%
0Not Relevant0 0.0%
1Relevant143 100.0%

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev2")
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev2 qrels --format tsv
[query_id]    [doc_id]    [relevance]    [iteration]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
from pyterrier.measures import *
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev2')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_dev2') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pt.Experiment(
    [pipeline],
    dataset.get_topics(),
    dataset.get_qrels(),
    [MAP, nDCG@20]
)

You can find more details about PyTerrier experiments here.

XPM-IR 
from datamaestro import prepare_dataset
qrels = prepare_dataset('irds.trec-tot.2025.dev2.qrels')  # AdhocAssessments
for topic_qrels in qrels.iter():
    print(topic_qrels)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocAssessments.

Metadata 
{
  "docs": {
    "count": 6407814,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 143
  },
  "qrels": {
    "count": 143,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 143
        }
      }
    }
  }
}

"trec-tot/2025/dev3"

(no description provided)

queries
536 queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev3")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev3 queries
[query_id]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev3')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_dev3') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics())

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2025.dev3.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
6.4M docs

Language: en

Document type:
TrecToT2025Doc: (namedtuple)
  1. doc_id: str
  2. title: str
  3. url: str
  4. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev3")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, url, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev3 docs
[doc_id]    [title]    [url]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev3')
# Index trec-tot/2025/dev3
indexer = pt.IterDictIndexer('./indices/trec-tot_2025_dev3')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'url', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2025.dev3')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

qrels
536 qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.DefinitionCount%
0Not Relevant0 0.0%
1Relevant536 100.0%

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/dev3")
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/dev3 qrels --format tsv
[query_id]    [doc_id]    [relevance]    [iteration]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
from pyterrier.measures import *
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/dev3')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_dev3') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pt.Experiment(
    [pipeline],
    dataset.get_topics(),
    dataset.get_qrels(),
    [MAP, nDCG@20]
)

You can find more details about PyTerrier experiments here.

XPM-IR 
from datamaestro import prepare_dataset
qrels = prepare_dataset('irds.trec-tot.2025.dev3.qrels')  # AdhocAssessments
for topic_qrels in qrels.iter():
    print(topic_qrels)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocAssessments.

Metadata 
{
  "docs": {
    "count": 6407814,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 536
  },
  "qrels": {
    "count": 536,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 536
        }
      }
    }
  }
}

"trec-tot/2025/test"

(no description provided)

queries
622 queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/test")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/test queries
[query_id]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/test')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_test') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics())

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2025.test.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
6.4M docs

Language: en

Document type:
TrecToT2025Doc: (namedtuple)
  1. doc_id: str
  2. title: str
  3. url: str
  4. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/test")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, url, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/test docs
[doc_id]    [title]    [url]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/test')
# Index trec-tot/2025/test
indexer = pt.IterDictIndexer('./indices/trec-tot_2025_test')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'url', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2025.test')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

Metadata 
{
  "docs": {
    "count": 6407814,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 622
  }
}

"trec-tot/2025/train"

(no description provided)

queries
143 queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/train")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/train queries
[query_id]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/train')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_train') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pipeline(dataset.get_topics())

You can find more details about PyTerrier retrieval here.

XPM-IR 
from datamaestro import prepare_dataset
topics = prepare_dataset('irds.trec-tot.2025.train.queries')  # AdhocTopics
for topic in topics.iter():
    print(topic)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.

docs
6.4M docs

Language: en

Document type:
TrecToT2025Doc: (namedtuple)
  1. doc_id: str
  2. title: str
  3. url: str
  4. text: str

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/train")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, title, url, text>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/train docs
[doc_id]    [title]    [url]    [text]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/train')
# Index trec-tot/2025/train
indexer = pt.IterDictIndexer('./indices/trec-tot_2025_train')
index_ref = indexer.index(dataset.get_corpus_iter(), fields=['title', 'url', 'text'])

You can find more details about PyTerrier indexing here.

XPM-IR 
from datamaestro import prepare_dataset
dataset = prepare_dataset('irds.trec-tot.2025.train')
for doc in dataset.iter_documents():
    print(doc)  # an AdhocDocumentStore
    break

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore

qrels
143 qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.DefinitionCount%
0Not Relevant0 0.0%
1Relevant143 100.0%

Examples:

Python API 
import ir_datasets
dataset = ir_datasets.load("trec-tot/2025/train")
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

You can find more details about the Python API here.

CLI 
ir_datasets export trec-tot/2025/train qrels --format tsv
[query_id]    [doc_id]    [relevance]    [iteration]
...

You can find more details about the CLI here.

PyTerrier 
import pyterrier as pt
from pyterrier.measures import *
pt.init()
dataset = pt.get_dataset('irds:trec-tot/2025/train')
index_ref = pt.IndexRef.of('./indices/trec-tot_2025_train') # assumes you have already built an index
pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25')
# (optionally other pipeline components)
pt.Experiment(
    [pipeline],
    dataset.get_topics(),
    dataset.get_qrels(),
    [MAP, nDCG@20]
)

You can find more details about PyTerrier experiments here.

XPM-IR 
from datamaestro import prepare_dataset
qrels = prepare_dataset('irds.trec-tot.2025.train.qrels')  # AdhocAssessments
for topic_qrels in qrels.iter():
    print(topic_qrels)  # An AdhocTopic

This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocAssessments.

Metadata 
{
  "docs": {
    "count": 6407814,
    "fields": {
      "doc_id": {
        "max_len": 8,
        "common_prefix": ""
      }
    }
  },
  "queries": {
    "count": 143
  },
  "qrels": {
    "count": 143,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 143
        }
      }
    }
  }
}