ir_datasets: NeuCLIR CorpusTo access the docuemnts of this dataset, you will need to download the documents from Common Crawl. The script for downloading and validating the documents are in NeuCLIR/download-collection . Please use the following command to download the documents:
git clone https://github.com/NeuCLIR/download-collection
cd download-collection
pip install -r requirements.txt
python download_documents.py --storage ~/.ir_datasets/neuclir/1 \
--zho ./resource/zho/ids.jsonl.gz \
--fas ./resource/fas/ids.jsonl.gz \
--rus ./resource/rus/ids.*.jsonl.gz \
--jobs {number of process}
After download, please also post-process the downloaded file to verify all and only specified documents are downloaded, and modify the ordering of the collection to match the original specified ordering in the id files.
for lang in zho fas rus; do
python fix_document_order.py --raw_download_file ~/.ir_datasets/neuclir/1/$lang/docs.jsonl \
--id_file ./resource/$lang/ids*.jsonl.gz \
--check_hash
done You can also store the documents in other directory and create a soft link for ~/.ir_datasets/neuclir/22/.
This is the dataset created for TREC 2022 NeuCLIR Track. Topics will be developed and released by June 2022 by NIST. Relevance judgements will be available after the evaluation (around November).
The collection designed to be similar to [HC4] and a large portion of documents from HC4 are ported to this collection. Users can conduct experiemnts on this collection with queries and qrels in HC4 for development.
Version 1 of the NeuCLIR corpus.
The Persian collection contains English queries (to be released) and Persian documents for retrieval. Human and machine translated queries will be provided in the query object for running monolingual retrieval or cross-language retrival assuming the machine query tranlstion into Persian is available.
Language: fa
Examples:
import ir_datasets
dataset = ir_datasets.load("neuclir/1/fa")
for doc in dataset.docs_iter():
doc # namedtuple<doc_id, title, text, url, time, cc_file>
You can find more details about the Python API here.
Subset of the Persian collection that intersect with HC4. The 60 queries are the hc4/fa/dev and hc4/fa/test sets combined.
Language: multiple/other/unknown
Examples:
import ir_datasets
dataset = ir_datasets.load("neuclir/1/fa/hc4-filtered")
for query in dataset.queries_iter():
query # namedtuple<query_id, title, description, ht_title, ht_description, mt_title, mt_description, narrative_by_relevance, report, report_url, report_date, translation_lang>
You can find more details about the Python API here.
The Russian collection contains English queries (to be released) and Russian documents for retrieval. Human and machine translated queries will be provided in the query object for running monolingual retrieval or cross-language retrival assuming the machine query tranlstion into Russian is available.
Language: ru
Examples:
import ir_datasets
dataset = ir_datasets.load("neuclir/1/ru")
for doc in dataset.docs_iter():
doc # namedtuple<doc_id, title, text, url, time, cc_file>
You can find more details about the Python API here.
Subset of the Russian collection that intersect with HC4. The 54 queries are the hc4/ru/dev and hc4/ru/test sets combined.
Language: multiple/other/unknown
Examples:
import ir_datasets
dataset = ir_datasets.load("neuclir/1/ru/hc4-filtered")
for query in dataset.queries_iter():
query # namedtuple<query_id, title, description, ht_title, ht_description, mt_title, mt_description, narrative_by_relevance, report, report_url, report_date, translation_lang>
You can find more details about the Python API here.
The Chinese collection contains English queries (to be released) and Chinese documents for retrieval. Human and machine translated queries will be provided in the query object for running monolingual retrieval or cross-language retrival assuming the machine query tranlstion into Chinese is available.
Language: zh
Examples:
import ir_datasets
dataset = ir_datasets.load("neuclir/1/zh")
for doc in dataset.docs_iter():
doc # namedtuple<doc_id, title, text, url, time, cc_file>
You can find more details about the Python API here.
Subset of the Chinse collection that intersect with HC4. The 60 queries are the hc4/zh/dev and hc4/zh/test sets combined.
Language: multiple/other/unknown
Examples:
import ir_datasets
dataset = ir_datasets.load("neuclir/1/zh/hc4-filtered")
for query in dataset.queries_iter():
query # namedtuple<query_id, title, description, ht_title, ht_description, mt_title, mt_description, narrative_by_relevance, report, report_url, report_date, translation_lang>
You can find more details about the Python API here.