← home
Github: datasets/codesearchnet.py

ir_datasets: CodeSearchNet

Index
  1. codesearchnet
  2. codesearchnet/challenge
  3. codesearchnet/test
  4. codesearchnet/train
  5. codesearchnet/valid

"codesearchnet"

A benchmark for semantic code search. Uses

docs

Language: multiple/other/unknown

Document type:
CodeSearchNetDoc: (namedtuple)
  1. doc_id: str
  2. repo: str
  3. path: str
  4. func_name: str
  5. code: str
  6. language: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, repo, path, func_name, code, language>
Citation
bibtex: @article{Husain2019CodeSearchNet, title={CodeSearchNet Challenge: Evaluating the State of Semantic Code Search}, author={Hamel Husain and Ho-Hsiang Wu and Tiferet Gazit and Miltiadis Allamanis and Marc Brockschmidt}, journal={ArXiv}, year={2019} }

"codesearchnet/challenge"

Official challenge set, with keyword queries and deep relevance assessments.

queries

Language: multiple/other/unknown

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/challenge')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: multiple/other/unknown

Document type:
CodeSearchNetDoc: (namedtuple)
  1. doc_id: str
  2. repo: str
  3. path: str
  4. func_name: str
  5. code: str
  6. language: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/challenge')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, repo, path, func_name, code, language>
qrels
Query relevance judgment type:
CodeSearchNetChallengeQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: str
  4. note: str

Relevance levels

Rel.Definition
0Irrelevant
1Weak Match
2String Match
3Exact Match

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/challenge')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, note>

"codesearchnet/test"

Official test set, using queries inferred from docstrings.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/test')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: multiple/other/unknown

Document type:
CodeSearchNetDoc: (namedtuple)
  1. doc_id: str
  2. repo: str
  3. path: str
  4. func_name: str
  5. code: str
  6. language: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/test')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, repo, path, func_name, code, language>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Matches docstring

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/test')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

"codesearchnet/train"

Official train set, using queries inferred from docstrings.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/train')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: multiple/other/unknown

Document type:
CodeSearchNetDoc: (namedtuple)
  1. doc_id: str
  2. repo: str
  3. path: str
  4. func_name: str
  5. code: str
  6. language: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/train')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, repo, path, func_name, code, language>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Matches docstring

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/train')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>

"codesearchnet/valid"

Official validation set, using queries inferred from docstrings.

queries

Language: en

Query type:
GenericQuery: (namedtuple)
  1. query_id: str
  2. text: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/valid')
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>
docs

Language: multiple/other/unknown

Document type:
CodeSearchNetDoc: (namedtuple)
  1. doc_id: str
  2. repo: str
  3. path: str
  4. func_name: str
  5. code: str
  6. language: str

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/valid')
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, repo, path, func_name, code, language>
qrels
Query relevance judgment type:
TrecQrel: (namedtuple)
  1. query_id: str
  2. doc_id: str
  3. relevance: int
  4. iteration: str

Relevance levels

Rel.Definition
1Matches docstring

Example

import ir_datasets
dataset = ir_datasets.load('codesearchnet/valid')
for qrel in dataset.qrels_iter():
    qrel # namedtuple<query_id, doc_id, relevance, iteration>