CodeSearchNet - ir

`"codesearchnet"`

A benchmark for semantic code search. Uses

Documents: Code functions in python, java, go, php, ruby, and javascript
Queries: Inferred from docstrings, or
Dataset Paper
Challenge Task Leaderboard

docs Citation Metadata

2.1M docs

Language: multiple/other/unknown

Document type:

CodeSearchNetDoc: (namedtuple)

doc_id: str
repo: str
path: str
func_name: str
code: str
language: str

Examples:

Python API CLI PyTerrier XPM-IR

import ir_datasets
dataset = ir_datasets.load("codesearchnet")
for doc in dataset.docs_iter():
    doc # namedtuple<doc_id, repo, path, func_name, code, language>

You can find more details about the Python API here.

ir_datasets export codesearchnet docs



[doc_id]    [repo]    [path]    [func_name]    [code]    [language]
...

You can find more details about the CLI here.

{
  "docs": {
    "count": 2070536,
    "fields": {
      "doc_id": {
        "max_len": 339,
        "common_prefix": "https://github.com/"
      }
    }
  }
}

`"codesearchnet/challenge"`

Official challenge set, with keyword queries and deep relevance assessments.

queries docs qrels Citation Metadata

99 queries

Language: multiple/other/unknown

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier XPM-IR

import ir_datasets
dataset = ir_datasets.load("codesearchnet/challenge")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export codesearchnet/challenge queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
0	Irrelevant	`1.3K`	32.8%
1	Weak Match	`982`	24.5%
2	String Match	`863`	21.5%
3	Exact Match	`847`	21.1%

{
  "docs": {
    "count": 2070536,
    "fields": {
      "doc_id": {
        "max_len": 339,
        "common_prefix": "https://github.com/"
      }
    }
  },
  "queries": {
    "count": 99
  },
  "qrels": {
    "count": 4006,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "0": 1314,
          "1": 982,
          "2": 863,
          "3": 847
        }
      }
    }
  }
}

`"codesearchnet/test"`

Official test set, using queries inferred from docstrings.

queries docs qrels Citation Metadata

101K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier XPM-IR

import ir_datasets
dataset = ir_datasets.load("codesearchnet/test")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export codesearchnet/test queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Matches docstring	`101K`	100.0%

{
  "docs": {
    "count": 2070536,
    "fields": {
      "doc_id": {
        "max_len": 339,
        "common_prefix": "https://github.com/"
      }
    }
  },
  "queries": {
    "count": 100529
  },
  "qrels": {
    "count": 100529,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 100529
        }
      }
    }
  }
}

`"codesearchnet/train"`

Official train set, using queries inferred from docstrings.

queries docs qrels Citation Metadata

1.9M queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier XPM-IR

import ir_datasets
dataset = ir_datasets.load("codesearchnet/train")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export codesearchnet/train queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Matches docstring	`1.9M`	100.0%

{
  "docs": {
    "count": 2070536,
    "fields": {
      "doc_id": {
        "max_len": 339,
        "common_prefix": "https://github.com/"
      }
    }
  },
  "queries": {
    "count": 1880853
  },
  "qrels": {
    "count": 1880853,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 1880853
        }
      }
    }
  }
}

`"codesearchnet/valid"`

Official validation set, using queries inferred from docstrings.

queries docs qrels Citation Metadata

89K queries

Language: en

Query type:

GenericQuery: (namedtuple)

query_id: str
text: str

Examples:

Python API CLI PyTerrier XPM-IR

import ir_datasets
dataset = ir_datasets.load("codesearchnet/valid")
for query in dataset.queries_iter():
    query # namedtuple<query_id, text>

You can find more details about the Python API here.

ir_datasets export codesearchnet/valid queries



[query_id]    [text]
...

You can find more details about the CLI here.

Rel.	Definition	Count	%
1	Matches docstring	`89K`	100.0%

{
  "docs": {
    "count": 2070536,
    "fields": {
      "doc_id": {
        "max_len": 339,
        "common_prefix": "https://github.com/"
      }
    }
  },
  "queries": {
    "count": 89154
  },
  "qrels": {
    "count": 89154,
    "fields": {
      "relevance": {
        "counts_by_value": {
          "1": 89154
        }
      }
    }
  }
}

ir_datasets: CodeSearchNet

"codesearchnet"

"codesearchnet/challenge"

"codesearchnet/test"

"codesearchnet/train"

"codesearchnet/valid"

`ir_datasets`: CodeSearchNet

`"codesearchnet"`

`"codesearchnet/challenge"`

`"codesearchnet/test"`

`"codesearchnet/train"`

`"codesearchnet/valid"`