Total tasks
53,080
Mean difficulty
6.27
Median difficulty
6.7
Avg patch lines
376.0
Avg patch files
6.79
Difficulty distribution SWE-Lego-Live-Instances
| Label breakdown | easy | medium | hard |
|---|---|---|---|
5637 / 26526 / 20916 |
5,637 | 26,526 | 20,916 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 53,079 | 1.0 | 5.5 | 6.7 | 6.27 | 7.4 | 8.0 |
Score bins
<=32,032 (3.8%)
3.1-57,830 (14.8%)
5.1-722,301 (42.0%)
7.1-820,916 (39.4%)
>80 (0.0%)
Languages
c9,458 (17.8%)
go8,664 (16.3%)
typescript7,088 (13.4%)
javascript6,940 (13.1%)
python5,983 (11.3%)
rust5,580 (10.5%)
java4,346 (8.2%)
cpp4,226 (8.0%)
ruby211 (0.4%)
lua116 (0.2%)
vue105 (0.2%)
php63 (0.1%)
solidity43 (0.1%)
c++21 (0.0%)
cmake21 (0.0%)
Area / tier
backend28,199 (53.1%)
library12,695 (23.9%)
frontend7,627 (14.4%)
cli4,525 (8.5%)
fullstack31 (0.1%)
framework2 (0.0%)
Top topics
react657 (1.2%)
kubernetes320 (0.6%)
http191 (0.4%)
grpc187 (0.4%)
cobra181 (0.3%)
aem165 (0.3%)
eslint158 (0.3%)
aws-sdk147 (0.3%)
prometheus138 (0.3%)
clap136 (0.3%)
cmake135 (0.3%)
mongoose133 (0.3%)
picoquic130 (0.2%)
git126 (0.2%)
electron125 (0.2%)
openssl116 (0.2%)
github-actions113 (0.2%)
redis112 (0.2%)
express112 (0.2%)
svelte107 (0.2%)
fastify107 (0.2%)
pgbackrest106 (0.2%)
igraph105 (0.2%)
nimble101 (0.2%)
fastapi100 (0.2%)
Bug classes
incomplete-validation3,233 (6.1%)
missing-feature2,987 (5.6%)
missing-fallback2,135 (4.0%)
missing-metadata-propagation1,982 (3.7%)
wrong-default1,682 (3.2%)
type-handling-inconsistency1,270 (2.4%)
missing-validation631 (1.2%)
missing-state-propagation574 (1.1%)
missing-implementation555 (1.0%)
race-condition505 (1.0%)
missing-api400 (0.8%)
missing-error-handling286 (0.5%)
missing-functionality269 (0.5%)
resource-leak245 (0.5%)
missing-method234 (0.4%)
incomplete-parsing232 (0.4%)
missing-config-option207 (0.4%)
missing-default148 (0.3%)
memory-leak146 (0.3%)
missing-function141 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
27,878
Mean difficulty
5.40
Median difficulty
5.7
Avg patch lines
117.2
Avg patch files
3.48
Difficulty distribution SWE-rebench
| Label breakdown | easy | medium | hard |
|---|---|---|---|
7986 / 12347 / 7545 |
7,986 | 12,347 | 7,545 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 27,878 | 1.0 | 3.7 | 5.7 | 5.40 | 7.1 | 9.2 |
Score bins
<=35,000 (17.9%)
3.1-56,298 (22.6%)
5.1-79,035 (32.4%)
7.1-84,753 (17.0%)
>82,792 (10.0%)
Languages
python27,728 (99.5%)
cpp51 (0.2%)
fortran48 (0.2%)
vyper11 (0.0%)
typescript6 (0.0%)
sql6 (0.0%)
c5 (0.0%)
javascript5 (0.0%)
rust5 (0.0%)
bash3 (0.0%)
yaml3 (0.0%)
sqf2 (0.0%)
go2 (0.0%)
solidity2 (0.0%)
java1 (0.0%)
Area / tier
library13,355 (47.9%)
backend11,097 (39.8%)
cli2,425 (8.7%)
frontend1,001 (3.6%)
Top topics
sqlglot748 (2.7%)
sympy740 (2.7%)
xarray349 (1.3%)
matplotlib289 (1.0%)
dvc285 (1.0%)
dask262 (0.9%)
conan234 (0.8%)
sphinx204 (0.7%)
argparse200 (0.7%)
pybamm191 (0.7%)
sqlfluff190 (0.7%)
pandas189 (0.7%)
pennylane186 (0.7%)
astropy181 (0.6%)
pydicom177 (0.6%)
pytorch-ignite176 (0.6%)
pillow173 (0.6%)
networkx160 (0.6%)
click158 (0.6%)
tox155 (0.6%)
numpy148 (0.5%)
geopandas147 (0.5%)
textual138 (0.5%)
streamlink135 (0.5%)
nilearn134 (0.5%)
Bug classes
type-handling-inconsistency2,138 (7.7%)
incomplete-validation1,915 (6.9%)
missing-metadata-propagation1,833 (6.6%)
missing-fallback1,801 (6.5%)
missing-feature1,663 (6.0%)
wrong-default1,320 (4.7%)
missing-validation317 (1.1%)
missing-implementation237 (0.9%)
missing-state-propagation231 (0.8%)
missing-error-handling230 (0.8%)
incomplete-parsing224 (0.8%)
missing-method206 (0.7%)
missing-default176 (0.6%)
index-out-of-bounds172 (0.6%)
missing-exception-handling104 (0.4%)
missing-method-implementation104 (0.4%)
off-by-one93 (0.3%)
missing-functionality89 (0.3%)
missing-parameter84 (0.3%)
deprecated-api81 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
32,079
Mean difficulty
5.80
Median difficulty
6.2
Avg patch lines
86.9
Avg patch files
4.22
Difficulty distribution SWE-rebench-V2
| Label breakdown | easy | medium | hard |
|---|---|---|---|
7319 / 13490 / 11270 |
7,319 | 13,490 | 11,270 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 32,079 | 1.0 | 4.3 | 6.2 | 5.80 | 7.5 | 9.2 |
Score bins
<=34,590 (14.3%)
3.1-55,972 (18.6%)
5.1-710,247 (31.9%)
7.1-86,984 (21.8%)
>84,286 (13.4%)
Languages
python7,223 (22.5%)
go6,147 (19.2%)
javascript4,371 (13.6%)
typescript3,990 (12.4%)
rust2,933 (9.1%)
java1,750 (5.5%)
php1,406 (4.4%)
kotlin857 (2.7%)
julia793 (2.5%)
elixir410 (1.3%)
scala399 (1.2%)
swift359 (1.1%)
dart251 (0.8%)
r159 (0.5%)
cpp158 (0.5%)
Area / tier
backend12,928 (40.3%)
library11,598 (36.2%)
frontend4,125 (12.9%)
cli3,419 (10.7%)
fullstack8 (0.0%)
framework1 (0.0%)
Top topics
swc407 (1.3%)
kubernetes356 (1.1%)
eslint297 (0.9%)
react268 (0.8%)
pandas251 (0.8%)
moto232 (0.7%)
serverless198 (0.6%)
statamic188 (0.6%)
cfn-lint162 (0.5%)
diktat158 (0.5%)
rust-analyzer157 (0.5%)
cobra153 (0.5%)
helm150 (0.5%)
pennylane135 (0.4%)
aiohttp133 (0.4%)
qiskit132 (0.4%)
swift-syntax117 (0.4%)
sveltekit117 (0.4%)
hypothesis114 (0.4%)
obsidian-linter111 (0.3%)
clap109 (0.3%)
laravel107 (0.3%)
knative103 (0.3%)
kiota101 (0.3%)
scalameta97 (0.3%)
Bug classes
incomplete-validation2,494 (7.8%)
missing-metadata-propagation1,525 (4.8%)
type-handling-inconsistency1,376 (4.3%)
missing-feature1,271 (4.0%)
missing-fallback1,125 (3.5%)
wrong-default1,060 (3.3%)
missing-validation363 (1.1%)
missing-state-propagation279 (0.9%)
incomplete-parsing223 (0.7%)
missing-implementation215 (0.7%)
false-positive163 (0.5%)
race-condition157 (0.5%)
missing-error-handling150 (0.5%)
index-out-of-bounds140 (0.4%)
missing-method128 (0.4%)
missing-default110 (0.3%)
missing-method-implementation104 (0.3%)
nil-pointer-dereference100 (0.3%)
missing-function-implementation88 (0.3%)
off-by-one85 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
22,806
Mean difficulty
5.36
Median difficulty
5.8
Avg patch lines
61.8
Avg patch files
2.41
Difficulty distribution OpenSWE-filtered
| Label breakdown | easy | medium | hard |
|---|---|---|---|
6209 / 11129 / 5468 |
6,209 | 11,129 | 5,468 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 22,806 | 1.0 | 3.8 | 5.8 | 5.36 | 7.0 | 9.1 |
Score bins
<=34,214 (18.5%)
3.1-54,627 (20.3%)
5.1-78,497 (37.3%)
7.1-84,547 (19.9%)
>8921 (4.0%)
Languages
python22,685 (99.5%)
fortran41 (0.2%)
cpp20 (0.1%)
cython10 (0.0%)
java8 (0.0%)
typescript7 (0.0%)
solidity5 (0.0%)
basic3 (0.0%)
c3 (0.0%)
julia3 (0.0%)
yaml3 (0.0%)
javascript3 (0.0%)
rust3 (0.0%)
html2 (0.0%)
elixir1 (0.0%)
Area / tier
backend11,767 (51.6%)
library8,251 (36.2%)
cli1,751 (7.7%)
frontend1,036 (4.5%)
fullstack1 (0.0%)
Top topics
django445 (2.0%)
pandas284 (1.2%)
pydantic280 (1.2%)
fastapi202 (0.9%)
click178 (0.8%)
xarray150 (0.7%)
numpy147 (0.6%)
argparse139 (0.6%)
pytorch138 (0.6%)
scikit-learn116 (0.5%)
matplotlib111 (0.5%)
sqlalchemy103 (0.5%)
flask100 (0.4%)
jax99 (0.4%)
scipy98 (0.4%)
home-assistant98 (0.4%)
opentelemetry96 (0.4%)
qiskit91 (0.4%)
ansible80 (0.4%)
jinja273 (0.3%)
dask71 (0.3%)
conan65 (0.3%)
requests65 (0.3%)
kubernetes65 (0.3%)
fonttools63 (0.3%)
Bug classes
incomplete-validation1,830 (8.0%)
missing-metadata-propagation1,648 (7.2%)
missing-fallback1,613 (7.1%)
type-handling-inconsistency1,506 (6.6%)
missing-feature1,448 (6.3%)
wrong-default1,050 (4.6%)
missing-validation418 (1.8%)
missing-implementation283 (1.2%)
missing-error-handling254 (1.1%)
missing-state-propagation242 (1.1%)
missing-method168 (0.7%)
missing-default146 (0.6%)
index-out-of-bounds119 (0.5%)
missing-method-implementation112 (0.5%)
incomplete-parsing105 (0.5%)
missing-parameter-propagation92 (0.4%)
race-condition89 (0.4%)
missing-parameter88 (0.4%)
missing-config-option72 (0.3%)
resource-leak72 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
20,181
Mean difficulty
6.30
Median difficulty
6.5
Avg patch lines
321.4
Avg patch files
6.06
Difficulty distribution Scale-SWE
| Label breakdown | easy | medium | hard |
|---|---|---|---|
2311 / 10055 / 7815 |
2,311 | 10,055 | 7,815 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 20,181 | 1.0 | 5.2 | 6.5 | 6.30 | 7.6 | 9.1 |
Score bins
<=3795 (3.9%)
3.1-53,782 (18.7%)
5.1-77,789 (38.6%)
7.1-84,506 (22.3%)
>83,309 (16.4%)
Languages
python19,289 (95.6%)
c170 (0.8%)
cpp159 (0.8%)
typescript68 (0.3%)
java63 (0.3%)
scala61 (0.3%)
javascript47 (0.2%)
yaml46 (0.2%)
rust43 (0.2%)
cmake38 (0.2%)
dart32 (0.2%)
csharp31 (0.2%)
kotlin24 (0.1%)
go23 (0.1%)
shell17 (0.1%)
Area / tier
library12,366 (61.3%)
backend5,826 (28.9%)
cli1,393 (6.9%)
frontend595 (2.9%)
fullstack1 (0.0%)
Top topics
fonttools312 (1.5%)
click268 (1.3%)
globus-sdk220 (1.1%)
poetry-core213 (1.1%)
more-itertools208 (1.0%)
flatbuffers175 (0.9%)
aioesphomeapi167 (0.8%)
beets160 (0.8%)
xsdata160 (0.8%)
pymongo154 (0.8%)
coloraide151 (0.7%)
jsonargparse148 (0.7%)
geopandas147 (0.7%)
mesa146 (0.7%)
pydantic144 (0.7%)
ops143 (0.7%)
grizz131 (0.6%)
pymarkdown128 (0.6%)
django127 (0.6%)
traitlets126 (0.6%)
mcuboot116 (0.6%)
scikit-rf110 (0.5%)
troposphere108 (0.5%)
latex2mathml107 (0.5%)
sherpa-onnx103 (0.5%)
Bug classes
missing-feature2,087 (10.3%)
missing-fallback1,316 (6.5%)
incomplete-validation1,187 (5.9%)
missing-metadata-propagation1,056 (5.2%)
type-handling-inconsistency1,004 (5.0%)
wrong-default721 (3.6%)
missing-method335 (1.7%)
missing-validation222 (1.1%)
missing-error-handling164 (0.8%)
missing-implementation149 (0.7%)
missing-functionality143 (0.7%)
incomplete-parsing119 (0.6%)
performance-optimization109 (0.5%)
missing-parameter106 (0.5%)
missing-type-annotations105 (0.5%)
missing-state-propagation102 (0.5%)
missing-default101 (0.5%)
deprecated-api-usage86 (0.4%)
missing-function83 (0.4%)
race-condition75 (0.4%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)