Total tasks
53,080
Mean difficulty
6.27
Median difficulty
6.7
Avg patch lines
376.0
Avg patch files
6.79
Difficulty distribution SWE-Lego-Live-Instances
| Label breakdown | easy | medium | hard |
|---|---|---|---|
5637 / 26526 / 20916 |
5,637 | 26,526 | 20,916 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 53,079 | 1.0 | 5.5 | 6.7 | 6.27 | 7.4 | 8.0 |
Score bins
<=32,032 (3.8%)
3.1-57,830 (14.8%)
5.1-722,301 (42.0%)
7.1-820,916 (39.4%)
>80 (0.0%)
Languages
c9,458 (17.8%)
go8,664 (16.3%)
typescript7,088 (13.4%)
javascript6,940 (13.1%)
python5,983 (11.3%)
rust5,580 (10.5%)
java4,346 (8.2%)
cpp4,226 (8.0%)
ruby211 (0.4%)
lua116 (0.2%)
vue105 (0.2%)
php63 (0.1%)
solidity43 (0.1%)
c++21 (0.0%)
cmake21 (0.0%)
Area / tier
backend28,199 (53.1%)
library12,695 (23.9%)
frontend7,627 (14.4%)
cli4,525 (8.5%)
fullstack31 (0.1%)
framework2 (0.0%)
Top topics
react657 (1.2%)
kubernetes320 (0.6%)
http191 (0.4%)
grpc187 (0.4%)
cobra181 (0.3%)
aem165 (0.3%)
eslint158 (0.3%)
aws-sdk147 (0.3%)
prometheus138 (0.3%)
clap136 (0.3%)
cmake135 (0.3%)
mongoose133 (0.3%)
picoquic130 (0.2%)
git126 (0.2%)
electron125 (0.2%)
openssl116 (0.2%)
github-actions113 (0.2%)
redis112 (0.2%)
express112 (0.2%)
svelte107 (0.2%)
fastify107 (0.2%)
pgbackrest106 (0.2%)
igraph105 (0.2%)
nimble101 (0.2%)
fastapi100 (0.2%)
Bug classes
incomplete-validation3,233 (6.1%)
missing-feature2,987 (5.6%)
missing-fallback2,135 (4.0%)
missing-metadata-propagation1,982 (3.7%)
wrong-default1,682 (3.2%)
type-handling-inconsistency1,270 (2.4%)
missing-validation631 (1.2%)
missing-state-propagation574 (1.1%)
missing-implementation555 (1.0%)
race-condition505 (1.0%)
missing-api400 (0.8%)
missing-error-handling286 (0.5%)
missing-functionality269 (0.5%)
resource-leak245 (0.5%)
missing-method234 (0.4%)
incomplete-parsing232 (0.4%)
missing-config-option207 (0.4%)
missing-default148 (0.3%)
memory-leak146 (0.3%)
missing-function141 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
27,878
Mean difficulty
5.08
Median difficulty
5.3
Avg patch lines
86.3
Avg patch files
2.91
Difficulty distribution SWE-rebench
| Label breakdown | easy | medium | hard |
|---|---|---|---|
3943 / 5582 / 2371 |
3,943 | 5,582 | 2,371 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 11,896 | 1.0 | 3.4 | 5.3 | 5.08 | 6.7 | 9.2 |
Score bins
<=32,438 (20.5%)
3.1-53,104 (26.1%)
5.1-73,983 (33.5%)
7.1-81,621 (13.6%)
>8750 (6.3%)
Languages
python11,859 (99.7%)
cpp16 (0.1%)
bash3 (0.0%)
fortran3 (0.0%)
vyper3 (0.0%)
sqf2 (0.0%)
c2 (0.0%)
yaml2 (0.0%)
sql2 (0.0%)
typescript1 (0.0%)
javascript1 (0.0%)
go1 (0.0%)
rust1 (0.0%)
Area / tier
library5,481 (46.1%)
backend4,871 (40.9%)
cli1,154 (9.7%)
frontend390 (3.3%)
Top topics
sqlglot296 (2.5%)
pennylane184 (1.5%)
astropy150 (1.3%)
textual138 (1.2%)
pyupgrade110 (0.9%)
briefcase92 (0.8%)
pandas83 (0.7%)
argparse80 (0.7%)
pydantic75 (0.6%)
numpy73 (0.6%)
dask73 (0.6%)
conan71 (0.6%)
sqlfluff67 (0.6%)
click66 (0.6%)
dvc66 (0.6%)
gspread64 (0.5%)
pytorch-ignite63 (0.5%)
charmcraft60 (0.5%)
anyio57 (0.5%)
cekit57 (0.5%)
tox57 (0.5%)
pyflakes55 (0.5%)
esmvalcore55 (0.5%)
borgbackup54 (0.5%)
xarray53 (0.4%)
Bug classes
type-handling-inconsistency919 (7.7%)
incomplete-validation817 (6.9%)
missing-fallback769 (6.5%)
missing-metadata-propagation755 (6.3%)
missing-feature632 (5.3%)
wrong-default597 (5.0%)
missing-validation127 (1.1%)
missing-error-handling105 (0.9%)
incomplete-parsing98 (0.8%)
missing-state-propagation92 (0.8%)
missing-implementation86 (0.7%)
index-out-of-bounds82 (0.7%)
missing-default82 (0.7%)
missing-method66 (0.6%)
off-by-one53 (0.4%)
missing-exception-handling47 (0.4%)
missing-parameter42 (0.4%)
missing-parameter-propagation40 (0.3%)
race-condition40 (0.3%)
false-positive38 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
32,079
Mean difficulty
5.80
Median difficulty
6.2
Avg patch lines
86.9
Avg patch files
4.22
Difficulty distribution SWE-rebench-V2
| Label breakdown | easy | medium | hard |
|---|---|---|---|
7319 / 13490 / 11270 |
7,319 | 13,490 | 11,270 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 32,079 | 1.0 | 4.3 | 6.2 | 5.80 | 7.5 | 9.2 |
Score bins
<=34,590 (14.3%)
3.1-55,972 (18.6%)
5.1-710,247 (31.9%)
7.1-86,984 (21.8%)
>84,286 (13.4%)
Languages
python7,223 (22.5%)
go6,147 (19.2%)
javascript4,371 (13.6%)
typescript3,990 (12.4%)
rust2,933 (9.1%)
java1,750 (5.5%)
php1,406 (4.4%)
kotlin857 (2.7%)
julia793 (2.5%)
elixir410 (1.3%)
scala399 (1.2%)
swift359 (1.1%)
dart251 (0.8%)
r159 (0.5%)
cpp158 (0.5%)
Area / tier
backend12,928 (40.3%)
library11,598 (36.2%)
frontend4,125 (12.9%)
cli3,419 (10.7%)
fullstack8 (0.0%)
framework1 (0.0%)
Top topics
swc407 (1.3%)
kubernetes356 (1.1%)
eslint297 (0.9%)
react268 (0.8%)
pandas251 (0.8%)
moto232 (0.7%)
serverless198 (0.6%)
statamic188 (0.6%)
cfn-lint162 (0.5%)
diktat158 (0.5%)
rust-analyzer157 (0.5%)
cobra153 (0.5%)
helm150 (0.5%)
pennylane135 (0.4%)
aiohttp133 (0.4%)
qiskit132 (0.4%)
swift-syntax117 (0.4%)
sveltekit117 (0.4%)
hypothesis114 (0.4%)
obsidian-linter111 (0.3%)
clap109 (0.3%)
laravel107 (0.3%)
knative103 (0.3%)
kiota101 (0.3%)
scalameta97 (0.3%)
Bug classes
incomplete-validation2,494 (7.8%)
missing-metadata-propagation1,525 (4.8%)
type-handling-inconsistency1,376 (4.3%)
missing-feature1,271 (4.0%)
missing-fallback1,125 (3.5%)
wrong-default1,060 (3.3%)
missing-validation363 (1.1%)
missing-state-propagation279 (0.9%)
incomplete-parsing223 (0.7%)
missing-implementation215 (0.7%)
false-positive163 (0.5%)
race-condition157 (0.5%)
missing-error-handling150 (0.5%)
index-out-of-bounds140 (0.4%)
missing-method128 (0.4%)
missing-default110 (0.3%)
missing-method-implementation104 (0.3%)
nil-pointer-dereference100 (0.3%)
missing-function-implementation88 (0.3%)
off-by-one85 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
22,806
Mean difficulty
5.31
Median difficulty
5.7
Avg patch lines
60.2
Avg patch files
2.37
Difficulty distribution OpenSWE-filtered
| Label breakdown | easy | medium | hard |
|---|---|---|---|
3021 / 5101 / 2550 |
3,021 | 5,101 | 2,550 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 10,672 | 1.6 | 3.7 | 5.7 | 5.31 | 7.0 | 9.0 |
Score bins
<=32,085 (19.5%)
3.1-52,165 (20.3%)
5.1-73,872 (36.3%)
7.1-82,127 (19.9%)
>8423 (4.0%)
Languages
python10,618 (99.5%)
fortran22 (0.2%)
cpp7 (0.1%)
cython5 (0.0%)
java4 (0.0%)
solidity4 (0.0%)
typescript3 (0.0%)
c3 (0.0%)
elixir1 (0.0%)
basic1 (0.0%)
markdown1 (0.0%)
r1 (0.0%)
fish1 (0.0%)
julia1 (0.0%)
Area / tier
backend5,473 (51.3%)
library3,863 (36.2%)
cli831 (7.8%)
frontend503 (4.7%)
fullstack2 (0.0%)
Top topics
django196 (1.8%)
pandas156 (1.5%)
pydantic129 (1.2%)
click92 (0.9%)
fastapi83 (0.8%)
scikit-learn77 (0.7%)
xarray76 (0.7%)
argparse70 (0.7%)
numpy69 (0.6%)
ansible69 (0.6%)
jax66 (0.6%)
qiskit57 (0.5%)
pytorch56 (0.5%)
sqlalchemy55 (0.5%)
conan53 (0.5%)
home-assistant53 (0.5%)
opentelemetry51 (0.5%)
moto50 (0.5%)
cirq49 (0.5%)
scipy46 (0.4%)
matplotlib46 (0.4%)
flask42 (0.4%)
modin41 (0.4%)
dask41 (0.4%)
astroid40 (0.4%)
Bug classes
incomplete-validation871 (8.2%)
missing-metadata-propagation800 (7.5%)
missing-fallback793 (7.4%)
type-handling-inconsistency741 (6.9%)
missing-feature648 (6.1%)
wrong-default489 (4.6%)
missing-validation176 (1.6%)
missing-error-handling127 (1.2%)
missing-state-propagation118 (1.1%)
missing-implementation111 (1.0%)
missing-default80 (0.7%)
missing-method76 (0.7%)
missing-method-implementation54 (0.5%)
index-out-of-bounds45 (0.4%)
incomplete-parsing43 (0.4%)
race-condition38 (0.4%)
resource-leak34 (0.3%)
missing-parameter-propagation34 (0.3%)
missing-functionality30 (0.3%)
missing-exception-handling29 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
20,181
Mean difficulty
6.29
Median difficulty
6.5
Avg patch lines
315.1
Avg patch files
6.07
Difficulty distribution Scale-SWE
| Label breakdown | easy | medium | hard |
|---|---|---|---|
1545 / 6668 / 5168 |
1,545 | 6,668 | 5,168 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 13,381 | 1.0 | 5.2 | 6.5 | 6.29 | 7.6 | 9.1 |
Score bins
<=3577 (4.3%)
3.1-52,471 (18.5%)
5.1-75,165 (38.6%)
7.1-83,069 (22.9%)
>82,099 (15.7%)
Languages
python12,561 (93.9%)
c168 (1.3%)
cpp155 (1.2%)
scala61 (0.5%)
typescript60 (0.4%)
yaml44 (0.3%)
java44 (0.3%)
rust43 (0.3%)
cmake37 (0.3%)
javascript33 (0.2%)
dart32 (0.2%)
csharp31 (0.2%)
kotlin24 (0.2%)
go22 (0.2%)
shell17 (0.1%)
Area / tier
library8,092 (60.5%)
backend4,078 (30.5%)
cli795 (5.9%)
frontend415 (3.1%)
fullstack1 (0.0%)
Top topics
fonttools311 (2.3%)
globus-sdk220 (1.6%)
more-itertools208 (1.6%)
flatbuffers175 (1.3%)
aioesphomeapi167 (1.2%)
beets160 (1.2%)
pymongo154 (1.2%)
coloraide151 (1.1%)
ops143 (1.1%)
geopandas141 (1.1%)
grizz131 (1.0%)
pymarkdown128 (1.0%)
traitlets126 (0.9%)
jsonargparse121 (0.9%)
mcuboot116 (0.9%)
troposphere108 (0.8%)
sherpa-onnx103 (0.8%)
pybids102 (0.8%)
openshift94 (0.7%)
param93 (0.7%)
evidently89 (0.7%)
bioregistry87 (0.7%)
responses83 (0.6%)
wechatpy83 (0.6%)
compas80 (0.6%)
Bug classes
missing-feature1,347 (10.1%)
missing-fallback873 (6.5%)
missing-metadata-propagation724 (5.4%)
incomplete-validation723 (5.4%)
type-handling-inconsistency652 (4.9%)
wrong-default426 (3.2%)
missing-method229 (1.7%)
missing-validation144 (1.1%)
missing-error-handling120 (0.9%)
missing-functionality116 (0.9%)
missing-implementation112 (0.8%)
missing-type-annotations82 (0.6%)
missing-parameter78 (0.6%)
performance-optimization76 (0.6%)
missing-state-propagation73 (0.5%)
missing-function73 (0.5%)
missing-default64 (0.5%)
incomplete-parsing62 (0.5%)
race-condition57 (0.4%)
deprecated-api-usage53 (0.4%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)