Total tasks
53,080
Mean difficulty
6.27
Median difficulty
6.7
Avg patch lines
376.0
Avg patch files
6.79
Difficulty distribution SWE-Lego-Live-Instances
| Label breakdown | easy | medium | hard |
|---|---|---|---|
5637 / 26526 / 20916 |
5,637 | 26,526 | 20,916 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 53,079 | 1.0 | 5.5 | 6.7 | 6.27 | 7.4 | 8.0 |
Score bins
<=32,032 (3.8%)
3.1-57,830 (14.8%)
5.1-722,301 (42.0%)
7.1-820,916 (39.4%)
>80 (0.0%)
Languages
c9,458 (17.8%)
go8,664 (16.3%)
typescript7,088 (13.4%)
javascript6,940 (13.1%)
python5,983 (11.3%)
rust5,580 (10.5%)
java4,346 (8.2%)
cpp4,226 (8.0%)
ruby211 (0.4%)
lua116 (0.2%)
vue105 (0.2%)
php63 (0.1%)
solidity43 (0.1%)
c++21 (0.0%)
cmake21 (0.0%)
Area / tier
backend28,199 (53.1%)
library12,695 (23.9%)
frontend7,627 (14.4%)
cli4,525 (8.5%)
fullstack31 (0.1%)
framework2 (0.0%)
Top topics
react657 (1.2%)
kubernetes320 (0.6%)
http191 (0.4%)
grpc187 (0.4%)
cobra181 (0.3%)
aem165 (0.3%)
eslint158 (0.3%)
aws-sdk147 (0.3%)
prometheus138 (0.3%)
clap136 (0.3%)
cmake135 (0.3%)
mongoose133 (0.3%)
picoquic130 (0.2%)
git126 (0.2%)
electron125 (0.2%)
openssl116 (0.2%)
github-actions113 (0.2%)
redis112 (0.2%)
express112 (0.2%)
svelte107 (0.2%)
fastify107 (0.2%)
pgbackrest106 (0.2%)
igraph105 (0.2%)
nimble101 (0.2%)
fastapi100 (0.2%)
Bug classes
incomplete-validation3,233 (6.1%)
missing-feature2,987 (5.6%)
missing-fallback2,135 (4.0%)
missing-metadata-propagation1,982 (3.7%)
wrong-default1,682 (3.2%)
type-handling-inconsistency1,270 (2.4%)
missing-validation631 (1.2%)
missing-state-propagation574 (1.1%)
missing-implementation555 (1.0%)
race-condition505 (1.0%)
missing-api400 (0.8%)
missing-error-handling286 (0.5%)
missing-functionality269 (0.5%)
resource-leak245 (0.5%)
missing-method234 (0.4%)
incomplete-parsing232 (0.4%)
missing-config-option207 (0.4%)
missing-default148 (0.3%)
memory-leak146 (0.3%)
missing-function141 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
8,443
Mean difficulty
0.00
Median difficulty
0.0
Avg patch lines
0.0
Avg patch files
0.00
Difficulty distribution SWE-rebench
| Label breakdown | easy | medium | hard |
|---|---|---|---|
no data |
0 | 0 | 0 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 0 | 0.0 | 0.0 | 0.0 | 0.00 | 0.0 | 0.0 |
Score bins
<=30 (0.0%)
3.1-50 (0.0%)
5.1-70 (0.0%)
7.1-80 (0.0%)
>80 (0.0%)
Languages
no tags data
Area / tier
no tags data
Top topics
no tags data
Bug classes
no tags data
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
32,079
Mean difficulty
5.80
Median difficulty
6.2
Avg patch lines
86.9
Avg patch files
4.22
Difficulty distribution SWE-rebench-V2
| Label breakdown | easy | medium | hard |
|---|---|---|---|
7319 / 13490 / 11270 |
7,319 | 13,490 | 11,270 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 32,079 | 1.0 | 4.3 | 6.2 | 5.80 | 7.5 | 9.2 |
Score bins
<=34,590 (14.3%)
3.1-55,972 (18.6%)
5.1-710,247 (31.9%)
7.1-86,984 (21.8%)
>84,286 (13.4%)
Languages
python7,223 (22.5%)
go6,147 (19.2%)
javascript4,371 (13.6%)
typescript3,990 (12.4%)
rust2,933 (9.1%)
java1,750 (5.5%)
php1,406 (4.4%)
kotlin857 (2.7%)
julia793 (2.5%)
elixir410 (1.3%)
scala399 (1.2%)
swift359 (1.1%)
dart251 (0.8%)
r159 (0.5%)
cpp158 (0.5%)
Area / tier
backend12,928 (40.3%)
library11,598 (36.2%)
frontend4,125 (12.9%)
cli3,419 (10.7%)
fullstack8 (0.0%)
framework1 (0.0%)
Top topics
swc407 (1.3%)
kubernetes356 (1.1%)
eslint297 (0.9%)
react268 (0.8%)
pandas251 (0.8%)
moto232 (0.7%)
serverless198 (0.6%)
statamic188 (0.6%)
cfn-lint162 (0.5%)
diktat158 (0.5%)
rust-analyzer157 (0.5%)
cobra153 (0.5%)
helm150 (0.5%)
pennylane135 (0.4%)
aiohttp133 (0.4%)
qiskit132 (0.4%)
swift-syntax117 (0.4%)
sveltekit117 (0.4%)
hypothesis114 (0.4%)
obsidian-linter111 (0.3%)
clap109 (0.3%)
laravel107 (0.3%)
knative103 (0.3%)
kiota101 (0.3%)
scalameta97 (0.3%)
Bug classes
incomplete-validation2,494 (7.8%)
missing-metadata-propagation1,525 (4.8%)
type-handling-inconsistency1,376 (4.3%)
missing-feature1,271 (4.0%)
missing-fallback1,125 (3.5%)
wrong-default1,060 (3.3%)
missing-validation363 (1.1%)
missing-state-propagation279 (0.9%)
incomplete-parsing223 (0.7%)
missing-implementation215 (0.7%)
false-positive163 (0.5%)
race-condition157 (0.5%)
missing-error-handling150 (0.5%)
index-out-of-bounds140 (0.4%)
missing-method128 (0.4%)
missing-default110 (0.3%)
missing-method-implementation104 (0.3%)
nil-pointer-dereference100 (0.3%)
missing-function-implementation88 (0.3%)
off-by-one85 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
22,806
Mean difficulty
5.31
Median difficulty
5.7
Avg patch lines
60.2
Avg patch files
2.37
Difficulty distribution OpenSWE-filtered
| Label breakdown | easy | medium | hard |
|---|---|---|---|
3021 / 5101 / 2550 |
3,021 | 5,101 | 2,550 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 10,672 | 1.6 | 3.7 | 5.7 | 5.31 | 7.0 | 9.0 |
Score bins
<=32,085 (19.5%)
3.1-52,165 (20.3%)
5.1-73,872 (36.3%)
7.1-82,127 (19.9%)
>8423 (4.0%)
Languages
python10,618 (99.5%)
fortran22 (0.2%)
cpp7 (0.1%)
cython5 (0.0%)
java4 (0.0%)
solidity4 (0.0%)
typescript3 (0.0%)
c3 (0.0%)
elixir1 (0.0%)
basic1 (0.0%)
markdown1 (0.0%)
r1 (0.0%)
fish1 (0.0%)
julia1 (0.0%)
Area / tier
backend5,473 (51.3%)
library3,863 (36.2%)
cli831 (7.8%)
frontend503 (4.7%)
fullstack2 (0.0%)
Top topics
django196 (1.8%)
pandas156 (1.5%)
pydantic129 (1.2%)
click92 (0.9%)
fastapi83 (0.8%)
scikit-learn77 (0.7%)
xarray76 (0.7%)
argparse70 (0.7%)
numpy69 (0.6%)
ansible69 (0.6%)
jax66 (0.6%)
qiskit57 (0.5%)
pytorch56 (0.5%)
sqlalchemy55 (0.5%)
conan53 (0.5%)
home-assistant53 (0.5%)
opentelemetry51 (0.5%)
moto50 (0.5%)
cirq49 (0.5%)
scipy46 (0.4%)
matplotlib46 (0.4%)
flask42 (0.4%)
modin41 (0.4%)
dask41 (0.4%)
astroid40 (0.4%)
Bug classes
incomplete-validation871 (8.2%)
missing-metadata-propagation800 (7.5%)
missing-fallback793 (7.4%)
type-handling-inconsistency741 (6.9%)
missing-feature648 (6.1%)
wrong-default489 (4.6%)
missing-validation176 (1.6%)
missing-error-handling127 (1.2%)
missing-state-propagation118 (1.1%)
missing-implementation111 (1.0%)
missing-default80 (0.7%)
missing-method76 (0.7%)
missing-method-implementation54 (0.5%)
index-out-of-bounds45 (0.4%)
incomplete-parsing43 (0.4%)
race-condition38 (0.4%)
resource-leak34 (0.3%)
missing-parameter-propagation34 (0.3%)
missing-functionality30 (0.3%)
missing-exception-handling29 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
20,181
Mean difficulty
0.00
Median difficulty
0.0
Avg patch lines
0.0
Avg patch files
0.00
Difficulty distribution Scale-SWE
| Label breakdown | easy | medium | hard |
|---|---|---|---|
no data |
0 | 0 | 0 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 0 | 0.0 | 0.0 | 0.0 | 0.00 | 0.0 | 0.0 |
Score bins
<=30 (0.0%)
3.1-50 (0.0%)
5.1-70 (0.0%)
7.1-80 (0.0%)
>80 (0.0%)
Languages
no tags data
Area / tier
no tags data
Top topics
no tags data
Bug classes
no tags data
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)