Total tasks
45,881
Mean difficulty
6.30
Median difficulty
6.7
Avg patch lines
313.3
Avg patch files
6.41
Difficulty distribution SWE-Lego-Live-Instances
| Label breakdown | easy | medium | hard |
|---|---|---|---|
4617 / 22809 / 18455 |
4,617 | 22,809 | 18,455 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 45,881 | 1.0 | 5.6 | 6.7 | 6.30 | 7.4 | 8.0 |
Score bins
<=31,576 (3.4%)
3.1-56,648 (14.5%)
5.1-719,202 (41.9%)
7.1-818,455 (40.2%)
>80 (0.0%)
Languages
go7,487 (16.3%)
javascript6,592 (14.4%)
c6,466 (14.1%)
typescript6,426 (14.0%)
python5,535 (12.1%)
rust5,059 (11.0%)
java3,915 (8.5%)
cpp3,730 (8.1%)
ruby194 (0.4%)
vue102 (0.2%)
lua63 (0.1%)
php56 (0.1%)
solidity42 (0.1%)
yaml20 (0.0%)
cmake18 (0.0%)
Area / tier
backend24,598 (53.6%)
library10,855 (23.7%)
frontend6,639 (14.5%)
cli3,760 (8.2%)
fullstack27 (0.1%)
framework2 (0.0%)
Top topics
react606 (1.3%)
kubernetes284 (0.6%)
http197 (0.4%)
cobra196 (0.4%)
grpc161 (0.4%)
aem160 (0.3%)
aws-sdk145 (0.3%)
eslint145 (0.3%)
clap122 (0.3%)
mongoose122 (0.3%)
github-actions120 (0.3%)
git119 (0.3%)
electron117 (0.3%)
cmake117 (0.3%)
prometheus112 (0.2%)
pgbackrest110 (0.2%)
docker107 (0.2%)
redis106 (0.2%)
picoquic106 (0.2%)
igraph105 (0.2%)
openssl104 (0.2%)
fastify102 (0.2%)
svelte101 (0.2%)
express100 (0.2%)
ava96 (0.2%)
Bug classes
incomplete-validation3,075 (6.7%)
missing-feature2,405 (5.2%)
missing-fallback2,130 (4.6%)
missing-metadata-propagation1,723 (3.8%)
wrong-default1,627 (3.5%)
type-handling-inconsistency1,173 (2.6%)
missing-validation601 (1.3%)
missing-implementation507 (1.1%)
missing-state-propagation495 (1.1%)
race-condition407 (0.9%)
missing-api339 (0.7%)
missing-functionality273 (0.6%)
missing-error-handling241 (0.5%)
missing-method225 (0.5%)
resource-leak209 (0.5%)
incomplete-parsing207 (0.5%)
missing-config-option174 (0.4%)
missing-default148 (0.3%)
missing-function132 (0.3%)
feature-addition126 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
21,336
Mean difficulty
5.64
Median difficulty
6.0
Avg patch lines
142.2
Avg patch files
3.87
Difficulty distribution SWE-rebench
| Label breakdown | easy | medium | hard |
|---|---|---|---|
5349 / 9133 / 6854 |
5,349 | 9,133 | 6,854 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 21,336 | 1.0 | 4.0 | 6.0 | 5.64 | 7.4 | 9.2 |
Score bins
<=33,346 (15.7%)
3.1-54,316 (20.2%)
5.1-76,820 (32.0%)
7.1-84,191 (19.6%)
>82,663 (12.5%)
Languages
python21,216 (99.4%)
fortran43 (0.2%)
cpp41 (0.2%)
javascript7 (0.0%)
sql5 (0.0%)
vyper4 (0.0%)
typescript3 (0.0%)
rust3 (0.0%)
java2 (0.0%)
bash2 (0.0%)
c2 (0.0%)
solidity2 (0.0%)
cython1 (0.0%)
terraform1 (0.0%)
cmake1 (0.0%)
Area / tier
backend9,601 (45.0%)
library8,992 (42.1%)
cli1,784 (8.4%)
frontend958 (4.5%)
fullstack1 (0.0%)
Top topics
sympy732 (3.4%)
sqlglot454 (2.1%)
xarray340 (1.6%)
matplotlib272 (1.3%)
dvc215 (1.0%)
sphinx213 (1.0%)
dask198 (0.9%)
pybamm191 (0.9%)
argparse172 (0.8%)
pandas172 (0.8%)
pillow170 (0.8%)
astropy162 (0.8%)
conan156 (0.7%)
numpy153 (0.7%)
sqlfluff139 (0.7%)
pydicom137 (0.6%)
click135 (0.6%)
geopandas124 (0.6%)
networkx122 (0.6%)
nilearn109 (0.5%)
pennylane108 (0.5%)
pytorch-ignite108 (0.5%)
tox104 (0.5%)
narwhals95 (0.4%)
textual89 (0.4%)
Bug classes
incomplete-validation1,752 (8.2%)
type-handling-inconsistency1,465 (6.9%)
missing-metadata-propagation1,042 (4.9%)
missing-fallback1,036 (4.9%)
wrong-default930 (4.4%)
missing-feature838 (3.9%)
missing-validation247 (1.2%)
incomplete-parsing171 (0.8%)
missing-implementation163 (0.8%)
missing-state-propagation151 (0.7%)
missing-method125 (0.6%)
missing-default116 (0.5%)
missing-error-handling108 (0.5%)
index-out-of-bounds97 (0.5%)
missing-functionality91 (0.4%)
feature-addition88 (0.4%)
missing-method-implementation79 (0.4%)
incomplete-implementation65 (0.3%)
missing-exception-handling57 (0.3%)
race-condition55 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
32,079
Mean difficulty
5.80
Median difficulty
6.2
Avg patch lines
86.9
Avg patch files
4.22
Difficulty distribution SWE-rebench-V2
| Label breakdown | easy | medium | hard |
|---|---|---|---|
7319 / 13490 / 11268 |
7,319 | 13,490 | 11,268 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 32,077 | 1.0 | 4.3 | 6.2 | 5.80 | 7.5 | 9.2 |
Score bins
<=34,590 (14.3%)
3.1-55,972 (18.6%)
5.1-710,247 (31.9%)
7.1-86,982 (21.8%)
>84,286 (13.4%)
Languages
python7,222 (22.5%)
go6,147 (19.2%)
javascript4,371 (13.6%)
typescript3,989 (12.4%)
rust2,933 (9.1%)
java1,750 (5.5%)
php1,406 (4.4%)
kotlin857 (2.7%)
julia793 (2.5%)
elixir410 (1.3%)
scala399 (1.2%)
swift359 (1.1%)
dart251 (0.8%)
r159 (0.5%)
cpp158 (0.5%)
Area / tier
backend12,927 (40.3%)
library11,597 (36.2%)
frontend4,125 (12.9%)
cli3,419 (10.7%)
fullstack8 (0.0%)
framework1 (0.0%)
Top topics
swc407 (1.3%)
kubernetes356 (1.1%)
eslint297 (0.9%)
react268 (0.8%)
pandas250 (0.8%)
moto232 (0.7%)
serverless198 (0.6%)
statamic188 (0.6%)
cfn-lint162 (0.5%)
diktat158 (0.5%)
rust-analyzer157 (0.5%)
cobra153 (0.5%)
helm150 (0.5%)
pennylane135 (0.4%)
aiohttp133 (0.4%)
qiskit132 (0.4%)
swift-syntax117 (0.4%)
sveltekit117 (0.4%)
hypothesis114 (0.4%)
obsidian-linter111 (0.3%)
clap109 (0.3%)
laravel107 (0.3%)
knative103 (0.3%)
kiota101 (0.3%)
scalameta97 (0.3%)
Bug classes
incomplete-validation2,494 (7.8%)
missing-metadata-propagation1,525 (4.8%)
type-handling-inconsistency1,376 (4.3%)
missing-feature1,271 (4.0%)
missing-fallback1,124 (3.5%)
wrong-default1,060 (3.3%)
missing-validation363 (1.1%)
missing-state-propagation279 (0.9%)
incomplete-parsing223 (0.7%)
missing-implementation215 (0.7%)
false-positive163 (0.5%)
race-condition157 (0.5%)
missing-error-handling150 (0.5%)
index-out-of-bounds140 (0.4%)
missing-method128 (0.4%)
missing-default110 (0.3%)
missing-method-implementation104 (0.3%)
nil-pointer-dereference100 (0.3%)
missing-function-implementation88 (0.3%)
off-by-one85 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
22,806
Mean difficulty
5.36
Median difficulty
5.8
Avg patch lines
61.8
Avg patch files
2.41
Difficulty distribution OpenSWE-filtered
| Label breakdown | easy | medium | hard |
|---|---|---|---|
6209 / 11129 / 5468 |
6,209 | 11,129 | 5,468 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 22,806 | 1.0 | 3.8 | 5.8 | 5.36 | 7.0 | 9.1 |
Score bins
<=34,214 (18.5%)
3.1-54,627 (20.3%)
5.1-78,497 (37.3%)
7.1-84,547 (19.9%)
>8921 (4.0%)
Languages
python22,686 (99.5%)
fortran37 (0.2%)
cpp25 (0.1%)
java9 (0.0%)
cython8 (0.0%)
typescript4 (0.0%)
c4 (0.0%)
javascript4 (0.0%)
solidity4 (0.0%)
rust3 (0.0%)
julia2 (0.0%)
django2 (0.0%)
go2 (0.0%)
elixir1 (0.0%)
markdown1 (0.0%)
Area / tier
backend11,616 (50.9%)
library8,096 (35.5%)
cli1,928 (8.5%)
frontend1,165 (5.1%)
fullstack1 (0.0%)
Top topics
django507 (2.2%)
pandas300 (1.3%)
pydantic275 (1.2%)
fastapi200 (0.9%)
click184 (0.8%)
numpy154 (0.7%)
xarray150 (0.7%)
argparse149 (0.7%)
pytorch145 (0.6%)
scikit-learn123 (0.5%)
scipy111 (0.5%)
flask108 (0.5%)
matplotlib107 (0.5%)
jax103 (0.5%)
qiskit103 (0.5%)
sqlalchemy101 (0.4%)
opentelemetry97 (0.4%)
home-assistant85 (0.4%)
ansible81 (0.4%)
jinja278 (0.3%)
requests77 (0.3%)
kubernetes74 (0.3%)
dask74 (0.3%)
asyncio72 (0.3%)
fonttools65 (0.3%)
Bug classes
incomplete-validation1,833 (8.0%)
type-handling-inconsistency1,268 (5.6%)
missing-metadata-propagation1,016 (4.5%)
missing-fallback993 (4.4%)
missing-feature895 (3.9%)
wrong-default815 (3.6%)
missing-validation318 (1.4%)
missing-implementation227 (1.0%)
missing-state-propagation193 (0.8%)
missing-error-handling164 (0.7%)
missing-default131 (0.6%)
missing-method113 (0.5%)
index-out-of-bounds99 (0.4%)
incomplete-implementation87 (0.4%)
incomplete-parsing86 (0.4%)
missing-method-implementation85 (0.4%)
missing-functionality81 (0.4%)
race-condition78 (0.3%)
feature-addition78 (0.3%)
missing-parameter-propagation70 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
20,181
Mean difficulty
5.98
Median difficulty
6.3
Avg patch lines
321.4
Avg patch files
6.06
Difficulty distribution Scale-SWE
| Label breakdown | easy | medium | hard |
|---|---|---|---|
2314 / 12361 / 5506 |
2,314 | 12,361 | 5,506 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 20,181 | 1.0 | 5.2 | 6.3 | 5.98 | 7.1 | 7.8 |
Score bins
<=3795 (3.9%)
3.1-53,808 (18.9%)
5.1-710,072 (49.9%)
7.1-85,506 (27.3%)
>80 (0.0%)
Languages
python19,284 (95.6%)
c176 (0.9%)
cpp152 (0.8%)
typescript72 (0.4%)
scala61 (0.3%)
java60 (0.3%)
javascript50 (0.2%)
yaml43 (0.2%)
rust42 (0.2%)
cmake36 (0.2%)
dart31 (0.2%)
csharp28 (0.1%)
kotlin24 (0.1%)
go23 (0.1%)
shell19 (0.1%)
Area / tier
library10,187 (50.5%)
backend7,988 (39.6%)
cli1,275 (6.3%)
frontend726 (3.6%)
fullstack4 (0.0%)
framework1 (0.0%)
Top topics
fonttools310 (1.5%)
click283 (1.4%)
globus-sdk213 (1.1%)
more-itertools207 (1.0%)
flatbuffers175 (0.9%)
beets164 (0.8%)
django164 (0.8%)
aioesphomeapi164 (0.8%)
pydantic159 (0.8%)
xsdata155 (0.8%)
geopandas148 (0.7%)
pymongo148 (0.7%)
jsonargparse146 (0.7%)
mesa144 (0.7%)
coloraide131 (0.6%)
pymarkdown125 (0.6%)
poetry-core125 (0.6%)
poetry124 (0.6%)
traitlets121 (0.6%)
ops119 (0.6%)
pytest118 (0.6%)
grizz115 (0.6%)
mcuboot112 (0.6%)
troposphere108 (0.5%)
latex2mathml107 (0.5%)
Bug classes
incomplete-validation1,472 (7.3%)
missing-feature1,429 (7.1%)
missing-fallback1,128 (5.6%)
type-handling-inconsistency920 (4.6%)
missing-metadata-propagation832 (4.1%)
wrong-default694 (3.4%)
missing-validation205 (1.0%)
missing-method203 (1.0%)
missing-functionality183 (0.9%)
missing-implementation145 (0.7%)
missing-error-handling128 (0.6%)
feature-addition115 (0.6%)
incomplete-parsing112 (0.6%)
performance-optimization101 (0.5%)
missing-state-propagation97 (0.5%)
missing-type-annotations90 (0.4%)
missing-parameter89 (0.4%)
missing-default79 (0.4%)
race-condition67 (0.3%)
missing-api-method60 (0.3%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)