Total tasks
45,881
Tagged (LLM)
45,875 100.0%
Mean difficulty
6.30
Median difficulty
6.7
Avg patch lines
313.3
Avg patch files
6.41
Difficulty distribution Self-Made
| Label breakdown | easy | medium | hard |
|---|---|---|---|
4617 / 22805 / 18453 |
4,617 | 22,805 | 18,453 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 45,875 | 1.0 | 5.6 | 6.7 | 6.30 | 7.4 | 8.0 |
Score bins
<=31,576 (3.4%)
3.1-56,648 (14.5%)
5.1-719,198 (41.8%)
7.1-818,453 (40.2%)
>80 (0.0%)
Languages 45875 tagged
go7,487 (16.3%)
javascript6,592 (14.4%)
c6,465 (14.1%)
typescript6,423 (14.0%)
python5,534 (12.1%)
rust5,059 (11.0%)
java3,914 (8.5%)
cpp3,730 (8.1%)
ruby194 (0.4%)
vue102 (0.2%)
lua63 (0.1%)
php56 (0.1%)
solidity42 (0.1%)
yaml20 (0.0%)
cmake18 (0.0%)
Area / tier 45875 tagged
backend24,594 (53.6%)
library10,855 (23.7%)
frontend6,637 (14.5%)
cli3,760 (8.2%)
fullstack27 (0.1%)
framework2 (0.0%)
Top topics 45875 tagged
react606 (1.3%)
kubernetes284 (0.6%)
http197 (0.4%)
cobra196 (0.4%)
grpc161 (0.4%)
aem160 (0.3%)
aws-sdk145 (0.3%)
eslint145 (0.3%)
clap122 (0.3%)
mongoose122 (0.3%)
github-actions120 (0.3%)
git119 (0.3%)
cmake117 (0.3%)
electron116 (0.3%)
prometheus112 (0.2%)
pgbackrest110 (0.2%)
docker107 (0.2%)
redis106 (0.2%)
picoquic106 (0.2%)
igraph105 (0.2%)
openssl104 (0.2%)
fastify102 (0.2%)
svelte101 (0.2%)
express100 (0.2%)
ava96 (0.2%)
Bug classes 45875 tagged
incomplete-validation3,075 (6.7%)
missing-feature2,405 (5.2%)
missing-fallback2,130 (4.6%)
missing-metadata-propagation1,723 (3.8%)
wrong-default1,627 (3.5%)
type-handling-inconsistency1,173 (2.6%)
missing-validation601 (1.3%)
missing-implementation507 (1.1%)
missing-state-propagation495 (1.1%)
race-condition407 (0.9%)
missing-api339 (0.7%)
missing-functionality273 (0.6%)
missing-error-handling241 (0.5%)
missing-method225 (0.5%)
resource-leak209 (0.5%)
incomplete-parsing207 (0.5%)
missing-config-option174 (0.4%)
missing-default148 (0.3%)
missing-function132 (0.3%)
feature-addition126 (0.3%)
Tag schema harbor task_analysis
Every task is tagged with the 4-tuple
area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).
[language, area, topic, bug_class]area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).
Total tasks
20,181
Tagged (LLM)
20,178 100.0%
Mean difficulty
5.98
Median difficulty
6.3
Avg patch lines
321.4
Avg patch files
6.06
Difficulty distribution Scale-SWE
| Label breakdown | easy | medium | hard |
|---|---|---|---|
2314 / 12360 / 5504 |
2,314 | 12,360 | 5,504 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 20,178 | 1.0 | 5.2 | 6.3 | 5.98 | 7.1 | 7.8 |
Score bins
<=3795 (3.9%)
3.1-53,808 (18.9%)
5.1-710,071 (49.9%)
7.1-85,504 (27.3%)
>80 (0.0%)
Languages 20178 tagged
python19,281 (95.6%)
c176 (0.9%)
cpp152 (0.8%)
typescript72 (0.4%)
scala61 (0.3%)
java60 (0.3%)
javascript50 (0.2%)
yaml43 (0.2%)
rust42 (0.2%)
cmake36 (0.2%)
dart31 (0.2%)
csharp28 (0.1%)
kotlin24 (0.1%)
go23 (0.1%)
shell19 (0.1%)
Area / tier 20178 tagged
library10,186 (50.5%)
backend7,986 (39.6%)
cli1,275 (6.3%)
frontend726 (3.6%)
fullstack4 (0.0%)
framework1 (0.0%)
Top topics 20178 tagged
fonttools310 (1.5%)
click283 (1.4%)
globus-sdk213 (1.1%)
more-itertools207 (1.0%)
flatbuffers175 (0.9%)
beets164 (0.8%)
django164 (0.8%)
aioesphomeapi164 (0.8%)
pydantic159 (0.8%)
xsdata155 (0.8%)
geopandas148 (0.7%)
pymongo148 (0.7%)
jsonargparse146 (0.7%)
mesa144 (0.7%)
coloraide131 (0.6%)
pymarkdown125 (0.6%)
poetry-core125 (0.6%)
poetry124 (0.6%)
traitlets121 (0.6%)
ops119 (0.6%)
pytest118 (0.6%)
grizz115 (0.6%)
mcuboot112 (0.6%)
troposphere108 (0.5%)
latex2mathml107 (0.5%)
Bug classes 20178 tagged
incomplete-validation1,472 (7.3%)
missing-feature1,429 (7.1%)
missing-fallback1,128 (5.6%)
type-handling-inconsistency920 (4.6%)
missing-metadata-propagation832 (4.1%)
wrong-default694 (3.4%)
missing-validation205 (1.0%)
missing-method203 (1.0%)
missing-functionality183 (0.9%)
missing-implementation145 (0.7%)
missing-error-handling128 (0.6%)
feature-addition115 (0.6%)
incomplete-parsing112 (0.6%)
performance-optimization101 (0.5%)
missing-state-propagation97 (0.5%)
missing-type-annotations90 (0.4%)
missing-parameter89 (0.4%)
missing-default79 (0.4%)
race-condition67 (0.3%)
missing-api-method60 (0.3%)
Tag schema harbor task_analysis
Every task is tagged with the 4-tuple
area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).
[language, area, topic, bug_class]area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).
Total tasks
6,542
Tagged (LLM)
6,540 100.0%
Mean difficulty
4.60
Median difficulty
4.7
Avg patch lines
36.0
Avg patch files
2.20
Difficulty distribution SWE-rebench (filtered)
| Label breakdown | easy | medium | hard |
|---|---|---|---|
2637 / 3213 / 690 |
2,637 | 3,213 | 690 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 6,540 | 1.0 | 3.0 | 4.7 | 4.60 | 6.1 | 8.8 |
Score bins
<=31,654 (25.3%)
3.1-51,982 (30.3%)
5.1-72,214 (33.9%)
7.1-8561 (8.6%)
>8129 (2.0%)
Languages 6540 tagged
python6,506 (99.5%)
cpp18 (0.3%)
fortran3 (0.0%)
c2 (0.0%)
javascript2 (0.0%)
sqf1 (0.0%)
shell1 (0.0%)
clojure1 (0.0%)
yaml1 (0.0%)
go1 (0.0%)
rust1 (0.0%)
sql1 (0.0%)
php1 (0.0%)
vyper1 (0.0%)
Area / tier 6540 tagged
backend3,067 (46.9%)
library2,617 (40.0%)
cli612 (9.4%)
frontend244 (3.7%)
Top topics 6540 tagged
sqlglot294 (4.5%)
pennylane76 (1.2%)
dask72 (1.1%)
conan67 (1.0%)
dvc63 (1.0%)
pytorch-ignite60 (0.9%)
pandas57 (0.9%)
sqlfluff57 (0.9%)
tox56 (0.9%)
streamlink54 (0.8%)
textual49 (0.7%)
pre-commit44 (0.7%)
click43 (0.7%)
pydicom42 (0.6%)
argparse40 (0.6%)
zarr40 (0.6%)
numba39 (0.6%)
networkx36 (0.6%)
numpy34 (0.5%)
scrapy33 (0.5%)
pint33 (0.5%)
starlette33 (0.5%)
pyupgrade32 (0.5%)
sqllineage32 (0.5%)
wemake-python-styleguide31 (0.5%)
Bug classes 6540 tagged
type-handling-inconsistency511 (7.8%)
incomplete-validation495 (7.6%)
wrong-default379 (5.8%)
missing-fallback331 (5.1%)
missing-metadata-propagation258 (3.9%)
missing-feature169 (2.6%)
incomplete-parsing75 (1.1%)
missing-validation55 (0.8%)
index-out-of-bounds49 (0.7%)
missing-default47 (0.7%)
missing-error-handling37 (0.6%)
missing-state-propagation36 (0.6%)
parsing-error28 (0.4%)
missing-exception-handling26 (0.4%)
off-by-one26 (0.4%)
race-condition24 (0.4%)
false-positive23 (0.4%)
missing-parameter20 (0.3%)
missing-implementation17 (0.3%)
missing-syntax-support16 (0.2%)
Tag schema harbor task_analysis
Every task is tagged with the 4-tuple
area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).
[language, area, topic, bug_class]area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).
Total tasks
21,336
Tagged (LLM)
21,331 100.0%
Mean difficulty
5.64
Median difficulty
6.0
Avg patch lines
142.1
Avg patch files
3.87
Difficulty distribution SWE-rebench (test)
| Label breakdown | easy | medium | hard |
|---|---|---|---|
5349 / 9132 / 6850 |
5,349 | 9,132 | 6,850 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 21,331 | 1.0 | 4.0 | 6.0 | 5.64 | 7.4 | 9.2 |
Score bins
<=33,346 (15.7%)
3.1-54,316 (20.2%)
5.1-76,819 (32.0%)
7.1-84,189 (19.6%)
>82,661 (12.5%)
Languages 21331 tagged
python21,211 (99.4%)
fortran43 (0.2%)
cpp41 (0.2%)
javascript7 (0.0%)
sql5 (0.0%)
vyper4 (0.0%)
typescript3 (0.0%)
rust3 (0.0%)
java2 (0.0%)
bash2 (0.0%)
c2 (0.0%)
solidity2 (0.0%)
cython1 (0.0%)
terraform1 (0.0%)
cmake1 (0.0%)
Area / tier 21331 tagged
backend9,598 (45.0%)
library8,990 (42.1%)
cli1,784 (8.4%)
frontend958 (4.5%)
fullstack1 (0.0%)
Top topics 21331 tagged
sympy732 (3.4%)
sqlglot454 (2.1%)
xarray340 (1.6%)
matplotlib272 (1.3%)
dvc215 (1.0%)
sphinx213 (1.0%)
dask198 (0.9%)
pybamm191 (0.9%)
argparse172 (0.8%)
pandas172 (0.8%)
pillow170 (0.8%)
astropy161 (0.8%)
conan156 (0.7%)
numpy153 (0.7%)
sqlfluff139 (0.7%)
pydicom137 (0.6%)
click135 (0.6%)
geopandas124 (0.6%)
networkx122 (0.6%)
nilearn109 (0.5%)
pennylane108 (0.5%)
pytorch-ignite108 (0.5%)
tox104 (0.5%)
narwhals95 (0.4%)
textual89 (0.4%)
Bug classes 21331 tagged
incomplete-validation1,752 (8.2%)
type-handling-inconsistency1,465 (6.9%)
missing-metadata-propagation1,042 (4.9%)
missing-fallback1,036 (4.9%)
wrong-default930 (4.4%)
missing-feature838 (3.9%)
missing-validation247 (1.2%)
incomplete-parsing171 (0.8%)
missing-implementation163 (0.8%)
missing-state-propagation151 (0.7%)
missing-method125 (0.6%)
missing-default116 (0.5%)
missing-error-handling108 (0.5%)
index-out-of-bounds96 (0.5%)
missing-functionality91 (0.4%)
feature-addition88 (0.4%)
missing-method-implementation79 (0.4%)
incomplete-implementation65 (0.3%)
missing-exception-handling57 (0.3%)
race-condition55 (0.3%)
Tag schema harbor task_analysis
Every task is tagged with the 4-tuple
area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).
[language, area, topic, bug_class]area ∈ {backend, frontend, fullstack, cli, library, framework}.
Difficulty is a 5-dimension weighted score (1–10).