SWE-gen/instance
Updated 2026-06-25 22:12:00 BJT · next 2026-06-25 23:12:01 BJT
Overview

Live Dashboard of PRs collections and SWE Tasks Generation

Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.

Total PRs collected
589,514
23,810 unique repos · 1h 0 / 24h +9
Total valid SWE
50,207
9,933 unique repos · 1h +8 / 24h +123
Overall success rate
9.6%
Valid SWE / processed 520,950
Mean difficulty_score
5.89
median 5.8, count 50,157

Language Progress

LanguagePRs collectedLast 1hLast 24hValid SWELast 1hLast 24hProcessedSuccess rate
Cc32,357009,7140+432,357
30.0%
C++cpp48,1740+14,0460+424,478
16.5%
Gogo132,103008,150+1+2495,634
8.5%
Javajava89,0310+24,052+3+2388,907
4.6%
JavaScriptjs39,682007,191+2+2739,682
18.1%
Pythonpy105,7730+45,068+1+8105,659
4.8%
Rustrust71,9020+15,5200+1071,857
7.7%
TypeScriptts70,4920+16,466+1+2362,376
10.4%

Run Parameters

LanguageEval model (OPENAI)Completion model (ANTHROPIC)Concurrencymin_source_filesmax_source_files
Cgpt-5.4claude-sonnet-4-612215
C++Qwen3.6-35B-A3BQwen3.6-35B-A3B8215
GoQwen3.6-35B-A3BQwen3.6-35B-A3B12210
Javagpt-5.4claude-sonnet-4-68210
JavaScriptQwen3.6-35B-A3BQwen3.6-35B-A3B12210
Pythonglm-5claude-sonnet-4-612315
RustQwen3.6-35B-A3BQwen3.6-35B-A3B8210
TypeScriptQwen3.6-35B-A3BQwen3.6-35B-A3B12210

Failure Reason Breakdown

click to expand
LanguageProcessedValid SWEFailedtrivial_prvalidationinfra_errortimeoutworkflow_errorOther
C32,3579,71422,64315,9012,8733,913112301
C++24,4784,04620,4324,3492,19615,244159464266
Go95,6348,15087,48425,35311,46247,6541,604993406
Java88,9074,05284,85523,2737,55743,5631,2408,0251,601
JavaScript39,6827,19132,49116,8512,65613,6564532570
Python105,6595,068100,59134,77610,44455,634948387120
Rust71,8575,52066,33721,6176,87835,0301,2311,0061,234
TypeScript62,3766,46655,91017,1217,60328,8641,76287611

trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.

validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).

infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).

timeout: processing timed out (per-PR total timeout or Claude Code session timeout).

workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).

fix.patch Complexity

click to expand
LanguageValid SWE CountAvg fix.patch linesAvg fix.patch hunksAvg fix.patch files
C9,714336.5217.945.85
C++4,046286.5413.765.10
Go8,150212.3512.634.35
Java4,052163.1210.504.23
JavaScript7,19177.116.302.80
Python5,068152.1011.043.85
Rust5,520225.6613.174.10
TypeScript6,466157.869.614.13

Metric Definitions

click to expand

Difficulty score (difficulty_score)

Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.

The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.

Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.

Tag generation and display

tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.

The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.

fix.patch statistics

Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.

Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.

difficulty_label Distribution

click to expand
Languageeasy / medium / hardeasymediumhard
C
869 / 6416 / 2421
8696,4162,421
C++
430 / 2514 / 1097
4302,5141,097
Go
645 / 5887 / 1612
6455,8871,612
Java
449 / 2623 / 976
4492,623976
JavaScript
1086 / 5327 / 777
1,0865,327777
Python
273 / 3252 / 1519
2733,2521,519
Rust
382 / 3269 / 1867
3823,2691,867
TypeScript
583 / 4778 / 1105
5834,7781,105

difficulty_score Overview

click to expand
Languagecountminp25medianmeanp75max
C9,7062.44.96.05.977.09.2
C++4,0412.54.96.05.997.29.1
Go8,1442.64.95.85.866.89.1
Java4,0482.84.85.95.897.09.2
JavaScript7,1902.64.45.25.366.29.2
Python5,0442.65.26.26.227.39.1
Rust5,5182.75.26.36.267.49.0
TypeScript6,4662.74.75.65.726.69.2

Global Top Tags

click to expand
library23,376 (46.6%)
backend15,448 (30.8%)
cli6,794 (13.5%)
missing-feature3,504 (7.0%)
frontend3,087 (6.2%)
testing2,186 (4.4%)
http1,543 (3.1%)
react1,403 (2.8%)
incomplete-validation1,310 (2.6%)
framework1,226 (2.4%)
missing-implementation1,188 (2.4%)
missing-metadata-propagation926 (1.8%)
missing-fallback846 (1.7%)
kubernetes714 (1.4%)
async676 (1.3%)
wrong-default647 (1.3%)
embedded637 (1.3%)
networking616 (1.2%)
cpp594 (1.2%)
missing-validation551 (1.1%)
missing-functionality445 (0.9%)
missing-configuration409 (0.8%)
typescript389 (0.8%)
type-handling-inconsistency371 (0.7%)
parsing358 (0.7%)
eslint343 (0.7%)
graphql336 (0.7%)
race-condition328 (0.7%)
missing-configuration-option299 (0.6%)
postgresql298 (0.6%)

Per-Language Tag Distribution

click to expand

C c

library5,097 (52.5%)
backend2,512 (25.9%)
cli1,120 (11.5%)
missing-feature647 (6.7%)
embedded611 (6.3%)
cpp591 (6.1%)
testing415 (4.3%)
networking400 (4.1%)
framework293 (3.0%)
missing-implementation283 (2.9%)
incomplete-validation272 (2.8%)
ruby213 (2.2%)
postgresql212 (2.2%)
http202 (2.1%)
firmware183 (1.9%)
kernel177 (1.8%)
quic168 (1.7%)
missing-metadata-propagation150 (1.5%)
rust140 (1.4%)
missing-functionality133 (1.4%)

C++ cpp

library2,786 (68.9%)
backend735 (18.2%)
testing492 (12.2%)
cli357 (8.8%)
missing-feature268 (6.6%)
framework186 (4.6%)
missing-implementation144 (3.6%)
http129 (3.2%)
incomplete-validation118 (2.9%)
boost114 (2.8%)
async91 (2.3%)
parsing76 (1.9%)
qt63 (1.6%)
compiler54 (1.3%)
serialization54 (1.3%)
geometry51 (1.3%)
missing-fallback50 (1.2%)
networking49 (1.2%)
missing-metadata-propagation48 (1.2%)
formatting45 (1.1%)

Go go

backend4,272 (52.5%)
library2,047 (25.1%)
cli1,977 (24.3%)
missing-feature727 (8.9%)
kubernetes652 (8.0%)
http523 (6.4%)
incomplete-validation241 (3.0%)
testing225 (2.8%)
missing-metadata-propagation202 (2.5%)
missing-implementation177 (2.2%)
missing-fallback173 (2.1%)
docker134 (1.6%)
wrong-default129 (1.6%)
aws120 (1.5%)
grpc119 (1.5%)
terraform119 (1.5%)
missing-validation116 (1.4%)
prometheus104 (1.3%)
networking97 (1.2%)
database90 (1.1%)

Java java

backend1,967 (48.6%)
library1,828 (45.1%)
missing-feature220 (5.4%)
testing214 (5.3%)
spring160 (4.0%)
framework152 (3.8%)
aem147 (3.6%)
http138 (3.4%)
android122 (3.0%)
incomplete-validation92 (2.3%)
missing-metadata-propagation92 (2.3%)
missing-implementation82 (2.0%)
json67 (1.7%)
missing-configuration67 (1.7%)
cli66 (1.6%)
missing-null-check64 (1.6%)
wrong-default61 (1.5%)
maven59 (1.5%)
sling57 (1.4%)
kafka54 (1.3%)

JavaScript js

library3,875 (53.9%)
backend1,235 (17.2%)
frontend1,027 (14.3%)
cli856 (11.9%)
missing-feature490 (6.8%)
react352 (4.9%)
typescript326 (4.5%)
testing295 (4.1%)
eslint262 (3.6%)
incomplete-validation219 (3.0%)
framework199 (2.8%)
http181 (2.5%)
fastify148 (2.1%)
webpack146 (2.0%)
missing-fallback122 (1.7%)
missing-metadata-propagation113 (1.6%)
mongoose104 (1.4%)
nodejs104 (1.4%)
svelte103 (1.4%)
missing-implementation100 (1.4%)

Python py

library2,141 (42.4%)
backend2,101 (41.6%)
cli698 (13.8%)
missing-feature408 (8.1%)
fastapi206 (4.1%)
missing-implementation145 (2.9%)
django143 (2.8%)
testing115 (2.3%)
missing-fallback113 (2.2%)
missing-metadata-propagation106 (2.1%)
pytorch106 (2.1%)
incomplete-validation101 (2.0%)
ansible100 (2.0%)
async100 (2.0%)
framework99 (2.0%)
http93 (1.8%)
aws74 (1.5%)
aiohttp65 (1.3%)
missing-parameter63 (1.2%)
pydantic60 (1.2%)

Rust rust

library3,095 (56.1%)
backend1,195 (21.7%)
cli1,128 (20.4%)
missing-feature407 (7.4%)
testing308 (5.6%)
async223 (4.0%)
http207 (3.8%)
missing-implementation160 (2.9%)
incomplete-validation131 (2.4%)
compiler120 (2.2%)
git117 (2.1%)
missing-metadata-propagation97 (1.8%)
macros93 (1.7%)
parsing91 (1.6%)
graphql79 (1.4%)
substrate76 (1.4%)
blockchain75 (1.4%)
missing-fallback71 (1.3%)
serde71 (1.3%)
sql65 (1.2%)

TypeScript ts

library2,507 (38.8%)
frontend1,786 (27.6%)
backend1,431 (22.1%)
react1,043 (16.1%)
cli592 (9.2%)
missing-feature337 (5.2%)
angular203 (3.1%)
graphql162 (2.5%)
framework149 (2.3%)
missing-fallback143 (2.2%)
javascript140 (2.2%)
electron137 (2.1%)
incomplete-validation136 (2.1%)
fullstack126 (1.9%)
testing122 (1.9%)
missing-metadata-propagation118 (1.8%)
wrong-default117 (1.8%)
vue99 (1.5%)
missing-implementation97 (1.5%)
github-actions84 (1.3%)

Global Top Bug Classes

click to expand

Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.

missing-feature3,504 (7.0%)
incomplete-validation1,310 (2.6%)
missing-implementation1,188 (2.4%)
missing-metadata-propagation926 (1.8%)
missing-fallback846 (1.7%)
wrong-default647 (1.3%)
missing-validation551 (1.1%)
missing-functionality445 (0.9%)
missing-configuration409 (0.8%)
type-handling-inconsistency371 (0.7%)
race-condition328 (0.7%)
missing-configuration-option299 (0.6%)
missing-method278 (0.6%)
missing-api234 (0.5%)
missing-null-check226 (0.5%)
missing-type-support204 (0.4%)
missing-parameter182 (0.4%)
missing-error-handling176 (0.4%)
missing-initialization172 (0.3%)
missing-cleanup165 (0.3%)
missing-field135 (0.3%)
missing-error-propagation130 (0.3%)
missing-context-propagation125 (0.2%)
missing-bounds-check123 (0.2%)
missing-function107 (0.2%)
missing-format-support107 (0.2%)
missing-cli-option107 (0.2%)
missing-input-validation105 (0.2%)
off-by-one104 (0.2%)
missing-configuration-propagation102 (0.2%)

Per-Language Bug-Class Distribution

click to expand

Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.

C c9,714 tagged

missing-feature647 (6.7%)
missing-implementation283 (2.9%)
incomplete-validation272 (2.8%)
missing-metadata-propagation150 (1.5%)
missing-functionality133 (1.4%)
missing-fallback127 (1.3%)
missing-api108 (1.1%)
missing-validation108 (1.1%)
wrong-default97 (1.0%)
missing-initialization80 (0.8%)
race-condition74 (0.8%)
missing-configuration68 (0.7%)
missing-bounds-check55 (0.6%)
missing-null-check52 (0.5%)
memory-leak49 (0.5%)

C++ cpp4,044 tagged

missing-feature268 (6.6%)
missing-implementation144 (3.6%)
incomplete-validation118 (2.9%)
missing-fallback50 (1.2%)
missing-metadata-propagation48 (1.2%)
type-handling-inconsistency40 (1.0%)
missing-functionality38 (0.9%)
missing-api37 (0.9%)
wrong-default36 (0.9%)
missing-validation35 (0.9%)
missing-type-support29 (0.7%)
race-condition25 (0.6%)
missing-configuration20 (0.5%)
missing-method17 (0.4%)
missing-bounds-check16 (0.4%)

Go go8,144 tagged

missing-feature727 (8.9%)
incomplete-validation241 (3.0%)
missing-metadata-propagation202 (2.5%)
missing-implementation177 (2.2%)
missing-fallback173 (2.1%)
wrong-default129 (1.6%)
missing-validation116 (1.4%)
race-condition83 (1.0%)
missing-configuration80 (1.0%)
missing-functionality76 (0.9%)
missing-configuration-option69 (0.8%)
missing-field60 (0.7%)
type-handling-inconsistency59 (0.7%)
missing-nil-check53 (0.7%)
missing-method41 (0.5%)

Java java4,049 tagged

missing-feature220 (5.4%)
incomplete-validation92 (2.3%)
missing-metadata-propagation92 (2.3%)
missing-implementation82 (2.0%)
missing-configuration67 (1.7%)
missing-null-check64 (1.6%)
wrong-default61 (1.5%)
missing-fallback47 (1.2%)
type-handling-inconsistency42 (1.0%)
missing-validation41 (1.0%)
race-condition39 (1.0%)
missing-method32 (0.8%)
missing-configuration-option24 (0.6%)
missing-functionality21 (0.5%)
missing-type-support19 (0.5%)

JavaScript js7,191 tagged

missing-feature490 (6.8%)
incomplete-validation219 (3.0%)
missing-fallback122 (1.7%)
missing-metadata-propagation113 (1.6%)
missing-implementation100 (1.4%)
wrong-default91 (1.3%)
missing-validation82 (1.1%)
missing-configuration-option71 (1.0%)
missing-method61 (0.8%)
type-handling-inconsistency56 (0.8%)
missing-functionality52 (0.7%)
missing-configuration47 (0.7%)
missing-null-check46 (0.6%)
missing-error-handling40 (0.6%)
missing-option40 (0.6%)

Python py5,048 tagged

missing-feature408 (8.1%)
missing-implementation145 (2.9%)
missing-fallback113 (2.2%)
missing-metadata-propagation106 (2.1%)
incomplete-validation101 (2.0%)
missing-parameter63 (1.2%)
wrong-default59 (1.2%)
missing-validation55 (1.1%)
type-handling-inconsistency51 (1.0%)
missing-functionality50 (1.0%)
missing-configuration44 (0.9%)
missing-error-handling31 (0.6%)
missing-method29 (0.6%)
missing-configuration-option23 (0.5%)
missing-type-support23 (0.5%)

Rust rust5,518 tagged

missing-feature407 (7.4%)
missing-implementation160 (2.9%)
incomplete-validation131 (2.4%)
missing-metadata-propagation97 (1.8%)
missing-fallback71 (1.3%)
missing-validation59 (1.1%)
wrong-default57 (1.0%)
missing-functionality50 (0.9%)
missing-type-support49 (0.9%)
missing-method48 (0.9%)
missing-configuration38 (0.7%)
missing-api37 (0.7%)
missing-syntax-support31 (0.6%)
type-handling-inconsistency26 (0.5%)
missing-cli-option24 (0.4%)

TypeScript ts6,466 tagged

missing-feature337 (5.2%)
missing-fallback143 (2.2%)
incomplete-validation136 (2.1%)
missing-metadata-propagation118 (1.8%)
wrong-default117 (1.8%)
missing-implementation97 (1.5%)
type-handling-inconsistency56 (0.9%)
missing-validation55 (0.9%)
race-condition48 (0.7%)
missing-configuration45 (0.7%)
missing-configuration-option42 (0.6%)
missing-method38 (0.6%)
missing-null-check27 (0.4%)
missing-prop27 (0.4%)
missing-functionality25 (0.4%)