SWE Task Progress Dashboard

Overview

Live Dashboard of PRs collections and SWE Tasks Generation

Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.

Total PRs collected

557,089

21,827 unique repos · 1h -32,193 / 24h -32,193

Total valid SWE

52,847

10,082 unique repos · 1h +2,882 / 24h +2,882

Overall success rate

9.6%

Valid SWE / processed 553,153

Mean difficulty_score

5.90

median 5.9, count 52,797

Language Progress

Language	PRs collected	Last 1h	Last 24h	Valid SWE	Last 1h	Last 24h	Processed	Success rate
Cc	29,471	-2,886	-2,886	10,271	+562	+562	29,171	35.2%
C++cpp	45,625	-2,363	-2,363	4,062	+23	+23	45,552	8.9%
Gogo	126,449	-5,654	-5,654	8,473	+378	+378	125,524	6.8%
Javajava	84,869	-4,160	-4,160	4,372	+355	+355	84,386	5.2%
JavaScriptjs	36,409	-3,273	-3,273	7,257	+120	+120	36,109	20.1%
Pythonpy	98,883	-6,885	-6,885	5,971	+913	+913	98,321	6.1%
Rustrust	68,650	-3,215	-3,215	5,595	+90	+90	68,018	8.2%
TypeScriptts	66,733	-3,757	-3,757	6,846	+441	+441	66,072	10.4%

Run Parameters

Language	Eval model (OPENAI)	Completion model (ANTHROPIC)	Concurrency	min_source_files	max_source_files
C	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	12	2	15
C++	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	8	2	15
Go	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	12	2	10
Java	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	8	2	10
JavaScript	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	12	2	10
Python	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	12	3	15
Rust	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	8	2	10
TypeScript	`Qwen3.6-35B-A3B`	`claude-opus-4-8`	12	2	10

Failure Reason Breakdown

click to expand

Language	Processed	Valid SWE	Failed	trivial_pr	validation	infra_error	timeout	workflow_error	Other
C	29,171	10,271	18,900	16,541	2,748	259	1	127	2
C++	45,552	4,062	41,490	8,660	5,316	28,905	159	452	266
Go	125,524	8,473	117,051	39,618	25,849	49,173	1,393	1,006	18
Java	84,386	4,372	80,014	25,867	10,835	37,672	1,155	3,364	1,599
JavaScript	36,109	7,257	28,852	19,465	1,535	9,392	1	103	0
Python	98,321	5,971	92,350	43,623	12,359	36,744	857	469	120
Rust	68,018	5,595	62,423	29,531	2,480	28,109	1,071	665	1,233
TypeScript	66,072	6,846	59,226	19,981	11,751	25,551	1,776	541	8

trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.

validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).

infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).

timeout: processing timed out (per-PR total timeout or Claude Code session timeout).

workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).

fix.patch Complexity

click to expand

Language	Valid SWE Count	Avg fix.patch lines	Avg fix.patch hunks	Avg fix.patch files
C	10,271	333.26	17.96	5.82
C++	4,062	285.93	13.74	5.09
Go	8,473	210.55	12.49	4.32
Java	4,372	166.81	10.83	4.31
JavaScript	7,257	77.11	6.31	2.80
Python	5,971	156.52	11.23	3.95
Rust	5,595	226.35	13.19	4.11
TypeScript	6,846	154.60	9.52	4.11

Metric Definitions

click to expand

Difficulty score (difficulty_score)

Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.

The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.

Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.

Tag generation and display

tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.

The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.

fix.patch statistics

Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.

Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.

difficulty_label Distribution

click to expand

Language	easy / medium / hard	easy	medium	hard
C	897 / 6768 / 2598	897	6,768	2,598
C++	433 / 2524 / 1100	433	2,524	1,100
Go	705 / 6101 / 1661	705	6,101	1,661
Java	472 / 2824 / 1072	472	2,824	1,072
JavaScript	1096 / 5376 / 784	1,096	5,376	784
Python	281 / 3764 / 1902	281	3,764	1,902
Rust	391 / 3312 / 1890	391	3,312	1,890
TypeScript	627 / 5056 / 1163	627	5,056	1,163

difficulty_score Overview

click to expand

Language	count	min	p25	median	mean	p75	max
C	10,263	2.4	4.9	6.0	5.99	7.1	9.2
C++	4,057	2.5	4.9	6.0	5.99	7.2	9.1
Go	8,467	2.6	4.9	5.8	5.84	6.8	9.1
Java	4,368	2.8	4.8	5.9	5.91	7.0	9.2
JavaScript	7,256	2.6	4.4	5.2	5.36	6.2	9.2
Python	5,947	2.6	5.3	6.3	6.29	7.3	9.1
Rust	5,593	2.7	5.2	6.3	6.26	7.4	9.0
TypeScript	6,846	2.7	4.7	5.6	5.72	6.6	9.2

Global Top Tags

click to expand

library24,617 (46.6%)

backend16,400 (31.1%)

cli7,063 (13.4%)

missing-feature3,740 (7.1%)

frontend3,211 (6.1%)

testing2,237 (4.2%)

http1,619 (3.1%)

react1,455 (2.8%)

incomplete-validation1,386 (2.6%)

framework1,258 (2.4%)

missing-implementation1,199 (2.3%)

missing-metadata-propagation961 (1.8%)

missing-fallback867 (1.6%)

kubernetes752 (1.4%)

async698 (1.3%)

wrong-default668 (1.3%)

embedded643 (1.2%)

networking629 (1.2%)

cpp607 (1.1%)

missing-validation579 (1.1%)

missing-functionality453 (0.9%)

missing-configuration429 (0.8%)

typescript402 (0.8%)

type-handling-inconsistency392 (0.7%)

parsing365 (0.7%)

eslint350 (0.7%)

graphql346 (0.7%)

race-condition337 (0.6%)

missing-configuration-option318 (0.6%)

postgresql301 (0.6%)

Per-Language Tag Distribution

click to expand

C c

library5,450 (53.1%)

backend2,634 (25.6%)

cli1,174 (11.4%)

missing-feature710 (6.9%)

embedded617 (6.0%)

cpp601 (5.9%)

testing417 (4.1%)

networking409 (4.0%)

framework294 (2.9%)

incomplete-validation288 (2.8%)

missing-implementation284 (2.8%)

http224 (2.2%)

ruby221 (2.2%)

postgresql213 (2.1%)

firmware184 (1.8%)

kernel182 (1.8%)

quic180 (1.8%)

missing-metadata-propagation153 (1.5%)

rust141 (1.4%)

tls137 (1.3%)

C++ cpp

library2,793 (68.8%)

backend741 (18.3%)

testing496 (12.2%)

cli359 (8.8%)

missing-feature268 (6.6%)

framework186 (4.6%)

missing-implementation144 (3.5%)

http129 (3.2%)

incomplete-validation119 (2.9%)

boost114 (2.8%)

async92 (2.3%)

parsing76 (1.9%)

qt65 (1.6%)

serialization55 (1.4%)

compiler54 (1.3%)

geometry51 (1.3%)

missing-fallback50 (1.2%)

networking49 (1.2%)

missing-metadata-propagation48 (1.2%)

formatting45 (1.1%)

Go go

backend4,486 (53.0%)

library2,097 (24.8%)

cli2,046 (24.2%)

missing-feature745 (8.8%)

kubernetes672 (7.9%)

http542 (6.4%)

incomplete-validation255 (3.0%)

testing239 (2.8%)

missing-metadata-propagation208 (2.5%)

missing-fallback179 (2.1%)

missing-implementation179 (2.1%)

docker134 (1.6%)

wrong-default134 (1.6%)

terraform126 (1.5%)

grpc125 (1.5%)

aws124 (1.5%)

missing-validation120 (1.4%)

prometheus119 (1.4%)

networking97 (1.1%)

database90 (1.1%)

Java java

backend2,124 (48.6%)

library1,980 (45.3%)

missing-feature243 (5.6%)

testing223 (5.1%)

spring165 (3.8%)

framework158 (3.6%)

http148 (3.4%)

aem147 (3.4%)

android129 (3.0%)

incomplete-validation101 (2.3%)

missing-metadata-propagation97 (2.2%)

missing-implementation83 (1.9%)

missing-configuration72 (1.6%)

cli71 (1.6%)

json68 (1.6%)

missing-null-check64 (1.5%)

wrong-default63 (1.4%)

maven61 (1.4%)

kafka57 (1.3%)

sling57 (1.3%)

JavaScript js

library3,913 (53.9%)

backend1,239 (17.1%)

frontend1,043 (14.4%)

cli859 (11.8%)

missing-feature496 (6.8%)

react354 (4.9%)

typescript334 (4.6%)

testing297 (4.1%)

eslint263 (3.6%)

incomplete-validation221 (3.0%)

framework200 (2.8%)

http183 (2.5%)

fastify149 (2.1%)

webpack148 (2.0%)

missing-fallback124 (1.7%)

missing-metadata-propagation113 (1.6%)

svelte106 (1.5%)

mongoose104 (1.4%)

nodejs104 (1.4%)

missing-implementation100 (1.4%)

Python py

library2,589 (43.5%)

backend2,425 (40.7%)

cli800 (13.4%)

missing-feature519 (8.7%)

fastapi235 (3.9%)

django174 (2.9%)

missing-implementation147 (2.5%)

pytorch144 (2.4%)

incomplete-validation124 (2.1%)

testing119 (2.0%)

missing-fallback117 (2.0%)

missing-metadata-propagation117 (2.0%)

async112 (1.9%)

framework104 (1.7%)

http102 (1.7%)

ansible100 (1.7%)

aws80 (1.3%)

pydantic73 (1.2%)

aiohttp66 (1.1%)

flask66 (1.1%)

Rust rust

library3,132 (56.0%)

backend1,224 (21.9%)

cli1,138 (20.3%)

missing-feature411 (7.3%)

testing315 (5.6%)

async225 (4.0%)

http211 (3.8%)

missing-implementation162 (2.9%)

incomplete-validation133 (2.4%)

compiler120 (2.1%)

git119 (2.1%)

missing-metadata-propagation97 (1.7%)

macros93 (1.7%)

parsing92 (1.6%)

graphql81 (1.4%)

blockchain80 (1.4%)

substrate77 (1.4%)

serde73 (1.3%)

missing-fallback71 (1.3%)

sql65 (1.2%)

TypeScript ts

library2,663 (38.9%)

frontend1,869 (27.3%)

backend1,527 (22.3%)

react1,090 (15.9%)

cli616 (9.0%)

missing-feature348 (5.1%)

angular204 (3.0%)

graphql165 (2.4%)

framework164 (2.4%)

missing-fallback150 (2.2%)

electron146 (2.1%)

javascript146 (2.1%)

incomplete-validation145 (2.1%)

fullstack135 (2.0%)

testing131 (1.9%)

missing-metadata-propagation128 (1.9%)

wrong-default122 (1.8%)

vue106 (1.5%)

missing-implementation100 (1.5%)

nextjs88 (1.3%)

Global Top Bug Classes

click to expand

Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.

missing-feature3,740 (7.1%)

incomplete-validation1,386 (2.6%)

missing-implementation1,199 (2.3%)

missing-metadata-propagation961 (1.8%)

missing-fallback867 (1.6%)

wrong-default668 (1.3%)

missing-validation579 (1.1%)

missing-functionality453 (0.9%)

missing-configuration429 (0.8%)

type-handling-inconsistency392 (0.7%)

race-condition337 (0.6%)

missing-configuration-option318 (0.6%)

missing-method285 (0.5%)

missing-api245 (0.5%)

missing-null-check229 (0.4%)

missing-type-support210 (0.4%)

missing-error-handling187 (0.4%)

missing-parameter185 (0.4%)

missing-initialization177 (0.3%)

missing-cleanup173 (0.3%)

missing-field138 (0.3%)

missing-error-propagation136 (0.3%)

missing-context-propagation128 (0.2%)

missing-bounds-check125 (0.2%)

missing-input-validation112 (0.2%)

missing-format-support112 (0.2%)

missing-function108 (0.2%)

missing-configuration-propagation108 (0.2%)

off-by-one107 (0.2%)

missing-cli-option107 (0.2%)

Per-Language Bug-Class Distribution

click to expand

Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.

C c10,271 tagged

missing-feature710 (6.9%)

incomplete-validation288 (2.8%)

missing-implementation284 (2.8%)

missing-metadata-propagation153 (1.5%)

missing-functionality135 (1.3%)

missing-fallback128 (1.2%)

missing-api118 (1.1%)

missing-validation112 (1.1%)

wrong-default98 (1.0%)

missing-initialization80 (0.8%)

race-condition77 (0.7%)

missing-configuration70 (0.7%)

missing-bounds-check55 (0.5%)

memory-leak54 (0.5%)

missing-null-check53 (0.5%)

C++ cpp4,060 tagged

missing-feature268 (6.6%)

missing-implementation144 (3.5%)

incomplete-validation119 (2.9%)

missing-fallback50 (1.2%)

missing-metadata-propagation48 (1.2%)

type-handling-inconsistency40 (1.0%)

missing-functionality38 (0.9%)

missing-api37 (0.9%)

wrong-default36 (0.9%)

missing-validation35 (0.9%)

missing-type-support29 (0.7%)

race-condition25 (0.6%)

missing-configuration20 (0.5%)

missing-method17 (0.4%)

missing-bounds-check16 (0.4%)

Go go8,467 tagged

missing-feature745 (8.8%)

incomplete-validation255 (3.0%)

missing-metadata-propagation208 (2.5%)

missing-fallback179 (2.1%)

missing-implementation179 (2.1%)

wrong-default134 (1.6%)

missing-validation120 (1.4%)

missing-configuration87 (1.0%)

race-condition85 (1.0%)

missing-functionality79 (0.9%)

missing-configuration-option69 (0.8%)

type-handling-inconsistency63 (0.7%)

missing-field62 (0.7%)

missing-nil-check58 (0.7%)

missing-method43 (0.5%)

Java java4,369 tagged

missing-feature243 (5.6%)

incomplete-validation101 (2.3%)

missing-metadata-propagation97 (2.2%)

missing-implementation83 (1.9%)

missing-configuration72 (1.6%)

missing-null-check64 (1.5%)

wrong-default63 (1.4%)

missing-fallback48 (1.1%)

missing-validation44 (1.0%)

type-handling-inconsistency44 (1.0%)

race-condition39 (0.9%)

missing-method33 (0.8%)

missing-configuration-option24 (0.5%)

missing-functionality21 (0.5%)

missing-type-support20 (0.5%)

JavaScript js7,257 tagged

missing-feature496 (6.8%)

incomplete-validation221 (3.0%)

missing-fallback124 (1.7%)

missing-metadata-propagation113 (1.6%)

missing-implementation100 (1.4%)

wrong-default93 (1.3%)

missing-validation82 (1.1%)

missing-configuration-option71 (1.0%)

missing-method63 (0.9%)

type-handling-inconsistency58 (0.8%)

missing-functionality52 (0.7%)

missing-configuration47 (0.6%)

missing-null-check46 (0.6%)

missing-error-handling40 (0.6%)

missing-option40 (0.6%)

Python py5,951 tagged

missing-feature519 (8.7%)

missing-implementation147 (2.5%)

incomplete-validation124 (2.1%)

missing-fallback117 (2.0%)

missing-metadata-propagation117 (2.0%)

missing-validation66 (1.1%)

missing-parameter63 (1.1%)

wrong-default62 (1.0%)

type-handling-inconsistency59 (1.0%)

missing-functionality51 (0.9%)

missing-configuration45 (0.8%)

feature-implementation39 (0.7%)

missing-error-handling36 (0.6%)

missing-configuration-option35 (0.6%)

missing-method29 (0.5%)

Rust rust5,593 tagged

missing-feature411 (7.3%)

missing-implementation162 (2.9%)

incomplete-validation133 (2.4%)

missing-metadata-propagation97 (1.7%)

missing-fallback71 (1.3%)

wrong-default60 (1.1%)

missing-validation59 (1.1%)

missing-functionality52 (0.9%)

missing-method49 (0.9%)

missing-type-support49 (0.9%)

missing-configuration38 (0.7%)

missing-api37 (0.7%)

missing-syntax-support31 (0.6%)

type-handling-inconsistency26 (0.5%)

missing-cli-option24 (0.4%)

TypeScript ts6,837 tagged

missing-feature348 (5.1%)

missing-fallback150 (2.2%)

incomplete-validation145 (2.1%)

missing-metadata-propagation128 (1.9%)

wrong-default122 (1.8%)

missing-implementation100 (1.5%)

missing-validation61 (0.9%)

type-handling-inconsistency59 (0.9%)

missing-configuration50 (0.7%)

race-condition49 (0.7%)

missing-configuration-option43 (0.6%)

missing-method39 (0.6%)

missing-null-check28 (0.4%)

missing-prop27 (0.4%)

missing-functionality25 (0.4%)