Live Dashboard of PRs collections and SWE Tasks Generation
Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.
Language Progress
| Language | PRs collected | Last 1h | Last 24h | Valid SWE | Last 1h | Last 24h | Processed | Success rate |
|---|---|---|---|---|---|---|---|---|
| Cc | 32,357 | 0 | 0 | 9,715 | 0 | 0 | 32,357 | |
| C++cpp | 48,175 | 0 | +1 | 4,054 | +1 | +7 | 27,821 | |
| Gogo | 132,104 | 0 | +1 | 8,198 | +1 | +33 | 101,325 | |
| Javajava | 89,032 | 0 | +1 | 4,084 | +2 | +23 | 88,911 | |
| JavaScriptjs | 39,682 | 0 | 0 | 7,221 | +1 | +18 | 39,682 | |
| Pythonpy | 105,777 | 0 | +2 | 5,087 | 0 | +13 | 105,670 | |
| Rustrust | 71,903 | 0 | +1 | 5,541 | 0 | +15 | 71,862 | |
| TypeScriptts | 70,493 | 0 | 0 | 6,507 | +2 | +27 | 65,718 |
Run Parameters
| Language | Eval model (OPENAI) | Completion model (ANTHROPIC) | Concurrency | min_source_files | max_source_files |
|---|---|---|---|---|---|
| C | glm-5 | claude-sonnet-4-6 | 12 | 2 | 15 |
| C++ | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 8 | 2 | 15 |
| Go | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 12 | 2 | 10 |
| Java | glm-5 | claude-sonnet-4-6 | 8 | 2 | 10 |
| JavaScript | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 12 | 2 | 10 |
| Python | glm-5 | claude-sonnet-4-6 | 12 | 3 | 15 |
| Rust | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 8 | 2 | 10 |
| TypeScript | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 12 | 2 | 10 |
Failure Reason Breakdown
click to expand
| Language | Processed | Valid SWE | Failed | trivial_pr | validation | infra_error | timeout | workflow_error | Other |
|---|---|---|---|---|---|---|---|---|---|
| C | 32,357 | 9,715 | 22,642 | 18,028 | 4,217 | 602 | 2 | 77 | 2 |
| C++ | 27,821 | 4,054 | 23,767 | 5,901 | 3,787 | 15,376 | 159 | 527 | 266 |
| Go | 101,325 | 8,198 | 93,127 | 28,036 | 14,444 | 47,667 | 1,576 | 1,060 | 332 |
| Java | 88,911 | 4,084 | 84,827 | 24,055 | 8,571 | 42,092 | 1,182 | 7,733 | 1,601 |
| JavaScript | 39,682 | 7,221 | 32,461 | 16,993 | 3,428 | 12,789 | 372 | 261 | 0 |
| Python | 105,670 | 5,087 | 100,583 | 37,486 | 11,861 | 51,557 | 899 | 380 | 120 |
| Rust | 71,862 | 5,541 | 66,321 | 22,303 | 7,976 | 33,274 | 1,174 | 1,020 | 1,234 |
| TypeScript | 65,718 | 6,507 | 59,211 | 19,149 | 10,099 | 27,625 | 1,764 | 891 | 10 |
trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.
validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).
infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).
timeout: processing timed out (per-PR total timeout or Claude Code session timeout).
workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).
fix.patch Complexity
click to expand
| Language | Valid SWE Count | Avg fix.patch lines | Avg fix.patch hunks | Avg fix.patch files |
|---|---|---|---|---|
| C | 9,715 | 336.48 | 17.94 | 5.85 |
| C++ | 4,054 | 286.20 | 13.75 | 5.09 |
| Go | 8,198 | 211.56 | 12.59 | 4.34 |
| Java | 4,084 | 162.80 | 10.50 | 4.22 |
| JavaScript | 7,221 | 77.08 | 6.30 | 2.80 |
| Python | 5,087 | 152.58 | 11.07 | 3.85 |
| Rust | 5,541 | 225.29 | 13.15 | 4.10 |
| TypeScript | 6,507 | 157.28 | 9.59 | 4.13 |
Metric Definitions
click to expand
Difficulty score (difficulty_score)
Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.
The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.
Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.
Tag generation and display
tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.
The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.
fix.patch statistics
Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.
Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.
difficulty_label Distribution
click to expand
| Language | easy / medium / hard | easy | medium | hard |
|---|---|---|---|---|
| C | 869 / 6417 / 2421 | 869 | 6,417 | 2,421 |
| C++ | 432 / 2518 / 1099 | 432 | 2,518 | 1,099 |
| Go | 653 / 5920 / 1619 | 653 | 5,920 | 1,619 |
| Java | 452 / 2644 / 984 | 452 | 2,644 | 984 |
| JavaScript | 1094 / 5348 / 778 | 1,094 | 5,348 | 778 |
| Python | 273 / 3261 / 1529 | 273 | 3,261 | 1,529 |
| Rust | 386 / 3282 / 1871 | 386 | 3,282 | 1,871 |
| TypeScript | 588 / 4810 / 1109 | 588 | 4,810 | 1,109 |
difficulty_score Overview
click to expand
| Language | count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|---|
| C | 9,707 | 2.4 | 4.9 | 6.0 | 5.97 | 7.0 | 9.2 |
| C++ | 4,049 | 2.5 | 4.9 | 6.0 | 5.99 | 7.2 | 9.1 |
| Go | 8,192 | 2.6 | 4.9 | 5.8 | 5.86 | 6.8 | 9.1 |
| Java | 4,080 | 2.8 | 4.8 | 5.9 | 5.90 | 7.0 | 9.2 |
| JavaScript | 7,220 | 2.6 | 4.4 | 5.2 | 5.36 | 6.2 | 9.2 |
| Python | 5,063 | 2.6 | 5.2 | 6.2 | 6.22 | 7.3 | 9.1 |
| Rust | 5,539 | 2.7 | 5.2 | 6.3 | 6.26 | 7.4 | 9.0 |
| TypeScript | 6,507 | 2.7 | 4.7 | 5.6 | 5.72 | 6.6 | 9.2 |
Global Top Tags
click to expand
Per-Language Tag Distribution
click to expand
Global Top Bug Classes
click to expand
Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.
Per-Language Bug-Class Distribution
click to expand
Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.