Live Dashboard of PRs collections and SWE Tasks Generation
Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.
Language Progress
| Language | PRs collected | Last 1h | Last 24h | Valid SWE | Last 1h | Last 24h | Processed | Success rate |
|---|---|---|---|---|---|---|---|---|
| Cc | 32,357 | 0 | 0 | 9,715 | 0 | 0 | 32,357 | |
| C++cpp | 48,176 | 0 | 0 | 4,060 | 0 | +2 | 32,768 | |
| Gogo | 132,107 | 0 | +1 | 8,270 | +2 | +29 | 109,603 | |
| Javajava | 89,033 | 0 | +1 | 4,131 | 0 | +19 | 88,922 | |
| JavaScriptjs | 39,682 | 0 | 0 | 7,257 | 0 | +18 | 39,682 | |
| Pythonpy | 105,781 | +1 | +1 | 5,114 | 0 | +15 | 105,686 | |
| Rustrust | 71,905 | 0 | +1 | 5,583 | +1 | +16 | 71,864 | |
| TypeScriptts | 70,494 | 0 | +1 | 6,569 | 0 | +23 | 70,408 |
Run Parameters
| Language | Eval model (OPENAI) | Completion model (ANTHROPIC) | Concurrency | min_source_files | max_source_files |
|---|---|---|---|---|---|
| C | glm-5 | claude-sonnet-4-6 | 12 | 2 | 15 |
| C++ | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 8 | 2 | 15 |
| Go | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 12 | 2 | 10 |
| Java | glm-5 | claude-sonnet-4-6 | 8 | 2 | 10 |
| JavaScript | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 12 | 2 | 10 |
| Python | glm-5 | claude-sonnet-4-6 | 12 | 3 | 15 |
| Rust | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 8 | 2 | 10 |
| TypeScript | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B | 12 | 2 | 10 |
Failure Reason Breakdown
click to expand
| Language | Processed | Valid SWE | Failed | trivial_pr | validation | infra_error | timeout | workflow_error | Other |
|---|---|---|---|---|---|---|---|---|---|
| C | 32,357 | 9,715 | 22,642 | 18,841 | 3,370 | 649 | 1 | 65 | 2 |
| C++ | 32,768 | 4,060 | 28,708 | 8,161 | 6,166 | 15,629 | 159 | 578 | 266 |
| Go | 109,603 | 8,270 | 101,333 | 31,843 | 18,950 | 47,673 | 1,532 | 1,073 | 255 |
| Java | 88,922 | 4,131 | 84,791 | 25,366 | 9,904 | 39,854 | 1,089 | 7,395 | 1,599 |
| JavaScript | 39,682 | 7,257 | 32,425 | 17,225 | 4,725 | 11,351 | 270 | 236 | 0 |
| Python | 105,686 | 5,114 | 100,572 | 41,050 | 13,751 | 46,182 | 829 | 364 | 120 |
| Rust | 71,864 | 5,583 | 66,281 | 23,271 | 9,450 | 30,889 | 1,095 | 1,008 | 1,232 |
| TypeScript | 70,408 | 6,569 | 63,839 | 22,004 | 13,714 | 25,842 | 1,764 | 834 | 8 |
trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.
validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).
infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).
timeout: processing timed out (per-PR total timeout or Claude Code session timeout).
workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).
fix.patch Complexity
click to expand
| Language | Valid SWE Count | Avg fix.patch lines | Avg fix.patch hunks | Avg fix.patch files |
|---|---|---|---|---|
| C | 9,715 | 336.48 | 17.94 | 5.85 |
| C++ | 4,060 | 286.03 | 13.75 | 5.09 |
| Go | 8,270 | 210.21 | 12.55 | 4.33 |
| Java | 4,131 | 163.37 | 10.50 | 4.22 |
| JavaScript | 7,257 | 77.11 | 6.31 | 2.80 |
| Python | 5,114 | 152.56 | 11.08 | 3.86 |
| Rust | 5,583 | 226.55 | 13.19 | 4.10 |
| TypeScript | 6,569 | 156.29 | 9.55 | 4.11 |
Metric Definitions
click to expand
Difficulty score (difficulty_score)
Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.
The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.
Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.
Tag generation and display
tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.
The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.
fix.patch statistics
Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.
Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.
difficulty_label Distribution
click to expand
| Language | easy / medium / hard | easy | medium | hard |
|---|---|---|---|---|
| C | 869 / 6417 / 2421 | 869 | 6,417 | 2,421 |
| C++ | 433 / 2522 / 1100 | 433 | 2,522 | 1,100 |
| Go | 668 / 5968 / 1628 | 668 | 5,968 | 1,628 |
| Java | 457 / 2677 / 993 | 457 | 2,677 | 993 |
| JavaScript | 1096 / 5376 / 784 | 1,096 | 5,376 | 784 |
| Python | 273 / 3282 / 1535 | 273 | 3,282 | 1,535 |
| Rust | 390 / 3304 / 1887 | 390 | 3,304 | 1,887 |
| TypeScript | 602 / 4852 / 1115 | 602 | 4,852 | 1,115 |
difficulty_score Overview
click to expand
| Language | count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|---|
| C | 9,707 | 2.4 | 4.9 | 6.0 | 5.97 | 7.0 | 9.2 |
| C++ | 4,055 | 2.5 | 4.9 | 6.0 | 5.99 | 7.2 | 9.1 |
| Go | 8,264 | 2.6 | 4.9 | 5.8 | 5.85 | 6.8 | 9.1 |
| Java | 4,127 | 2.8 | 4.8 | 5.9 | 5.89 | 7.0 | 9.2 |
| JavaScript | 7,256 | 2.6 | 4.4 | 5.2 | 5.36 | 6.2 | 9.2 |
| Python | 5,090 | 2.6 | 5.2 | 6.2 | 6.22 | 7.3 | 9.1 |
| Rust | 5,581 | 2.7 | 5.2 | 6.3 | 6.26 | 7.4 | 9.0 |
| TypeScript | 6,569 | 2.7 | 4.7 | 5.6 | 5.71 | 6.6 | 9.2 |
Global Top Tags
click to expand
Per-Language Tag Distribution
click to expand
Global Top Bug Classes
click to expand
Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.
Per-Language Bug-Class Distribution
click to expand
Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.