Update (November 17, 2026): Harness Hub ships a public Pi LoRA adapter
Harness Hub released Qwen3.8-27B-pi-adapted on November 17, 2026—a LoRA trained with async GRPO inside real Pi sandboxes, not a new base checkpoint from Alibaba.
The model card publishes paired pass@1 numbers at temperature 0.8 with thinking off:
| Benchmark | Base Qwen3.8-27B on Pi | Adapter (step 160) |
|---|---|---|
| SWE-bench Pro (hard 51) | 29.4% | 62.7% |
| Terminal Bench 2.1 | 39.5% | 55.8% |
The card also reports 55–62% fewer tool calls per task on those suites. That is a different headline from the stray 63% social screenshot: the verified adapter line is 62.7% on the hard-51 subset, while the official Alibaba card still lists 61.7% on full SWE-bench Pro with the Claude Code harness.
Caveats builders should note. Checkpoint 160 was selected using scores on SWE-bench Pro and an 18-task Terminal-Bench subset tracked during training—so the headline benchmarks were not blind holdouts. The model card points to an upcoming Harness Hub release blog post for the full training recipe; until that ships, treat contamination risk like any other RL-on-benchmark release.
How to run it. The card documents vLLM with --enable-lora, transformers serve, llama.cpp GGUF, and Llama.app on macOS, with explicit warnings that Pi must use openai-completions, not the Responses API, because the adapter was not trained on that path. See also the Harness Hub adapters blog for the multi-harness safeagents story.
Social timelines in November 2026 pushed a crisp headline: Qwen3.8-27B hits 63% on SWE-bench Pro thanks to a new Pi harness adapter. The official model card still prints 61.7% with an explicit footnote: Claude Code harness, temp=1.0, top_p=0.95, 256K context, after refined tasks.
explainx.ai covered the base launch in
Qwen3.8-27B open-weight HN reaction and harness tier debates. This post is the score reconciliation builders need before they retune spend on Alibaba/Qwen APIs.
Verified table row from Qwen
| Benchmark | Qwen3.8-27B (card) | Notes |
|---|---|---|
| SWE-bench Pro | 61.7% | Claude Code harness |
| Terminal Bench 2.1 | 73.0% | Terminus harness |
| DeepSWE 1.1 | 42.2% | Claude Code harness |
The card also lists Qwen3.6-27B at 53.5% on the same benchmark—so +8 points is the vendor-certified generational jump, not the +1.3 implied by 63% vs 61.7% social math.
An issue card linked to a patch tile, illustrating SWE-bench Pro fixes and harness-dependent scores
What the 63% claim assumes
Aggregator posts this week pointed at a Pi adapter without always linking:
- Harness repo + commit hash
- Temperature / top_p
- Context window actually used
- Whether tasks match the refined SWE-bench Pro set
Until those appear in a public log, label 63% as claimed. Our vLLM Rubin throughput post shows why explainx.ai separates headline metrics from filings.
Why Pi numbers diverge from Claude Code
Third-party guides already document wide spreads on the same checkpoint:
- Official Claude Code eval: 42.2% DeepSWE on card.
- Community Pi xhigh patches reported ~46% DeepSWE in some configs (Regolo harness notes, secondary).
Pi emphasizes minimal tool surfaces; Claude Code ships vendor-default tool schemas. Qwen3.8-27B-pi adds SFT + GRPO to fix effort-level ordering on Terminal-Bench, not necessarily the same recipe as a one-line Pi adapter rumor.
Mechanisms that move SWE-bench Pro without changing weights:
| Lever | Effect |
|---|---|
| Tool-error retries | Fewer false failures on flaky bash |
supportsDeveloperRole flags | API routing differences in Pi models.json |
| Context truncation | Silent task loss past window |
| Timeout / max turns | Agent gives up early |
See Pi harness basics and Agent Lightning harness RL for why harness engineering is half the benchmark.
What developers should run locally
- Freeze the checkpoint:
Qwen/Qwen3.8-27B. - Pick harness: Pi or Claude Code—do not compare apples to social screenshots.
- Log git SHA of harness config.
- Run a slice of SWE-bench Pro before committing to a full cloud eval bill.
- Compare against Qwen Code terminal agent if you want Alibaba’s official CLI path.
If you need cheap API, cross-check RightNow YC inference and Together prefix cache—price per solved issue, not per marketing percent.
Terminal Bench vs SWE-bench Pro: do not mix headlines
Social posts sometimes conflate benchmarks. The official card separates:
| Bench | Qwen3.8-27B | Harness named on card |
|---|---|---|
| Terminal Bench 2.1 | 73.0% | Terminus |
| SWE-bench Pro | 61.7% | Claude Code |
| QwenSWEBench (in-house) | 79.0% | Claude Code |
A Pi-tuned artifact like Qwen3.8-27B-pi reports effort ordering fixes on Terminal-Bench, not an official SWE-bench Pro line on the base card. When someone says “63% Pi”, ask which benchmark and which checkpoint (base vs -pi merge).
A green baton directs gears along a curved shaft, illustrating harness routing for coding agents
Reproducing Alibaba’s 61.7% row
From the model README footnote:
- Claude Code harness
- temp=1.0, top_p=0.95
- 256K context window
- Refined SWE-bench Pro task set (problematic tasks corrected; baselines re-run)
If your harness uses 8K context, temperature 0, or different tool schemas, you are not reproducing the 61.7% row—you are running a new experiment. Publish all four knobs when you tweet scores.
Community Pi tuning: what would change the number?
The bytkim Pi blog documents:
- SFT on successful Pi sessions
- GRPO with effort-ordering rewards
- Terminal-Bench gains at lower token use
That recipe might lift SWE-bench Pro too, but it is not the same as a rumored same-day Pi adapter without public weights. Until a repo pins eval YAML, treat 63% as hypothesis.
Cost per solved issue beats leaderboard points
Together’s agent eval reporting and SWE-Cowork benchmarks emphasize dollars per merged PR, not single-percent bragging.
Run this worksheet on your monorepo:
- Pick harness you will standardize on in Q1.
- Measure median turns, tokens, and wall clock per issue.
- Multiply by API list price or amortized GPU.
- Compare Qwen3.8-27B vs Tencent Hy4 Flash or Qwen Code CLI only after equal harness.
When a 63% Pi score would matter
A +1.3 point lift is noise if variance across reruns is ±2%. A +5 point lift with public trajectories would signal Pi-specific tool prompts that generalize—worth forking.
Watch for:
- Hugging Face discussion attachments on the model repo.
- Pi monorepo releases citing SWE-bench configs.
- Alibaba updates to the model card (they edited eval footnotes before).
Until then, procurement slides should cite 61.7% with Claude Code footnotes, and social 63% as unverified Pi.
OpenCode, Kilo Code, and Pi JSON configs
Secondary guides document provider-specific models.json quirks—supportsDeveloperRole: false, reasoning flags, and compat blocks for Qwen on OpenAI-shaped endpoints (Regolo harness guide, secondary). A single boolean mis-set can drop tool calls and crater SWE-bench pass rates without retraining.
Standardize one config repo per team; pin it in Docker images used for eval workers.
Goodhart risk on coding leaderboards
Goodhart’s law posts apply to SWE-bench forks too: when harness patches become public, trainers overfit tool traces. A 63% screenshot may reflect leaked task tooling rather than general coding.
Prefer private issue suites and real-repo eval posts for hiring and vendor bake-offs.
Alibaba cloud vs local: same score?
Qwen Code and DashScope may ship different default harnesses than your local Pi setup. The 61.7% row is not a promise for every endpoint.
Before migrating production agents, run A/B on twenty internal tickets with frozen prompts and identical timeouts.
DeepSWE and NL2Repo rows matter too
The same card shows 42.2% DeepSWE 1.1 and 42.3 NL2Repo-Bench under Claude Code. Teams building repo synthesis tools should log all three—SWE-bench Pro, DeepSWE, NL2Repo—not whichever number looks best in a screenshot.
If Pi adapters claim gains, demand paired runs on the same trio with published trajectories.
Token and timeout settings
Qwen’s footnote uses 8-hour timeout on QwenSWEBench averages—far longer than Twitter demos. Your CI budget may kill agents before they match vendor scores.
Set wall-clock caps explicitly in harness JSON and compare pass rate vs cost.
Hardware locality
Local Unsloth GGUF paths differ from FP8 cloud SKUs. A Pi config tuned for API Qwen may fail on MLX builds—version compat tables belong in internal wikis.
When to update this post
We will append a section if Alibaba edits the model card above 61.7% or a public Pi eval repo publishes 63% with commit SHAs. Until then, share this link instead of unsourced percents in Slack.
Procurement language you can paste
Use this RFP snippet to block harness laundering:
Vendor must disclose harness name, version, temperature, context limit, and tool schema for any quoted SWE-bench Pro or Terminal-Bench figure. Screenshots without logs are non-responsive.
Workshop tie-in for agent leads
Teams standardizing on Pi or Claude Code should run a half-day harness audit before Q4 renewals. Our Claude Code workshop coverage lists formats where eval hygiene is already on the agenda.
Incident response when benchmarks drive headcount
When executives treat a viral percent as a hiring trigger, platform teams get surprise deadlines. Pre-empt this by publishing an internal scoreboard with:
- Verified vendor rows (61.7% baseline).
- Your harness results with variance bands.
- Cost per merged PR as the decision metric.
That dashboard prevents Pi adapter rumors from becoming roadmap law overnight.
Checklist before you quote any SWE-bench number in a deck
- Link the model card row or eval commit.
- Name the harness and version.
- State temperature, top_p, max tokens, context.
- Note timeout and max agent turns.
- Attach cost per run from billing export.
- Mark claimed social figures red until logs arrive.
Why explainx.ai separates 61.7% from 63%
We are not saying Pi cannot beat 61.7%—only that unlogged deltas should not move budget. The August HN launch thread already showed community harness tweaks beating vendor defaults on other benches. Replicate that discipline here: celebrate open evals, not screenshots.
Status: November 17, 2026; official score from Qwen/Qwen3.8-27B README; 63% Pi treated as unverified social claim until a public eval log appears.
Bottom line: cite 61.7% with Claude Code footnotes; treat 63% Pi as a hypothesis until someone ships reproducible configs. Ping explainx.ai if Alibaba updates the card or a Pi repo publishes eval YAML.
