explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • Update (November 17, 2026): Harness Hub ships a public Pi LoRA adapter
  • Verified table row from Qwen
  • What the 63% claim assumes
  • Why Pi numbers diverge from Claude Code
  • What developers should run locally
  • Terminal Bench vs SWE-bench Pro: do not mix headlines
  • Reproducing Alibaba’s 61.7% row
  • Community Pi tuning: what would change the number?
  • Cost per solved issue beats leaderboard points
  • When a 63% Pi score would matter
  • OpenCode, Kilo Code, and Pi JSON configs
  • Goodhart risk on coding leaderboards
  • Alibaba cloud vs local: same score?
  • DeepSWE and NL2Repo rows matter too
  • Token and timeout settings
  • Hardware locality
  • When to update this post
  • Procurement language you can paste
  • Workshop tie-in for agent leads
  • Incident response when benchmarks drive headcount
  • Checklist before you quote any SWE-bench number in a deck
  • Why explainx.ai separates 61.7% from 63%
  • Related reading
← Back to blog

explainx / blog

Qwen3.8-27B SWE-Bench Pro 63% on Pi: Claimed vs Official 61.7%

Qwen, Coding Agents, Benchmarks, Claimed vs Verified, Open Models

Part of Open-Weight Models

Social posts cite 63% SWE-bench Pro for Qwen3.8-27B with a Pi harness. Alibaba’s model card lists 61.7% with the Claude Code harness—here is what differs.

Nov 17, 2026·9 min read·Yash Thakker
add explainx.ai
go deep
Qwen3.8-27B SWE-Bench Pro 63% on Pi: Claimed vs Official 61.7%

Update (November 17, 2026): Harness Hub ships a public Pi LoRA adapter

Harness Hub released Qwen3.8-27B-pi-adapted on November 17, 2026—a LoRA trained with async GRPO inside real Pi sandboxes, not a new base checkpoint from Alibaba.

The model card publishes paired pass@1 numbers at temperature 0.8 with thinking off:

table · 3 cols
BenchmarkBase Qwen3.8-27B on PiAdapter (step 160)
SWE-bench Pro (hard 51)29.4%62.7%
Terminal Bench 2.139.5%55.8%

The card also reports 55–62% fewer tool calls per task on those suites. That is a different headline from the stray 63% social screenshot: the verified adapter line is 62.7% on the hard-51 subset, while the official Alibaba card still lists 61.7% on full SWE-bench Pro with the Claude Code harness.

Caveats builders should note. Checkpoint 160 was selected using scores on SWE-bench Pro and an 18-task Terminal-Bench subset tracked during training—so the headline benchmarks were not blind holdouts. The model card points to an upcoming Harness Hub release blog post for the full training recipe; until that ships, treat contamination risk like any other RL-on-benchmark release.

How to run it. The card documents vLLM with --enable-lora, transformers serve, llama.cpp GGUF, and Llama.app on macOS, with explicit warnings that Pi must use openai-completions, not the Responses API, because the adapter was not trained on that path. See also the Harness Hub adapters blog for the multi-harness safeagents story.

Social timelines in November 2026 pushed a crisp headline: Qwen3.8-27B hits 63% on SWE-bench Pro thanks to a new Pi harness adapter. The official model card still prints 61.7% with an explicit footnote: Claude Code harness, temp=1.0, top_p=0.95, 256K context, after refined tasks.

explainx.ai covered the base launch in

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Qwen3.8-27B open-weight HN reaction and harness tier debates. This post is the score reconciliation builders need before they retune spend on Alibaba/Qwen APIs.

Verified table row from Qwen

table · 3 cols
BenchmarkQwen3.8-27B (card)Notes
SWE-bench Pro61.7%Claude Code harness
Terminal Bench 2.173.0%Terminus harness
DeepSWE 1.142.2%Claude Code harness

The card also lists Qwen3.6-27B at 53.5% on the same benchmark—so +8 points is the vendor-certified generational jump, not the +1.3 implied by 63% vs 61.7% social math.

An issue card linked to a patch tile, illustrating SWE-bench Pro fixes and harness-dependent scoresAn issue card linked to a patch tile, illustrating SWE-bench Pro fixes and harness-dependent scores

What the 63% claim assumes

Aggregator posts this week pointed at a Pi adapter without always linking:

  • Harness repo + commit hash
  • Temperature / top_p
  • Context window actually used
  • Whether tasks match the refined SWE-bench Pro set

Until those appear in a public log, label 63% as claimed. Our vLLM Rubin throughput post shows why explainx.ai separates headline metrics from filings.

Why Pi numbers diverge from Claude Code

Third-party guides already document wide spreads on the same checkpoint:

  • Official Claude Code eval: 42.2% DeepSWE on card.
  • Community Pi xhigh patches reported ~46% DeepSWE in some configs (Regolo harness notes, secondary).

Pi emphasizes minimal tool surfaces; Claude Code ships vendor-default tool schemas. Qwen3.8-27B-pi adds SFT + GRPO to fix effort-level ordering on Terminal-Bench, not necessarily the same recipe as a one-line Pi adapter rumor.

Mechanisms that move SWE-bench Pro without changing weights:

table · 2 cols
LeverEffect
Tool-error retriesFewer false failures on flaky bash
supportsDeveloperRole flagsAPI routing differences in Pi models.json
Context truncationSilent task loss past window
Timeout / max turnsAgent gives up early

See Pi harness basics and Agent Lightning harness RL for why harness engineering is half the benchmark.

What developers should run locally

  1. Freeze the checkpoint: Qwen/Qwen3.8-27B.
  2. Pick harness: Pi or Claude Code—do not compare apples to social screenshots.
  3. Log git SHA of harness config.
  4. Run a slice of SWE-bench Pro before committing to a full cloud eval bill.
  5. Compare against Qwen Code terminal agent if you want Alibaba’s official CLI path.

If you need cheap API, cross-check RightNow YC inference and Together prefix cache—price per solved issue, not per marketing percent.

Terminal Bench vs SWE-bench Pro: do not mix headlines

Social posts sometimes conflate benchmarks. The official card separates:

table · 3 cols
BenchQwen3.8-27BHarness named on card
Terminal Bench 2.173.0%Terminus
SWE-bench Pro61.7%Claude Code
QwenSWEBench (in-house)79.0%Claude Code

A Pi-tuned artifact like Qwen3.8-27B-pi reports effort ordering fixes on Terminal-Bench, not an official SWE-bench Pro line on the base card. When someone says “63% Pi”, ask which benchmark and which checkpoint (base vs -pi merge).

A green baton directs gears along a curved shaft, illustrating harness routing for coding agentsA green baton directs gears along a curved shaft, illustrating harness routing for coding agents

Reproducing Alibaba’s 61.7% row

From the model README footnote:

  • Claude Code harness
  • temp=1.0, top_p=0.95
  • 256K context window
  • Refined SWE-bench Pro task set (problematic tasks corrected; baselines re-run)

If your harness uses 8K context, temperature 0, or different tool schemas, you are not reproducing the 61.7% row—you are running a new experiment. Publish all four knobs when you tweet scores.

Community Pi tuning: what would change the number?

The bytkim Pi blog documents:

  • SFT on successful Pi sessions
  • GRPO with effort-ordering rewards
  • Terminal-Bench gains at lower token use

That recipe might lift SWE-bench Pro too, but it is not the same as a rumored same-day Pi adapter without public weights. Until a repo pins eval YAML, treat 63% as hypothesis.

Cost per solved issue beats leaderboard points

Together’s agent eval reporting and SWE-Cowork benchmarks emphasize dollars per merged PR, not single-percent bragging.

Run this worksheet on your monorepo:

  1. Pick harness you will standardize on in Q1.
  2. Measure median turns, tokens, and wall clock per issue.
  3. Multiply by API list price or amortized GPU.
  4. Compare Qwen3.8-27B vs Tencent Hy4 Flash or Qwen Code CLI only after equal harness.

When a 63% Pi score would matter

A +1.3 point lift is noise if variance across reruns is ±2%. A +5 point lift with public trajectories would signal Pi-specific tool prompts that generalize—worth forking.

Watch for:

  • Hugging Face discussion attachments on the model repo.
  • Pi monorepo releases citing SWE-bench configs.
  • Alibaba updates to the model card (they edited eval footnotes before).

Until then, procurement slides should cite 61.7% with Claude Code footnotes, and social 63% as unverified Pi.

OpenCode, Kilo Code, and Pi JSON configs

Secondary guides document provider-specific models.json quirks—supportsDeveloperRole: false, reasoning flags, and compat blocks for Qwen on OpenAI-shaped endpoints (Regolo harness guide, secondary). A single boolean mis-set can drop tool calls and crater SWE-bench pass rates without retraining.

Standardize one config repo per team; pin it in Docker images used for eval workers.

Goodhart risk on coding leaderboards

Goodhart’s law posts apply to SWE-bench forks too: when harness patches become public, trainers overfit tool traces. A 63% screenshot may reflect leaked task tooling rather than general coding.

Prefer private issue suites and real-repo eval posts for hiring and vendor bake-offs.

Alibaba cloud vs local: same score?

Qwen Code and DashScope may ship different default harnesses than your local Pi setup. The 61.7% row is not a promise for every endpoint.

Before migrating production agents, run A/B on twenty internal tickets with frozen prompts and identical timeouts.

DeepSWE and NL2Repo rows matter too

The same card shows 42.2% DeepSWE 1.1 and 42.3 NL2Repo-Bench under Claude Code. Teams building repo synthesis tools should log all three—SWE-bench Pro, DeepSWE, NL2Repo—not whichever number looks best in a screenshot.

If Pi adapters claim gains, demand paired runs on the same trio with published trajectories.

Token and timeout settings

Qwen’s footnote uses 8-hour timeout on QwenSWEBench averages—far longer than Twitter demos. Your CI budget may kill agents before they match vendor scores.

Set wall-clock caps explicitly in harness JSON and compare pass rate vs cost.

Hardware locality

Local Unsloth GGUF paths differ from FP8 cloud SKUs. A Pi config tuned for API Qwen may fail on MLX builds—version compat tables belong in internal wikis.

When to update this post

We will append a section if Alibaba edits the model card above 61.7% or a public Pi eval repo publishes 63% with commit SHAs. Until then, share this link instead of unsourced percents in Slack.

Procurement language you can paste

Use this RFP snippet to block harness laundering:

Vendor must disclose harness name, version, temperature, context limit, and tool schema for any quoted SWE-bench Pro or Terminal-Bench figure. Screenshots without logs are non-responsive.

Workshop tie-in for agent leads

Teams standardizing on Pi or Claude Code should run a half-day harness audit before Q4 renewals. Our Claude Code workshop coverage lists formats where eval hygiene is already on the agenda.

Incident response when benchmarks drive headcount

When executives treat a viral percent as a hiring trigger, platform teams get surprise deadlines. Pre-empt this by publishing an internal scoreboard with:

  • Verified vendor rows (61.7% baseline).
  • Your harness results with variance bands.
  • Cost per merged PR as the decision metric.

That dashboard prevents Pi adapter rumors from becoming roadmap law overnight.

Checklist before you quote any SWE-bench number in a deck

  1. Link the model card row or eval commit.
  2. Name the harness and version.
  3. State temperature, top_p, max tokens, context.
  4. Note timeout and max agent turns.
  5. Attach cost per run from billing export.
  6. Mark claimed social figures red until logs arrive.

Why explainx.ai separates 61.7% from 63%

We are not saying Pi cannot beat 61.7%—only that unlogged deltas should not move budget. The August HN launch thread already showed community harness tweaks beating vendor defaults on other benches. Replicate that discipline here: celebrate open evals, not screenshots.

Status: November 17, 2026; official score from Qwen/Qwen3.8-27B README; 63% Pi treated as unverified social claim until a public eval log appears.

Bottom line: cite 61.7% with Claude Code footnotes; treat 63% Pi as a hypothesis until someone ships reproducible configs. Ping explainx.ai if Alibaba updates the card or a Pi repo publishes eval YAML.

Related reading

  • Qwen3.8-27B launch and HN tests
  • Pi minimal harness guide
  • Qwen Code open-source terminal agent
  • Agent Lightning SWE-bench harness training
  • Real-repo coding agent evals
  • Together AI agent evals reporting
  • SWE-Cowork digital employee benchmark
Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 11, 2026

Qwen4-Max Preview Lands on Alibaba Token Plan: What Builders Can Try Today

On November 11, 2026 Alibaba’s Qwen team said Qwen4 is arriving and put Qwen4-Max into preview on Token Plan and four consumer apps. There is still no public weight drop, API list price row, or full four-model rollout—but builders with a Token Plan can call the preview id today.

Nov 10, 2026

Naive-N0.5-Flash: 2,100 tok/s Open Model and Latent Space Coding Test

NaiveAI markets Naive-N0.5-Flash as an open 309B MoE coding model with NaiveRT inference up to roughly 2,000 tokens per second. On November 10, 2026 Latent Space published an unbiased hands-on versus DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash. We separate NaiveAI's peak-speed claims from end-to-end latency and app quality.

Oct 30, 2026

Co-Design Bench: Best Coding Agent Fully Solves Only 26.4% of Real App Edits

Co-Design Bench tests whether coding agents can turn real users multimodal requests into working application revisions. The authors, from the Qwen team and collaborators, report a large gap between partial progress and finished tasks, and publish the code and dataset.