If you run open-weight models in a coding agent, the harness — tools, compaction, permissions, prompts — can move benchmark scores by double digits without changing the weights. Researcher Joel Niklaus made that concrete with a public Harness Leaderboard on Hugging Face. Version 0.1.1 (November 6, 2026) adds reliability bars for three-attempt success, the statistic that matters when you leave an agent running overnight.
TL;DR: how to read the board
| Question | Answer |
|---|---|
| What is held fixed? | Model, provider, reasoning effort, task set |
| What varies? | The coding-agent harness only |
| Benchmarks | Terminal-Bench 2.1 (89 tasks) and DeepSWE v1.1 (113 tasks) via Harbor on HF infrastructure |
| Headline score | Macro-average resolve rate with 95% confidence intervals |
| New in v0.1.1 | Share solved on all three attempts vs at least one attempt |
| Where to verify | Live Space plus CHANGELOG.md |
What the leaderboard measures
Agent harness scaffold illustrating fixed-model coding harness comparisons on the leaderboard
The README states the goal plainly: which coding-agent harness should you run your open model in? Same model, same provider, same reasoning effort, three attempts, every trajectory public.
Harnesses on the board (v0.1.0): Claude Code, Codex, Pi, OpenCode, Hermes Agent (Nous Research), mini-swe-agent, Kimi Code, Qwen Code, and OpenClaw.
Models: GLM-5.3-Flash and DeepSeek-V4.1-Flash through Hugging Face Inference Providers at low and high reasoning effort, plus Flint on dedicated H200 endpoints with default effort.
Infrastructure: Trials run in HF Sandboxes, orchestrated by HF Jobs, with a metering proxy inside each sandbox so harnesses cannot self-report cost. Results land in the public bucket joelniklaus/harness-leaderboard.
For what each harness actually does — edit format, compaction, tool list — Niklaus points to Harness Lens, a dictionary of 34 harnesses with sourced claims. That pairing matters: the leaderboard scores outcomes; Harness Lens documents mechanisms. If you are comparing Hermes Agent versus OpenClaw in production, read both.
Version 0.1.1 and the reliability section
Coding terminal gears metaphor for three-attempt reliability metrics on agent harnesses
The changelog dated November 6, 2026 lists v0.1.1 changes:
- A reliability section after the main table showing, per harness, the share of tasks solved on all three attempts and the share solved on at least one attempt.
- Models still running stay off the board until their full run finishes and trajectories pass checks.
- A visible version number next to the Space title, linked to the changelog.
Niklaus writes that for unattended work, solving a task every time matters more than solving it once. That is the gap social posts about "Hermes leads reliability" are pointing at: headline averages hide flaky harnesses that get lucky on one attempt.
The unreleased changelog section (November 7 entries) also documents methodology fixes — Claude Code reasoning visibility on router models, provider default output limits, and upcoming harness versions — without changing the reliability framing. Treat unreleased notes as in-flight, not scored yet.
Scores readers are citing (verify live)
explainx.ai did not recompute the board. Numbers below come from the Space plus secondary summaries cross-checked on November 7, 2026. Open the Space before you ship a workflow.
Community reads of the published v0.1.x board report roughly:
- Macro average across models: mini-swe-agent about 51.4%, with Kimi Code, Hermes Agent (~50.3%), Pi, and Claude Code within bootstrap confidence intervals — marked tied for first.
- DeepSeek-V4.1-Flash column: Hermes Agent ~71.3%, slightly ahead of mini-swe-agent at ~70.7%, with a wider tied group above 68%.
- OpenClaw sits far lower on average (~27% in summaries), illustrating harness spread on the same weights.
- Reliability band (top harnesses): summaries cite roughly 32–33% of tasks solved on all three attempts and 65–69% on at least one for the leading pack.
Those figures can shift as re-runs land (OpenClaw and OpenCode updates are already queued in the changelog). This is why we treat the leaderboard as a living benchmark, not a one-day rank stat.
Method details builders should trust
From the README method section:
- Fairness: Pinned provider, pinned harness versions, pinned reasoning effort enforced by a metering proxy.
- Cheating fails: Looking up published solutions, copying transcripts, or calling another model counts as failure, checked with atif-scan.
- Harness vs noise: With three attempts, the board compares how often rerunning the same harness flips a task versus swapping harnesses, marking ties when bootstrap CIs overlap.
- DeepSWE networking: DeepSWE runs offline since October 23, 2026; older online trials were rerun or flagged.
- Cost and time: Measured outside the harness via Harbor and the proxy; agents only see inference-only tokens, redacted before upload.
That methodology is stricter than most viral tier lists. When a viral harness tier list omitted Hermes entirely, the complaint was about missing a major open harness. This board explicitly includes Hermes and holds the model constant.
What Hermes Agent is on this chart
Hermes Agent is Nous Research's open-source coding and general agent stack. explainx.ai has covered its OpenRouter usage rank, v0.22.0 connector release, and VPS plus Telegram setups. The leaderboard does not endorse Hermes for every model; on Flint, summaries show much lower absolute scores and different winners (Kimi Code and mini-swe-agent in the ~21% range), reinforcing that harness rankings transfer poorly across models.
If you are on DeepSeek-V4.1-Flash locally or via a provider, Hermes's column is the one driving today's discussion. If you are on Flint or GLM-5.3-Flash, read the per-model tab instead of the macro average.
How much harness choice actually moves outcomes
Niklaus's broader research theme — also visible in his harness-optimization repository — is that swapping harnesses can move pass rates by tens of points on held-out coding tasks while Spearman rank correlation across models can be near zero. The leaderboard's harness-vs-noise section makes the same point for Terminal-Bench and DeepSWE: switching harness flips a minority of tasks, but rerunning the same harness also flips a similar minority, so single-attempt leaderboard screenshots overfit luck.
Practical takeaway: pick a harness, then budget retries and verification like any long-horizon agent workflow. The reliability bars exist to quantify that need.
Security and unattended agents
Unattended coding agents are a security surface, not only a benchmark puzzle. Tools, shell access, and compaction policies decide whether an agent can exfiltrate secrets on attempt two after failing attempt one. If you run agents on shared infrastructure, pair harness choice with session monitoring. The explainx.ai team ships AgentBeam to stop agent sessions before dangerous tool calls execute — use it when an agent has credentials, not only when a leaderboard score looks good.
For a wider view of how wrappers affect safety, see our report on Check Point's Artex autopilot exposure study — unattended agents with shell access are an operational risk, not only a scoreboard question.
How to use the board this week
- Open the Harness Leaderboard Space, pick your model and reasoning effort, and note the version string.
- Read average resolve for ranking, then reliability if you plan unattended runs.
- Click through to a harness trajectory when a task fails — compare tool loops between Hermes, Pi, and mini-swe-agent on the same prompt.
- Cross-check harness internals on Harness Lens before you change production config.
- Revisit after changelog entries ship; OpenClaw and OpenCode reruns are explicitly pending.
If you are new to the vocabulary, start with What is an agent harness? and our DeepSeek Harness desktop guide for another vendor-built wrapper on open weights.
How this compares to other harness rankings
| Source | What it tests | Harness vs model |
|---|---|---|
| Harness Leaderboard (Niklaus) | Terminal-Bench 2.1 plus DeepSWE on open models | Harness varies, model fixed |
| Terminal-Bench official rows | Often one harness per vendor blog post | Mixed |
| Viral tier lists | Opinion buckets | Not controlled |
| OpenRouter token leaderboard | Production traffic | Usage, not quality |
Niklaus's board is closest to a controlled experiment among the three November 2026 conversation threads. It still covers only the harnesses and models listed — no claim about Claude Code on Anthropic's own weights, for example.
Limitations the changelog already admits
Builders should copy three caveats straight from Niklaus's November 7 changelog notes before trusting a screenshot.
Context windows were wrong for several harnesses on day one. Pi assumed 128k tokens while board models expose 262k to 1M tokens, which changes cost and compaction behavior. Kimi Code assumed 1M tokens on Flint and blew past the real 262k window on dozens of trials. Upcoming runs fix configuration, but historical rows stay frozen until reruns complete.
Claude Code on router models missed reasoning streams. Saved trajectories on GLM and DeepSeek runs may lack reasoning text Claude Code would normally see on native endpoints. Flint trajectories differ. That matters if you compare tool-loop quality harness to harness.
Pending harness reruns will reshuffle ranks. OpenClaw moves to a version with proper output limits; OpenCode jumps to 2.x; Claude Code minimal mode joins as a sibling entry. A rank that looks decisive on November 7 may read differently after those jobs finish.
Treat the leaderboard like a versioned dataset: note the version string, download trajectories for tasks you care about, and rerun locally when your stack diverges from HF Sandboxes.
Mini walkthrough: reading one row
Suppose you run DeepSeek-V4.1-Flash at high reasoning and wonder whether Hermes or mini-swe-agent deserves your weekend experiment.
- Open the Space, select that model stage, and sort by average resolve.
- Open the reliability panel: if Hermes shows a higher all-three rate but a similar at-least-one rate, Hermes is less flaky for unattended batches.
- Pick one task both solved on attempt one but Hermes failed on attempt three — download trajectories and compare whether Hermes retried a bad shell command.
- Cross-read the Harness Lens entries for tool lists; Hermes speaks agentskills.io skills many readers already use in loops.
- Port the winning config to your machine with Hermes VPS guidance only after you accept sandbox differences.
That five-step loop beats treating social posts about "Hermes leads reliability" as a permanent ranking.
Related reading
- A viral agent harness tier list — does it hold up?
- Hermes Agent vs OpenClaw
- Hermes Agent v0.22.0 connectors release
- What is an agent harness?
- Harness is the company
- Agent harness DAG planner worker critic
Harness Leaderboard versions, harness versions, and cited percentages are accurate as of November 7, 2026, per the Hugging Face Space README and changelog; live scores may change as new runs complete.