Nebius published Longtail-SWE on November 6, 2026 (arXiv 2611.10304) with immediate artifacts on Hugging Face: nebius/Longtail-SWE, nebius/Longtail-SWE-trajectories, and a Harbor task bundle. The benchmark fills a gap mainstream SWE-bench forks leave open: repository-level repair in languages whose communities rarely get agent eval seats.
TL;DR
| Item | Detail |
|---|---|
| Tasks | 202 issue-resolution instances |
| Languages | 13 long-tail ecosystems |
| Source | Real GitHub issues and PRs |
| Pipeline | Solver → judge → repairer on tests and prompts |
| Artifacts | Datasets, envs, tooling, agent trajectories |
| Extra | Easier task subset for smaller models |
Why another SWE benchmark exists
Leaderboard-centric benchmarks concentrate on Python and JavaScript monorepos. When constructors filter ambiguous issue text or flaky tests, they often delete the only public tasks for Clojure, Erlang, or OCaml ecosystems entirely. Longtail-SWE argues that discards real engineering difficulty and makes polyglot agents look better than they are.
The construction insight is repair instead of reject. A solver attempts each candidate task. A judge reads the solver trajectory plus test output and proposes test and description fixes that:
- Stop rejecting valid alternative implementations
- Stop accepting incorrect solutions
- Keep tasks hard while preserving original intent
That mirrors human maintainer work more than one-shot SWE-bench importers.
Issue card linked to a patch illustrating Longtail-SWE repository-level agent tasks
Image: Nebius Longtail-SWE project
Human hand and gripper sharing a beam for polyglot coding agent evaluation
Language coverage
The thirteen languages are not decorative tags—they change build graphs, package managers, and test runners your agent must orchestrate:
| Language cluster | Why agents stumble |
|---|---|
| BEAM family (Elixir, Erlang) | Supervision trees, dynamic modules |
| FP web (Clojure, Haskell, OCaml, F#) | Rich type errors and macro layers |
| Mobile / JVM (Kotlin, Scala, Swift) | Gradle or Xcode-sized environments |
| Stats / DSL (R, Julia, Lua) | Domain libraries with sparse LLM priors |
If your product claims “any language” support, a single Python SWE-bench score is insufficient evidence.
Artifacts you can run this week
Dataset: nebius/Longtail-SWE
Trajectories: nebius/Longtail-SWE-trajectories
Project site: longtail-swe.com
GitHub: github.com/SWE-rebench/LongTail-SWE
Harbor: longtail-swe/longtail-swe on the Harbor framework hub
Harbor matters because teams already running agent eval playbooks or custom CI sandboxes can slot Longtail-SWE beside SWE-bench Verified without rewriting orchestration.
Suggested first run
- Pull Harbor task definitions and inspect one language you actually ship (for example Kotlin).
- Run a single-task smoke with your default coding agent—log tool calls, patch size, and test iterations.
- Compare against a Python SWE-bench task of similar star-count repo to see token and time inflation.
- Only then schedule a full 202 pass; container cold starts multiply by language toolchains.
Evaluation design in the paper
Authors test nine model configurations, including frontier models, and analyze:
- Task solve rate (with language breakdowns in the paper body)
- Consistency across repeated attempts—important for noisy agents
- Resource usage—wall clock and spend proxies
They also release easier companion tasks so smaller open models are not blocked from participation. That lowers the barrier for teams fine-tuning 9B–32B coders who still need non-Python signal.
Community comments on the Hugging Face paper page highlight the judge repairing tests that rejected valid implementations—a failure mode anyone who has maintained OSS will recognize.
How Longtail-SWE fits the 2026 eval stack
| Benchmark family | What it stresses |
|---|---|
| WorkflowArena | Multi-agent orchestration |
| Remote Labor Index | Economics of automation |
| Longtail-SWE | Polyglot repo repair |
Pair Longtail-SWE with internal tickets from your customers’ least common language. If the agent fails Longtail-SWE OCaml tasks but passes Python, your marketing should say so.
Limitations to plan around
- Construction cost: judge-repairer loops use LLM labor; tasks may inherit judge blind spots.
- Star counts: LongTail-SWE’s GitHub mirror showed modest stars at launch—expect rough edges in docs.
- License mix: repos retain their upstream licenses; commercial eval hosts must comply per task.
- Not a security bench: passing tests does not imply safe diffs—run separate policy checks if agents touch prod configs.
Per-language notes for harness engineers
The paper’s value is per-ecosystem friction, not a single solve-rate meme. When you schedule eval batches, expect:
Clojure and Scala (JVM cousins): Dependency inference agents often hallucinate Leiningen versus deps.edn versus sbt paths. Long tasks reward agents that read project.clj before editing.
Elixir and Erlang (BEAM): Tests may assume supervision semantics. Patches that compile but break OTP patterns will fail in ways Python agents rarely see.
Haskell, OCaml, and F#: Type errors are long. Allocate higher turn budgets or your agent will stop mid-derivation even when close.
Kotlin and Swift: Mobile toolchains drag Gradle or Xcode weight into containers. Cold-start minutes dominate cost more than LLM tokens.
Julia and R: Numeric stacks pull heavy native deps. Mirror Nebius executable environments exactly—do not slim Dockerfiles for speed.
Lua and Dart: Smaller communities mean sparser LLM priors. Expect more tool calls to read docs inside the repo.
Harbor operators: Import longtail-swe/longtail-swe and pin one language per nightly job so failures are attributable. Mixing thirteen toolchains in one CI matrix obscures regressions.
Judge–repairer failure modes (what reviewers should watch)
The pipeline fixes bad tests, not bad agents. Watch for:
- Overfitted tests that accept one hacky patch pattern
- Over-repaired descriptions that leak solution hints
- Judge variance if the judge model drifts between construction batches
Nebius mitigates by keeping human-readable trajectories public. Audit a handful before trusting leaderboard numbers in vendor decks.
SWE-bench family positioning (conceptual, not ranked)
| Bench style | Strength | Longtail gap it leaves |
|---|---|---|
| Python-centric SWE-bench forks | Volume and community replicas | Drops rare languages when filters bite |
| Polyglot ports | Wider language tags | Often still Python-majority issues |
| Longtail-SWE | Retention via repair | Smaller N, higher setup cost |
If your COO asks “are we SOTA?”, answer with language coverage, not one percentage.
Cost model for a full 202-task pass
Budget container minutes more than tokens:
- 202 environments × average 15–40 minutes cold start (varies by language)
- Agent turns × model price
- Storage for trajectory logs if you archive for compliance
Start with 20 stratified tasks (two per language) before burning a full cluster weekend.
Security note for autonomous eval
Longtail-SWE runs arbitrary project test suites inside containers. Treat harness hosts like CI: network egress limits, no production secrets, and pinned base images. Our AI usage fragile codebase thread shows why unreviewed agent patches compound tech debt even when tests pass.
Integrating Longtail-SWE with Claude Code, Codex, and custom harnesses
Most teams do not run Harbor natively on day one. A practical bridge:
- Export one task JSON from the dataset and craft a DESIGN.md-style prompt with repo layout, issue text, and test command—patterns from Claude Code ownership guides.
- Run the agent in a fresh container per attempt; Longtail-SWE assumes executable roots, not partial checkouts.
- Capture trajectory format compatible with Nebius releases so you can compare against published baselines later.
- If the agent proposes test changes, flag that as a judge-repairer scenario—human reviewers should decide whether the agent cheated by weakening assertions.
For eBay blocking third-party agents, remember benchmarks are offline repos—you are not scraping commerce sites. Keep network egress off in eval sandboxes.
What a good internal scorecard looks like
| Metric | Definition |
|---|---|
| Solve rate | Tests green without human edit |
| Turn count | Tool calls until patch or abort |
| Revert rate | Tasks where agent broke unrelated modules |
| Language delta | Worst language vs median |
| Cost per solve | GPU container minutes + model spend |
Report language delta to leadership; it exposes “Python-only SOTA” marketing lies.
Paper metadata and citation
- arXiv: 2611.10304
- Authors: Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev (Nebius)
- Submitted to HF daily papers: November 10, 2026
- Community upvotes on HF: trending modestly—signal for researchers, not mass virality
Cite the dataset version hash when publishing internal numbers; Nebius may add tasks in v2.
FAQ for engineering managers
Will this replace our private eval set?
No. Use Longtail-SWE for coverage gaps in public languages; keep customer tickets for revenue-linked skills.
Can we fine-tune on trajectories?
Technically yes, ethically thorny. Read licenses on upstream repos before training on logged patches.
Does Nebius benefit commercially?
Nebius sells GPU cloud. A harder public bench nudges buyers toward serious infra—disclose that bias when interpreting their baselines.
How often should we re-run?
Quarterly, or whenever you bump frontier model versions. Agent regressions hide between releases.
Nebius and the eval transparency trend
Cloud GPU vendors sponsoring benchmarks is not new, but releasing trajectories is. Trajectories let outsiders detect judge gaming and tool abuse—aligned with Hugging Face security tradeoff debates about open eval data.
If your roadmap includes non-English issue trackers, Longtail-SWE still uses English issue text from GitHub—the languages are code, not locale. Plan separate evals for multilingual product copy. Archive raw logs under your internal experiment id so you can compare against Nebius trajectory releases later.
Related reading
- Amazon Bedrock AgentCore evaluation playbook
- WorkflowArena multi-agent benchmark
- Jane Street agent skills on GitHub
- Git project vote on LLM contributions
- AI peer review planted errors benchmark
- Opus 5.5 CPython formalization bugs
Dataset URLs and paper id are accurate as of November 11, 2026.