MLExpert: An Open 35B Model Trained to Do Machine Learning Engineering Nearly Doubles Its MLE-Bench Lite Medal Rate

A new open model aims at a narrow, valuable job: doing machine learning engineering by itself. MLExpert is a 35B-parameter mixture-of-experts agent model, released under Apache-2.0, that its authors say lifts a base model's medal rate on MLE-Bench Lite from 37.9 percent to 72.7 percent. The paper is on Hugging Face, the code is on GitHub and the weights are on Hugging Face.

The authors frame it as a step toward AI systems that improve AI systems. That framing is ambitious; what the release actually offers is a trained open model, a harness and a reproducible recipe for long-horizon experiment loops. This post explains the recipe, the numbers, how to run it, and what to treat with caution.

TL;DR

What problem does it target?

Machine learning engineering is a long-horizon job. You read a competition or research task, explore data, pick a model, write training code, run it, read the metrics, and revise. Each step may take minutes or hours, and the useful feedback arrives late. General coding agents struggle because the reward for a decision, such as feature engineering or a validation split, shows up only after several experiments.

The paper's answer is training that matches the structure of the job instead of treating every token as equal.

How was MLExpert trained?

Environment. The authors built Kaggle-MLE Factory, which generates executable tasks, and an interaction harness that lets a model direct experiments.

Stage 1: hierarchical agentic SFT. Supervised fine-tuning transfers general tool-use and coding skills in layers, so the model learns to operate in an agent loop before it faces ML-specific problems.

Stage 2: Windowed Interaction RL (WIRL). Instead of rewarding only the final submission, WIRL assigns a shared outcome-based reward to the decisions made between successive experiment results. In plain terms, the model gets credit or blame for the stretch of choices that led from one result to the next. That gives denser learning signal than a single end-of-episode score and fits how experiments report back.

If you follow reinforcement learning for agents, this is a variation on credit assignment: split a long trajectory into windows that line up with feedback events. We have seen similar thinking in agent research such as Microsoft's idea-basins work on autoresearch agents.

What are the results?

The model card says 72.7 percent is the highest among same-scale models evaluated and comparable to GLM-5.3 and Kimi-K3. The README says MLExpert matches published results on seven NatureBench tasks and surpasses them on five. The paper also mentions gains on agent scaffold optimization and algorithm optimization, but the pages we read give no numbers for those.

Two details make the result more believable. The gain appears under two different harnesses, the authors' and Claude Code, so it is not only an artifact of one scaffold. And the base-to-trained comparison uses the same model family, isolating what the training added.

What are people likely to ask?

Is a 35B model really a "3B-active" model? The base model name, Qwen3.6-35B-A3B, suggests about 3B parameters active per token, which is why a 35B mixture-of-experts can be served on two GPUs. The memory cost still reflects the full 35B of weights, so plan for the weights to fit, not just the active slice.

Why does the harness matter so much? An ML engineering agent needs to launch jobs, read logs, keep notes across experiments and recover from failures. The 72.7 percent figure is with the MLExpert harness, and the 69.7 percent figure is with Claude Code, so the model is not tied to one scaffold, but the scaffold still affects results by a few points.

What is a medal on MLE-Bench Lite? The benchmark takes real Kaggle competitions and awards bronze, silver or gold based on where an agent's submission would have ranked on the leaderboard. A medal rate of 72.7 percent therefore means the agent reached medal territory on roughly three of every four of the 22 tasks. Because the set is small, each task is worth about 4.5 percentage points.

Is the 37.9 to 72.7 jump surprising? For a post-training recipe aimed at the target benchmark's task style, large gains from a modest base are common. The more informative check is whether the improvement holds for tasks the model was not trained on, which the NatureBench Lite results try to address with scientific problems of a different kind. Even there, ten tasks is a thin sample.

Should I use it instead of a frontier API? It depends on what you need. For repeated, well-specified tuning loops on private data, a local Apache-2.0 model is attractive for cost and control. For open-ended or unusual problems, closed frontier models remain the safer first choice until independent evaluations exist. A sensible approach is to run both on a handful of your own tasks and compare medals, cost and failure modes.

What would a fair test look like? Pick five tasks from your own backlog with a measurable score, fix the time and compute budget, run each system three times to see variance, and review the code the agent wrote rather than only the metric. Watch for shortcuts such as training on the test split, which scored benchmarks can accidentally reward.

What should you be careful about?

• Small benchmarks. MLE-Bench Lite has 22 Kaggle competitions and NatureBench Lite 10 tasks. A few tasks flipping changes a percentage by several points.
• Self-reported. These are the authors' numbers, not an independent evaluation, and "comparable to larger frontier models" is a comparison in their paper and model card, not something we reproduced.
• Harness dependence. Performance with the dedicated harness is higher than with Claude Code, so deployment details matter.
• Medals are not discovery. A medal rate on Kaggle-style tasks measures performance on well-defined, scored problems. It does not show open-ended research ability.
• Early project. The repository had 2 stars when we checked, and no inference provider hosts the model.
• License detail. The model card says Apache-2.0, while the README links to a license file; read the file before commercial use.

How to run it

Per the README and model card:

1. Create a Python 3.12 environment (the README uses uv) and install vLLM or SGLang.
2. Serve InternScience/MLExpert as an OpenAI-compatible API at http://localhost:8000/v1 with tensor parallelism across two GPUs.
3. Use the recommended sampling settings: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0. Machine-readable profiles are in inference_profiles.json.
4. Both servers support a 262,144-token context, a qwen3 reasoning parser and tool calling via the qwen3_coder parser. Toggle thinking with chat_template_kwargs.enable_thinking.
5. Start with a small, scored task, such as an MLE-Bench Lite competition, before pointing it at your own data.

How does it fit the wider picture?

Open models specialized for agent work are multiplying. We have covered Hugging Face's ML intern that trained six custom models for about $103, Meta's gold-medal Kaggle result with NVIDIA tools and TastEval, a benchmark of research taste. MLExpert's distinction is that it is an open-weight model trained specifically for the experiment loop, which makes it a candidate for teams that cannot send data to a closed API.

It also touches the idea of AI improving AI. The paper frames MLExpert as a step in that direction, but it is worth separating that aspiration from the facts: this is a model that does ML engineering tasks under a harness, with human-defined benchmarks. For the larger debate see our explainer on recursive self-improvement and the Brin claim-versus-evidence post.

What does it mean for builders?

• A cheap baseline for AutoML agents. If you are building an agent that tunes models, MLExpert is a ready open candidate to compare against closed APIs.
• Data stays local. Because it runs on two GPUs under Apache-2.0, sensitive datasets need not leave your infrastructure.
• The training idea is portable. Rewarding windows of decisions between feedback events applies to any slow-feedback agent task, from data pipelines to simulation tuning.
• Verify before trusting. Run it on your own held-out tasks and watch for leakage, such as a model that has seen the benchmark competitions during training.

What we still do not know

• Independent reproductions of the MLE-Bench Lite and NatureBench Lite numbers.
• The numbers behind the scaffold and algorithm optimization claims.
• How the model behaves on tasks unlike Kaggle competitions, including messy real data.
• The full training data composition and whether any benchmark tasks overlap with training.
• Cost per task in GPU hours and tokens.

Related reading

• Hugging Face ML intern: six custom models for $103
• Meta AIRA3 gold medal with NVIDIA Kaggle tools
• Microsoft idea basins for autoresearch agents
• TastEval: research taste benchmark
• What is recursive self-improvement?
• Small models comparison, 2026

Based on the MLExpert paper, repository and model card as of October 16, 2026; figures are the authors' and unreplicated.