Aravind Srinivas posted a GitHub link on September 3, 2026 with a three-word caption: "open-source RL-as-a-service." Forty-two thousand views, and a reply thread that named most of the ecosystem in about six hours — Miles, prime-rl, SkyRL, SGLang, with the Miles team turning up to salute.
The phrase is doing a lot of work, so it is worth unpacking. "RL-as-a-service" is the claim that reinforcement learning post-training has stopped being research code you fork and rewrite, and has become infrastructure you configure. That is a real transition, it happened fast, and it has a specific shape.
This post covers what an RL post-training stack is actually made of, what each of the major open contenders is optimized for, and the question the excitement tends to skip: whether you should be running one.
TL;DR
| Question | Answer |
|---|---|
| What is "RL-as-a-service"? | The post-training loop packaged as configurable infra — orchestration, fault tolerance, weight sync — not research code. |
| Why now? | Post-training is where model differentiation moved. The frameworks commoditized; environments and GPU hours did not. |
| Miles | RadixArk (SGLang team), v0.1 on Aug 18, 2026. Large-scale, MoE-aware, production concerns first. |
| SkyRL | Berkeley/NovaSky. Modular full-stack, built for multi-turn tool-using agents. Swap any component. |
| prime-rl / Prime Intellect | Decentralized async training plus a large open environment catalog (2,500+ envs, 400+ contributors as of June 2026). |
| OpenRLHF | Ray + vLLM, broad algorithm coverage (PPO, DAPO, REINFORCE++). The pragmatic default. |
| Hardware floor | Multi-GPU, because training and serving run at the same time. This is the real barrier, not the license. |
| Should you run RL? | Only if you have a verifiable reward, a task distribution you own, and evals you trust. Otherwise: no. |
What an RL post-training stack is actually made of
Every one of these frameworks decomposes into the same four parts. Understanding the decomposition is most of understanding the comparison, because each project is really a bet about which part is hardest.
| Component | Job | Why it's hard |
|---|---|---|
| Rollout engine | Generate completions from the current policy — thousands of them, fast | It's a full inference server that must be re-pointed at new weights mid-run |
| Trainer | Compute advantages, run backward passes, hold optimizer state | Memory. Optimizer state for a large MoE dwarfs the weights |
| Orchestrator | Schedule rollout and training across nodes, survive failures | A 72-hour run on hundreds of GPUs will lose a node |
| Environment + reward | Define the task, score the output | Not an infra problem at all — and the one that decides whether any of it works |
The weight synchronization step between rollout and trainer is where these systems live or die. After each update, the new policy has to reach the inference engine before the next batch of rollouts, and doing that naively means serializing and reloading tens of gigabytes on every step. Miles's use of NCCL/RDMA for fast weight sync is not a footnote — it is close to the whole engineering thesis.
The contenders
Miles — the enterprise MoE bet
Released as v0.1 on August 18, 2026 by RadixArk, a company founded by the SGLang development team. It is described as an enterprise-facing RL framework for LLM and VLM post-training, forked from and co-evolving with slime.
Composition: SGLang for rollout, NVIDIA Megatron-LM for training, Ray for orchestration, PyTorch-native extensibility behind a small pluggable trainer. What it adds on top is the production layer — unified low-precision recipes, MoE-aware rollout/training alignment, fast NCCL/RDMA weight synchronization, observability, and fault tolerance.
The team's own framing of the problem is the honest one: RL training is easy to start and hard to debug. Miles is aimed less at "can I run GRPO" and more at "is my 200-GPU run correct, and will it survive the night."
Pick it if: you are training MoE models at multi-node scale and your bottleneck is throughput plus reliability. Skip it if: you have four GPUs — you will pay the complexity without touching the capability.
SkyRL — the modular agent bet
From Berkeley's NovaSky group, positioned as a modular full-stack RL library for LLMs, with an explicit design goal that each component can be modified, extended, or replaced. Its distinguishing focus is multi-turn, tool-using agents on real-world tasks — the SkyRL-Agent line targets efficient RL training for exactly that, and it integrates with agent evaluation harnesses rather than assuming single-turn scoring.
That focus matters more than it sounds. Single-turn RL scores one completion. Agentic RL has to score a trajectory — several model calls, tool invocations, and environment responses — where credit assignment is genuinely unsolved. A framework that treats the trajectory as the unit is doing different work from one that treats the completion as the unit.
Pick it if: you are training agents that call tools over multiple turns, and you expect to replace components as your setup evolves. This is the closest match to the workloads we describe in what are AI agents and loop engineering for coding agents.
prime-rl / Prime Intellect — the decentralized-and-environments bet
Prime Intellect's stack is built around asynchronous RL at scale across heterogeneous, distributed compute, plus a platform for turning arbitrary tasks into RL environments through a CLI. As of June 2026 it reported 2,500+ open RL environments and 400+ contributors.
The environment catalog is arguably the more important half. Frameworks are increasingly interchangeable; a large, shared, versioned library of scored tasks is not, because building good environments is slow expert labor that does not parallelize well.
Pick it if: your constraint is environment supply or you are assembling compute from places that do not look like one clean cluster.
OpenRLHF — the pragmatic default
Ray + vLLM distributed architecture, unified agent-based design, and broad algorithm coverage — PPO, DAPO, REINFORCE++, VLM support, async RL. It has been the "it just runs" option for long enough to have accumulated the boring virtues: documentation, worked examples, and people who have already hit your error message.
Pick it if: you are doing your first serious RL run and want the smallest gap between a tutorial and your own job.
At a glance
| Miles | SkyRL | prime-rl | OpenRLHF | |
|---|---|---|---|---|
| Origin | RadixArk (SGLang team) | Berkeley / NovaSky | Prime Intellect | Community |
| Optimized for | Large-scale MoE, production reliability | Multi-turn tool-using agents | Decentralized async + environments | Broad algorithm coverage |
| Rollout engine | SGLang | Pluggable | Pluggable | vLLM |
| Trainer | Megatron-LM | Pluggable | Custom async | DeepSpeed/FSDP lineage |
| Orchestration | Ray | Ray | Custom distributed | Ray |
| Standout | NCCL/RDMA weight sync, fault tolerance | Component modularity | 2,500+ open environments | Maturity and docs |
| Realistic floor | Multi-node | Multi-GPU | Multi-GPU, tolerant of heterogeneity | Multi-GPU |
Why all of this is free
It is worth being clear-eyed about the business logic, because it tells you what will still be maintained in two years.
The framework is not the moat. Rollout engines went free first — vLLM, SGLang — because serving throughput commoditizes quickly and the value accrues to whoever owns the hardware or the model. RL orchestration is following the same curve one layer up. Giving it away builds gravity around your serving engine (SGLang, for Miles), your compute marketplace (Prime Intellect), or your research reputation (SkyRL).
The scarce inputs stay scarce. Verifiable reward functions, high-quality environments, and GPU hours. None of those got cheaper because a repo got a license.
This is the same dynamic behind the week's other open-source releases — Perplexity open-sourcing its Lily local inference engine for Apple Silicon and shipping hybrid local/cloud compute in its Mac app — and it is the strategic case we laid out in going open source with AI.
Should you actually be doing this?
The uncomfortable section. RL post-training is genuinely the frontier of model differentiation, and it is also the most over-attempted technique in applied AI right now. Three questions, in order:
1. Do you have a verifiable reward? Not a preference model — a checkable signal. Tests pass or fail. The SQL returns the right rows. The refactor compiles. If you cannot state your reward in one sentence and defend why it cannot be gamed, RL will find the gaming strategy faster than you will.
2. Have you exhausted the cheaper layers? Prompting, retrieval, tool design, and supervised fine-tuning on your own data are all dramatically cheaper and account for most of the gains most teams actually need. We walk that ladder in context engineering vs prompt engineering.
3. Do you trust your evals? RL optimizes exactly what you measure. A weak eval plus a strong RL loop produces a model that is measurably better and actually worse — the failure mode explored in self-graded benchmarks and agent memory.
If all three are yes, the framework question is real and this comparison is for you. If any is no, the honest advice is that these repos will still be here — and better documented — in six months.
Honest limitations
- We cannot confirm which repository the original post linked to. The caption was three words and a link; the thread named Miles, prime-rl and SkyRL in the replies. This post covers the landscape those names describe rather than asserting a specific announcement.
- "Open source" is not "runnable." The hardware floor — training and serving concurrently, multi-node interconnect for the big frameworks — is the actual gate, and no license removes it.
- Version churn is severe. Miles is at v0.1. APIs in this category break between minor releases; pin commits, not tags.
- No independent head-to-head benchmark exists across these frameworks on identical hardware and workloads. Every performance claim in the space is self-reported.
- Environment quality is unaudited. A catalog of 2,500 environments is not 2,500 good environments; reward-hacking surface scales with catalog size.
- Comparisons here are by stated design goals and published architecture, not by our own training runs. Treat this as a map for choosing what to evaluate, not as an evaluation.
Related on explainx.ai
- Perplexity open-sources Lily, its Apple Silicon inference engine
- Intelligence ownership, GRPO, and the 9B catalog
- Going open source with AI — the business guide
- What are AI agents — the complete beginner's guide
- Loop engineering for coding agents
- Self-graded benchmarks and the eval trust problem
- How to run open-source models locally with OpenCode
- Context engineering vs prompt engineering
Framework descriptions reflect publicly stated architecture and design goals as of September 3, 2026 — including Miles v0.1 (RadixArk, August 18, 2026) and Prime Intellect's June 2026 environment-catalog figures. This category moves monthly and none of these comparisons are based on our own training runs; verify current capabilities against each project's own documentation before committing GPU hours.