Model components / Fine-tuning code

OpenRLHF

OpenRLHF is a Ray-based distributed RLHF and agentic-RL framework that separates Actor, Reward, Reference, and Critic models across GPUs for training at 70B+ parameters. It combines Ray for scheduling, vLLM for fast rollout generation, and DeepSpeed ZeRO-3 for memory-efficient training, and supports PPO, REINFORCE++, GRPO, RLOO, and DAPO-style dynamic filtering. A published design paper (arXiv:2405.11143) documents the architecture.

Verified 2026-08-09 via GitHub and the LICENSE body.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public(OpenRLHF/OpenRLHF)
core-gated: ungated

Apache-2.0, an OSI license, with the full source public and no proprietary tier. The license text, the README and the repository page all agree, and nothing is held back from the published code.

https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/LICENSE recorded 2026-08-09
The unmodified Apache License, Version 2.0 template -- opens "Apache License" / "Version 2.0, January 2004" and the standard TERMS AND CONDITIONS, ending with the generic placeholder "Copyright [yyyy] [name of copyright owner]" rather than a custom notice. No altered terms, no NOASSERTION trap.
https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/README.md recorded 2026-08-09
The Installation section gives "pip install openrlhf" and, for the source route, "git clone https://github.com/OpenRLHF/OpenRLHF.git" followed by "pip install -e ." -- the thing you run builds from this repo. No enterprise, pricing, hosted-service, or paid-tier section appears anywhere in the file.
https://github.com/OpenRLHF/OpenRLHF recorded 2026-08-09
Repo page's embedded data records "license":{"spdxId":"Apache-2.0","name":"Apache License 2.0"} and "isArchived":false, with a top-level tree containing the openrlhf package directory itself (plus dockerfile, docs, examples, tests) -- the repo is the shipped code, not a stand-in for it.

Adoption

2 medium confidence

2.0

~9.6K GitHub stars, and the repository lists named adopters: Google, ByteDance, Tencent, Alibaba, Baidu, China Telecom, Vivo, Allen AI, NexusFlow and Jülich. This is a specialist RLHF tool used by research and training teams rather than a mass-download library, and no headline PyPI figure is published for it, so the level rests on those named adopters rather than on the star count.

https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/README.md recorded 2026-08-09
The "Companies and Organizations using OpenRLHF" section still lists Google, ByteDance, Tencent, Alibaba, Baidu, China Telecom, Vivo, Allen AI, NexusFlow, and Jülich Supercomputing Centre (JSC), among others -- the same named adopters the score's note cites, unchanged. No download-count badge or PyPI figure appears anywhere in the file.
https://github.com/OpenRLHF/OpenRLHF recorded 2026-08-09
Repo page reports "stargazerCount":9896 -- attention in the high thousands, well short of a mass-adoption tier, and consistent with the named-traction basis the score rests on.

Capability

4 medium confidence

4.0

Deep, production-grade distributed RLHF: PPO, REINFORCE++, GRPO and RLOO, orchestrated with Ray scheduling, vLLM-accelerated rollout and DeepSpeed, scaling to 70B+ models across many GPUs, with multi-turn agent RL as well. Its method scope is RL-centric rather than full-spectrum, and it does not reach Megatron-LM's frontier scale, so it sits one band below the top.

https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/README.md recorded 2026-08-09
Architecture section still describes Ray separating Actor/Reward/Reference/Critic models "up to 70B+ parameters," vLLM for high-throughput rollout generation, and DeepSpeed ZeRO-3/AutoTP/RingAttention for training; the algorithm table still lists PPO, REINFORCE++, REINFORCE++-baseline, RLOO, GRPO, and Dr. GRPO. A new experimental "Molt" backend is noted as an alternative to DeepSpeed, not yet the primary path.

Verified 2026-08-09