PPO
Clipped policy optimization
ModuRL is a deep reinforcement learning framework for Rust, with training algorithms, environments, models, policy distributions, storage, schedules, and logging.
Select an algorithm, and for PPO or A2C a network topology, to see an assembly example.
// CartPole · separate models · PPO
use burn::tensor::Device;
use modurl::prelude::*;
use modurl_gym::classic_control::cartpole::CartPoleV1;
let device = Device::flex().autodiff();
let env = CartPoleV1::builder().seed(7).build()?;
let env = VectorizedGymWrapper::from(env);
let mut env = DeviceMultiGymWrapper::new(env, Device::flex(), device.clone());
let observation_space = env.observation_space();
let action_space = env.action_space();
// actor_model, critic and optimizers: setup omitted; see the guide.
let policy = ProbabilisticPolicyModel::new(
actor_model,
CategoricalDistribution,
);
let networks = SeparatePPONetwork::builder()
.actor_network(policy)
.critic_network(critic)
.actor_optimizer(actor_optimizer)
.critic_optimizer(critic_optimizer)
.actor_learning_rate(3e-4)
.critic_learning_rate(3e-4)
.build()?;
let mut agent = PPOAgent::builder()
.observation_space(observation_space)
.action_space(action_space)
.network_info(networks)
.batch_size(2_048)
.mini_batch_size(64)
.clip_range(ConstantSchedule::new(0.2))
.training_horizon(100_000)
.build()?;
agent.learn(&mut env, 100_000)?;
Implementations for on-policy and off-policy training.
End-to-end Breakout PPO throughput.
40,960 transitions with matched preprocessing and training
workloads.
CUDA · Throughput only, not final reward or sample efficiency. Emulator versions differ. See the benches README for methodology and fairness controls.
Train PPO on CartPole.
git clone https://github.com/ModuRL/ModuRL.git
cd ModuRL
cargo run --release -p examples --example ppo_cartpole
Requires a Rust toolchain with edition 2024 support. Read the getting started guide