In reinforcement learning, a policy chooses actions from observations to earn rewards. PPO trains a policy from batches of experience while limiting how much its action probabilities change per update. While PPO can be applied to robot control, games, language-model post-training, as well as various other problems, this program uses navigation as an example for demonstrating PPO.
One robot navigates a randomly generated square grid. It observes its
position, the goal’s absolute position, and the full obstacle map. It
moves in four directions. Reaching the goal gives +1, each step costs
0.01, and hitting a wall costs another 0.02. There is no reward for
getting closer. --size sets the grid width (default 5),
--obstacles defaults to 30% of its area, and
--horizon defaults to size*size steps. Each
start and goal are distinct and connected by a route within that limit.
A shortest-path search checks reachability when generating maps but does
not guide the robot.
ppo.cc uses only the C++17 standard library. It runs
--envs independent environments
sequentially, collecting --rollout steps
per environment with a fixed policy. Each step records the observation,
action, reward, and the policy’s action probability and value prediction
(expected future reward). One rollout may cross an episode boundary.
The program uses generalized advantage estimation (GAE) to estimate
whether actions did better than predicted; it stops the calculation at
episode ends and uses a value prediction if a rollout ends mid-episode.
It then shuffles the batch and trains for --epochs passes
in groups of --minibatch steps. PPO clips its training
objective when an action’s probability ratio moves too far from the
probability recorded during collection. The shared model has ReLU hidden
layers (--num-layers layers of --hidden units)
and separate action-probability and value outputs. Gradients and Adam
updates are implemented directly, without a machine-learning
library.
Every --log-every updates, the program prints and
flushes training_s (cumulative rollout and learning time,
excluding evaluation), train_return (mean reward of
recently finished training episodes), eval_return (mean
reward on separate fixed-seed maps), and eval_success
(fraction reaching the goal). Evaluation uses the most probable action
and does not train the model.
In the notebook’s compilation cell, after cd code and
module setup:
clang++ -O3 -Wall -std=c++17 ppo.cc -o ppoIn its %%sub run cell, after cd code:
./ppo --size 5 --updates 20000 --envs 16 --rollout 16 --epochs 3 --minibatch 128 --hidden 64 --num-layers 2 --log-every 1000An additional --seed parameter sets the random seed for
reproducibility.
In addition to the usual GPU acceleration of neural-network matrix operations,