OakEmu-RL v0.1.0

Appearance

PPOLearner ​

python
PPOLearner(context: RuntimeContext[PPOExperimentConfiguration], learner_model: ThreadSafeModel, rollout_buffer: RolloutBuffer, optimizer: torch.optim.Optimizer, lr_schedule: Schedule, action_mask: torch.Tensor | None = None, bc_sampler: BCSampler | None = None) -> None

PPO Learner that performs clipped policy gradient updates.

Consumes completed rollouts from the RolloutBuffer, running multiple epochs of PPO updates with clipped surrogate objective, value loss, and entropy bonus.

Initialize the PPO Learner.

Parameters:

  • context (RuntimeContext[PPOExperimentConfiguration]) – The shared runtime context.
  • learner_model (ThreadSafeModel) – The learner model container (with lock).
  • rollout_buffer (RolloutBuffer) – The rollout buffer to consume.
  • optimizer (Optimizer) – The PyTorch optimizer.
  • lr_schedule (Schedule) – Schedule for the learning rate.
  • action_mask (Tensor | None) – Optional action mask tensor on the learner device.
  • bc_sampler (BCSampler | None) – Optional learner-side behaviour-cloning data source.

Source

Properties ​

iterations ​

python
iterations: int

Get the total number of learner iterations.

Source

rollouts_collected ​

python
rollouts_collected: int

Get the total number of rollouts collected.

Source

Methods ​

restore_state ​

python
restore_state(state: PPOLearnerRuntimeState) -> None

Restore checkpointed learner counters.

Parameters:

  • state (PPOLearnerRuntimeState) – The learner runtime state to restore.

Source

run ​

python
run(stop_event: Event, current_step: Callable[[], int], rollout_handshake: RolloutHandshake, policy: CheckpointPolicy) -> None

Main learner loop: wait for a published rollout, run a PPO update, notify the tester, feed automatic rollback monitoring, and manage periodic/final checkpointing.

Parameters:

  • stop_event (Event) – Signals when training should stop.
  • current_step (Callable[[], int]) – Returns the current global actor step count.
  • rollout_handshake (RolloutHandshake) – Synchronizes rollout hand-off with the actor.
  • policy (CheckpointPolicy) – The checkpoint/rollback seam this loop drives.

Source

step ​

python
step(actor_steps: int) -> None

Perform a full PPO update on the current rollout buffer contents.

Runs ppo_epochs epochs when on-policy PPO data is present. Replay-only BC rollouts use a single epoch so the replay bootstrap phase is not multiplied by the PPO epoch count.

Parameters:

  • actor_steps (int) – The current actor step count (for metrics logging).

Source