OakEmu-RL v0.1.0

Appearance

PPOActor ​

python
PPOActor(context: RuntimeContext[PPOExperimentConfiguration], actor_model: PPOModel, learner_model: ThreadSafeModel, rollout_buffer: RolloutBuffer, rollout_handshake: RolloutHandshake, episode_observer: Callable[[dict[str, float], int, bool], None] | None = None, replay_observation_cache: ReplayObservationCache | None = None, env_factory: Callable[[], OakEmuMultiEnv] | None = None) -> None

PPO actor that collects rollouts using the stochastic policy.

Unlike the DQN Trainer, this does not use epsilon-greedy exploration or a replay buffer. Instead, it collects complete rollouts (observations, actions, log probabilities, values, rewards, dones) into a RolloutBuffer. When the buffer is full, it signals the learner.

Note: Unlike Trainer/Tester, this does not inherit from Agent. PPO requires (action, log_prob, value) per step — not just greedy actions, so action selection is performed directly via PPOModel.get_action_bundle() in _select_actions, never through Agent's batched-inference path (_queue_for_inference / _flush_pending_inference). Composes EpisodeMetrics and EnvLifecycle directly instead -- the two pieces of Agent it actually needs.

Initialize the PPO actor.

Parameters:

  • context (RuntimeContext[PPOExperimentConfiguration]) – The shared runtime context.
  • actor_model (PPOModel) – The local PPO model for action selection.
  • learner_model (ThreadSafeModel) – The learner model container (for weight sync).
  • rollout_buffer (RolloutBuffer) – The rollout buffer to fill.
  • rollout_handshake (RolloutHandshake) – Synchronizes rollout hand-off with the learner.
  • episode_observer (Callable[[dict[str, float], int, bool], None] | None) – Optional observer invoked with each completed episode's data.
  • replay_observation_cache (ReplayObservationCache | None) – Optional shared replay observation cache.
  • env_factory (Callable[[], OakEmuMultiEnv] | None) – Optional multi-environment factory; defaults to _default_env_factory.

Source

Properties ​

replay_observation_cache ​

python
replay_observation_cache: ReplayObservationCache | None

Get the replay observation cache shared with the learner-side BC sampler, if enabled.

Source

restart_pool ​

python
restart_pool: RestartPool | None

Get the restart pool, or None when restarts are not configured.

Source

result_queue_depth ​

python
result_queue_depth: int

Get the current depth of the underlying environment result queue.

Source

steps ​

python
steps: int

Get the total number of environment steps.

Source

total_episodes ​

python
total_episodes: int

Get the total number of episodes completed.

Source

total_losses ​

python
total_losses: int

Get the total number of losses.

Source

total_truncated ​

python
total_truncated: int

Get the total number of truncated episodes.

Source

total_wins ​

python
total_wins: int

Get the total number of wins.

Source

trajectory_runtime ​

python
trajectory_runtime: TrajectoryEpisodeCoordinator

Get the trajectory coordinator shared with the learner-side BC sampler.

Source

Methods ​

restore_runtime_state ​

python
restore_runtime_state(state: ActorRuntimeState) -> None

Restore checkpointed actor counters and trajectory runtime state.

Not named restore_state (unlike the old Agent.restore_state): the actor restores a strictly wider payload than the base episode-totals state, so this is a distinct method rather than an incompatibly-typed override.

Parameters:

  • state (ActorRuntimeState) – The actor runtime state to restore.

Source

run ​

python
run() -> None

Main loop for the PPO Trainer.

Collects rollouts by stepping all environments synchronously and storing transitions in the rollout buffer. When the buffer is full, signals the learner.

Source

set_bc_materialisation_source ​

python
set_bc_materialisation_source(source: BCMaterialisationSource | None) -> None

Attach the learner-side BC sampler's materialisation queue.

Parameters:

  • source (BCMaterialisationSource | None) – The request source, or None to disable materialisation.

Source