RolloutFinalizer

RolloutFinalizer(config: PPOExperimentConfiguration, actor_model: PPOModel, learner_model_container: ThreadSafeModel, rollout_buffer: RolloutBuffer, rollout_handshake: RolloutHandshake, observation_workspace: ObservationTransferWorkspace) -> None

Finalizes one completed PPO rollout: bootstraps GAE from the current observations, hands the buffer to the learner, waits for it to be consumed, and syncs the actor's weights.

Initialize the finaliser.

Parameters:

• config (PPOExperimentConfiguration) – The PPO experiment configuration (for gamma/gae_lambda).
• actor_model (PPOModel) – The local PPO model, used to bootstrap the value at the rollout's end.
• learner_model_container (ThreadSafeModel) – The learner model container, synced to after each rollout.
• rollout_buffer (RolloutBuffer) – The rollout buffer to finalize and reset.
• rollout_handshake (RolloutHandshake) – Synchronizes rollout hand-off with the learner.
• observation_workspace (ObservationTransferWorkspace) – Transfers the bootstrap observations to the model's device.

Source

Methods

finalize

finalize(env_states: list[PPOActorEnvironmentState], dones: np.ndarray, trainable: np.ndarray) -> None

Compute GAE, signal the learner, wait for consumption, sync weights, and reset the buffer.

Parameters:

• env_states (list[PPOActorEnvironmentState]) – The actor's current per-environment state, used to bootstrap the value of each environment's next observation.
• dones (ndarray) – Whether each environment's current episode ended on the rollout's last step.
• trainable (ndarray) – Whether each environment contributed a trainable step.

Source