OakEmu-RL v0.1.0

Appearance

PPOExperimentConfiguration ​

Configuration for PPO experiments.

Source

Values ​

activation_statistics_interval ​

python
activation_statistics_interval: int | None = Field(default=20, gt=0)

Log per-layer activation health (dead / near-dead unit percentage, effective rank) every this many rollouts, measured on up to activation_statistics_observations observations of the last minibatch. None disables. A long run can lose most of a layer to dead ReLUs without any loss metric moving; this is the gauge that shows it happening.

Source

activation_statistics_observations ​

python
activation_statistics_observations: int = Field(default=1024, gt=0)

Observations used for each activation health measurement.

Source

bc_label_smoothing ​

python
bc_label_smoothing: float = Field(default=0.0, ge=0.0, lt=1.0)

Label smoothing for the behaviour-cloning cross-entropy. 0 disables it.

Cross-entropy against a one-hot target has no finite optimum, so BC drives the policy logits apart without bound for as long as it is applied. Measured on the cavern-15 run (2026-08-08): 104 of the seeded route's 316 steps reached a max action probability above 0.999 -- top-2 logit gaps of 7 to 24, so an alternative action was sampled with probability 1e-6 to 1e-11. At that point neither the policy gradient nor the entropy bonus can recover the action (both vanish as p -> 1), and the saturated steps survived 40M steps of BC-free training at entropy_coef=0.01 unchanged. PPO could no longer try alternatives at exactly the decisions that defined the route.

Smoothing gives the cross-entropy a finite optimum and so bounds the logit gap: with n actions the target becomes 1 - s + s/n for the demonstrated action and s/n for the rest, capping the gap at ln((1 - s + s/n) / (s/n)). At s=0.05 over 6 actions that is a gap of 4.75 (max probability 0.958, ~0.8% on each alternative) -- firm commitment that still leaves every decision samplable, which is what makes a permanent low-weight BC feature extractor safe to leave on indefinitely.

Source

clip_coef ​

python
clip_coef: float = Field(default=0.2, gt=0.0, le=1.0)

PPO clip coefficient (epsilon).

Source

entropy_coef ​

python
entropy_coef: NonNegativeScheduledFloat = Field(default_factory=lambda: Schedule.starting(0.01))

Entropy bonus coefficient. Accepts a bare number for a constant, or a Schedule.

Schedulable because the right value is not constant across a run: what matters is the entropy term's weight relative to the policy gradient, and that ratio moves as training progresses. Measured on cavern 15 (2026-08-09), both directions fail -- 0.005 flat: fine while advantages are strong, but at the backward curriculum's deep end the tail is ~300 actions, the win rate plateaus, Policy_Loss falls to -0.003, and 0.005 * 1.3 = 0.0065 becomes TWICE the policy gradient. Entropy is then the dominant objective, the policy diffuses toward uniform, and every episode becomes identical (Advantages_Std 0.33 -> 0.04). 0.001 flat: too weak to sustain exploration early -- entropy fell to 0.307 inside 650k steps, the policy went near-deterministic before finding anything worth committing to, and the first depth never got above a 2% win rate. A schedule lets one run explore early and commit late instead of trading one failure for the other.

Source

gae_lambda ​

python
gae_lambda: float = Field(default=0.95, ge=0.0, le=1.0)

GAE lambda parameter.

Source

max_kl ​

python
max_kl: float | None = None

Optional KL divergence threshold for early stopping. None disables.

Source

normalize_advantages ​

python
normalize_advantages: bool = True

Whether to normalise advantages.

Source

num_minibatches ​

python
num_minibatches: int = Field(default=4, gt=0)

Number of minibatches per epoch.

Source

potential_reward_shaping ​

python
potential_reward_shaping: PotentialRewardShapingConfiguration | None = None

Optional potential-based reward shaping from a trajectory potential utility. None disables it.

Source

ppo_epochs ​

python
ppo_epochs: int = Field(default=4, gt=0)

Number of epochs over each rollout.

Source

reset_parameter_curriculum ​

python
reset_parameter_curriculum: ResetParameterCurriculumConfiguration | None = None

Optional curriculum that varies reset parameters over training, one staggered stage at a time (generic over whatever start parameters a game exposes, for example start_cavern for Manic Miner). The chosen stage's parameters are sent per-episode as reset options. None => single configuration, no reset parameters sent (identical to before). Pair with store.archive_group_field (the field identifying the stage, for example "cavern") so the best-route archive is per stage, and an OakEmu build that supports unsafe reset parameters.

Source

rollout_length ​

python
rollout_length: int = Field(default=128, gt=0)

Number of steps per environment per rollout.

Source

value_clip ​

python
value_clip: float | None = None

Optional value function clipping range. None disables clipping.

Source

value_loss_coef ​

python
value_loss_coef: float = Field(default=0.5, gt=0.0)

Value loss coefficient.

Source