OakEmu-RL v0.1.0

Appearance

BackwardCurriculum ​

python
BackwardCurriculum(configuration: BackwardCurriculumConfiguration) -> None

A one-way receding front over a single archived route's segments (Salimans & Chen, done strictly).

Midpoint restarts are confined to the route's LAST route_length - front segments. The front moves backward only once the policy has mastered the current depth, so the tail is optimised before the prefix is ever touched -- dynamic programming by curriculum ordering. Because the policy carries a learned tail forward, improvements compound without needing archive banking.

Why this replaces the utility-blend frontier: that blend (win_proximity x (1 - local_progress)) was measured inert on cavern 15 -- Midpoint_Utility_Min/Max did not move between 7M and 98.7M steps while local_progress struggling-cell counts went backwards (86 -> 120 of 226). It spreads restarts across every depth at once, so PPO optimises a moving target: the tail keeps changing while it works on the prefix. A front is guaranteed to advance or to stall visibly at a known segment.

Restart states come directly from the archived route's path in the repository, which the archive pins for as long as the route is archived. An earlier version instead filtered the segment store's cell buckets for the route's own trajectory_id; that population is fixed at the route's segment count on the day it is ingested, is never replenished (every later episode gets a new id), and is ground away by max_per_key priority retention. It emptied within 100k steps on two consecutive cavern-15 runs, collapsing midpoint restarts to 0.1-0.4% of episodes with no error anywhere. Sourcing from the path removes that failure mode by construction rather than by tuning a filter.

Advancement is on a PLATEAU, not an absolute win rate. An absolute bar cannot work: the front recedes but the tail grows, so 80% means a 25-action tail at front 37 and a ~300-action tail at front 2, which needs ~99% per segment. The same number therefore becomes unreachable at some depth in every game, and the curriculum is forced through a wall it was never going to clear -- measured on cavern 15, where the collapse simply moved (front 2 on one run, front 24 on the next). S&C's premise is that each step back adds a small increment, so what is measured here is whether the increment still costs anything: the front holds while the win rate improves and moves when it stops.

Three rules keep it honest:

  • A floor, not a target. A plateau below minimum_advance_rate is a wall, not mastery, and the curriculum STOPS rather than advancing. Handing a policy that cannot solve the shorter problem a longer one is how every earlier run died. A visible stall at a known segment is the design goal.
  • A retreat rail. If a depth fails outright the front steps back toward the exit. One-way was right against the 2026-06-19 oscillation, but one-way plus forcing guarantees destruction: at 0% the returns are uniform and there is no gradient back. Firing only on catastrophic failure is a different thing from oscillating around a threshold, and max_advance_attempts bounds it.
  • No step-based patience. It existed to escape restart starvation, which direct path sourcing made impossible, and its only remaining effect was to force advances into walls.

Tail lengths at each depth are compared against the reference route's own remaining actions, which is the direct expression of "better than, or equal to, the trajectory" and the signal for whether a depth still has anything left to give.

Initialize the curriculum.

Parameters:

  • configuration (BackwardCurriculumConfiguration) – The curriculum's tuning parameters.

Source

Properties ​

action_granular ​

python
action_granular: bool

Whether rungs are single actions (per-action minted restarts) rather than marker segments.

Source

advances ​

python
advances: int

How many times the front has receded.

Source

at_start ​

python
at_start: bool

Whether the front has receded past the first segment, so episodes now run from the true start.

A restart resumes from the END of its segment, so a front of 0 still skips the route's opening segment. The front therefore recedes one step further, to -1, at which point the curriculum stops supplying restart states and the episode begins where the game does. That final phase is the natural place for reset jitter: the policy has a robust route by then, and jitter widens the set of opening conditions it holds under.

Source

best_tail_delta ​

python
best_tail_delta: int | None

Best tail length at the current front minus the reference route's own tail from there.

Negative beats the reference, zero matches it, None means no win yet at this depth.

Source

best_window_rate ​

python
best_window_rate: float | None

The best win rate achieved at the current depth, or None before the first full window.

Source

enabled ​

python
enabled: bool

Whether the curriculum is active.

Source

episodes_at_front ​

python
episodes_at_front: int

How many qualifying episodes have run at the current front.

Source

front ​

python
front: int | None

The current front segment index, -1 once it has receded to the true start, or None before a route has been seen.

Source

front_rung_success_rate ​

python
front_rung_success_rate: float | None

The success rate over the window for episodes that restarted AT the front rung, or None when empty.

Rising = the front is being learned (be patient); flat near zero while episodes pile up = a wall. Distinct from window_win_rate, which under Salimans & Chen averages the whole sampled band.

Source

over_budget_penalty ​

python
over_budget_penalty: float

The reward applied to an episode that exceeds its action budget.

Source

pulse_active ​

python
pulse_active: bool

Whether a pulse is currently teaching or fading at the front.

Source

pulse_weight ​

python
pulse_weight: float

The just-in-time pulse's current BC loss weight (0 when idle or disabled).

Source

pulses_fired ​

python
pulses_fired: int

The number of pulses fired across the run.

Source

retreats ​

python
retreats: int

How many times a depth failed outright and the front stepped back toward the exit.

Source

salimans_chen ​

python
salimans_chen: bool

Whether the front follows the Salimans & Chen rule (band ahead of the front, success >= rho advances).

Source

stalled ​

python
stalled: bool

Whether the curriculum has hit a wall and stopped advancing.

A visible stall at a known segment is the whole point of a strict front, and is far more useful than forcing through: it preserves a working policy and localises the difficulty exactly.

Source

starved ​

python
starved: int

How many times the front's filter yielded no candidate, so the episode fell back to a true start.

Source

total_tail_delta ​

python
total_tail_delta: int

Cumulative actions saved against the reference across all mastered depths.

Source

window_win_rate ​

python
window_win_rate: float | None

The win rate over the current front's outcome window, or None until the window is FULL.

A partial window is not a rate: after an advance or a resume the first few votes read as wild swings (one win in three is 33%) that mean nothing, so nothing is reported until window episodes have voted at this depth.

Source

Methods ​

action_budget ​

python
action_budget(segment_index: int) -> int | None

Return the action budget for a restart at segment_index, or None when uncapped.

Scaled by the reference route's remaining actions from that segment, so the budget is tight where the problem is small and loosens automatically as the front recedes and the policy is asked to play more of the route.

Parameters:

  • segment_index (int) – The segment the episode restarts from.

Returns:

  • int | None – The budget in actions, or None when the cap is disabled or the route is unknown.

Source

load_state_dict ​

python
load_state_dict(state: dict[str, Any]) -> None

Restore the curriculum from a checkpoint.

Parameters:

  • state (dict[str, Any]) – A previously serialised curriculum state.

Source

observe_route ​

python
observe_route(route_length: int, remaining_actions: list[int]) -> None

Register the archived route's shape, seeding the front on first sight.

The front starts initial_offset segments back from the exit so the first depth is a short, winnable tail rather than the terminal segment alone.

Parameters:

  • route_length (int) – The archived route's segment count.
  • remaining_actions (list[int]) – Per segment, the reference route's action count from the END of that segment (where a restart resumes) to the win. Used to score tails as better than, equal to, or worse than the reference.

Source

record_outcome ​

python
record_outcome(reference: TrajectorySliceReference, *, won: bool, tail_actions: int | None = None) -> None

Record one midpoint episode's outcome, moving the front when this depth has given what it can.

Episodes count only when they start within credit_window segments of the front -- normally the front itself. Restarts are drawn from a wider sample_window for the difficulty gradient, but a shallower, easier restart must never vote on the front's competence.

Once the window is full the win rate is tracked, but no judgement is made until the depth has had min_episodes_at_front episodes: a depth the policy has never seen reads 0% because it has not learned it yet. After that, three outcomes are possible -- the depth is failing outright and the front retreats; the rate is still improving and the front holds; or it has plateaued, so the depth has given what it can and the front either advances or declares a wall.

Parameters:

  • reference (TrajectorySliceReference) – The slice reference the episode restarted from.
  • won (bool) – Whether the episode ended in a win.
  • tail_actions (int | None) – Actions taken from the restart to the win, when it won.

Source

record_restart_served ​

python
record_restart_served() -> None

Note that a curriculum restart was served, ending any starvation streak.

Source

record_starved ​

python
record_starved() -> None

Record that the front admitted no sampleable candidate.

Starvation is silent by construction -- the store simply returns no sample and the episode becomes a true start ; it must be counted AND announced. On cavern 15 an unmetered starvation switched the entire midpoint mechanism off for 9.6 hours before anyone noticed; on 2026-08-13 a metered but unannounced one ran 43k episodes into a reset storm. Tiered so it cannot swamp the console: the first starvation warns, then every STARVATION_WARNING_INTERVAL at WARNING, escalating to ERROR once STARVATION_ERROR_THRESHOLD is passed. A single starvation is legitimate (the gap between a route displacement and its re-mint), so this never throws.

Source

sample_segment_index ​

python
sample_segment_index(rng: random.Random) -> int | None

Draw the route segment to restart from, uniformly over the front band.

Drawn from sample_window segments, which is a difficulty gradient: the shallower draws are short, winnable tails the policy can bootstrap from. This is deliberately WIDER than the band that votes on mastery -- see credit_window for why letting the easy draws vote is what made the front advance on competence it did not have.

Parameters:

  • rng (Random) – The selector's random source.

Returns:

  • int | None – A segment index, or None before a route has been seen.

Source

state_dict ​

python
state_dict() -> dict[str, Any]

Serialise the curriculum for checkpointing.

Source