OakEmu-RL v0.1.0

Appearance

RestartPool ​

python
RestartPool(configuration: RestartPoolConfiguration, rng: random.Random | None = None) -> None

Bucketed collection of restart points harvested from played episodes.

A restart point is a marker the recording confirmed: the grace period has already dropped anything that led straight into a terminal, so every point here is one the policy survived. Points are keyed by the configured marker values and sampled over non-empty buckets -- uniformly, or weighted towards the buckets the policy is failing from -- which is what spreads practice over the game rather than over the states episodes happen to visit most.

Initialize the pool.

Parameters:

  • configuration (RestartPoolConfiguration) – The pool configuration.
  • rng (Random | None) – Optional random source; defaults to one seeded from configuration.random_seed.

Source

Properties ​

configuration ​

python
configuration: RestartPoolConfiguration

Get the pool's configuration.

Source

Methods ​

advance_rates ​

python
advance_rates() -> list[BucketAdvanceRate]

Get the advance rate of every bucket that has ever been restarted from, shallowest first.

Returns:

  • list[BucketAdvanceRate] – The per-bucket advance rates, ordered by key.

Source

choose ​

python
choose(step: int, env_id: int) -> RestartPoint | None

Decide whether the next episode restarts, and from where.

Parameters:

  • step (int) – The current global step, for the probability schedule.
  • env_id (int) – The environment the episode will run on, so a later timeout can be attributed back.

Returns:

  • RestartPoint | None – The restart point to start from, or None to start the episode normally.

Source

diagnostics ​

python
diagnostics() -> RestartPoolDiagnostics

Get the pool's current occupancy and lifetime counters.

Returns:

  • RestartPoolDiagnostics – The diagnostics snapshot.

Source

export_points ​

python
export_points(per_key: int = 0) -> list[RestartPoint]

Get the newest points of every bucket, for checkpointing.

Parameters:

  • per_key (int) – How many of each bucket's newest points to include; 0 for the whole bucket.

Returns:

  • list[RestartPoint] – The points, bucket by bucket in key order and oldest first within a bucket, so storing them back in that order reproduces each bucket's newest-wins ordering.

Source

import_points ​

python
import_points(points: Iterable[RestartPoint]) -> int

Store previously exported points, for example from a checkpoint.

Each point goes into the bucket its key names, subject to the usual per-bucket cap; the bounds in the configuration are not re-applied because the key was built under them when the point was harvested. Advance windows and timeout counts are untouched.

Parameters:

  • points (Iterable[RestartPoint]) – The points to store, oldest first within a bucket.

Returns:

  • int – How many were stored.

Source

record_episode ​

python
record_episode(env_id: int, episode: RestartEpisode | None, labels: tuple[str, ...] = (), *, won: bool = False) -> int

Note that an environment's episode finished, score the restart it came from, and harvest its points.

If the episode was restarted from the pool, it counts towards its bucket's advance rate: advanced when it won, or confirmed a marker in a deeper bucket than the one it started from. A win has to count explicitly because it ends the episode at once, before any marker at the finish can survive the grace period -- and from the last bucket, winning is the only advance there is. Markers are confirmed only after the grace period, so a band crossed in the last few steps before dying does not count; that bias is the same at every step of training, so the trend is unaffected.

Parameters:

  • env_id (int) – The environment whose episode ended.
  • episode (RestartEpisode | None) – The completed episode's trajectory, or None when there is no recording.
  • labels (tuple[str, ...]) – The captured-value labels, in marker value order, from the episode block.
  • won (bool) – Whether the episode ended in a win.

Returns:

  • int – The number of restart points stored.

Source

record_timeout ​

python
record_timeout(env_id: int) -> None

Note that an environment's episode was abandoned for running out of actions.

A restart point that repeatedly fails to produce an episode that ENDS is a point the game cannot play on from -- an emulator or game bug, not a hard section ; it is dropped once it has done so max_timeouts_per_point times. This needs no advance knowledge of what a broken state looks like, which is the point: filters only catch the breakage you predicted.

Only timeouts count, never losses. A genuinely valuable deep point loses constantly -- that is what practising the hard part of a game means -- and counting losses would evict the best points in the pool first.

Parameters:

  • env_id (int) – The environment whose episode was abandoned.

Source

selection_weights ​

python
selection_weights() -> list[BucketSelectionWeight]

Get how the next restart's bucket would be drawn, shallowest first.

Under uniform selection every bucket weighs 1. Under failure_rate the weights follow the advance rates, so this is the share of restarts each section is about to get.

Returns:

  • list[BucketSelectionWeight] – One entry per non-empty bucket, ordered by key.

Source