OakEmu-RL v0.1.0

Appearance

NStepBuffer ​

python
NStepBuffer(n_step: int, gamma: float) -> None

Buffer to handle n-step transitions and reward calculations.

Properly distinguishes between terminated (episode ended due to environment rules) and truncated (episode ended due to time limit) for correct n-step bootstrapping:

  • Terminated: Bootstrap should NOT happen (terminal state value = 0)
  • Truncated: Bootstrap SHOULD happen from next state's value

Initialize the NStepBuffer.

Parameters:

  • n_step (int) – Number of steps to consider.
  • gamma (float) – Discount factor.

Source

Methods ​

add ​

python
add(state: np.ndarray, action: int, reward: float, next_state: np.ndarray, terminated: bool, truncated: bool = False, reward_shaping_metadata: dict[str, int | float] | None = None) -> list[BufferedTransition]

Add a transition to the n-step buffer and return any completed n-step transitions.

Parameters:

  • state (ndarray) – Current state.
  • action (int) – Action taken.
  • reward (float) – Reward received.
  • next_state (ndarray) – Next state.
  • terminated (bool) – Whether the episode ended due to environment rules (goal, death, and similar values).
  • truncated (bool) – Whether the episode ended due to time limit or artificial cutoff.
  • reward_shaping_metadata (dict[str, int | float] | None) – Optional learner metadata associated with next_state.

Returns:

  • list[BufferedTransition] – The completed n-step transitions. The done flag will be True only for terminated episodes, ensuring truncated episodes still bootstrap from the next state.

Source

clear ​

python
clear() -> None

Clear the n-step buffer.

Source

Values ​

buffer ​

python
buffer: deque[tuple[np.ndarray, int, float, bool]] = deque(maxlen=n_step)

Source

gamma ​

python
gamma = gamma

Source

n_step ​

python
n_step = n_step

Source