One episode: a model stepped against an environment until it concludes.
The loop is MontyExperiment.run_episode_steps, which this port also has as
Monty.Frameworks.Experiments.MontyExperiment.run_episode_steps/2 — that is
the one a studio run goes through, because a studio run also has a caller who
may stop it and steps whose reports are recorded. This module is the same loop
for a caller that has neither, and it answers a different question: it reports
done? and timed_out? separately, so a test or a probe can say which ending
an episode got, rather than only that it ended.
What the loop does, without the experiment's data loading, epochs and loggers — those belong to whatever is running episodes, and this module is what is left when they are taken away:
- clear the model's short-term memory and tell it whether the episode trains or evaluates;
- step the environment and the model until the model says it is done, or until the step limit the experiment allows;
- hand a training episode to long-term memory.
The environment
An episode takes a function from the actions of the previous step to
{observations, proprioceptive_state} — Monty's Interface.step/1. That is
all the model needs of a world, so a recorded stream of observations, a
simulator, and a Python environment behind a port are all the same thing here.
Why the step limit is a policy
Monty's experiment does not stop an episode by counting steps; it asks a
recognition policy (MaxTotalSteps, MinEvalMaxTotalSteps,
ObjectRecognition, and AnyPolicy over them), and the policy also decides
whether undecided modules should be timed out and whether the episode should
start exploring. So this loop asks the same question, through the same
policies: :max_steps is sugar for MaxTotalSteps, and :policy takes any
of them.
Summary
Functions
Runs one episode of model_id against environment.
Types
@type mode() :: Monty.Frameworks.Models.AbstractMontyClasses.LearningModule.mode()
@type result() :: %{ steps: non_neg_integer(), done?: boolean(), timed_out?: boolean(), terminal_states: %{ optional(atom()) => Monty.Runtime.Step.State.terminal_state() }, infos: [map()] }
Functions
@spec run(atom() | String.t(), environment(), keyword()) :: result()
Runs one episode of model_id against environment.
Options:
:mode—:trainor:eval; training writes to memory at the end;:policy— a recognition policy, as{module, state}; defaults toMaxTotalStepsover:max_steps;:max_steps— how many steps the episode may take before it is timed out;:seed— the episode's seed, carried in the context.
A time-out is not a failure: it is the experiment giving up on an episode, and the model is told so, so its columns are not left open.