Monty.Frameworks.Actions.ActionSamplers.ActionSampler (ExTbpMonty.Core v0.1.0)

Copy Markdown View Source

The shared part of a sampler: which actions it can make, and how to ask.

Mirrors tbp/monty/frameworks/actions/action_samplers.py::ActionSampler. Monty's base class keeps the list of action types and, in sample, picks one at random and calls its own sample_<name>; everything else lives in the samplers built on it, which is why this module holds only those two calls.

A sampler here is a struct with an :actions list of the action modules it can make — Monty.Frameworks.Actions.LookDown and friends — and the fourteen sample_<name>/3 functions that the pick dispatches to. The struct's own module is what gets called, so this works for any sampler that keeps that list, without a behaviour to satisfy.

The stream is threaded, not looked up

Monty writes self.action_sampler.sample(self.agent_id, ctx.rng) and mutates ctx.rng in place, so consecutive draws consume one advancing stream. Elixir has no mutable state, so sample/3 answers {action, rng} and the caller threads it. stream/3 is where that thread starts: a policy that has none yet takes one from the context.

Answering the stream rather than reseeding it from the step is not a matter of taste — a policy draws several times in a single step (SurfacePolicy samples a direction, then a distance), and a per-step stream would answer those draws with the same action every time. Neither is it a matter of cost: see scripts/bench_samplers.exs. See docs/monty-differences.md §2.5 for the deviation this leaves and §2.1 for the streams themselves.

Summary

Types

t()

Any struct with an :actions list, answered by a module with the sample_<name>/3 functions this module dispatches to.

Functions

One randomly chosen action, and the random stream after it.

The stream to draw from: the one being threaded, or a new one from ctx.

Types

t()

@type t() :: %{:actions => [module()], optional(atom()) => term()}

Any struct with an :actions list, answered by a module with the sample_<name>/3 functions this module dispatches to.

Functions

sample(sampler, agent_id, rng)

One randomly chosen action, and the random stream after it.

The choice is uniform over the configured actions — including over repeats, as numpy's rng.choice is, so a list naming an action twice makes it twice as likely.

stream(map, rng, ctx)

@spec stream(t(), :rand.state() | nil, Monty.Context.t()) :: :rand.state()

The stream to draw from: the one being threaded, or a new one from ctx.

A policy holds the stream it is threading; before its first draw it holds nil and derives one here, from the episode seed, its own module and the step. The sampler's module is what names the stream, so two kinds of sampler working in the same step do not draw from the same one, and two samplers of one kind deliberately do.