A learning module whose memory is a graph of what it has seen.
Mirrors tbp/monty/frameworks/models/graph_matching.py::GraphLM, the class
DisplacementGraphLM, FeatureGraphLM and EvidenceGraphLM are all built on
in Monty. It is the part they share: a buffer of what this episode has seen, a
GraphMemory of what past episodes left, and the bookkeeping that turns one
into the other — which graphs are possible, where on the object the sensor is,
and which objects a graph was built from.
What it does not decide is how a possible match is narrowed; that is
compute_possible_matches/4, which every subclass in Monty overrides. In Monty
it is a private method called on self, and the subclass's version replaces it.
A module here composes instead of inheriting (docs/monty-differences.md
§2.9), so the same function is public and takes the module it works on.
Why the fields are the module's
Monty's __init__ sets them and its methods read them, so they are the state
this module is: the buffer, the memory, the possible matches and their paths,
the object this episode is about, and what the module concluded. A class that
composes this one keeps the same shape, which is why LoggingUtils can read an
episode's statistics off either.
Where it deliberately differs
get_unique_pose_if_available/2raises. Monty raisesNotImplementedError: a graph module cannot say where the object is without a matching scheme, and a subclass supplies one. A silentnilwould read as "no pose yet", which is a different answer.- Errors are values, not exceptions. A percept set with no sensor
location does not raise out of a step:
add_displacements/2answers{:error, :no_sensor_module_location}, which the caller turns into a step that changed nothing.
Summary
Functions
Monty's _add_detailed_stats: nothing extra here; subclasses add their own.
Monty's _add_displacements: how far the sensor moved since the last percept.
Monty's add_lm_processing_to_buffer_stats.
Monty's _agent_moved_since_reset: whether the buffer has anything in it.
The step's percepts, in the buffer — twice, as Monty writes them.
Monty's wipe for a module the experiment does not supervise.
Monty's collect_stats_to_save: what this step is worth logging.
Monty's _compute_possible_matches: which observations are still possible.
What this module concluded, as the model reads it.
Monty's exploratory_step: buffer the observation, learn nothing yet.
Monty's fixme_update_ground_truth: which objects the graph was built from.
Monty's GraphLM(**config): a module that learns by being told, and does not
match.
Monty's get_all_known_object_ids: everything this module's memory holds.
Monty's get_graph: one object's graph in one input channel.
Monty's get_input_channels_in_graph.
Monty's get_object_scale: no scale detection, so the object is as stored.
Monty's get_output: only an evidence module has a hypothesis to publish.
Monty's get_possible_matches: the objects still in the running.
Monty's get_possible_paths, for a log or a plot.
Monty's get_unique_pose_if_available: the subclass's to answer.
The long-term memory this module holds.
Monty's init_from_ltm: the hypotheses the memory already supports.
Monty's load_state_dict: a memory, and the module state that follows it.
Monty's matching_step: what the observation makes possible, and what it does
not.
The same, narrowing with narrow rather than with this class's own scheme.
A module over an empty memory, with its own goal generator.
Whether the last buffered observation was one this module processed.
Monty's propose_goals: what the goal generator last asked for.
Monty's propose_region: the region its goals are in.
Monty's receive_votes: another module ruling an object out.
Monty's statistics for one step, written into the buffer.
Monty's reset_stm: forget the episode, keep what was learned.
Monty's _select_features_to_use: the features this module matches on.
Monty's send_out_vote: the objects this module knows this one is not.
Monty's set_detected_object: what a conclusion says the module was looking at.
Monty's set_experiment_mode: whether this episode trains or evaluates.
Monty's set_individual_ts: conclude, and record the pose if it was a match.
Records which object the module's sensor is on this step.
Monty's fixme_reset_ground_truth: the object this episode is about.
Sets the conclusion without deciding one.
Monty's state_dict: the memory, and which objects it was built from.
One step of the goal generator, pointed at the module it now belongs to.
Monty's _pass_target_info_to_model: the conclusion an experiment states.
Monty's update_ltm_from_stm: a training episode's observations, into memory.
Monty's update_terminal_condition: has this module concluded?
The same, asking pose_of whether it can narrow the pose.
Types
@type graph_id() :: String.t()
@type input_channel() :: String.t()
@type pose() :: [float()]
@type t() :: %Monty.Frameworks.Models.GraphMatching.GraphLM{ buffer: term(), detected_object: term(), detected_pose: term(), detected_rotation_r: term(), goals: term(), graph_id_to_target: term(), graph_memory: term(), gsg: term(), has_detailed_logger: term(), learning_module_id: term(), mode: term(), possible_matches: term(), possible_paths: term(), primary_target: term(), primary_target_rotation_quat: term(), stepwise_target_object: term(), stepwise_targets_list: term(), symmetry_evidence: term(), target_to_graph_id: term(), terminal_state: term(), tolerances: term() }
@type terminal_state() :: :match | :no_match | :time_out | :pose_time_out | nil
Functions
Monty's _add_detailed_stats: nothing extra here; subclasses add their own.
@spec add_displacements(t(), [Monty.Cmp.Message.t()]) :: {:ok, [Monty.Cmp.Message.t()]} | {:error, term()}
Monty's _add_displacements: how far the sensor moved since the last percept.
One displacement, from the mean of the sensor modules' locations, set on every percept — a learning module's own output moves with the sensor that produced it. The first step of an episode is a zero displacement, which is what makes the first observation a starting point rather than a move.
Monty's add_lm_processing_to_buffer_stats.
Monty's _agent_moved_since_reset: whether the buffer has anything in it.
A module that has buffered nothing has not moved, and its first observation is the place the displacements start from.
@spec buffer_percepts([Monty.Cmp.Message.t()], t()) :: t()
The step's percepts, in the buffer — twice, as Monty writes them.
Once as the step's own percepts, which is what a memory update reads later, and once per displaced percept as what this module matched against. A class that composes this one runs this after its own displacement is set.
Monty's wipe for a module the experiment does not supervise.
Such a module matched on its own during the episode, so what it concluded is a
guess — and a guess written into memory is worse than nothing learned.
detected_object is what gates the write, so clearing it is the whole
decision; the hypotheses go with it so that the episode's statistics do not
claim a match nobody made.
Monty's collect_stats_to_save: what this step is worth logging.
@spec compute_possible_matches( t(), Monty.Context.t(), [Monty.Cmp.Message.t()], boolean() ) :: {:error, :not_implemented}
Monty's _compute_possible_matches: which observations are still possible.
The base class has nothing to say — Monty raises NotImplementedError, and
every subclass in the tree answers this instead. The answer is an error rather
than a guess because a graph module with no matching scheme would narrow
nothing and look like one that had not decided yet.
@spec conclusion(t()) :: terminal_state()
What this module concluded, as the model reads it.
@spec exploratory_step([Monty.Cmp.Message.t()], Monty.Context.t(), t()) :: {:ok, t()}
Monty's exploratory_step: buffer the observation, learn nothing yet.
Monty's fixme_update_ground_truth: which objects the graph was built from.
Monty's GraphLM(**config): a module that learns by being told, and does not
match.
The base class is buildable in Monty — graph_1lm.yaml names it, and a
supervised pretraining run builds it and trains with it, because a supervised
episode is handed the object and pose it saw and never asks the module to
narrow anything. What the base class cannot do it does not do: _compute_possible_matches
raises NotImplementedError upstream, and compute_possible_matches/4 here
answers {:error, :not_implemented} — which is where a config that evaluates
with this module finds out, instead of at its second step.
Refusing to build it was this port saying more than Monty does: it turned a module that trains perfectly well into a configuration nobody could run.
Monty's get_all_known_object_ids: everything this module's memory holds.
@spec get_graph(t(), graph_id(), input_channel() | nil) :: term()
Monty's get_graph: one object's graph in one input channel.
@spec get_input_channels_in_graph(t(), graph_id()) :: [input_channel()]
Monty's get_input_channels_in_graph.
Monty's get_object_scale: no scale detection, so the object is as stored.
Monty's get_output: only an evidence module has a hypothesis to publish.
Monty's get_possible_matches: the objects still in the running.
Monty's get_possible_paths, for a log or a plot.
Monty's get_unique_pose_if_available: the subclass's to answer.
A graph module on its own has no matching scheme, so there is nothing to turn a possible path into a pose with. Two answers are honest: no path means no pose yet, and a path only a subclass can read means the class is incomplete — which is said out loud rather than answered with "not yet", because the model reads that as an episode that has not finished.
@spec graph_memory(t()) :: Monty.Frameworks.Models.GraphMatching.GraphMemory.t() | :none
The long-term memory this module holds.
Monty's init_from_ltm: the hypotheses the memory already supports.
GraphMemory.get_initial_hypotheses/1 answers every object it holds and no
paths — there is nothing to be possible about until an observation arrives.
Monty's load_state_dict: a memory, and the module state that follows it.
@spec matching_step([Monty.Cmp.Message.t()], Monty.Context.t(), t()) :: {:ok, t()}
Monty's matching_step: what the observation makes possible, and what it does
not.
A location-only step is nothing to match on and returns the module unchanged;
a step with features narrows the possible matches, concludes :no_match when
nothing is left, steps the goal generator and records the step's statistics.
@spec matching_step([Monty.Cmp.Message.t()], Monty.Context.t(), t(), (t(), Monty.Context.t(), [ Monty.Cmp.Message.t() ], boolean() -> {:ok, t()} | {:error, term()})) :: {:ok, t()}
The same, narrowing with narrow rather than with this class's own scheme.
A matching scheme is what DisplacementGraphLM, FeatureGraphLM and the
evidence module each supply, so it is a callback here for the same reason the
pose is: Monty inherits the method and overrides it, and a class that
composes must be able to name what it composes.
A module over an empty memory, with its own goal generator.
Options are Monty's __init__ arguments plus the two this port's modules all
take: :learning_module_id (Monty's default is "LM_0", and a config's id
replaces it), :tolerances (what a feature has to match within, by input
channel), :has_detailed_logger, and :gsg — a goal generator, built per
module because it holds a copy of the module it points at.
Whether the last buffered observation was one this module processed.
@spec propose_goals(t()) :: {[Monty.Cmp.Goal.t()], t()}
Monty's propose_goals: what the goal generator last asked for.
Only after a step the module processed, so a sensor that has moved without looking does not redirect the agent.
@spec propose_region(t()) :: {Monty.Cmp.AttentionRegion.t(), t()}
Monty's propose_region: the region its goals are in.
Monty's receive_votes: another module ruling an object out.
A module that has not been on the object this episode has no hypothesis to remove, and joining a vote against an object it never considered would be a conclusion it did not reach.
Monty's statistics for one step, written into the buffer.
extra is for a class that composes this one: the numbers only it knows —
the paths a displacement match came from, say — are merged over what a graph
module records by itself. Monty gets the same effect by asking the subclass
from inside collect_stats_to_save; a struct that is not this one cannot be
asked, so it says instead.
Monty's reset_stm: forget the episode, keep what was learned.
@spec select_features_to_use(t(), [Monty.Cmp.Message.t()]) :: %{ optional(input_channel()) => map() }
Monty's _select_features_to_use: the features this module matches on.
A feature is matchable when the module has a tolerance for it, and
pose_fully_defined is always carried because a pose that is not defined
changes how the pose it came with may be read.
Monty's send_out_vote: the objects this module knows this one is not.
A negative vote, so two modules on one object need not know the same objects: what one rules out, the other need not check.
@spec set_detected_object(t(), terminal_state()) :: t()
Monty's set_detected_object: what a conclusion says the module was looking at.
A conclusion that is not a recognition names a new object — new_object<n> —
which is what makes unsupervised learning possible: a training episode that
matched nothing still learned something, and it becomes the next graph.
@spec set_experiment_mode( t(), Monty.Frameworks.Models.AbstractMontyClasses.LearningModule.mode() ) :: t()
Monty's set_experiment_mode: whether this episode trains or evaluates.
@spec set_individual_ts(t(), terminal_state()) :: t()
Monty's set_individual_ts: conclude, and record the pose if it was a match.
The buffer keeps the first conclusion only — an episode ends once, and the first ending is the one that mattered.
Records which object the module's sensor is on this step.
Monty's fixme_reset_ground_truth: the object this episode is about.
Named "fixme" upstream because a learning module cannot work the object out;
the experiment knows it and passes it in. A nil target leaves the one this
module already had, which is what an episode of a run without resetting
boundaries relies on.
@spec set_terminal_state(t(), terminal_state()) :: t()
Sets the conclusion without deciding one.
Monty's _set_terminal_state, which a run without episode boundaries uses to
forget a conclusion it no longer stands by.
Monty's state_dict: the memory, and which objects it was built from.
Not the buffer: what an episode saw is over, and what a later one needs is what it left.
@spec step_goal_generator(t(), Monty.Context.t(), [Monty.Cmp.Message.t()]) :: t()
One step of the goal generator, pointed at the module it now belongs to.
A class that composes this one runs the generator itself, because its own fields are what the generator was told about.
Monty's _pass_target_info_to_model: the conclusion an experiment states.
A pretraining experiment does not match. It knows which object it put in front of the agent, so the episode's answer is that knowledge rather than something the module worked out — and the six numbers a matching episode would have concluded for itself (where on the model, where the sensor was, the turn, the scale) are the ones it is told instead.
location_on_model is the object's place in the world the first time this
experiment showed it, not this episode's: a later epoch may place the object
somewhere else, and the graph must stay where it was learned. That map is
Monty's first_epoch_object_location.
Monty's update_ltm_from_stm: a training episode's observations, into memory.
An evaluation episode learns nothing, and an episode that saw nothing has
nothing to write — which is what keeps detected_object the gate on both.
@spec update_terminal_condition(t()) :: {t(), terminal_state()}
Monty's update_terminal_condition: has this module concluded?
Three answers, in Monty's order: nothing possible is :no_match (and the
place the buffer last held is where it was); exactly one possible object with
a unique pose is :match; anything else is not a conclusion, and a previous
:match is withdrawn because the evidence no longer says it.
The same, asking pose_of whether it can narrow the pose.
A matching scheme is what turns a possible path into a pose, so the question
is a callback rather than a method: DisplacementGraphLM and the evidence
module each answer it, and GraphLM on its own cannot.