# `Monty.Frameworks.Models.GraphMatching.GraphMemory`
[🔗](https://github.com/d-led/ex.tbp.monty/blob/v0.1.0/packages/ex_tbp_monty_core/lib/monty/frameworks/models/graph_matching/graph_memory.ex#L1)

A library of object models, with a librarian.

Mirrors `tbp/monty/frameworks/models/graph_matching.py::GraphMemory`. Memory
holds one model per `graph_id`, each of which holds one graph per input
channel — a mug seen through a patch sensor and through a view finder is one
object with two graphs, and they are matched separately.

Alongside the models it keeps, per graph and channel, the same features laid
out as rows (`feature_array`) with the names of those columns
(`feature_order`). That is a cache, not a second truth: matching compares one
observation against every node at once, and doing it column-wise is what makes
that affordable — Monty's own note calls the per-node loop one of the three
slowest things in the loop.

## What is a `nil` here

Monty pre-fills the feature array with NaN so a node that lacks a feature is
distinguishable from a node whose feature is zero. This port keeps that
distinction by keeping `nil` for a missing value rather than by filling a
buffer: a node with no stored value has no evidence (see
`...FeatureEvidence.Calculator`), which is exactly what the NaN was for.

## Where the layout comes from

Monty reads the column order from the first node's feature dictionary, which
in Python is insertion-ordered. An Elixir map has a stable but not
insertion-based order, so the order here is the map's — returned alongside the
array, never assumed by a caller, and therefore identical in effect.

# `graph_id`

```elixir
@type graph_id() :: String.t()
```

# `input_channel`

```elixir
@type input_channel() :: String.t()
```

# `t`

```elixir
@type t() :: %Monty.Frameworks.Models.GraphMatching.GraphMemory{
  feature_array: term(),
  feature_order: term(),
  feature_rows: term(),
  features_to_use: term(),
  graph_delta_thresholds: term(),
  k: term(),
  models_in_memory: term(),
  node_arrays: term(),
  object_model: term(),
  object_model_options: term(),
  rotation_features: term()
}
```

# `add_graph_to_memory`

```elixir
@spec add_graph_to_memory(t(), map(), graph_id()) :: t()
```

Store a model under `graph_id`, replacing anything with that id.

# `build_graph`

```elixir
@spec build_graph(t(), [[number()]], map(), graph_id(), input_channel()) :: t()
```

Build a graph for a channel that memory has nothing for yet.

No edges are added here: a graph grows observation by observation, and edges
are what `update_memory/7` adds once there is a model to compare against.

# `extend_graph`

```elixir
@spec extend_graph(
  t(),
  [[number()]],
  map(),
  graph_id(),
  input_channel(),
  [number()],
  [number()],
  Monty.Geometry.Rotation.t()
) :: t()
```

Add observations to a graph that already exists for this channel.

The new observations are in the body's frame; they are transformed into the
model's using the two reference points and the detected rotation.

# `flatten_rows`

```elixir
@spec flatten_rows([[number() | nil]]) :: {binary(), non_neg_integer()}
```

The rows as one flat binary of little-endian f64, with the width of a row.

The same bytes a native kernel reads: one row after another, with a NaN where
a node has no value — Monty pre-fills a missing feature with NaN, which is what
`nil` means here. `{<<>>, 0}` for no rows. Rows are written where they are
built, which is why a reader never rebuilds them: a step reads the same bytes
once per node, and the rows do not change while an episode is being matched.

# `get_all_models_in_memory`

```elixir
@spec get_all_models_in_memory(t()) :: %{optional(graph_id()) =&gt; map()}
```

Every model in memory, by object id.

# `get_feature_array`

```elixir
@spec get_feature_array(t(), graph_id(), input_channel()) :: [[float() | nil]]
```

The feature rows for a graph and channel, as `{rows, column_order}`.

# `get_feature_order`

```elixir
@spec get_feature_order(t(), graph_id(), input_channel()) :: [String.t()]
```

The feature names, in the order their columns appear.

# `get_feature_rows`

```elixir
@spec get_feature_rows(t(), graph_id(), input_channel()) ::
  {binary(), non_neg_integer()} | nil
```

The same rows as one flat binary of little-endian f64, with the width of a row.

`nil` is a NaN, which is the missing value Monty pre-fills with. `nil` when no
rows have been built for the graph yet, which is a graph that has never been
asked for its features.

# `get_features_at_node`

```elixir
@spec get_features_at_node(
  t(),
  graph_id(),
  input_channel(),
  non_neg_integer() | [non_neg_integer()],
  [String.t()] | nil
) :: %{optional(String.t()) =&gt; [float()] | [[float()]]}
```

The features stored at one node, or at each of several nodes.

`feature_keys` of `nil` uses the channel's configured features, which is how
the learning module reads a node; naming keys reads exactly those, which is
how a caller asks for one feature without knowing the rest.

# `get_graph`

```elixir
@spec get_graph(t(), graph_id(), input_channel() | :first | nil) ::
  %{optional(input_channel()) =&gt; Monty.Frameworks.Models.ObjectModel.model()}
  | Monty.Frameworks.Models.ObjectModel.model()
```

The models stored for `graph_id`: a map of input channel to model.

With an `input_channel` of `:first`, one channel is picked arbitrarily — it is
a placeholder for callers that need a graph rather than a particular channel,
and Monty makes the same arbitrary choice.

# `get_graph_node_ids`

```elixir
@spec get_graph_node_ids(t(), graph_id(), input_channel()) :: [non_neg_integer()]
```

The node ids of a graph: `0..num_nodes-1`.

# `get_initial_hypotheses`

```elixir
@spec get_initial_hypotheses(t()) :: {%{optional(graph_id()) =&gt; map()}, map()}
```

What a learning module starts from: every object in memory, and no paths yet.

At the first step nothing is ruled out, so every stored object is a candidate;
the hypotheses get narrowed by evidence from there.

# `get_input_channels_in_graph`

```elixir
@spec get_input_channels_in_graph(t(), graph_id()) :: [input_channel()]
```

The input channels a graph stores.

# `get_locations_in_graph`

```elixir
@spec get_locations_in_graph(t(), graph_id(), input_channel()) :: [[float()]]
```

The node locations of a graph.

# `get_memory_ids`

```elixir
@spec get_memory_ids(t()) :: [graph_id()]
```

The ids of every object in memory.

# `get_node_arrays`

```elixir
@spec get_node_arrays(t(), graph_id(), input_channel()) ::
  {binary(), binary(), binary()} | nil
```

Where every node is, how its frame sits, and whether that frame was defined —
as the bytes a kernel reads.

`{locations, poses, defined}`, each a binary: three f64 a node for its
position, six for its normal and first curvature direction, one for whether the
pose was fully defined. `nil` when a graph has no frames to read, which is a
graph the evidence kernel cannot be handed at all.

Written where the feature rows are written, and forgotten where they are
forgotten: all of it is what the graphs hold when an episode starts, and none of
it changes while one is being matched.

# `get_num_nodes_in_graph`

```elixir
@spec get_num_nodes_in_graph(t(), graph_id(), input_channel() | nil) ::
  non_neg_integer()
```

How many nodes a graph has — summed over its channels when none is named.

# `get_rotation_features_at_all_nodes`

```elixir
@spec get_rotation_features_at_all_nodes(t(), graph_id(), input_channel()) :: [
  [Monty.Frameworks.Utils.SpatialArithmetics.vector()]
]
```

The pose vectors stored at every node of a graph, one 3×3 set per node.

A burst samples rotations by aligning these against what the sensor sees, so
it wants the whole set in one place rather than a node at a time — Monty's
`get_rotation_features_at_all_nodes`, which reshapes the flattened pose
vectors back into the three directions they are.

Read from the cache the feature rows come with, and from the graph itself when
a graph holds only some of the features (a model built from a fixture).

# `get_rotation_features_at_nodes`

```elixir
@spec get_rotation_features_at_nodes(t(), graph_id(), input_channel(), [
  non_neg_integer()
]) :: [
  [Monty.Frameworks.Utils.SpatialArithmetics.vector()]
]
```

The pose vectors stored at the named nodes, one 3×3 set each.

The same rows as `get_rotation_features_at_all_nodes/3`, for a chosen few. A
burst samples rotations from a few hundred nodes of a graph that may hold
twenty thousand, and reading every node's pose vectors to keep a handful built
a row list and a tuple for the whole graph on every step, per channel.

Those rows never change while a graph is being matched — the graph is written
once per episode at most — so this answers from a cache built when the graph's
rows were: ten graphs of a ten-object library cost 9.5 ms a step to re-read,
and 0.2 ms to look up.

# `initialize_feature_arrays`

```elixir
@spec initialize_feature_arrays(t()) :: t()
```

Ensures every graph's feature rows and column names are present.

A pretrained library is read once and its rows travel with the memory, so the
learning module's `reset_stm/1` — which asks for the rows at the start of
*every* episode, Monty's own order — finds them already there. Only rows that
are missing are built, and rows go missing in exactly one way: a graph is
written (`add_graph_to_memory/3`, `build_graph/5`, `extend_graph/8`), which
drops the rows derived from it. A count is not that signal — a graph replaced
by one of the same size has rows that name the old nodes.

This is not only an optimisation. A seventy-seven-object library is a hundred
and fifty thousand nodes, and building their rows takes minutes; an experiment
of three rotations would spend those minutes two hundred and thirty times.

# `load_state_dict`

```elixir
@spec load_state_dict(t(), %{optional(graph_id()) =&gt; map()}) :: t()
```

Replace the contents of memory with a saved set of models.

This is how a pretrained memory arrives: every object is added as it was
stored, and nothing is rebuilt.

# `new`

```elixir
@spec new(keyword()) :: t()
```

An empty memory.

`:graph_delta_thresholds` decides when two observations are different enough
to both be stored; `:k` how many neighbours each new node links to.

`:object_model` is the kind of model memory stores — a graph of every
observation, or a grid thinned into one — and `:object_model_options` is what
that kind was configured with. Monty spells the same choice as a subclass;
here it is a module and a keyword list, so nothing has to know which model it
is holding.

# `remove_graph_from_memory`

```elixir
@spec remove_graph_from_memory(t(), graph_id()) :: t()
```

Forget an object.

# `size`

```elixir
@spec size(t()) :: non_neg_integer()
```

The number of objects in memory.

# `state_dict`

```elixir
@spec state_dict(t()) :: %{optional(graph_id()) =&gt; map()}
```

The models, as something that can be saved and reloaded.

# `update_memory`

```elixir
@spec update_memory(
  t(),
  [[number()]],
  %{optional(input_channel()) =&gt; map()},
  graph_id() | nil,
  [number()],
  [number()],
  Monty.Geometry.Rotation.t() | nil
) :: t()
```

Add an observation to the graph it belongs to, building it if it is new.

`graph_id` of `nil` means recognition did not settle on an object in time, and
a memory that learns from a guess is worse than one that learns nothing, so
nothing is stored.

How each channel's data is handled falls into one of three cases, and the third
is the one that is easy to get wrong:

  1. unknown object — the points are stored as they came, untransformed;
  2. known object, known channel — the graph is extended, which transforms the
     points into the model's own frame on the way in (`extend_graph/8`);
  3. known object, **unknown channel** — a channel first seen in a later
     episode. It has no graph to be extended, and building one from the
     untransformed points puts the new channel in the body's frame while every
     other channel of that object is in the model's: the two then never line up,
     which is the misalignment of Monty's PR #1280. So the points are
     transformed here, through the object's detected rotation, before the graph
     is built.

Whether the object is known is decided once, before the loop: a channel this
call adds must not make the object look like it already had a reference frame
for the channels after it (Monty's own note on the same bug).

---

*Consult [api-reference.md](api-reference.md) for complete listing*