Modules
The native index Monty.Frameworks.Utils.KdTree uses when it can.
The native kernels a matching step repeats once per node and once per hypothesis.
A PNG of pixels this port already has.
The processes the core needs, for a host to put in its own tree.
What attention does to a step — Monty's attention/attention_system.py.
Attention that learns where looking has paid off: Monty's
attention/attention_system.py::DefaultAttentionSystem.
Attention that has no opinion: Monty's
attention/attention_system.py::NoopAttentionSystem.
How a weight that was not refreshed this step loses its say: Monty's
attention/decay.py.
Move every weight a fixed distance toward zero each step: Monty's
attention/decay.py::LinearWeightDecay.
Decay that forgets nothing: Monty's attention/decay.py::NoopDecay.
The one method every decay has: Monty's attention/decay.py::VoxelGridWeightDecay.
Which goals attention leaves alone — Monty's attention/goal_filter.py.
Keep only the goals that land in a place attention wants looked at: Monty's
attention/goal_filter.py::HardGoalFilter.
Pass every goal through unchanged: Monty's
attention/goal_filter.py::NoopGoalFilter.
How two grids become one — Monty's attention/merge.py.
Merge by union, the second grid winning where they overlap: Monty's
attention/merge.py::Union.
The contract a merge satisfies — Monty's attention/merge.py::VoxelGridMerge.
What the attention system remembers about its own grids — Monty's
attention/telemetry.py.
The contract every attention telemetry satisfies — Monty's
attention/telemetry.py::AttentionSystemTelemetry.
Telemetry that keeps every grid it is told about — Monty's
attention/telemetry.py::DefaultAttentionSystemTelemetry.
Telemetry that records nothing: Monty's
attention/telemetry.py::NoopAttentionSystemTelemetry.
A sparse grid of per-voxel weights over 3-D space: Monty's
attention/voxel_grid.py::VoxelGrid.
How several weights about one voxel become one — Monty's
attention/weight_pooler.py.
The cortical messaging protocol: one payload for observed states, hypothesized states and goals.
Locations carrying an attention weight, proposed by a sensor or learning module to the attention system.
A target state on the cortical messaging protocol.
A message in Monty's cortical messaging protocol (CMP).
Evaluates the tiny arithmetic dialect Monty's configs use.
Resolves the OmegaConf interpolations Monty's configs are written in.
Reads Monty's own Hydra/OmegaConf config files.
The Elixir module a _target_ names.
What a module needs to know about the run it is part of.
Turning per-module terminal states into one system-level decision.
Satisfied once any count learning modules have recognised an object.
What a match criterion is.
Satisfied once every named learning module has recognised an object.
What counts as "the episode is over".
Done when any of its policies is, and it stops asking at the first yes.
Done at a hard step ceiling, and every lagging module timed out with it.
Done once the matching steps reach the limit for the current mode.
A wrapper that refuses to stop too early, and stops anyway at a ceiling.
Done once enough learning modules have concluded, whatever they concluded.
The original policy: the model says it is done, or it does not.
Done when a naive scan's spiral has completed.
Monty's own object-recognition policy: three ways to finish an episode.
The episode's counters, and the mode they are counting in.
What a policy decided: stop, time out the laggards, or start exploring.
Three limits in one policy: matching, exploring, and when to start exploring.
Something an agent can be told to do.
The action a map names.
An action as the map it crosses a boundary as.
Making actions from a random stream: the choices a policy draws.
The shared part of a sampler: which actions it can make, and how to ask.
Makes actions whose parameters never change.
Makes actions whose parameters are drawn uniformly from configured ranges.
Turn the agent downwards, by a number of degrees.
Turn the agent upwards, by a number of degrees.
Move the agent forward by a distance.
Move the agent sideways across the surface it is facing.
Move in the horizontal plane and turn to face where it went.
Move in the vertical plane and turn to face where it went.
Set how far the agent's body is pitched, in degrees.
Put the agent at an absolute place, facing an absolute way.
Set how far one sensor is pitched, in degrees.
Put one sensor at an absolute place, facing an absolute way.
Turn one sensor, relative to the agent it is attached to.
Set the agent body's yaw, in degrees.
Turn the agent to its left, by a number of degrees.
Turn the agent to its right, by a number of degrees.
Where the agent stands before an episode starts, and how it gets there.
Puts the agent somewhere it can see the object before the episode starts.
Builds a GetGoodView for whichever object an episode is about.
Raised when a single-object scene shows no object to position against.
How an experiment puts the agent somewhere sensible before the first step.
Where a procedure comes from, given the object an episode is about.
What a positioning procedure decided, and why it stopped.
Goals other learning modules and motor systems will try to achieve.
Behaviour for a learning module, mirroring
tbp/monty/frameworks/models/abstract_monty_classes.py::LearningModule.
Behaviour for a sensor module, mirroring
tbp/monty/frameworks/models/abstract_monty_classes.py::SensorModule.
Turns a buffer's contents into data a JSON encoder accepts.
Features at locations arriving into one learning module, and the statistics of an episode.
A learning module that recognises objects by the displacements between what it
senses: Monty's DisplacementGraphLM.
What a displacement module reads off a graph: Monty's DisplacementGraphMemory.
Spending steps on new hypotheses when the current ones stop learning.
Which hypotheses survive a step, and which new ones are worth trying.
Which of the observed input channels can be matched against the graph.
The slope of each hypothesis's evidence over a sliding window.
Which hypotheses a slope threshold keeps, as a mask and as indices.
How much of a match a node's stored features are — the calculator family.
Feature evidence as "how far inside the tolerance is this value".
What a feature-evidence calculator has to be able to do.
Turning one observation into evidence per node — the scorer family.
Feature evidence per node for one channel of one graph.
What a feature-evidence scorer has to be able to do.
Features are matched whenever the experiment weighted the channel at all.
The AllFeaturesForMatchingSelector class of
features_for_matching/all_selector.py.
Deciding which channels take part in feature matching — the selector family.
Features are matched when the experiment asked for them.
What a features-for-matching selector has to be able to do.
The memory an evidence-based learning module learns into.
Memory that stores grid models, which is what an evidence-based learning module learns into.
A set of hypotheses: their evidence, the location each implies, the pose each implies, and whether each can still be possible.
Where hypotheses move to, and how much evidence they earn there.
What a displacing step reports about itself.
How a learning module decides which hypotheses to keep and how they earn.
The straightforward updater: accumulate evidence, start channels, displace.
How much evidence a hypothesis needs before this step is worth spending on it.
The learning module that recognises objects by accumulating evidence.
An evidence update threshold setting that Monty does not define.
Monty's model for evidence-graph matching:
tbp/monty/frameworks/models/evidence_matching/model.py.
A learning module that recognises objects by the features it senses at
locations: Monty's FeatureGraphLM.
Monty's FeatureGraphMemory: the same library, read by feature.
Goal-state generators: what a learning module wants to look at next.
Look where the candidates disagree most, then look again.
Generate sub-goals until the goal it is driving towards is achieved.
A goal generator was asked about its learning module before one was provided.
Matching against a library of learned objects — the graph-matching family.
A learning module whose memory is a graph of what it has seen.
A library of object models, with a librarian.
What the graph-matching model does rather than owns: the vote round.
One graph's hypotheses, as the updater left them: what a chart draws.
What a learning module's hypotheses were worth, in the terms a paper uses.
Policies: how a body decides what to do with the space it is in.
A policy that takes one sampled action, whatever the step looked like.
A policy that reads the percept: stay on the object, or step back onto it.
The random walk InformedPolicy does, on its own.
A policy that teleports the agent to a goal, and takes it back if that lands somewhere useless.
The scaffold every motor policy is built on.
What one call to a motor policy decided: the actions, and how finished it is.
A policy that scans the object in a square spiral and nothing else.
Raised when a policy needs a goal and was given none.
Whether a policy is done, or still working on the goal in hand.
A policy that replays a written-down sequence of actions, for ever.
What the runtime asks a policy: given this step, what should move?
A policy that walks the surface of an object, feeling along it.
A surface policy that follows the principal curvature directions when it can.
Why a surface policy chose the action it chose.
Choosing which policy drives the body on a given step.
Three policies, chosen by who proposed the goal.
The motor system's question to a selector: given this step, what should move?
One policy, given the goal the rest of the run believes in most.
The body: turns a step and the goals it was given into actions.
What the rest of the runtime asks a motor system: this step, what should move?
What a surface policy wants remembered about the actions it just chose.
Where the body is: the motor system's view of the agents it moves.
Where one agent is, how it is turned, and where its sensors are.
What the environment says about every agent it contains.
Where one sensor is, and how it is turned.
The model an unsupervised-inference run starts: one that is never told an episode has ended.
An evidence learning module that is never told an episode has ended.
How a learned object is stored — the object-model family.
An object model as a graph: one node per distinct observation, edges to the nearest neighbours.
A model whose nodes are the busiest voxels of a grid.
An object does not fit in the space the model was given to represent it.
Raised when a goal is where the sensor already is.
A policy that turns to look at a goal: the agent yaws, the sensor pitches.
A view, reshaped so that every array is addressed by pixel.
Inhibition of return: do not propose the place you just looked at.
Every place the sensor has been, as kernels that fade.
One place the sensor has been, and how much it still says "not here again".
Makes the kernels a field is filled with, all configured alike.
Turning a view into a region worth attending to.
How a salience module turns a colour image into one region.
A wide view that proposes where to look next.
How a salience module decides where in a view is worth looking.
How a salience module decides where in a view is worth looking.
Every pixel is equally interesting.
VOCUS2: where in a view is worth looking, as one map of a number per pixel.
How much one colour channel stands out from its own surroundings.
How much a patch of surface stands out in depth.
The colour and blur operations VOCUS2's attention model is built from.
How a salience strategy turns the camera's colour image into the channels it reasons about.
How several feature maps become one salience map.
No limits at all: whatever sigmas the config names are the sigmas used.
How a finished salience map is brought into a usable range.
What an image has to be for an attention model to mean anything on it.
How much a patch looks like an edge, at several orientations.
Multi-scale images, and the comparisons VOCUS2 makes across them.
An image's scales, by octave and by blur.
How a pyramid becomes one image again.
How several pyramids become one.
The sigmas VOCUS2 was actually tried with, as fractions of the image.
The numbers a VOCUS2 strategy is configured from.
Averages the feature maps, each weighted by how much it is trusted.
Adds the feature maps up, each scaled by its weight.
What a salience module writes down about the views it judged.
A recorder that keeps every step of a salience module's reasoning.
A recorder that records nothing, and says so with an empty schema.
Where a salience module records what it judged, and why.
A sensor module: one camera observation in, one percept out.
The noise layer a config asks for with noise_params.
Sends a point to the learning modules only when something about it has changed enough to be worth learning.
Something that adds noise to a percept, or does not.
The noise layer that adds nothing.
Turns one camera observation into a CMP message: a location, a surface, and the features a learning module matches on.
The filter that passes everything on.
Something that decides whether a percept carries features or only a location.
Keeps the raw observations a sensor saw, and where it was when it saw them.
How a sensor module estimates a surface normal from a patch.
The sensor module that reads a 2-D model pressed onto a 3-D surface: Monty's
TwoDSensorModule.
Reading what came in over one channel.
Finding the edge in a patch of pixels, and saying which way it points.
One patch in, one edge out.
What one patch of an image says about the edge in it.
The three OpenCV calls Monty's edge detector is built on, written out.
A structure tensor at one point: the 2×2 symmetric matrix [[xx, xy], [xy, yy]].
The bookkeeping around matching: which k, which poses, which distances.
A k-nearest-neighbour index over points.
tbp/monty/frameworks/utils/logging_utils.py: what the loggers write with.
The Python file is the largest in Monty's framework, and most of it is pandas
shape-shifting and plotting (lm_stats_to_dataframe, get_rgba_frames_single_sm,
the buffer-size recipe total_size) that exists to feed a notebook. What is
here is the part a logger cannot do without, ported as it is needed rather than
as it appears
Grid and point-cloud helpers used while building an object model.
What a sensor module makes of a depth image.
Vector and rotation arithmetic on pose vectors.
An orthonormal (u, v, n) frame riding on a surface.
Small 3×3 matrix arithmetic and pose helpers.
A gaze goal expressed as a matrix whose first row points opposite the surface normal.
The linear algebra Monty's sensors ask numpy for.
A rotation matrix representing object orientation.
A rotation, as the quaternion wxyz Monty passes around.
Surface normal and the two principal curvature directions.
The constants Monty's arithmetic agrees on.
A snapshot of an object's internal state, as an opaque map: Monty's
memento.py.
Objects that can save and restore their internal state: Monty's
memento.py::Snapshotable.
The contract that keeps this rewrite module for module.
The module map itself: one entry per Python file, generated from the checkouts.
What this application runs in place of a Monty class it has not written.
The map as /modules reads it: one row per class, with the two files a reader
wants to open and the one fact the map alone cannot show.
One view of the world, as a sensor module receives it: Monty's
observations.py::SensorObservation.
NumPy's RandomState, as far as Monty draws from it.
The one random stream an episode draws from.
What the runtime asks about a set of percepts.
One phase of a step: ask every module the same kind of question, and wait.
One episode: a model stepped against an environment until it concludes.
One learning module, one process — a cortical column with a mailbox.
One running Monty model: its modules as supervised processes, and the step coordinator.
Puts the agent in front of the object before an episode's first step.
Names for module processes.
One sensor module, one process.
The timestep coordinator: Monty's MontyBase.step as a sequence of barriers.
What a step coordinator needs to know between steps, with the wiring
normalised once at start, and the model-level bookkeeping Monty keeps on
MontyBase.
A sensor module that is a pipeline of transforms rather than a class of its own.
What one sensor module step carries between its transforms.
One stage of a sensor module's pipeline.
What a transform knows about the step it is transforming.
What a world has to be: something a step can be taken against.
A simulator process on the other side of this port.
What this port says to a simulator, and what it expects back.
A sensor that replays a recording instead of sensing.
What a sensor saw, as a picture a page can show.
A stand-in world for developing the runtime, driven by the actions it is given.
Where the stand-in world puts the agent after an action.
A learning module that counts what it is shown, so the runtime can be exercised before evidence matching is ported.
A motor system that always takes one small step forward.
The one object the stand-in world has, and what a sensor sees of it.
A sensor module for a sensor whose config names a class this port has not written.