Monty.Frameworks.Models.SensorModules.ObservationProcessor (ExTbpMonty.Core v0.1.0)

Copy Markdown View Source

Turns one camera observation into a CMP message: a location, a surface, and the features a learning module matches on.

Mirrors tbp/monty/frameworks/models/sensor_modules.py::ObservationProcessor. It is the pure part of CameraSM — an observation in, a percept out, no state — which is why Monty has it as a class of its own too.

What it reads

The observation has been through the experiment's transforms, so it carries "semantic_3d" (one row per pixel: three coordinates and the on-surface mask), "cam_to_world", "rgba" and "depth". The middle pixel is half + width * half of the point cloud — Monty's own arithmetic for a square patch — and on_object is that row's mask.

Which features

:features is the list a config declares, and a feature not in it is not computed. That is the point of declaring it: feature_weights in a hidden layer and the tolerances in graph matching are written against exactly these names, and a percept carrying more would be one Monty's modules never see. :object_coverage, :min_depth and :mean_depth are taken over the pixels on the surface rather than over the whole patch.

One difference from Monty

Monty reads features["object_coverage"] when a surface sensor's middle pixel is off the object, and raises KeyError if the config did not ask for that feature. Here a missing coverage counts as zero (docs/monty-differences.md §2.10).

Summary

Functions

The processor a config describes.

The percept one observation makes.

Types

t()

@type t() :: %Monty.Frameworks.Models.SensorModules.ObservationProcessor{
  features: [String.t()],
  is_surface_sm: boolean(),
  pc1_is_pc2_threshold: float(),
  sensor_module_id: atom() | String.t(),
  surface_normal_method:
    Monty.Frameworks.Models.SensorModules.SurfaceNormalMethod.t(),
  weight_curvature: boolean()
}

Functions

new(options)

@spec new(keyword()) :: t()

The processor a config describes.

Options: :features and :sensor_module_id, then :pc1_is_pc2_threshold, :surface_normal_method, :weight_curvature and :is_surface_sm, as Monty's takes them.

process(processor, observation)

@spec process(t(), map()) :: {:ok, Monty.Cmp.Message.t()} | {:error, term()}

The percept one observation makes.

{:error, :no_point_cloud} when the observation never went through the depth-to-points transform: there is no surface to report, and inventing one would be worse than saying so.