Monty.Frameworks.Models.SensorModules.CameraSM (ExTbpMonty.Core v0.1.0)

Copy Markdown View Source

A sensor module: one camera observation in, one percept out.

Mirrors tbp/monty/frameworks/models/sensor_modules.py::CameraSM, which is what every config in this revision of Monty targets for its patch sensor. It is four things in a row, and each of them is a module of its own here:

  • ObservationProcessor — turns the pixels into a location, a surface and the features the config asked for;
  • the noise layer — DefaultMessageNoise when the config gives noise_params, NoMessageNoise when it does not;
  • the percept filter — FeatureChangeFilter when the config gives delta_thresholds, PassthroughPerceptFilter when it does not;
  • SnapshotTelemetry — the raw observations, kept only when the config asks with save_raw_obs: true.

The random stream

Monty's step draws from ctx.rng, one stream per runtime that every module and every step shares. The noise layer draws from that same stream here, so a noised percept consumes the draws Monty's does and the run stays comparable. The sensor's transforms and the environment still derive a stream of their own from the context and the module's id (Monty.Context.rng_for/2) — a deviation from Monty's single object that docs/monty-differences.md §2.1 lists. A sensor with no noise_params draws nothing and is unaffected.

When there is no percept

Monty's processor raises if the observation never went through the depth-to-points transform. Here step/4 answers nil — no point cloud, no percept — and the runtime treats a silent sensor as it treats a silent step, which is what a view that sees nothing is (docs/monty-differences.md §2.11).

Summary

Types

t()

Functions

clear_buffers(state)

Drops what was seen, keeping the delta filter's memory.

from_config(config)

The sensor a config entry describes, or why it is not this class's.

new(sensor_module_id, options)

A sensor module for one sensor.

state_dict(sm)

Everything recorded so far: the raw observations, and every percept this sensor made, as Monty's state_dict reports it.

Types

t()

@type t() :: %Monty.Frameworks.Models.SensorModules.CameraSM{
  features: [String.t()],
  filter: {module(), struct()},
  filter_memory: term(),
  noise: {module(), struct()},
  pose: Monty.Frameworks.Models.MotorSystemState.SensorState.t() | nil,
  processed_observations: [Monty.Cmp.Message.t()],
  processor: Monty.Frameworks.Models.SensorModules.ObservationProcessor.t(),
  save_raw_obs: boolean(),
  sensor_module_id: atom() | String.t(),
  telemetry: Monty.Frameworks.Models.SensorModules.SnapshotTelemetry.t()
}

Functions

clear_buffers(state)

@spec clear_buffers(t()) :: t()

Drops what was seen, keeping the delta filter's memory.

Monty's _reset_modules_buffers: the observations of the object that has gone are never read again, but filter_memory is what decides whether the next observation is new, so clearing it would make the first patch on the next object look like a change of scene rather than a look at something else.

from_config(config)

@spec from_config(map()) :: {:ok, t()} | {:error, term()}

The sensor a config entry describes, or why it is not this class's.

Options are Monty's: features, then save_raw_obs, noise_params, delta_thresholds, is_surface_sm and pc1_is_pc2_threshold when the config gives them. sensor_module_id falls back to the config's sensor_id, which is how a patch sensor is named in the configs that carry both.

new(sensor_module_id, options)

@spec new(atom() | String.t(), keyword()) :: t()

A sensor module for one sensor.

Options: :features (required) and :save_raw_obs, :noise_params, :delta_thresholds, :is_surface_sm, :pc1_is_pc2_threshold.

state_dict(sm)

@spec state_dict(t()) :: map()

Everything recorded so far: the raw observations, and every percept this sensor made, as Monty's state_dict reports it.