# `Monty.Frameworks.Utils.SensorProcessing`
[🔗](https://github.com/d-led/ex.tbp.monty/blob/v0.1.0/packages/ex_tbp_monty_core/lib/monty/frameworks/utils/sensor_processing.ex#L1)

What a sensor module makes of a depth image.

Mirrors `tbp/monty/frameworks/utils/sensor_processing.py`, which is a file of
functions rather than classes — the curvature estimation, the patch
normalisation, the surface normalisation. What is ported so far:

  * `log_sign/1` — **ported**: a signed logarithm, which is how a curvature
    becomes a feature a learning module can average;
  * `pixel_dist_to_center/3`, `weight_matrix/3`, `center_neighbors/3` —
    **ported**: which pixels of a patch take part in a fit, and how much;
  * `surface_normal_total_least_squares/4` — **ported**: the plane through
    those pixels, whose least-spread direction is the surface normal;
  * `principal_curvatures/4` — **ported**: the quadric fitted to the same
    pixels, whose shape operator answers the two principal curvatures and
    their directions.

  * `arc_from_projection/3`, `directional_curvature/5`,
    `arc_length_corrected_displacement/6`, `unit_vector?/2` and
    `coplanar?/4` — **ported**: how far along a curved surface a movement
    really went, which is what a 2-D model is built from.

Not ported: `surface_normal_naive/2` and
`surface_normal_ordinary_least_squares/4` (Monty defaults to total least
squares), and `surface_normal_sliding_window/…`.

And one function that file does not have but its module uses:

  * `rgb2hsv/1` — reproduced from `skimage.color.rgb2hsv`, which is what
    `CameraSM` calls to turn the centre pixel into Monty's `hsv` feature. It
    is here rather than in the simulator because a simulator that invented its
    own hue would train a learning module on a different feature from Monty's.

The two estimators take a point cloud in the shape the depth-to-points
transform leaves it: one row per pixel, `[x, y, z, on_surface]`, in Monty's
order — so `center_id` is a row index and a patch's width is the square root of
its row count. `Monty.Simulators.Synthetic.SensorModule` still reads the
geometry a world reports rather than calling these; wiring them in is a step of
its own.

# `arc_from_projection`

```elixir
@spec arc_from_projection(number(), number(), number()) :: float()
```

The true distance along a curved surface, from its straight-line projection.

A sensor that moves along a surface measures a chord in the tangent plane, and
a chord is shorter than the arc it subtends. On a circle of curvature `k`,
`projection = sin(k · arc) / k`, so the arc is `asin(k · projection) / k` —
Monty's `arc_from_projection`.

Two guards, both about not believing the circle where it cannot apply: a
product below `threshold` is flat enough that the correction is noise, and one
at or above 1 says the step reached the far side of the circle, which is not a
circle any more. Both return the projection unchanged.

    iex> Monty.Frameworks.Utils.SensorProcessing.arc_from_projection(0.5, 1.0)
    0.5235987755982989

# `arc_length_corrected_displacement`

```elixir
@spec arc_length_corrected_displacement(
  float(),
  float(),
  [number()],
  [number()],
  [number()],
  Monty.Geometry.SurfacePoseVectors.t()
) :: {float(), float()}
```

A 2-D movement on a curved surface, corrected along each basis direction.

Monty's `arc_length_corrected_displacement`: `{du, dv}` is what a flat surface
would have measured, and the answer is how far the sensor really travelled
along each direction, given the curvature in that direction. A 2-D model built
from the chords instead would shrink every step on a curved patch.

# `center_neighbors`

```elixir
@spec center_neighbors([[float()]], non_neg_integer(), float()) :: [
  Monty.Geometry.Rotation.vector()
]
```

The points of the patch within a neighbourhood of its middle, as coordinates.

They are answered relative to the middle pixel, and only the ones on the
object: a normal fitted across points that are not on it is a normal of
nothing. `neighbor_patch_frac` is how wide the neighbourhood is, as a fraction
of the patch's width.

# `coplanar?`

```elixir
@spec coplanar?([number()], [number()], [number()], float()) :: boolean()
```

Whether a vector lies in the plane two others span.

`|v · (b1 × b2)|` against `tolerance`: a direction with any component out of
the plane has a curvature that the two principal curvatures do not describe.

# `directional_curvature`

```elixir
@spec directional_curvature([number()], float(), float(), [number()], [number()]) ::
  float()
```

The normal curvature in one direction, from the two principal curvatures.

Euler's formula, `k(θ) = k₁cos²θ + k₂sin²θ`, where `θ` is the angle between the
direction and the first principal direction. It holds only for the *principal*
directions, which is why the two guards exist: directions that are not unit
vectors, or a movement that is not in their plane, have no answer here and are
refused rather than approximated.

A movement of zero length has no direction, so it curves by nothing.

# `flat_threshold`

```elixir
@spec flat_threshold() :: float()
```

Monty's `FLAT_THRESHOLD`, for a caller that wants to pass its own.

# `log_sign`

```elixir
@spec log_sign(number() | [number()]) :: float() | [float()]
```

A signed logarithm: `sign(x) * ln(|x| + 1)`.

Monty's `log_sign`, which scales a curvature without losing which way the
surface bends — a convex surface has negative principal curvatures, and a
plain logarithm would lose that. The `+ 1` keeps values near zero from
running off to negative infinity.

    iex> Monty.Frameworks.Utils.SensorProcessing.log_sign(0.0)
    0.0

    iex> Monty.Frameworks.Utils.SensorProcessing.log_sign(-1.0)
    -0.6931471805599453

# `pixel_dist_to_center`

```elixir
@spec pixel_dist_to_center(pos_integer(), pos_integer(), non_neg_integer()) :: [
  float()
]
```

Every pixel's distance from the middle of the patch, in pixels.

The patch is square, and its rows are one pixel each, so the width is the
square root of the row count — Monty's own assumption. Distances are what a
neighbourhood and a weight are decided by.

# `point_pair_features`

```elixir
@spec point_pair_features([number()], [number()], [number()], [number()]) :: [float()]
```

Point pair features: the four numbers that describe how two surface points sit
relative to each other — Monty's `point_pair_features`.

The line between them, its length, and the three angles it makes with the two
surface normals and they with each other: `[length, angle(n_i, line),
angle(n_j, line), angle(n_i, n_j)]`, in radians, with the line taken from `i`
to `j`.

A displacement module keeps these on every edge of a graph, so a step's
displacement can be compared with a stored one however the object is turned.

    iex> Monty.Frameworks.Utils.SensorProcessing.point_pair_features(
    ...>   [0.0, 0.0, 0.0],
    ...>   [1.0, 0.0, 0.0],
    ...>   [1.0, 0.0, 0.0],
    ...>   [0.0, 1.0, 0.0]
    ...> )
    [1.0, 0.0, 1.5707963267948966, 1.5707963267948966]

# `principal_curvatures`

```elixir
@spec principal_curvatures([[float()]], non_neg_integer(), [float()], keyword()) ::
  {float(), float(), [float()], [float()], boolean()}
```

The two principal curvatures at the middle of a patch, and their directions.

A quadric is fitted to the patch's points in the frame of the surface normal —
`n = a u² + b v² + c u v + d u + e v + f` — and the shape operator that falls
out of it answers the curvatures. Negative means convex, as Monty's are, and
the largest is answered first.

Options: `:neighbor_patch_frac` (the width of the Gaussian that weights the
fit), `:weighted` and `:fit_intercept`, as Monty's own are.

Answers `{k1, k2, dir1, dir2, valid?}`: zeros and zero vectors when the middle
pixel is not on the object or the fit has too little to work with.

# `rgb2hsv`

```elixir
@spec rgb2hsv([number()]) :: [float()]
```

One `rgb` pixel as hue, saturation and value, each in `0..1`.

`skimage.color.rgb2hsv`, whose conventions this follows: hue is the angle
round the colour wheel divided by 360, so red is 0, green a third and blue two
thirds; saturation is how far the colour is from grey; value is the brightest
channel. A grey pixel has no hue to speak of and skimage answers 0 for it, as
this does.

What it also follows is `img_as_float`, which skimage applies before any of
that: a pixel whose channels are integers is a `uint8` image and means
`n / 255`, while a pixel of floats is already in `0..1`. Monty's `CameraSM`
hands `rgb2hsv` a camera image, so this is the branch that is used in a run —
and skipping it leaves the value channel on a 0..255 scale while hue and
saturation are already unit-scaled, which is a percept no learning module can
score against the graph it has.

    iex> Monty.Frameworks.Utils.SensorProcessing.rgb2hsv([1.0, 0.0, 0.0])
    [0.0, 1.0, 1.0]

    iex> Monty.Frameworks.Utils.SensorProcessing.rgb2hsv([255, 0, 0])
    [0.0, 1.0, 1.0]

# `scale_clip`

```elixir
@spec scale_clip(number(), number()) :: float()
```

A value clipped into a range and square-rooted, keeping its sign.

`scale_clip`, which brings a Gaussian or mean curvature into a range a learning
module can use and softens the outliers: the square root of anything clipped to
`[-clip, clip]`, signed as the input was.

# `surface_normal_total_least_squares`

```elixir
@spec surface_normal_total_least_squares(
  [[float()]],
  non_neg_integer(),
  [float()],
  keyword()
) ::
  {[float()], boolean()}
```

The surface normal at the middle of a patch, and whether it is worth having.

Total least squares: the plane the neighbourhood's points lie in least badly,
whose normal direction is the one with the least spread. The viewing direction
only decides which way along that line the answer points — Monty aligns it
with the camera, so a normal always faces the sensor that saw it.

Options: `:neighbor_patch_frac`, how wide the neighbourhood is as a fraction
of the patch's width.

Answers `{[0.0, 0.0, 1.0], false}` when the middle pixel is not on the object,
or when the neighbourhood is too small or too degenerate to fit.

# `unit_vector?`

```elixir
@spec unit_vector?([number()], float()) :: boolean()
```

Whether a vector has unit length, to `tolerance` — Monty's `is_unit_vector`.

# `weight_matrix`

```elixir
@spec weight_matrix(pos_integer(), non_neg_integer(), keyword()) :: [float()]
```

A Gaussian weight per pixel of the patch, summing to one.

The fit is a local one: a pixel `neighbor_patch_frac`-th of the patch away
from the middle counts `e^-2` as much as the pixel at the middle, so the
quadric follows the surface near the point the sensor is reporting rather than
the whole patch.

---

*Consult [api-reference.md](api-reference.md) for complete listing*