What a sensor module makes of a depth image.
Mirrors tbp/monty/frameworks/utils/sensor_processing.py, which is a file of
functions rather than classes — the curvature estimation, the patch
normalisation, the surface normalisation. What is ported so far:
log_sign/1— ported: a signed logarithm, which is how a curvature becomes a feature a learning module can average;pixel_dist_to_center/3,weight_matrix/3,center_neighbors/3— ported: which pixels of a patch take part in a fit, and how much;surface_normal_total_least_squares/4— ported: the plane through those pixels, whose least-spread direction is the surface normal;principal_curvatures/4— ported: the quadric fitted to the same pixels, whose shape operator answers the two principal curvatures and their directions.arc_from_projection/3,directional_curvature/5,arc_length_corrected_displacement/6,unit_vector?/2andcoplanar?/4— ported: how far along a curved surface a movement really went, which is what a 2-D model is built from.
Not ported: surface_normal_naive/2 and
surface_normal_ordinary_least_squares/4 (Monty defaults to total least
squares), and surface_normal_sliding_window/….
And one function that file does not have but its module uses:
rgb2hsv/1— reproduced fromskimage.color.rgb2hsv, which is whatCameraSMcalls to turn the centre pixel into Monty'shsvfeature. It is here rather than in the simulator because a simulator that invented its own hue would train a learning module on a different feature from Monty's.
The two estimators take a point cloud in the shape the depth-to-points
transform leaves it: one row per pixel, [x, y, z, on_surface], in Monty's
order — so center_id is a row index and a patch's width is the square root of
its row count. Monty.Simulators.Synthetic.SensorModule still reads the
geometry a world reports rather than calling these; wiring them in is a step of
its own.
Summary
Functions
The true distance along a curved surface, from its straight-line projection.
A 2-D movement on a curved surface, corrected along each basis direction.
The points of the patch within a neighbourhood of its middle, as coordinates.
Whether a vector lies in the plane two others span.
The normal curvature in one direction, from the two principal curvatures.
Monty's FLAT_THRESHOLD, for a caller that wants to pass its own.
A signed logarithm: sign(x) * ln(|x| + 1).
Every pixel's distance from the middle of the patch, in pixels.
Point pair features: the four numbers that describe how two surface points sit
relative to each other — Monty's point_pair_features.
The two principal curvatures at the middle of a patch, and their directions.
One rgb pixel as hue, saturation and value, each in 0..1.
A value clipped into a range and square-rooted, keeping its sign.
The surface normal at the middle of a patch, and whether it is worth having.
Whether a vector has unit length, to tolerance — Monty's is_unit_vector.
A Gaussian weight per pixel of the patch, summing to one.
Functions
The true distance along a curved surface, from its straight-line projection.
A sensor that moves along a surface measures a chord in the tangent plane, and
a chord is shorter than the arc it subtends. On a circle of curvature k,
projection = sin(k · arc) / k, so the arc is asin(k · projection) / k —
Monty's arc_from_projection.
Two guards, both about not believing the circle where it cannot apply: a
product below threshold is flat enough that the correction is noise, and one
at or above 1 says the step reached the far side of the circle, which is not a
circle any more. Both return the projection unchanged.
iex> Monty.Frameworks.Utils.SensorProcessing.arc_from_projection(0.5, 1.0)
0.5235987755982989@spec arc_length_corrected_displacement( float(), float(), [number()], [number()], [number()], Monty.Geometry.SurfacePoseVectors.t() ) :: {float(), float()}
A 2-D movement on a curved surface, corrected along each basis direction.
Monty's arc_length_corrected_displacement: {du, dv} is what a flat surface
would have measured, and the answer is how far the sensor really travelled
along each direction, given the curvature in that direction. A 2-D model built
from the chords instead would shrink every step on a curved patch.
@spec center_neighbors([[float()]], non_neg_integer(), float()) :: [ Monty.Geometry.Rotation.vector() ]
The points of the patch within a neighbourhood of its middle, as coordinates.
They are answered relative to the middle pixel, and only the ones on the
object: a normal fitted across points that are not on it is a normal of
nothing. neighbor_patch_frac is how wide the neighbourhood is, as a fraction
of the patch's width.
Whether a vector lies in the plane two others span.
|v · (b1 × b2)| against tolerance: a direction with any component out of
the plane has a curvature that the two principal curvatures do not describe.
The normal curvature in one direction, from the two principal curvatures.
Euler's formula, k(θ) = k₁cos²θ + k₂sin²θ, where θ is the angle between the
direction and the first principal direction. It holds only for the principal
directions, which is why the two guards exist: directions that are not unit
vectors, or a movement that is not in their plane, have no answer here and are
refused rather than approximated.
A movement of zero length has no direction, so it curves by nothing.
@spec flat_threshold() :: float()
Monty's FLAT_THRESHOLD, for a caller that wants to pass its own.
A signed logarithm: sign(x) * ln(|x| + 1).
Monty's log_sign, which scales a curvature without losing which way the
surface bends — a convex surface has negative principal curvatures, and a
plain logarithm would lose that. The + 1 keeps values near zero from
running off to negative infinity.
iex> Monty.Frameworks.Utils.SensorProcessing.log_sign(0.0)
0.0
iex> Monty.Frameworks.Utils.SensorProcessing.log_sign(-1.0)
-0.6931471805599453@spec pixel_dist_to_center(pos_integer(), pos_integer(), non_neg_integer()) :: [ float() ]
Every pixel's distance from the middle of the patch, in pixels.
The patch is square, and its rows are one pixel each, so the width is the square root of the row count — Monty's own assumption. Distances are what a neighbourhood and a weight are decided by.
Point pair features: the four numbers that describe how two surface points sit
relative to each other — Monty's point_pair_features.
The line between them, its length, and the three angles it makes with the two
surface normals and they with each other: [length, angle(n_i, line), angle(n_j, line), angle(n_i, n_j)], in radians, with the line taken from i
to j.
A displacement module keeps these on every edge of a graph, so a step's displacement can be compared with a stored one however the object is turned.
iex> Monty.Frameworks.Utils.SensorProcessing.point_pair_features(
...> [0.0, 0.0, 0.0],
...> [1.0, 0.0, 0.0],
...> [1.0, 0.0, 0.0],
...> [0.0, 1.0, 0.0]
...> )
[1.0, 0.0, 1.5707963267948966, 1.5707963267948966]@spec principal_curvatures([[float()]], non_neg_integer(), [float()], keyword()) :: {float(), float(), [float()], [float()], boolean()}
The two principal curvatures at the middle of a patch, and their directions.
A quadric is fitted to the patch's points in the frame of the surface normal —
n = a u² + b v² + c u v + d u + e v + f — and the shape operator that falls
out of it answers the curvatures. Negative means convex, as Monty's are, and
the largest is answered first.
Options: :neighbor_patch_frac (the width of the Gaussian that weights the
fit), :weighted and :fit_intercept, as Monty's own are.
Answers {k1, k2, dir1, dir2, valid?}: zeros and zero vectors when the middle
pixel is not on the object or the fit has too little to work with.
One rgb pixel as hue, saturation and value, each in 0..1.
skimage.color.rgb2hsv, whose conventions this follows: hue is the angle
round the colour wheel divided by 360, so red is 0, green a third and blue two
thirds; saturation is how far the colour is from grey; value is the brightest
channel. A grey pixel has no hue to speak of and skimage answers 0 for it, as
this does.
What it also follows is img_as_float, which skimage applies before any of
that: a pixel whose channels are integers is a uint8 image and means
n / 255, while a pixel of floats is already in 0..1. Monty's CameraSM
hands rgb2hsv a camera image, so this is the branch that is used in a run —
and skipping it leaves the value channel on a 0..255 scale while hue and
saturation are already unit-scaled, which is a percept no learning module can
score against the graph it has.
iex> Monty.Frameworks.Utils.SensorProcessing.rgb2hsv([1.0, 0.0, 0.0])
[0.0, 1.0, 1.0]
iex> Monty.Frameworks.Utils.SensorProcessing.rgb2hsv([255, 0, 0])
[0.0, 1.0, 1.0]A value clipped into a range and square-rooted, keeping its sign.
scale_clip, which brings a Gaussian or mean curvature into a range a learning
module can use and softens the outliers: the square root of anything clipped to
[-clip, clip], signed as the input was.
@spec surface_normal_total_least_squares( [[float()]], non_neg_integer(), [float()], keyword() ) :: {[float()], boolean()}
The surface normal at the middle of a patch, and whether it is worth having.
Total least squares: the plane the neighbourhood's points lie in least badly, whose normal direction is the one with the least spread. The viewing direction only decides which way along that line the answer points — Monty aligns it with the camera, so a normal always faces the sensor that saw it.
Options: :neighbor_patch_frac, how wide the neighbourhood is as a fraction
of the patch's width.
Answers {[0.0, 0.0, 1.0], false} when the middle pixel is not on the object,
or when the neighbourhood is too small or too degenerate to fit.
Whether a vector has unit length, to tolerance — Monty's is_unit_vector.
@spec weight_matrix(pos_integer(), non_neg_integer(), keyword()) :: [float()]
A Gaussian weight per pixel of the patch, summing to one.
The fit is a local one: a pixel neighbor_patch_frac-th of the patch away
from the middle counts e^-2 as much as the pixel at the middle, so the
quadric follows the surface near the point the sensor is reporting rather than
the whole patch.