Monty.
Copy Markdown
View Source
Puts the agent somewhere it can see the object before the episode starts.
Mirrors
tbp/monty/frameworks/environments/positioning_procedures.py::GetGoodView, the
procedure every distant-agent evaluation episode starts with. Two things have
to be true before the first percept is worth anything: the object has to fill
enough of the view (or be close enough), and the sensor's centre pixel has to
be on it. So the procedure is a small state machine:
- in a scene with several objects, first turn towards the target, because the agent may start looking at the wrong thing;
- translate towards the object until it fills
good_view_percentageof the view or its nearest point is withingood_view_distance; - turn the sensor onto the object — no more than
max_orientation_attemptstimes, after which the procedure gives up and saystruncated, and the episode starts wherever the agent happens to be.
Why it is not a motor policy
It runs against the environment directly, before the motor system exists in the step, and it moves the camera by aiming at a pixel it found: the location to look at is the brightest point of a smoothed on-object mask, which needs no hypothesis and no memory.
The smoothing
Monty smooths the on-object mask with scipy.ndimage.gaussian_filter(image, size // 16, mode="constant") before taking its maximum, so that a single stray
on-object pixel does not attract the gaze. That is a different Gaussian from
the one the salience code uses (cv2.GaussianBlur, truncate 8σ+1,
reflect-101): scipy truncates at 4σ, pads with zeros, and takes sigma in
pixels. The rule below was measured against scipy's own answer to 2e-16
rather than recalled — see docs/monty-differences.md §1.15.
State
Monty's procedure mutates itself; this one is a value, so a step answers
{result, procedure} and the caller keeps what it gets back.
Summary
Functions
How far to look down and left to aim at a location relative to the sensor.
Where in the view to look, relative to the sensor: the middle of the target.
The distance Monty's factory uses when a config does not say otherwise.
The percentage Monty's factory uses when a config does not say otherwise.
The action that brings the object closer, or nil when it is close enough.
A procedure for the device :agent_id sees through, and the object to look at.
Whether the centre pixel of the view is on the target object.
The two actions that aim the sensor at the target.
Types
@type t() :: %Monty.Frameworks.Environments.PositioningProcedures.GetGoodView{ agent_id: atom() | String.t(), allow_translation: boolean(), executed_multiple_objects_orientation: boolean(), good_view_distance: float(), good_view_percentage: float(), max_orientation_attempts: integer(), multiple_objects_present: boolean(), num_orientation_attempts: integer(), sensor_id: atom() | String.t(), target_semantic_id: integer() }
Functions
How far to look down and left to aim at a location relative to the sensor.
The location is rotated back into the sensor's own frame first, so the answer is "relative to where this sensor already points". Looking up and turning right are negative amounts.
Where in the view to look, relative to the sensor: the middle of the target.
The on-object mask is smoothed with sigma rows // 16 before the brightest
point is taken — a sixteenth of the view is the scale a patch of object has to
be to matter — and the mask is multiplied back in so that the maximum cannot
land where smoothing spread brightness into the background. The answer is
relative to the sensor, because that is what the two rotations need.
The distance Monty's factory uses when a config does not say otherwise.
The percentage Monty's factory uses when a config does not say otherwise.
@spec move_close_enough(t(), map()) :: Monty.Frameworks.Actions.MoveForward.t() | nil
The action that brings the object closer, or nil when it is close enough.
Close enough means either that the object fills good_view_percentage of the
view, or that its nearest visible point is within good_view_distance — the
percentage for big objects, the distance for small ones, so that a small object
is not driven past and a big one is not entered.
In a scene with other objects it also refuses to step forward when another
object is already a quarter of the way inside good_view_distance, which is
how the agent avoids walking into the scenery on its way to the target.
A procedure for the device :agent_id sees through, and the object to look at.
Whether the centre pixel of the view is on the target object.
In a single-object scene every object pixel counts as the target, which is Monty's remapping: the scene has one object, so whatever is in view is it.
@spec orient_to_object(t(), map(), map()) :: [ Monty.Frameworks.Actions.LookDown.t() | Monty.Frameworks.Actions.TurnLeft.t() ]
The two actions that aim the sensor at the target.
The pixel to aim at is the maximum of the smoothed on-object mask, so the gaze goes to the middle of the visible surface rather than to its edge.