Monty.Simulators.Synthetic.Scene (ExTbpMonty.Core v0.1.0)

The one object the stand-in world has, and what a sensor sees of it.

A sphere, because a sphere is the simplest surface whose depth, normal and curvature are all exact: the ray from a sensor either misses it, and the sensor is looking at nothing, or hits it, and the distance, the outward normal and the two principal curvatures follow in closed form. Anything a surface policy does — following curvature, avoiding places it has been, re-orienting against a normal — can be exercised against it, which is the whole point of having it.

The conventions are Monty's, from frameworks/utils/sensor_processing.principal_curvatures:

a convex surface has negative principal curvatures (Monty's own TODO says it would rather they were positive; the sign is reproduced);
the curvatures are ordered largest first, which for a sphere is a tie;
dir2 is the right-hand partner of dir1 about the normal, so dir1 × dir2 = normal.

What it is not: an object. A run against this scene says nothing about recognition; it says whether the body and the world agree about where the agent is and what it is looking at.

Summary

Types

observation()

What one ray saw: whether it hit the object, how far away it is, and the colour at that point.

t()

Functions

new(options \\ [])

A scene with :radius metres of sphere at :centre, coloured :rgb.

observe(scene, pose)

The middle pixel of a view, as a camera would answer it.

radius()

The radius a scene has when nobody says: five centimetres.

render(scene, map, options \\ [])

What a sensor at pose sees through a pinhole camera.

Types

observation()

@type observation() :: %{
  on_object: boolean(),
  depth: float() | nil,
  rgba: [float()] | nil
}

What one ray saw: whether it hit the object, how far away it is, and the colour at that point.

t()

@type t() :: %Monty.Simulators.Synthetic.Scene{
  centre: Monty.Geometry.Rotation.vector(),
  radius: float(),
  rgb: [float()]
}

Functions

new(options \\ [])

@spec new(keyword()) :: t()

A scene with :radius metres of sphere at :centre, coloured :rgb.

A centimetre-radius ball would be too small for a 2.5 cm standoff; Monty's objects are of the order of a few centimetres, so five is a fair stand-in. The colour is arbitrary — there is one object and it is all one colour — but it exists because Monty's sensor modules report hsv, and experiments weight it: a world with no colour would leave that feature empty.

observe(scene, pose)

@spec observe(t(), %{
  position: Monty.Geometry.Rotation.vector(),
  rotation: Monty.Geometry.Rotation.t()
}) ::
  observation()

The middle pixel of a view, as a camera would answer it.

Answers the depth, colour and semantic id of one pixel — what Monty.Frameworks.EnvironmentUtils.Transforms.DepthTo3DLocations unprojects, and what a policy asking "how far away is the thing in front of me?" reads. The middle pixel of a wide patch is not its axis — only a one-pixel camera or an odd count puts a pixel there — so this renders rather than assuming.

radius()

@spec radius() :: float()

The radius a scene has when nobody says: five centimetres.

render(scene, map, options \\ [])

@spec render(
  t(),
  %{
    position: Monty.Geometry.Rotation.vector(),
    rotation: Monty.Geometry.Rotation.t()
  },
  keyword()
) :: %{
  depth: [[float()]],
  rgba: [[[float()]]],
  semantic: [[float()]],
  centre: observation()
}

What a sensor at pose sees through a pinhole camera.

The camera is described the way Monty's own experiments describe it, so the studio's world is the experiment's camera rather than one chosen here: a :resolution of {height, width} pixels (a bare integer for a square patch), a :zoom, and a field of view :hfov in degrees. Monty's Habitat agents give a resolution and a zoom per sensor, and DepthTo3DLocations unprojects the image with fx = tan(hfov / 2) / zoom, so a world that rendered a different camera would make the transform disagree with it about which ray a pixel came from.

A sensor looks along its own −z, as Monty's do, so its pose is enough: no separate gaze has to be passed in and kept consistent with it. Depth is measured along that axis, as a depth sensor measures it — not along the ray — so an off-axis pixel's depth is the distance to the plane through the surface point, which is what the transform assumes when it unprojects.

One pixel is the default, and it is a real camera rather than a summary: depth, colour and semantic id come back as images, which is the shape Monty's own code indexes. With one pixel the grid has a single sample at the axis, so the field of view does not matter.

A ray that misses sees nothing: depth 0, which is what a depth image has where the camera saw nothing, and a semantic id of 0. That is a difference from Monty's worlds rather than a simplification — theirs are enclosed, so a camera pointed anywhere sees a floor, a wall or a ceiling at some distance, and a surface agent's search for a lost object relies on that (docs/monty-differences.md §1.7). Here there is nothing behind the object.