One view of the world, as a sensor module receives it: Monty's observations.py::SensorObservation.

Monty spells the shape as a TypedDict with every field optional, because a sensor sends what it can and a consumer reads the keys it understands. Here it is a map with string keys — the same shape the depth-to-3D transform writes and every consumer in this application reads — plus the names for the keys a consumer cannot do without:

• "rgba" — the patch as rows of [r, g, b, a], 0..255.
• "depth" — one distance per pixel, in metres.
• "semantic" — one object id per pixel, 0 for background.
• "semantic_3d" — one row per pixel, [x, y, z, semantic_id], which is what a sensor module actually reads: the depth image is gone by then.
• "sensor_frame_data" — the same points in the sensor's own frame.
• "cam_to_world" — the 4x4 transform from camera to world coordinates.
• "pixel_loc" — where in the image the patch was taken, when it was cropped.
• "raw" — the untouched image, for logging.

Why the accessors raise

A missing "semantic_3d" is not an empty observation, it is an observation that never went through the depth-to-3D transform — and a filter that silently read nil there would report a surface at the origin. The accessors name the key they need, so the failure says which transform did not run.