Head up display with grounded language

In science fiction movies of the 1980s, a head up display (HUD) is shown. This allows the audience to see what the robot might see. It was assumed, that such a head up display doesn't has a purpose but its only a special effect. An example for such a head up display is shown on top of this blog post. There are multiple food items on a table and some text boxes with a description.

The surprising situation is, that a head up display and especially the textual labels have a use case from a scientific perspective. They are demonstrating the symbol grounding problem. The robot is able to think and fulfill tasks by using the information from the HUD. A command like "grasp the banana" is converted into an action like "grasp the object on top left which is a fruit and has a weight of 120g". These detail information are extracted from the hud display, because the banana item was recognized in the picture.

A head up display is a photo from the ego perspective of a robot with annotated objects. There is sometimes a status box available which shows additional information. These text boxes ensures, that the robot understands a situation. It makes the teleoperation of the robot smoother.