Semantic camera for grounded language

 

An entry level demonstration for the symbol grounding problem is a semantic camera. The human operator points with a mouse on the screen and the algorithm shows the detected sematnic tags. A tag might be "green, circle" or "blue, rectangle". So the algorithm describes in words what can be seen on the screen.

Implementing such a camera in python is not very hard. The newly created class takes the game engine and the mouse position as input and generates a list of tags as output.

There are many applications for such a semantic camera. A robot which is equipped with such a device can navigate with ease in a maze, because the robot can communicate much better with a human operator. The robot generates words to describe its perception, and it can execute commands from a human also formulated in high level natural language.