DIKW pyramid for motion capture annotation

 

Graph traversal with a head up display

The perhaps most simple example for a head up display is a graph traversal problem of a robot. The robot moves inside a graph and should reach a target node.

The AI for the robot works with a head up display. There is a text box at the bottom showing the inner voice of the robot. The inner voice determines at which position the robot is, which nodes are in the near, what the target node is, and which action should be taken next.

A mathematical problem, graph traversal, gets converted into a textual problem. Textual means, that the head up display is using words to describe the reality. possible words are [currentnode, goalnode, nextnode, distance_to_goal]. These words and events are used to describe the game state from a high level perspective. The text box ensures that the inner voice was implemented correctly. That means, the AI isn't solving an optimization problem and its not running an algorithm, but the main task for the AI is to generate textual output in the head up display and talk to the human operator.

The paradigm shift in robotics around the year 2010

According to published research papers, the year 2010 was a turning point in robotics research. After this year, higher effort was put into human to robot interaction with natural language. Projects from this time span were:

- M.I.T. forklift by Stefanie Tellex
- Marco route instruction following by Matt MacMahon
- Word2vec algorithm by Tomas Mikolov

These projects were started from around 2008 until 2013. Many smaller projects also tried to use natural language for robot control.

This development was different from robotics research until 2010. The years before this year, there was a search for sophisticated algorithms available like training algorithms for neural networks, path planning algorithms and SLAM algorithms. The search for novel algorithms was working with the same principle how common computer science is working. The idea was, that AI gets implemented on a computer, computers need an algorithm and the consequence is to develop dedicated robotics algorithms for solving tasks.

The problem with the algorithm centric perspective until 2010 was, that all these developed techniques were not powerful enough. Even advanced probabilistic path planning algorithms implemented on a multi core CPU are not able to control a warehouse robot. The problem is the reality gap. The robot assumes a different reality than the real reality and the algorithm can't bridge the gap. It makes no sense to program more additional software modules or create a larger database for the robot, but the principle of autonomous robotics in general has to be questioned.

This paradigm shift took place in published academic literature around the year 2010. Research papers written after this date put a higher emphasis on teleoperation and grounded language for robot control. The idea is, that the source of wisdom is located outside of the robot as a human operator and the task is to get access to this knowledge by asking the human in natural language.

Even if the principle sounds inaccurate, it can be scaled up towards more complex scenarios. In the easiest case the human to robot interface works with a list of predefined commands, in a more advanced setup a neural network can parse the instructions. In contrast to figure out algorithms for autonomous robots, the new paradigm is to build language parser and see a robot as an open system.

The paradigm shift around the year 2010 allows to create artificial intelligence. Nearly all the former problems in robot control can be solved with the open system paradigm. Its only a detail problem how to create a high level user interface, so that the human operator can provide general statements like "tidy up the kitchen" and the robot is doing the full task by its own.

Roughly spoken, robotics programmed after the year 2010 are entirely teleoperated. There is a human operator in the background who gives instructions, or the former human operator instruction list was translated into a computer program who talks with the robot as large language model. in all the cases the robot interacts with a higher instance outiside of the robot and the artificial intelligence is located in the language interface.

Engine for grounded language

One possible explanation why the symbol grounding problem has emerged late in the history of computer science is because the theory is difficult to realize on a computer. Suppose natural language is important for robot control, the problem is create a language parser which is working for a concrete domain.

A possible command for a robot might be "Move until obstacle and then stop". Each of the words is stored as a string, but it remains unclear who to process the instruction into actions for a robot. The reason is, that the sentence is formulated in English but computers need a programming language as input. Even if every word is encoded as a number, it doesn't make sense to submit an array with numbers to the robot because its not possible to add or subtract the values in a meaningful way.

In general the problem is how to convert natural language into a computer program. Without solving this issue, the symbol grounding problem remains only a philosophical problem without any practical consequences.

The good news is, that the problem of programming a parser can be solved. Not with tools from computer science but by using techniques from linguistics, namely language games. Instead of treating language parsing as an algorithm problem, the idea is to invent around words a puzzle game. Typical language games are:

- Name guessing game. Player1 points to an object in the reality, and Player2 has to tell the name
- NPC quest game, a non player character in a role playing game formulates a quest like "bring me the sword from the wood" and the player has to fulfill the task
- bounding box game, player1 says a word like "table" and player2 has to draw a bounding box around this object

All these language games are located outside of computer science. They have nothing to do with algorithms, programming language nor existing robotics libraries, but they are games played with 2 human players.

The interesting situation is, that its possible to simulate the games with a computer. The software encodes the rules of the language game, determines the score for the human player and then the player can take action inside this game.

The problem is not how to program a certain parser, but the problem is how to formulate the game outside of a computer first. A well formulated game can be implemented in a software with ease. The programmer needs only the specification of the game including its rule, and then its possible to program the game with python. The only requirement is, that the computer works like the original language game. The programming workflow is identical to implement card games and board games on a computer.

Information overload with head up display

 

Die späte Entdeckung der Schrift im Kontext von Robotik

 Die Schrift ist eine sehr alte Erfindung der Menschheit. Die erste Bilderschrift, die Ägyptischen Hieroglyphen entstanden um 3200 v. Chr. Insgesamt ist Schrift und natürliche Sprache sehr detailiert erforscht. Es gibt umfassende Wörterbücher, Darstellungen welche die Geschichte der Sprache zeigen und Untersuchungen bezüglich Wortherkunft.

Vereinfacht gesagt sind Wörter Referenzsysteme zur Realität. Substantive stehen für Objekte wie "Himmel, Tisch, Apfel", Adjektive stehen für Tätigkeiten wie "Laufen, springen, geben" und Adjektive werden als Eigenschaftswörter verwendet wie "gelb, groß, schnell, feucht". Das Wissen bezüglich Wortarten und die Nennung von Beispielwörtern ist banal, allerdings nur für die Sprachwissenschaft selber. Im Bereich Computerwissenschaft und Mathematik wurde natürliche Sprache lange Zeit ignoriert. Es gab zwar Versuche chatbots zu programmieren, aber das war nur ein Teilbereich der Künstlichen Intelligenz.

Erst in jüngerer Zeit stellte sich heraus, dass natürliche Sprache womöglich das fehlende Puzzleteil darstellt mit der man künstliche Intelligenz inbesamt realisieren kann. Und zwar indem man Sprache als Technologie verwendet. Insbesondere dessen Eigenschaft auf die Realität zu verweisen macht es zum idealen Abstraktionsmechanismus. Es müssen keine neuen Sprachen erfunden werden sondern vorhandene Sprachen wie English, Deutsch usw. bieten bereits ein umfassendes Vokabular was von Robotern ähnlich wie Menschen verwendet werden kann. Alles was eine Maschine dafür benötigt ist eine Übersetzungstabelle von Bildern zu Sprache und in umgekehrter Richtung. Mit Hilfe dieser Bild zu Wort Tabelle kann man einem Roboter ein Kommando geben wie "Fahre zum Tisch". Und der Roboter übersetzt den Satz dann in Bilder und in Aktionen.

Bis ungefähr zum Jahr 2000 hat die KI Forschung nach Algorithmen gesucht mit deren Hilfe sich denkende Maschinen konstruieren lassen. Typische Algorithmen waren Lernverfahren für neuronale Netze, SLAM Algorithmen zur Selbstlokalisierung, A* Pfadplanungsalgorithmen oder Momdel predictive control Algorithmen. Die Annahme lautete jeweils dass mit diesen Algorithmen denkende Maschinen konstruiert werden könnten. Diese Annahme ist jedoch falsch. Es liegt zusätzlich der Verdacht nahe, dass es generell keine Algorithmen gibt, die Künstliche Intelligenz erzeugen, weil ein Algorithmus per se nicht mächtig genug ist um Roboter zu steuern. Was man stattdessen verwenden könnte wäre natürliche Sprache als zentrales Koordinierungsinstrument. Sprache ist ein Interface zwischen den Wortsymbolen einerseits und der Realität andererseits. Dadurch kann die komplexe Realität in wenige Wörter komprimiert werden.

Erst durch diese Realitätskompression ist es möglich den Handlungsraum für Roboter zu verkleinern. Der Roboter plant nicht länger in einem 3d Raum mit Trajektorien sondern der Roboter plant mit Hilfe von Verben und Substantiven in einem abstrakten Sprachraum. Seit der Erfindung von Word embeddings wie Word2vec mag dieser Ansatz selbstverständlich klingen aber bis zum Jahr 2000 war der Fokus auf Sprache eine Revolution.

Noch immer steht natürliche Sprache ein wenig außerhalb der klassischen Computerwissenschaft. Es hat nichts zu tun mit Elektrotechnik, Mathematik oder Algorithmen, sondern die Sprachwissenschaft hat ihren Ursprung in den Geisteswissenschaften. Nicht nur in der Dewey Dezimalklassifikation wie sie in Bibliotheken verwendet wird, sondern auch in der Gliederung von Universitäten sind Geistes- und Naturwissenschaften unversöhnliche Gegensätze die getrennt betrachtet werden.

Open systems for robotics

 Robotics in the past was organized with a closed system paradigm. A robot was described as a machine which consists of hardware, software and algorithms and the task for the programmer was to improve the internal mechanism of the robot. It was ignored that robots are communicating with the outside world. For example a robot might receives commmands by teleoperation and submits a status code to the operator. Such kind of interaction was mostly described as wrong path towards robotics because such a machine isn't autonomous anymore. There decision making isn't determined by the internal algorithm but from the outside which was seen as anti pattern in Artificial intelligence.

It takes decades until computer science has questioned the self created bias. Modern robotics is working as open system which means, that the robot gets information from sensors and from remote control. Also the robot interacts with human operators and is able to answer questions like "What object is visible in the camera?".

The transition from closed to open systems in robotics can be seen as an important innovation. In contrast to invent yet another path planning algorithm or program a robot control software in C/C++ the open system paradigm reformulates the goals of a robot system. It puts a higher importance on the robot's environment and allows the enviornment to take influence on the robot. There are many examples available in the history of robotics with this background, e.g. Braitenberg vehicle, kismet social robot and SHRDLU. These projects have demonstrated interactive robotics. There is always a robot and a human operator who interacts with the robot.

From a technical perspectives, interactive robotics is equal to teleoperation. Teleoperation was recognized by computer science as opposite to artificial intelligence, because the machine doesn't decide by itself but is guided by external human wisdom. So the maschine can't be called a robot anymore but has more in common with a RC Car.

The rejection of teleoperation makes sense on the first look. If a human operator is in charge to control the RC car, then no artificial intelligence is needed. Therefor it has nothing to do with thinking machines and is located outside of robotics. Only autonomous robots are intelligent robots.

With a modern perspective, Artificial intelligence isn't located inside of a robot but its the interface between a robot and its environment. Such an interface can become smart in the sense that the interface understands natural language.

Practical demonstration of the Total turing test

Stevan Harnad coined around the year 1990 the term "total turing test" which is a philosophical description of an instruction following task in robotics. What is missing is a practical demonstration of such a test for a real robot.

Such a demonstration can be realized in a simple video game modeled as a language game. An entry level example is a navigation task in a graph. There are 8 nodes connected with lines and the robot has to move along the graph to reach a certain goal node. Possible interaction with the robot would be:
- what is your position?
- what is your battery status?
- which nodes are reachable from current position?
- Move north
- move to node #3.
- what is the shortest path to reach node #6?

The robot is in charge to answer these requests in natural language and with motor actions. The problem is easy enough to get implemented as normal computer code without using advanced large language models or vision language action models. The human to robot interaction can be simplified by using a codebook. The amount of possible commands is given in the menu and the human can select one of the commands. IN other words, the Total Turing Test (TTT) is some sort of speaker to hearer interaction game played between a human and a robot.