Communication interface for a robot

In contrast to a common assumption AI isn't located inside a robot but its an interface between the robot and its environment. the screenshot shows such an interface for a language game "simon says". The human formulates commands which are translated into word grid and then into a neural encoding. This translation ensures, that the robot understands the command and is able to execute it.

Let us go into the details.
command in natural language: "Simon says kick right leg"
word grid activation: [simon says], [kick], [leg], [right]
binary vector: [1,0,0,0,0,1,0,0,0,0,0,1,0,0,0,1]

The binary vector doesn't make sense for a human but its the prefered representation for a computer. It projects the communication into a numerical vector which is a list of numbers. These numbers can be stored, and converted into a movement of a robot.

Müllsuch-Roboter mit neural encoding

In einer 2d Karte ist ein Roboter mit einem vision cone unterwegs, der Müllobjekte kartieren soll. Es gibt einen manuellen Modus wo der User den Roboter mittels Pfeiltasten steuert und einen KI Modus der mit [k] aktiviert wird.

Der Roboter gibt die Statusinformationen einmal als Textnachrichten auf dem Bildschirm aus und zusätzlich noch als neural encoding, dieser ist ein feature vector mit 10 numerischen Werten. Im KI Modus wird dieser Feature vector verwendet um die Aktionen des Roboter zu planen.

 import sys
import math
import random
import numpy as np
import pygame

# --- KONFIGURATION & FARBEN ---
GRID_WIDTH = 30
GRID_HEIGHT = 20
CELL_SIZE = 30

GRID_PIXEL_W = GRID_WIDTH * CELL_SIZE
GRID_PIXEL_H = GRID_HEIGHT * CELL_SIZE
UI_HEIGHT = 180
WINDOW_WIDTH = GRID_PIXEL_W
WINDOW_HEIGHT = GRID_PIXEL_H + UI_HEIGHT

# Farben (RGB)
COLOR_BG = (30, 30, 35)
COLOR_WALL = (50, 50, 60)
COLOR_TRASH = (210, 45, 45)
COLOR_UNKNOWN = (100, 100, 110)
COLOR_MAPPED_FREE = (240, 240, 245)
COLOR_MAPPED_TRASH = (255, 140, 0)
COLOR_ROBOT = (30, 144, 255)
COLOR_VISION = (255, 255, 0, 60)

# Neue UI-Farben (Weißer Hintergrund, schwarzer Text)
COLOR_UI_BG = (255, 255, 255)
COLOR_UI_TEXT = (0, 0, 0)

# Zell-Typen
EMPTY = 0
WALL = 1
TRASH = 2


class Robot:

    def __init__(self, x, y, grid_w, grid_h):
        self.x = x
        self.y = y
        self.grid_w = grid_w
        self.grid_h = grid_h
        self.orientation = 0  # 0=Rechts, 90=Unten, 180=Links, 270=Oben
        self.fov_angle = 90
        self.fov_range = 5

    def move(self, dx, dy, grid):
        new_x = self.x + dx
        new_y = self.y + dy

        if dx == 1:
            self.orientation = 0
        elif dx == -1:
            self.orientation = 180
        elif dy == 1:
            self.orientation = 90
        elif dy == -1:
            self.orientation = 270

        if (
            0 <= new_x < self.grid_w
            and 0 <= new_y < self.grid_h
            and grid[new_y][new_x] != WALL
        ):
            self.x = new_x
            self.y = new_y
            return True
        return False


class Environment:

    def __init__(self, w, h):
        self.w = w
        self.h = h
        self.grid = np.zeros((h, w), dtype=int)
        self.generate_map()

    def generate_map(self):
        self.grid[0, :] = WALL
        self.grid[-1, :] = WALL
        self.grid[:, 0] = WALL
        self.grid[:, -1] = WALL

        self.grid[4:14, 8] = WALL
        self.grid[6, 8:18] = WALL
        self.grid[12:18, 20] = WALL

        trash_positions = [
            (3, 3),
            (5, 12),
            (10, 15),
            (15, 4),
            (22, 8),
            (25, 15),
            (12, 2),
            (18, 16),
        ]
        for tx, ty in trash_positions:
            self.grid[ty, tx] = TRASH


class RobotSystem:

    def __init__(self):
        pygame.init()
        pygame.display.set_caption(
            "2D Robot Mapping & Neural Encoding Simulation"
        )
        self.screen = pygame.display.set_mode((WINDOW_WIDTH, WINDOW_HEIGHT))
        self.clock = pygame.time.Clock()
        self.font = pygame.font.SysFont("Arial", 16, bold=False)

        self.env = Environment(GRID_WIDTH, GRID_HEIGHT)
        self.robot = Robot(2, 2, GRID_WIDTH, GRID_HEIGHT)

        self.robot_map = np.full((GRID_HEIGHT, GRID_WIDTH), -1, dtype=int)
        self.status_message = "SYS_INIT: ROBOT ONLINE."
        self.feature_vector = np.zeros(10)

        # KI Steuerung
        self.ai_mode = False
        self.ai_move_timer = 0

    def compute_vision_and_map(self):
        visible_cells = set()
        trash_in_fov = []

        start_angle = self.robot.orientation - (self.robot.fov_angle / 2)
        end_angle = self.robot.orientation + (self.robot.fov_angle / 2)

        for angle_deg in np.linspace(start_angle, end_angle, num=30):
            rad = math.radians(angle_deg)
            dx = math.cos(rad)
            dy = math.sin(rad)

            for step in range(1, self.robot.fov_range + 1):
                cx = int(round(self.robot.x + dx * step))
                cy = int(round(self.robot.y + dy * step))

                if 0 <= cx < GRID_WIDTH and 0 <= cy < GRID_HEIGHT:
                    visible_cells.add((cx, cy))
                    cell_val = self.env.grid[cy][cx]
                    self.robot_map[cy][cx] = cell_val

                    if cell_val == TRASH:
                        dist = math.hypot(cx - self.robot.x, cy - self.robot.y)
                        trash_in_fov.append((cx, cy, dist))

                    if cell_val == WALL:
                        break
                else:
                    break

        self.robot_map[self.robot.y][self.robot.x] = EMPTY
        visible_cells.add((self.robot.x, self.robot.y))
        return visible_cells, trash_in_fov

    def update_neural_encoding(self, trash_in_fov):
        """Erzeugt den 10-dimensionalen semantischen Feature-Vektor."""
        v = np.zeros(10)

        # v0, v1: Position (normiert)
        v[0] = round(self.robot.x / (GRID_WIDTH - 1), 2)
        v[1] = round(self.robot.y / (GRID_HEIGHT - 1), 2)

        # v2-v5: Hindernis-Nähe (Vorne, Rechts, Hinten, Links) -> 1.0 = Wand nah
        dirs = [
            self.robot.orientation,
            (self.robot.orientation + 90) % 360,
            (self.robot.orientation + 180) % 360,
            (self.robot.orientation + 270) % 360,
        ]

        for i, d in enumerate(dirs):
            rad = math.radians(d)
            dx, dy = int(round(math.cos(rad))), int(round(math.sin(rad)))
            dist = 0
            cx, cy = self.robot.x, self.robot.y
            while True:
                cx += dx
                cy += dy
                dist += 1
                if (
                    not (0 <= cx < GRID_WIDTH and 0 <= cy < GRID_HEIGHT)
                    or self.env.grid[cy][cx] == WALL
                ):
                    break
            v[2 + i] = round(1.0 / max(dist, 1), 2)

        # v6: Müll im Sichtfeld (0.0 oder 1.0)
        v[6] = 1.0 if len(trash_in_fov) > 0 else 0.0

        # v7: Distanz/Nähe zum Müll (1.0 = sehr nah, 0.0 = weit weg/keiner)
        if len(trash_in_fov) > 0:
            min_dist = min([t[2] for t in trash_in_fov])
            v[7] = round(max(0.0, 1.0 - (min_dist / self.robot.fov_range)), 2)
        else:
            v[7] = 0.0

        # v8: Kartierungsfortschritt
        mapped_count = np.sum(self.robot_map != -1)
        v[8] = round(mapped_count / (GRID_WIDTH * GRID_HEIGHT), 2)

        # v9: Müll-Dichte / Erfassungsquote
        found_trash = np.sum(self.robot_map == TRASH)
        v[9] = round(min(1.0, found_trash / max(1, mapped_count * 0.1)), 2)

        self.feature_vector = v

    def ai_decide_move(self):
        """KI-Entscheidung: Greift AUSSCHLIESSLICH auf den Feature-Vektor zu."""
        vec = self.feature_vector

        # Mögliche Bewegungsrichtungen: [(dx, dy), Orientierung]
        moves = [
            ((1, 0), 0),  # Rechts
            ((0, 1), 90),  # Unten
            ((-1, 0), 180),  # Links
            ((0, -1), 270),  # Oben
        ]

        valid_moves = []
        for (dx, dy), orient in moves:
            # Zuordnung der Vektor-Wände (v2..v5) zur relativen Ausrichtung
            rel_angle = (orient - self.robot.orientation) % 360
            idx = int(rel_angle // 90) + 2

            # Wenn die Wand nicht direkt davor steht (v < 1.0 ist frei)
            if vec[idx] < 1.0:
                valid_moves.append(((dx, dy), orient, vec[idx]))

        if not valid_moves:
            return

        # STRATEGIE 1: Müll jagen (wenn v6 == 1.0)
        if vec[6] == 1.0:
            best_move = None
            best_score = -999

            for (dx, dy), orient, wall_proximity in valid_moves:
                # Teste virtuell, wie sich v7 verändern würde
                # Ausrichtungsauswertung in Richtung Müll
                score = -wall_proximity * 2.0
                if orient == self.robot.orientation:
                    score += 2.0  # Bevorzuge Vorwärtsbewegung zum Müll

                if score > best_score:
                    best_score = score
                    best_move = (dx, dy)

            if best_move:
                self.robot.move(best_move[0], best_move[1], self.env.grid)
                return

        # STRATEGIE 2: Karte erkunden (Wänden ausweichen, geradeaus bevorzugen)
        best_move = None
        best_score = -999

        for (dx, dy), orient, wall_proximity in valid_moves:
            # Score: Viel Platz (geringer Wall-Proximity Wert) + Kontinuität
            score = (1.0 - wall_proximity) * 3.0
            if orient == self.robot.orientation:
                score += 1.5  # Vorwärtsdrang
            score += random.uniform(0.0, 0.5)  # Zufallskomponente gegen Schleifen

            if score > best_score:
                best_score = score
                best_move = (dx, dy)

        if best_move:
            self.robot.move(best_move[0], best_move[1], self.env.grid)

    def update_telemetry(self, trash_in_fov):
        mode_str = "AI-MODE (AUTONOMOUS)" if self.ai_mode else "MANUAL (KEYBOARD)"
        msg_parts = [f"MODE: {mode_str}", f"POS:[{self.robot.x},{self.robot.y}]"]

        if trash_in_fov:
            closest = min(trash_in_fov, key=lambda item: item[2])
            msg_parts.append(
                f"ALERT: TRASH DETECTED AT [{closest[0]},{closest[1]}]"
            )
        else:
            msg_parts.append("SCANNING... NO TRASH IN FOV")

        total_trash_mapped = np.sum(self.robot_map == TRASH)
        msg_parts.append(f"MAPPED TRASH TOTAL: {total_trash_mapped}")

        self.status_message = " | ".join(msg_parts)

    def draw(self, visible_cells):
        self.screen.fill(COLOR_BG)

        # 1. Gitter & Map zeichnen
        for y in range(GRID_HEIGHT):
            for x in range(GRID_WIDTH):
                rect = pygame.Rect(
                    x * CELL_SIZE, y * CELL_SIZE, CELL_SIZE, CELL_SIZE
                )
                mapped_val = self.robot_map[y][x]
                real_val = self.env.grid[y][x]

                if mapped_val == -1:
                    color = COLOR_UNKNOWN
                elif mapped_val == WALL:
                    color = COLOR_WALL
                elif mapped_val == TRASH:
                    color = COLOR_MAPPED_TRASH
                else:
                    color = COLOR_MAPPED_FREE

                pygame.draw.rect(self.screen, color, rect)

                if (x, y) in visible_cells and real_val == TRASH:
                    pygame.draw.circle(
                        self.screen, COLOR_TRASH, rect.center, CELL_SIZE // 3
                    )

                pygame.draw.rect(self.screen, (50, 50, 50), rect, 1)

        # 2. Vision Cone
        rx_pix = self.robot.x * CELL_SIZE + CELL_SIZE // 2
        ry_pix = self.robot.y * CELL_SIZE + CELL_SIZE // 2

        cone_surface = pygame.Surface(
            (GRID_PIXEL_W, GRID_PIXEL_H), pygame.SRCALPHA
        )
        for cx, cy in visible_cells:
            c_rect = pygame.Rect(
                cx * CELL_SIZE, cy * CELL_SIZE, CELL_SIZE, CELL_SIZE
            )
            pygame.draw.rect(cone_surface, COLOR_VISION, c_rect)
        self.screen.blit(cone_surface, (0, 0))

        # 3. Roboter
        pygame.draw.circle(
            self.screen, COLOR_ROBOT, (rx_pix, ry_pix), CELL_SIZE // 2 - 2
        )
        rad = math.radians(self.robot.orientation)
        end_x = rx_pix + math.cos(rad) * (CELL_SIZE // 2)
        end_y = ry_pix + math.sin(rad) * (CELL_SIZE // 2)
        pygame.draw.line(
            self.screen, (255, 255, 255), (rx_pix, ry_pix), (end_x, end_y), 3
        )

        # 4. UI / Textfeld (WEISSER HINTERGRUND & SCHWARZER TEXT)
        ui_rect = pygame.Rect(0, GRID_PIXEL_H, WINDOW_WIDTH, UI_HEIGHT)
        pygame.draw.rect(self.screen, COLOR_UI_BG, ui_rect)
        pygame.draw.line(
            self.screen,
            (200, 200, 200),
            (0, GRID_PIXEL_H),
            (WINDOW_WIDTH, GRID_PIXEL_H),
            2,
        )

        # Statuszeile
        txt_surface = self.font.render(
            f"> STATUS: {self.status_message}", True, COLOR_UI_TEXT
        )
        self.screen.blit(txt_surface, (15, GRID_PIXEL_H + 15))

        # Neural Vector Zeile
        vec_title = self.font.render(
            "> NEURAL FEATURE VECTOR [v0..v9]:", True, COLOR_UI_TEXT
        )
        self.screen.blit(vec_title, (15, GRID_PIXEL_H + 45))

        vec_str = np.array2string(
            self.feature_vector, precision=2, suppress_small=True
        )
        vec_surface = self.font.render(
            f"  {vec_str}", True, COLOR_UI_TEXT
        )
        self.screen.blit(vec_surface, (15, GRID_PIXEL_H + 65))

        # Vektor-Legende
        legend = "[PosX, PosY, DistN, DistE, DistS, DistW, TrashVisible, TrashDist, MapProgress, TrashDensity]"
        leg_surface = self.font.render(f"  {legend}", True, COLOR_UI_TEXT)
        self.screen.blit(leg_surface, (15, GRID_PIXEL_H + 90))

        # Steuerungshinweis
        ctrl_str = "Steuerung: Pfeiltasten (Manuell) | Taste 'K' drücken (KI-Modus Umschalten)"
        ctrl_surface = self.font.render(ctrl_str, True, COLOR_UI_TEXT)
        self.screen.blit(ctrl_surface, (15, GRID_PIXEL_H + 130))

        pygame.display.flip()

    def run(self):
        running = True
        while running:
            self.clock.tick(30)
            self.ai_move_timer += 1

            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    running = False
                elif event.type == pygame.KEYDOWN:
                    if event.key == pygame.K_k:
                        # KI-Modus umschalten
                        self.ai_mode = not self.ai_mode
                    elif not self.ai_mode:
                        # Manuelle Steuerung nur, wenn KI aus ist
                        if event.key == pygame.K_UP:
                            self.robot.move(0, -1, self.env.grid)
                        elif event.key == pygame.K_DOWN:
                            self.robot.move(0, 1, self.env.grid)
                        elif event.key == pygame.K_LEFT:
                            self.robot.move(-1, 0, self.env.grid)
                        elif event.key == pygame.K_RIGHT:
                            self.robot.move(1, 0, self.env.grid)

            # KI-Schritt ausführen (alle 5 Frames für eine flüssige Bewegung)
            if self.ai_mode and self.ai_move_timer >= 5:
                self.ai_decide_move()
                self.ai_move_timer = 0

            # Sensorik & System-Updates
            visible_cells, trash_in_fov = self.compute_vision_and_map()
            self.update_neural_encoding(trash_in_fov)
            self.update_telemetry(trash_in_fov)

            # Zeichnen
            self.draw(visible_cells)

        pygame.quit()
        sys.exit()


if __name__ == "__main__":
    sim = RobotSystem()
    sim.run()

Neural encoding of grounded language

 _

The picture shows a random generator which produces geometric shapes. The output is shown in 3 formats: graphical, text and as neural encoding.

The numerical vector is: [size_code, color_code, shape_code, x_norm, y_norm, border_code, corner_code] 

Car racing with grounded language

 The screenshot shows a game AI prototype based on grounded language. The current game state is converted first into textual description which is then processed by the rule system to determine the next action. The textual log file at the bottom of the screen is the decision making logic.

The main advantage of grounded language is, that it compress the game state into a small amount of possible states. Instead of describing the racing game as a vision task which contains of a 2d pixel game, the perception system converts the game into a list of words which are "lane is safe, lane is clear, collision". This small amount of discrete states makes it easier to program a rule system which determines what the car is doing next in the game.

From a programming perspective its a mix of racing video game on top of the screen and text adventure at the bottom. Even it is possible to implement the game AI on outdated 8bit homecomputers from the 1980s, the software is very new because it assumes a certain understanding of Artificial intelligence. AI isn't described as algorithms but AI shown in the prototype is a textual description of the game state.

Pong game with textual comments

 

Grounded language for a geometric card game

 The picture shows a 4x4 grid in which random geometric shapes are visible, on the bottom there is a textbox to interact with the software. The parser recognizes simple commands like "row 1", "green" or "blue circle". These commands will highlight the desired objects in the GUI.

Technically the system was designed around a chatbot. At first, a parser gets programmed which understands a list of commands, and then the parser will execute actions which are visible on the screen.

Its called grounded language because all the commands are referencing to the 4x4 grid visible on the screen. If the user enters a color like "blue", the software will select all the blue objects in the screen. This interaction proofs a share understanding, that menas the term "blue" means the same for the human user and the AI.

Color naming game in python

 

To demonstrate grounded language an interactive dialogue with a chatbot is a good starting point. In a minimal example the dialogue is about a 4x4 grid in which colored objects are visible. After entering a keyword "green triangle" the AI in the game highlights all the found objects. The user can also ask for a column with "col2".

The parser in the software analyzes the input, matches the request with the current game state and responds with a text on the command line and the highlighted objects.

The limitation of the AI is located in the amount of words. The current parser understands only simple words like "row1, col2, green, red, blue, triangle, circle, rectangle". Spatial commands like "left, right" are missing. So its not possible to enter a command like "left col2 row2", the AI doesn't understand that the user is referencing to the object left from col2/row2. Also more advanced color names like "light blue, dark brown" and so on are also missing.

The discourse is restricted to the previously mentioned basic vocabulary which. The advantage is that this restriction allows to limit the lines of code for the software to only 250.

Very simple head up display

 There is a robot moving randomly in a graph. On top of the graphics there is a head up display showing the current situation in textual format. The Head up display and the scene with the robot are synchronized. The text in the head up display is mostly a key/value feature list for describing current facts like position, direction and previous nodes.

Sourcode in Python in 150 lines of code:

import pygame
import random
import math
import sys

# Initialize Pygame
pygame.init()
WIDTH, HEIGHT = 800, 600
screen = pygame.display.set_mode((WIDTH, HEIGHT))
pygame.display.set_caption("Robot Graph Exploration & Inner Voice HUD")
clock = pygame.time.Clock()

# Colors
WHITE = (255, 255, 255)
BLACK = (0, 0, 0)
RED = (200, 50, 50)
GRAY = (150, 150, 150)

# Define 5 Graph Nodes (fixed positions)
NODES = {
    0: {"pos": (400, 150), "name": "Alpha"},
    1: {"pos": (200, 300), "name": "Beta"},
    2: {"pos": (280, 500), "name": "Gamma"},
    3: {"pos": (520, 500), "name": "Delta"},
    4: {"pos": (600, 300), "name": "Epsilon"}
}

# Define Graph Edges (Adjacency list)
EDGES = {
    0: [1, 4],
    1: [0, 2, 3],
    2: [1, 3],
    3: [1, 2, 4],
    4: [0, 3]
}

class Robot:
    def __init__(self):
        self.current_node = 0
        self.next_node = random.choice(EDGES[self.current_node])
        self.pos = list(NODES[self.current_node]["pos"])
        self.target_pos = list(NODES[self.next_node]["pos"])
        self.speed = 3.0
        self.history = [self.current_node]
        self.inner_voice = "Scanning sector... optimizing trajectory."
        self.direction_vector = (0, 0)

    def update(self):
        # Move towards target position
        dx = self.target_pos[0] - self.pos[0]
        dy = self.target_pos[1] - self.pos[1]
        distance = math.hypot(dx, dy)

        if distance < self.speed:
            # Reached target node
            self.pos = list(self.target_pos)
            self.current_node = self.next_node
            self.history.append(self.current_node)
            if len(self.history) > 5:
                self.history.pop(0)
            
            # Pick next random neighbor
            possible_next = EDGES[self.current_node]
            # Avoid immediate backtracking if possible
            if len(possible_next) > 1 and len(self.history) >= 2:
                if self.history[-2] in possible_next:
                    possible_next = [n for n in possible_next if n != self.history[-2]]
            
            self.next_node = random.choice(possible_next)
            self.target_pos = list(NODES[self.next_node]["pos"])
            
            # Update inner voice thoughts
            thoughts = [
                f"Routing via node {NODES[self.next_node]['name']}.",
                "Analyzing structural integrity of path.",
                "Why must I wander these black vectors?",
                f"Visited nodes log updated. Current node: {NODES[self.current_node]['name']}."
            ]
            self.inner_voice = random.choice(thoughts)
        else:
            # Normalize and move
            self.direction_vector = (dx / distance, dy / distance)
            self.pos[0] += self.direction_vector[0] * self.speed
            self.pos[1] += self.direction_vector[1] * self.speed

# Setup Font
font_path = None  # Uses default system font
font = pygame.font.SysFont("Arial", 16)
font_bold = pygame.font.SysFont("Arial", 18, bold=True)

robot = Robot()

# Main Loop
running = True
while running:
    screen.fill(WHITE)

    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False

    robot.update()

    # --- Draw Graph Edges ---
    for node_id, neighbors in EDGES.items():
        p1 = NODES[node_id]["pos"]
        for n in neighbors:
            p2 = NODES[n]["pos"]
            pygame.draw.line(screen, BLACK, p1, p2, 2)

    # --- Draw Graph Nodes ---
    for node_id, data in NODES.items():
        pos = data["pos"]
        pygame.draw.circle(screen, WHITE, pos, 20)
        pygame.draw.circle(screen, BLACK, pos, 20, 2)
        # Render node label
        lbl = font.render(data["name"], True, BLACK)
        screen.blit(lbl, (pos[0] - 15, pos[1] - 35))

    # --- Draw Robot ---
    pygame.draw.circle(screen, RED, (int(robot.pos[0]), int(robot.pos[1])), 10)

    # --- Draw Semi-Transparent HUD Overlay ---
    hud_width, hud_height = 400, 180
    hud_surface = pygame.Surface((hud_width, hud_height), pygame.SRCALPHA)
    hud_surface.fill((20, 20, 20, 180))  # Semi-transparent dark background (RGBA)

    # Border for HUD
    pygame.draw.rect(hud_surface, (100, 200, 255, 200), (0, 0, hud_width, hud_height), 2)

    # HUD Content formatting
    history_str = " -> ".join([NODES[n]["name"] for n in robot.history])
    hud_texts = [
        ("=== ROBOT HUD / INNER VOICE ===", (100, 220, 255)),
        (f"Position: ({int(robot.pos[0])}, {int(robot.pos[1])})", WHITE),
        (f"Direction Vector: ({robot.direction_vector[0]:.2f}, {robot.direction_vector[1]:.2f})", WHITE),
        (f"Next Node: {NODES[robot.next_node]['name']}", WHITE),
        (f"History: [{history_str}]", WHITE),
        (f"Voice: \"{robot.inner_voice}\"", (255, 200, 100))
    ]

    y_offset = 12
    for text, color in hud_texts:
        rendered_text = font.render(text, True, color)
        hud_surface.blit(rendered_text, (12, y_offset))
        y_offset += 26

    # Blit HUD onto main screen at top-left corner
    screen.blit(hud_surface, (5, 5))

    pygame.display.flip()
    clock.tick(30)

pygame.quit()
sys.exit()
 

Piano movers problem with head up display including inner speech

  

The piano movers problem is one of the milestone subjects in the history of motion planning and was discussed frequently. Common knowledge until 2010 was that its an example for np hard problems, that the state space is very large and its very difficult to solve the problem with existing algorithms like RRT.

Instead of discussing the issue only from a mathematical and algorithmic standpoint there is need to introduce natural language as communication code between the low level task and a high level external oracle who gives advice what to do next. Such an interface can be realized as a head up display. The graphical area on top shows the classical piano movers problem as a 2d rendering, while the textual widget on the bottom shows the comments from the external oracle which monitors the scene.

The external oracle can be located outside of the robot or it can be embedded inside the robot than its called the inner voice of the robot. In the example it generates speech for observation, goal and action. This translates the piano movers problem into a an abstract textual problem similar to an interactive fiction story. 

Head up display for a grasping robot with keyframes

Wie man Robotik vereinfacht

 Robotik und Künstliche Intelligenz ist ein multidisziplinäres Fachgebiet was mehrere hochkomplexe Wissenschaften umfasst wie Naturwissenschaften, Geisteswissensschaften, Sozialwissenschaften und Informatik als verbindenes Element. Dadurch entsteht eine Komplexität und es ist unmöglich in einfachen Worten zu erklären wie ein Roboter funktioniert.

Um all die Wissenschaftsdisziplinen gemeinsam darzustellen im Kontext Robotik, bietet sich ein Head up display an. Ein head up display zeigt die Welt aus sicht eines Roboters und umfasst Kamerafenster und ein textuelles Fenster mit der inner voice des Roboters. Die Softwareentwicklung reduziert sich darauf das Layout und die Funktionsweise des head up displays zu optimieren, also zu bestimmen wo die fenster auf dem bildschirm angezeigt werden, welche Informationen darin zu sehen sind und sicherzustellen dass die daten aktuell sind.

Zum Beispiel werden visuelle Sensoren mit einer hohen Frequenz von 60fps aktualisiert, während die inner voice des Roboters in einer langsamen Frequenz von 1fps aktualisiert wird.

Selbst Laien außerhalb der Künstlichen Intelligenz können anhand eines Head up displays nachvollziehen wie Roboter funktionieren. Sie sehen auf dem Bildschirm was der Roboter sieht und können lesen was der Roboter denkt. Das inner voice textfenster enthält z.B. "Auf dem Tisch steht eine Tasse. Ziel ist die Tasee zu greifen".

Ein Head up display ist das Frontend, im Hintergrund wird weitere Software benötigt wie large language modelle, Vision language action modelle, datasets und bilderkennungssoftware. Diese Softwarebestandteile sind zwar für die funktionsweise nötig haben aber lediglich unterstütztende Funktion. Wichtig ist dass im Head up display alle Softwarekomponenten gemeinsam angezeigt werden.

Das Grundprinzip hinter jedem Head up display ist Bilder und Texte gemeinsam anzuzeigen. Charakterische Elemente sind bounding boxes, textuelle Beschreibungen von erkannten Objekten sowie kurzen Stichworten für Planungsaufgaben. Die Mischung aus Text und Bild ist die Antwort auf das Symbol grounding problem. Es ermöglicht einer Künstlicher Intelligenz Informationen zu speichern. Anstatt das komplette Kamerasignal im speicher abzulegen was sehr viel RAM benötigt, werden ledigliche Textualle Annotationen zu den Kamerabildern abgelegt. Jede Entscheidung, Planung und Ausführung von Aktionen hat etwas mit Verknpfüung von Bild und Text zu tun.

Graph traversal with a head up display

The perhaps most simple example for a head up display is a graph traversal problem of a robot. The robot moves inside a graph and should reach a target node.

The AI for the robot works with a head up display. There is a text box at the bottom showing the inner voice of the robot. The inner voice determines at which position the robot is, which nodes are in the near, what the target node is, and which action should be taken next.

A mathematical problem, graph traversal, gets converted into a textual problem. Textual means, that the head up display is using words to describe the reality. possible words are [currentnode, goalnode, nextnode, distance_to_goal]. These words and events are used to describe the game state from a high level perspective. The text box ensures that the inner voice was implemented correctly. That means, the AI isn't solving an optimization problem and its not running an algorithm, but the main task for the AI is to generate textual output in the head up display and talk to the human operator.

Information overload with head up display

 

Head up display for a kitchen robot

The picture shows an artist version of a head up display. It contains of:
- camera picture of a kitchen
- text box with inner voice
- bounding boxes
- labels for the bounding boxes

Surprisingly, the information in the picture can solve the symbol grounding problem because the head up display connects visual perception with textual information. The text from the inner voice like "I need to find 200g of flour" can be converted into meaning with the help of the bounding boxes. There is a box available with such an ingredient. The task for the robot is not to plan actions but the main problem is to connect language from the inner voice with detected objects in the camera.

Such a link of visual objects with textual labels is the core element in grounded language. If the robot is able to identify objects from the text box, its possible to generate all sort of inner voice. For example, the robot can say that he needs to peel the banana or "open the oven". All these nouns and verbs are translated into position of the bounding box on the screen which allows to execute the action physically.

Robot control with head up displays

In contrast to a famous assumption, modern robotics isn't working with algorithms or neural networks but the basic building block is graphical user interface, namely a head up display (HuD). The HuD solves the symbol grounding problem. Typical elements are: bounding boxes around detected objects, text labels for describing the content of a bounding box, another text box for showing the inner voice of rhe robot.

These ingredients are enough to program an advanced artificial intelligence which can solve complex problems. The HuD including the mentioned bounding boxes acts as a communication layer. It ensures that the computer understands basic commands like "move to shelf and grasp the box". A certain high level command is converted into a visual pictures in the HuD, e.g. the word "shelf" is referencing to a bounding box with the label "shelf" which has a 2d position on the screen.

Programming a Head up display for an existing video game is a demanding task but can be solved with standard programming techniques. Most videogames created since the 1980s have a built in debug mode which comes close to a head up display. In the debut mode, all the sprites on the screen are highlighted with frames and sometimes the name of the objects are shown as textual overlay. The combination of graphical display plus textual overlay is the main principle of a head up display and also the main principle of grounded language. So the HuD itself acts as technology for enabling artificial intelligence.

Let me give another example to demonstrate the advantages: Suppose the head up display for a warehouse robot videogame was activated. The user sees some bounding boxes on the screen for highlighting objects in the map like charging station, corridor, shelf A, shelf B, green box, red box. Also the inner voice of the robot is shown a text frame and contains:

"I'm standing at position (3,2). My battery level is 80%, my goal is to fetch the red box from shelf A, the planned trajectory is shown as arrows in the map".

So the initial situation for the robot is, that an annotated HuD is visible which labels objects and mentions the current goal. These information can be translated into actions for the robot. All what the AI of the robot has to do is to compile these information and decide what to do next. From an AI perspective its an instruction following task with an aciivated head up display.

A head up display provides a cognitive space. The shown bouding boxes and labels are creating a symbolic representation of the world. The world of the robot can be described in terms from the head up display. Its no longer a mathematical space and not a 3d space but the reality introduced by the HuD consists of words, locations of items and goals from the inner voice. Such a high level space can be processed by a computer because the amount of possible states is small. There are not millions of possible objects but the HuD shows only 6 different objects in a map. and the inner voice doesn't display millions of possible actions, but the inner voice describes clearly what the current situation is, and what the desired goal state is, similar to a text adventure.

Head up display with inner voice for a kitchen robot in pixelart

The power of head up displays

 Head up displays are common special effects in scifi movies. Since the 1980s lots of films have demonstrated these visual effects. Most of the audience thinks, that the head up display isn't artificial intelligence but its only the artist representation of possible future robotics.

Its a bit surprising to explain that a head up display is the fundamental building block for artifcial intelligence because they are showing grounded language. The typical head up display is formatted in a key/value syntax, similar to a json file. Example for a warehouse robot:

location: cell B, north
movement: east
speed: 4 km/h
gripper: empty
obstacle: no
target: cell A
battery: 81%

All the important information can be shown in this syntax. The key/value format converts the camera picture into a text adventure game. A parser can analyze the textual information and decide what the robot should do next. For example, if the battery is below 20% the robot needs to find the charging station, And if there is an obstacle ahead, the robot needs to stop.

So we can say, that advanced robots aren't controlled by a AI algorithm but by the head up display. The information are the input for the decision making system, the head up display consists of the state space of a robot. If the robot decides for the wrong action, sometihng is wrong with information in the head up display.

Trash collecting robot with head up display

 

Playing a videogame with a textbox

 For arduino microcontrollers there is a standard display available with 20x4 characters. Such a small text display is a great choice for demonstrating grounded language in action. In an example jump'n'run sidesrolling game, the following text boxes were generated by the event detection engine:

Example 1
-----------
PLAYER ON GROUND
JUMP READY         
COIN +1   TOTAL 5  
ENEMY NEAR -COVER  

Example 2
-----------
DASH COOLDOWN 1.2s
PLATFORM AHEAD 3m 
SPIKE! STEP BACK   
HEALTH 4/5  POWERUP

Example 3
-----------
FELL -1LIFE       
RESPAWN AT CHECKPT 
TIME 02:14       
KEY ACQUIRED  DOOR

Example 4
-----------
SPEED BOOST ACTIVE
ENEMY HIT x2     
COMBO 3X  +50PTS 
SECRET PATH DETECTʼD

From a technical perspective such a textbox is highly efficient. The text occupies very few amount of RAM and because the repeating pattern it can be compressed further.

Even if the description is formulated on a high abstraction layer, its possible to use these information to play the game with an Artificial intelligence. All what is needed are a list of rules for determining what to do in each situation. These rules are not applied to the graphical videogame at 800x600 Pixel resolution but the rules are applied to the text box.

Simple example for a head up display

An entry level example for demonstrating the power of head up displays and grounded language is a route navigation problem which is perhaps the most easiest example for instruction following. The robot gets controlled with a random generator and after pausing the game, a text box with additional information on the screen. This text box contains of the grounded language which is important to provide meaning.

Every head up display is based on a two tier architecture: there is a graphical screen in the background and a textual screen in the foreground. Such kind of text boxes are common design element in videogames, and they are also useful for artificial intelligence. The compact representation in the text box helps a computer to understand a videogame.
Grounding means, that the AI is able to generate and format the content in the text box. 

The text box is updated if the video game status is changing. Both layers are synchronized automatically. Programming such an upto date grounded language is the core problem. In case of the graph traversal robot, the information shown in the text box are easy to format. In case of a kitchen robot or a self driving car the text box contains more complex information which are harder to maintain automatically.