The technology of Stanley self driving car in 2005

In addition to the previous blog post which describes the Darpa challenge in 2007, the technology for the competition held in 2005 should be described next. All the information are taken from the document [1].

At first it should mentioned, that the winning car Stanley was the most advanced robot car at this time. It was designed by a team of experienced engineers with a university background and has proven in an official competition its superior over alternative concepts. What makes the situation interesting from today's perspective (the year 2026) is, that the entire technology stack can be called outdated. Stanley including all the mentioned hardware and software has only a value for a museum and is very different from the technology used today.

Even for the timeline of computer science, which is known for its rapid development, such kind of fast aging process is a surprise. The usual assumption is, that at least some of the technology is valid in a modified version for later robotics projects. But this was not the case. It seems, that Artificial intelligence since 2005 has drastically reinvented itself. But let us take a closer look into the year 2005.

The physical hardware of the robot was a Volkswagen Touareg R5 TDI, with a diesel engine.[1] page 2. The engine was powered by gas which comes from a gas station. The track was provided in a waypoint text file, in the RDDF format [1] page 3. The vehicle was equipped with multiple sensors like SICK laser, GPS camera, compass [1] 4. The CPU was located in the trunk of the car and was an Intel Pentium M CPU which was used also for laptops. It was running the Linux operating system on six different machines. [1] 5.

The software stack was divided into 30 modules for sensory perception, path planning, logging and steering. The perhaps most advanced module was the self localisation module which was a particle filter, based on a kalman filter [1] page 8. Multiple incoming sensor streams were fusioned with a probabilistic estimation. The vision module was responsible to detect the drivable area in the map [1] page 12. Handcrafted computer vision algorithm were utilized. The steering controlled was realized as a PID control mathematical equation, [1] page 24.

In summary, the technology used in the Stanley self driving car was a classical combination of a diesel vehicle, a computer cluster in the trunk and a large amount of software which implemented algorithms for vision and steering. In other terms, existing and well known software engineering principle were adapted to robotics development. The idea was that a self driving car is some sort of Open source software project with additional mathematical algorithms for road navigation. Typical problems during the project were:

- how to connect the computer in the trunk with the CAN bus of the car
- how to write all the software modules
- how to make the sensory loop fast enough with C/C++ code

It was the same principle used 2 years later during the 2007 darpa urban challenge and it was valid by all of the teams during this time.

Software engineering has a certain name for such projects: rapid prototyping better known as Throwaway-Software. Its a software system that was written in a short amount of time and has a limited lifespan. None of the hardware and software developed for stanley was reused in later projects. In other words, despite that Stanley has won the challenge the technology was obsolete a few weeks after the race was over.

sources:  

[1] Thrun, Sebastian, et al. "Stanley: The robot that won the DARPA Grand Challenge." Journal of field Robotics 23.9 (2006): 661-692.

Darpa urban challenge 2007 -- the last great robot project

 Before the year 2010, artificial intelligence was mostly a niche discipline within computer science without any impact to society. The reason was, that most of the projects were in an early stage and lots of technical obstacle were visible. Because of this limitation its interesting from a science history perspective to take a closer look what the self understanding was of AI in the past.

The goal of the Darpa urban challenge 2097 was to program self driving cars for an urban environment. These normal size cars were able to stop at a junction and do some parking maneuvers. According to the large amount of documentation and some of the talks from the teams its possible to extract some general principle how the cars were realized. Building safe driving cars in the past was recognized as a hardware and software challenge. One problem was to squeeze high performance server racks into the car's trunk. A second and more serious problem was to write all the software.

One team has written software with 100k lines of code, the next one has even created a software with 500k lines of code. The idea in 2007 was to treat self driving car software similar to a large scale software project similar to the linux kernel. Therfor the existing toolchain was used, namely a C/C++ compiler and modern version control systems including bug trackers. The logic of the car was encoded in endless amount of path planning algorithms, C++ classes and dedicated particle filter for self-localization.

It should be mentioned that the outcome of these large scale projects was poor. Despite the fact that a team of experienced programmers have written all the code, the resulting autonomous car was unable to navigate on the street. Simple task like waypoint following was successful demonstrated, but more complex problems like a road block and unexpected situation have overwhelmed the car's AI software.

On the one hand, the shown self driving cars were more powerful than every attempt in the past to build such vehicles. At the same time it was obvious that these cars were not ready for real world traffic. One disappointed detail was that all the written C/C++ software was only working for the original car but can*t be adapted to other cars or another sort of robotics vehicle. In technical terms, the software wasn't scaling up to slightly different problems which is a sign of bad software design.

In 2007 it was unclear how to write better software which fits to the needs of self driving cars. The reason was a certain bias about the project which was: a) the car is designed as an autonomous vehicle b) the decision making process is implemented in software and planning algorithms. So it was a autonomous computational vehicle which is from a modern AI perspective a dead end. In 2007 nobody was able to see the limitation of these constraints but it was imagined, that AI has to be realized this way.

Let us go a step backward and describe the motivation for the darpa urban challenge. The self understanding in 2007 about robotics was, that robotics is a hardware and software problem and located within computer science. The goal was to make sure that the hardware of a self driving car is working, which means that the lidar is rotating fast enough and that the powerful server build into the car gets enough electricity. The second goal was to program the software which means to utilize the C/C++ and implement powerful algorithms in a robot control system. The hope was that the combination of hardware and software would enable a robot car to take its own decision.

Symbol grounding with a DIKW pyramid

 A possible model to explain the symbol grounding problem is the DIKW pyramid. Grounding means to translate a higher layer in the pyramid into a lower layer. The layers are representating the same reality in different formats. The perhaps most important transition is from the numerical data layer into the labeled data. For a warehouse robot a GPS sensor reading like (40,10) gets translated into [roomB]. So the low level sensor data gets annotated with tag.

The next layer is the knowledge graph which encodes the tagging information into a semantic network. The realations between the tags are explained, synonyms are introduced and the information are stored in a json file. If all the layers in the DIKW pyramid are established and if an automatic parser can translate upward and downward in the pyramid, its possible to communicate with a robot in natural language. A voice command like "go to roomB and bring me the yellow box" is understood by the robot and executed in the reality.

Minsky frames as communiation tool

 

A minsky frame is a list of key/value pairs as text overlay in a GUI window. Its not intended as an internal data structure within a robot but its GUI gadget which displays information about the game on the screen.

For the example of an intersection simulator the minsky frame was realized with the pygame command:

screen.blit(txt, (35, y))

Which draws a text string to the screen, for example the information "exit_target: WEST". A minsky frame is some sort of form which determines which aspects of the reality are important. The computer determines the value for each item and shows the result on the screen. This allows to solve the symbol grounding problem because the shown text overlay translates the data layer of the DIKW pyramid into the information of the DIKW pyramid.

Making robots more chatty with minsky frames

Autonomous robots in the past were mostly silent systems which aren't talking but processing information. This makes it hard to debug the AI software.

The screenshot shows an alternative which is a very chatty maze robot. His task is to move around and recharge its battery when its empty. The AI brain consists of mulitiple Minsky frames with key/value information. Technically it was realized as python dictionary shown in text overlay windows.

The surprising situation is, that even such a minimal robot game consists of huge amount of information. There are raw sensor data itself like the position but there are also semantic information like the location in the map and the planned actions.

Creating a Minsky frame itself is not very complicated because its a normal python dictionary. What makes the datastructure powerful is, that frames are written information. They are stored in a database and this allows to add information and translate the existing Minsky frames into new information. For example the planned actions of the robot can only be determined if the existing frames are available which are analyzed.

In other word, the AI isn't a list of algorithms, but the AI is a database distributed over hierarchical key/value data.

Minimal Grounded language

The symbol grounding problem and especially recent vision language models are very complicated to explain. What is missing is a simplified introduction which should given in the following blogpost. The core element of grounded language is a multimodal dictionary, see the picture before. There is one column with pictures and another column with textual annotation. Both columns are connected to each other which allows to translate back and forth between the modalities.

The availability of such a dictionary allows the computer to understand instructions and also the computer can describe the content of a picture. Let me give an example. Suppose the human operator types in "circle left". The computer starts a look up request in the database and retrieves the correct pictograms showing a circle and the left-arrow. The symbols are shown on the screen and the human operator can enter the next command.

Such a pipeline doesn't look very impressive but it can be scaled up dramatically. suppose the pictograms are replaced with motion capture trajectories, there is one trajectory for "stand up" and another one for "walking to the left". This allows a human operator to control a humanoid robot with words. He types in a word, and the lookup in the database with retrieve the correct trajectory which gets executed on the robot. Such a system is known as vision language action model and is the core element of advanced robotics.

Let us go back to the initial example with a pictogram. The task can be summarized as a translation problem between words and symbols. A word is common string like "[l][e][f][t]". Such a string can be generated by keystrokes on a computer keyboard. In contrast, the matching symbol is a picture which is a bit harder to generate. Pictures are usually drawn in a vector graphics program. The trick is to see both information as connected to each other. The matching is called grounding and allows to simplify the communication.


It should be mentioned that from a computer science perspective, the grounding problem can be called boring or trivial. Storing the pictogram including the textual annotation into a computer is a solved problem. There are many ways available for doing so. Its a simple database problem which can be implemented in python in under 100 lines of code. 

The reason why grounded language is at the same task and advanced task is because it is strongly connected with linguistics. So its an interdisciplinary approach between AI, computer science, cognitive science and linguistics. This makes grounded language to a very complex subject.

Gründe gegen Linux auf dem Desktop

 Innerhalb der Linux Community gibt es ein breites Unverständnis gegenüber Windows User. Linux wird als das technisch bessere System definiert und folglich wird das Winbdows Ecosystem verspottet und es wird unterstellt dass Windows ein Auslaufmodell wäre.

Anstatt an diesen Gedankengang anzudocken und gebetsmühlenartig zu wiederholen was die Vorteil von Linux auf dem Desktop sind, sollen an dieser Stelle einmal die nachteile von Linux genannt werden mit dem ziel dass Windows Benutzer gestärkt werden.

Der wichtigste Grund gegen Linux ist, dass dort die beliebte Datenkbank-Anwendung MS Access nicht lauffähig ist, es gibt ferner kein Opensource pendant was dem Funktionsumfang nahe kommt. Lediglich für MS Word und MS Excel gibt es etablierte Linux Alternativen, für die Desktop_datenbank fehlte bisher die nötige Manpower um ein Open Source Projekt durchzuführen.

Eine Desktop Datenbank wie MS Access ist ferne nicht irgendeine Büroanwendung ähnlich wie ein texteditor oder ein Malprogramm, sondern mit besagter Software kann man ohne Programmieren zu müssen leistungsfähige Frontend und Backend EDV Anwendungen erstellen. Datenbanken mögen für Physiker und Mathematiker entbehrlich sein die eher Programmiersprachen und Mathematik-Software einsetzen, jedoch sind Datenbanken im Geschäftsumfeld die Brot und Butter Anwendung. Der Computereinsatz von Unternehmen ist ausnahmslos datenbank orientiert. Ohne Übertreibung kann man sagen, dass MS Access die Killerapplikation für Compüuter im Büroumfeld darstellt, fehlt diese Applikation sind alle weiteren Vorteile unwichtig.

Linux hat jedoch noch weitere Nachteile. Zunächst einmal ist es auf Consumer-PC nicht vorinstalliert. Wer einen neuen PC im Handel kauft wird dort lediglich Windows 11 vorinstalliert vorfinden. Technisch kann man das Problem natürlich mit einem selbst erstellten USB Bootstick lösen, jedoch wird der Umsteiger schnell erkennen, dass es im PC Fachgeschäft für Linux generell keine Software verfügbar ist. Sämtliche kommerzielle Software von Softwarefirmen die an Privat und LBusinesskunden verkauft wird, wurde dezidiert für das MS Windows Betriebssystem entwickelt.

es gab früher einmal auch kommerzielle Linux software die im Fachladen als Package verkauft wurde, jedoch hat sich das konzept nie durchsetzen können. Die PC Welt ist deshalb zweigeteilt: es gibt einmal kommerzielle Software entwickelt von Firmen für das Windows system und dann gibt es noch Open Source software die es nur online im Internet gibt, die aber nicht kommerziell vermarktet wird. Dadurch stehen normale 'Anwender die auf Linux setzen, plötzlich allein da. Sie müssen sich das nötige Fachwissen mühsam anlesen, sie müssen die Software online herunterladen und wenn es dabei Probleme gibt erhalten sie in Foren den Rat doch einfach auf eine andere Linux Distribution zu wechseln, weil die angeblich viel besser wäre.

Für den Durchschnittsanwender, der einen PC mit vorinstallierten Windows verwendet gibt es keinen Grund auf Linux zu wechseln. Er wird dort nur Nachteile erleben, und wegen der steilen Lernkurve an einfachsten Aufgaben scheitern wie der Installation eines Spieles oder dem Öffnen eines Word Dokumentes. MS Windows ist in sämtlichen Punkten überlegen und bildet unverändert den Industriestandard für Desktop PC sowohl bei Privatanwendern als auch in der Geschäftswelt.

The information layer in the DIKW pyramid

The lowest layer in the DIKW pyramid is the data layer which can be desribed easily. There are raw sensor data like distance, temperature, gps coordinates which are stored in a numerical format. The next layer in the pyramid, the information layer, is harder to describe. A working thesis is, that the information layer consists of [tags].

For the example of a warehouse robot, the tag cloud would be: [roomA, roomB, roomC, shelfNorth, shelfsouth, shelf1, shelf2, obstacle, battery, chargingstation, barcode, path, left, right, speed, direction, batteryempty, order]

Of course the tag list is not complete, there are additional tags available but for reason of simplication this might be a starting point. These tags are providing context because after selecting one tag, possible alternative tags are not activated. For example, the goal for the robot might be [roomB] but not [roomA, roomC]. The robot might rotate to [left] but not to [right]. So the context of a tag are always the tag which might be possible but are not activated at the moment.

All the tags are creating a semantic network. In contrast to a full blown ontology or AI frames, a tag based information is more minimalist. Every tag can be activated or not similiar to the tags in a blogging post for annotating a document.

The interesting situation is, that there is an intersection available between low level sensor data and mid level tagging cloud. For example:

- gps sensor -> [roomA]
- gps sensor -> [direction]
- distance sensor -> [obstacle]

For desribing the robot's behavior both layers (data and information) are important. The robot needs to log the numerical raw sensor data and also the robot needs to annotate the current sensory perception with semantic information.

What we can say for sure is, that tagging information doesn't belong to the lowest data layer. A sensor like a gps sensor has no builtin tagging mechanism. The sensor doesn't know the position of a certain shelf, or doesn't know if the robot is in roomA or in roomB. What the gps sensor knows instead are precise x/y coordinates. The reason is, that the sensor hardware is able to generate such data. Its up to a higher instance in the DIKW pyramid to process these data.