Programming a pick&place robot is on the first look a problem for Artificial Intelligence. It has to do with creating an algorithm which is capable of learning grasping poses. A closer look into the problem will show, that AI isn't needed in the domain. Instead, it's an engineering project which has to do with programming a simulation.
The mindmap on top of the posting shows a rough concept of the idea. All the terms used in the chart are domain specific. It's an attempt to formalize the grasping workflow. The mindmap can't executed on a computer, but it's part of a software engineering process. The idea is to program a prototype with the Python language, and the mindmap helps to identify subparts of the project.
The chart is not complete, because a real grasping robot system contains of many more requirements and design principles. Even the task of pick&place looks not very complicated, it can be a demanding project to write a simulator for this purpose.
On the other hand the potential benefit is great. If the grasping domain can be realized in software, this is equal to automation. In many segments of economy, the same task is done millions of time. All supermarkets, all container terminals, all warehouses and most agriculture production facilities are confronted with the simple problem of grasping an object and release it at the target location. Today, most of the work is done by humans, and not by robots. The reason is, that reliable grasping robots are difficult to program. The task is not a toy problem which can be realized in 300 lines of code, but it's a large scale software projects which needs a lot of heuristics preprogrammed into the system.
But let us go into the details. The main idea is to focus on a simulator which is realized with the object oriented paradigm. The domain of “robot grasping” is converted into an UML chart which consists of many classes. The classes are used to store information about the events, the grasp pose, the trajectory of the robot arm, the position of the objects on the table, the result of the vision system and the planned high-level actions. Right now, it's unclear how many classes are needed to model the overall domain. I would guess 100 classes are the minimum requirement for this complex domain.
The problem is, that the pick&place task consists of many subproblems. One of them is called inverse kinematics. Inverse kinematics has to do with controlling a robot gripper indirectly. Even if the inverse kinematics problem was solve, lots of other problems are available for example the gripper speed during the grasp-phase or what to do if the robot gripper lost the object during the transit.
The overall grasping robot will fail, if only one of the subproblems isn't handled well enough. So there is need to structure the overall task hierarchical. I think, that for creating the prototype, it make sense to get an overview with the help of a mindmap. This helps to identify subdomains of the grasping pipeline which can be solved separately. The idea is, to handle the robot task similar to the problem of programming an operating system for an IBM PC. The idea is that every detail has to be handled with sourcecode, and if the projects consists of millions of codelines, the resulting software will run great.
From the perspective of Artificial Intelligence this sounds a bit disappointing, because the AI Community is interested in building simple but powerful system. The secret goal is to program in 500 lines of code an Artificial Intelligence which can learn by itself, which makes software engineering obsolete. This kind of vision can't be realized in reality. Robot projects in reality have the tendency to become complicated and looking similar to normal software engineering projects from game development or application programming.
To minimize the failure probability of the project, it's important to define some constraints in advance and make the domain easy to realize. The first question which has to answer is, which kind of hardware layout make sense for a grasping robot. The most reliable layout is a portal crane which is used in the reality by container terminals. The advantage is, that the overall system can transport heavy loads, it was tested many times for real problems and it's conservative by default.
Other example of robot grasping systems for example a robot arm or a delta robot are interesting for research projects but they were not tested in reality. Usually these designs are utilized for exploring new path and open new research fields. This kind of open ended problem is not needed here.
The second issue which can reduced in complexity are cluttered object grasps. For the beginning it's much easier to avoid these requirements and define that the robot has only grasp normal objects which are aligned in advanced. So it's not a universal grasping robot, but a simple container crane who is working with a repetive mode.
The resulting mindmap looks clearly, the amount of open tasks is small and it's possible to program the prototype with a low amount of codelines. It's important to mentioned, that even with the simplications, it won't become a toy problem but a large scale software project. That means, the task of programming a simulator for a portal crane is highly complex.