The main problem in robotics is, to extend a given teleoperating system to an autonomous system which is equal to lower the workload of the human operator. The baseline for highly complex tasks with a robot is teleoperation which means, that the human is using a joystick to control the gripper. Such technique is working great but is not very advanced. The more elaborated way in robot control would be a software which is working alone. One way in realizing such system is with the help of plan recognition.
Plan recognition means to formalize actions on a higher level. The simplest form of a plan is path through a maze. If a plan is already there it is much easier to solve the task. In most cases, the robot has only to follow the plan. In most situation the problem is not to generate a plan with a planner, but to realize a formal plan at all. Most domains like dexterous manipulation have the problem, that a formal description in a plan language is difficult. That means, it's not possible to give the system a predefined plan because the notation is unknown.
Let us introduce some plan notations. For a maze game, the plan is equal to a 2d trajectory. Which means it contains of points which are connected with edges. The plan is equal to a spline. In case of a pick&place scenario the plan is build with a plan library. Each node in the planning graph is equal to a skill for example “open gripper”.
If a robot is controlled with a teleoperating system the human user is executing a plan. He has a rough idea who to solve a task and the robot is used for executing the plan. The main challenge is to convert between plans and actions in reality. A given plan can be translated in control actions, and executed actions can be recognized as a plan.
Planning language
In the trivial case of a maze solving robot, the plan is equal to a 2d trajectory. It can be stored in a table in the form:
p1, p2, p3, p4, p5
The plan is that the robot moves across the waypoints. He drives from p1 to p2, then to p3 and so on. The more formal description includes the walkto action in the plan:
start, moveto p1, moveto p2, moveto p3, moveto p4, moveto p5
In more demanding domains like a manipulation task the plan is more complicated. Instead of following points the robot is doing something at each point in time. This requires a more elaborated plan description language. An example would be:
start, opengripper, moveto p1, graspobject o1, closegripper, moveto p2, end
Similar to a maze solving robot, the plan describes the single steps. But this time, the steps are more complicated than only a trajectory in the 2d space. Instead the actions are given by natural language instructions. The term “opengripper” describes a certain motion primitive which is executed by the robot's gripper. During executing the robot doesn't move to a different place, but he standsstill and is doing something with the hand.
I'd like to describe the minimum requirement for a plan recognition system. Such a system is not working autonomously, it can only monitor the execution of a human's plan. The teleoperated robot hand is trying to solve a pick&place task. The plan is known in advance. And the AI is monitoring if the human is following it's own plan. According to the plan, the first step is to open the gripper. Does the human operator doing so in reality? If not, then the AI can print out a warning message.