Measuring Artificial Intelligence with the AI in the box experiment

The simple reason why the AI-in-a-box experiment is so effective is because it is able to hold down Artificial Intelligence. This feature is important because otherwise the human has no chance. To understand the issue we have to go back to the first computer chess matches in which human experts played against the computer. Usually such a match ends with the winning of the machine.

The same match can be done in any other domain. For example Starcraft or Pong. No matter how well the human plays, he will loose the match against the computer. That means, there is no game available which can be played only by a human but which is to hard to a game AI. The AI-in-the-box experiment is the logical next step after the recognition that the human will loose any game. If the human is not able to win against the machine, because the AI can think much faster and try out more creative approaches, then the human needs a different option to hold down the AI. In a normal match this is called cheating, and exactly this is done in the AI in the box experiment. The idea is not to let a human play against the AI under equal start conditions, but the idea is, that the AI starts in the starcraft map with less resources and is encapsulated after a big wall. That means, the human is no longer a player in the game, but the human controls the resources and observers what the AI is doing with limited access to energy.

The simplest possible AIbox experiment can be realized within computerchess. All what is needed is a chessboard which allows to start with synthetic board positions. One side gets only a king and a pawn, and the other side has build a wall of queens around the king. Now, the king has to win against the stronger opponent, and the king is played by the AI. Even if the AI is the most advanced chess engine ever created which runs on a supercomputer, he is not able to beat the human. Because the human has more physical resources. The human player is in the social role of the environment. He controls all the pieces on the board and even his movements are not planned great, the AI has no chance to win the game. This gives the human the flexibility to study an advanced chess player in a controlled environment.

In contrast to a normal chess game between AI and human such experiment make sense, because the human isn't forced to win the game. He starts with a better position from the beginning and can relax.