October 20, 2026

Another inverse VQA game

In contrast to the previous blog post, this demo is not about answering question but about executing actions. So its an inverse instruction following. Not the AI is in charge to do something, but the human has to do actions in the game, here a sokoban clone.

the possible instructions are formalized in a BNF grammar:

<instruction>      ::= <move_instruction> | <push_instruction>

<move_instruction> ::= "move to " <destination>
<push_instruction> ::= "push box #" <box_id> " to " <destination>

<destination>      ::= <section> | <position>

<section>          ::= "top left" | "top right" | "bottom left" | "bottom right"
<position>         ::= "position " <digit_1_to_10> "/" <digit_1_to_10>

<box_id>           ::= <digit_0_to_9>

<digit_0_to_9>     ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
<digit_1_to_10>    ::= "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "10"


After the human operator has pushed a box to the target location or moved the robot to a certain location, he gets rewarded with +1 and the next quest is shown. The interaction has much in common with NPC quests (non player character quests) in classic role playing games. The challenge for the player is to translate a certain textual quest into the correct action.

No comments:

Post a Comment