Interaction between agents from recognition of face direction and hand gestures in the virtual space

Abstract

This paper describes a human interface (HI) system through which a user interacts with agents in the virtual space incorporated in the system. An implemented HI system extracts a head and hands regions after taking image sequence of an operator's continuous behavior. In the HI system, we construct the virtual space where an avatar incarnating the operator himself, an autonomous object like a puppy, and other objects are located. In the virtual space, the agent like an avatar can open and close the window and door, grab or move objects, give orders to a puppy and respond to the puppy's behavior. The agents' actions are determined by the human operator's head and hand motion. Each motion is expressed with states so that the system can understand human intention. The experimental HI system shows that the operator can interact with agents in the virtual space using the avatar.