This paper addresses the remote control of robot skeleton based on gesture recognition of a human operator’s body. The control motion procedure is facilitated by the use of the Kinect motion capture sensor, which transmits appropriate signals to the robot’s joints to adequately replicate the operator’s movement. Since the movement of the human operator is allowed to occupy the whole Kinect’s space field of view, the arm robot however might have limited range of operation and therefore is subjected to operate in constrained region of the task space. The concept used for the control of the arm robot relies on the employment of the potential energy of a specified geometric desired region. The controller also incorporates high filtered version of joint errors to regulate the system without joint velocity measurement. The approach is experimentally demonstrated on five degrees of freedom articulated robot arm.