Surgical robotics has enjoyed widespread clinical use over the last decade and brought a number of benefits and drawbacks. Surgeons benefit from improved dexterity and accuracy, as well as better visualization and a more intuitive interface, yet hospitals must contend with the cost and size of robotic systems [1].Surgical robotics requires registering with moving anatomy, such as a scalpel incision or a suturing task. This can be accom plished with visual servoing [2] and machine learning [3], which would require keeping track of a straight line on human anatomy such as a hand, even if the anatomy is not stationary. A solution to this problem is collaborative control, where humans provide intuition while the robots provide the precision.For this experiment, the end effector of a robot was fitted with both a tool, in this case a marker, and a 3D scanning skeletal tracker, in this case a Leap Motion controller (Leap Motion, Inc., San Francisco, CA) as shown in Fig. 1. The skeletal tracker is optimized for hand scanning, intended for use as a desktop computer interface device. The API gives a skeletal tracking model for a human hand based on stereoscopic infrared imaging. The relative positions of the scanner and the tool tip were known so the kinematics could be completely solved.The robot used was a CORVUS arm (complete operating room robotics for virtually unassisted surgery), a custom built six degree of freedom robotic arm with a PRRRRR setup (a prismatic joint followed by five revolute joints in series), however, for this experiment only three degrees of freedom were required, as only position was being controlled, so only the first three consecutive revolute joints were used. The arm is a testbed for surgical robotics research; its hardware design is provided elsewhere [4].A hand shape was used as the target object, and the task was first to stay stationary with respect to the hand, then to draw a line on that hand. In order to allow the human to interact with the robot, the trajectory is projected onto a horizontal plane at approximately chest height, allowing the human to lower a hand onto the tool, controlling the Z axis of the interaction, which controls the force of the interaction as seen in Fig. 2. The robot then measures the X,Y and orientation (θ) of the hand, and computes the appropriate point for the trajectory, controlling X, Y, and θ. For safety the user can disengage at any time by pulling his or her hand up and away. If the hand is pulled away, the robot will simply wait for the hand to be returned to resume the trajectory. Drawings of the felt tip marker path were then used to quantify the performance in terms of pixels occupied.The skeletal tracker allows visual servoing to compensate for the drifting and shaking of a human hand being held in the air. As is shown in Fig. 3, the extents of the drifting of the tool tip are reduced by approximately 50% in the X-direction and 25% in the Y-direction, as shown in Table 1, resulting in a 5 mm error bound.The skeletal tracker allows a correction of larger deviations as well, as can be seen in Fig. 4 where the hand is rotated by 45 deg and translated by 1 cm during a line drawing task, designed to simulate a linear scalpel incision or a suturing task, and the closed loop control is able to quickly compensate for the change (right column) with feedback from the movement, while the fixed trajectory does not (left column), corresponding to a higher coefficient of determination with respect to the target trajectory, and less error pixels outside the 5 mm extents, as shown in Table 2.The noncontact skeletal tracking allows an intuitive human robot interaction with the human in the control loop. This allows the robot to provide the precision, while the human provides the intuition. This could have applications for commanding movement of a robot without contact, or for robotic interaction with the hand itself, such as tattooing, hand surgery, or physical therapy.Future work includes testing more complicated geometries, as well as larger or more dynamic perturbations, possibly up to a completely free-moving hand, and ability to retract the tool while the error is known to be beyond a threshold.