Intuitive 3D Model Prototyping with Leap Motion and Microsoft HoloLens

Abstract

This paper presents an advanced human-computer interaction system which supports the design of 3D model and makes 3D prototypes by merging technologies such as augmented reality (AR), hand gesture recognition, and 3D printing. This proposed development provides a system to enhance the user’s experience in designing of 3D model (3D prototypes). Beginners or novice designers can design 3D model intuitively. The proposed system provides an ability of manipulating with 3D model in 3D space such as translating, rotating, and scaling. Leap Motion was used to detect and recognize the hand gesture using the skeleton-based algorithm. This device sends the data of hand positions and gesture commands to display the virtual hands on the Microsoft HoloLens. The graphic manager manages the registration between the coordinate frame of Leap Motion and Microsoft HoloLens’ frame. The gesture-based modeling technique allows the user to design and manipulates 3D holographic objects. In addition, the HoloLens application is used to visualize holograms in the real environment’s scale. The designed holographic objects can be assembled, disassembled, interacted with the real environment surface. The holographic objects can be exported into the CAD file format from the mesh rendering to the ASCII STL structure which can be printed by a 3D printer automatically. In the experiment, the holographic objects can be modeled relatively to the physical objects. The system has been tested by eight participants. The purpose of this experiment was to explore the intuitive interaction techniques which facilitated in the designing and validated the relation between physical objects and holographic object. The output of the experiment was the real 3D printed prototype obtained from the designed holographics. The results of the system performance covered the operations such as translation, rotation, and scaling of holographic objects with respective to the actual object, the averaged time of designing, the precision of hand gesture interaction, and the usability.