VR Controller Research Articles

VR-HandNet: A Visually and Physically Plausible Hand Manipulation System in Virtual Reality.

This study aims to allow users to perform dexterous hand manipulation of objects in virtual environments with hand-held VR controllers. To this end, the VR controller is mapped to the virtual hand and the hand motions are dynamically synthesized when the virtual hand approaches an object. At each frame, given the information about the virtual hand, VR controller input, and hand-object spatial relations, the deep neural network determines the desired joint orientations of the virtual hand model in the next frame. The desired orientations are then converted into a set of torques acting on hand joints and applied to a physics simulation to determine the hand pose at the next frame. The deep neural network, named VR-HandNet, is trained with a reinforcement learning-based approach. Therefore, it can produce physically plausible hand motion since the trial-and-error training process can learn how the interaction between hand and object is performed under the environment that is simulated by a physics engine. Furthermore, we adopted an imitation learning paradigm to increase visual plausibility by mimicking the reference motion datasets. Through the ablation studies, we validated the proposed method is effectively constructed and successfully serves our design goal. A live demo is demonstrated in the supplementary video.

3D sketching in immersive environments: Shape from disordered ribbon strokes

Immersive environments with head mounted displays (HMD) and hand-held controllers, either in Virtual or Augmented Reality (VR/AR), offer new possibilities for the creation of artistic 3D content. Some of them are exploited by mid-air drawing applications: the user’s hand trajectory generates a set of stylized curves or ribbons in space, giving the impression of painting or drawing in 3D. We propose a method to extend this approach to the sketching of surfaces with a VR controller. The idea is to favor shape exploration, offering a tool, where the user creates a surface just by painting ribbons. These ribbons are not constrained to form patch boundaries for example or to completely cover the shape. They can be very sparse, disordered, overlap or not, intersect or not. The shape is computed simultaneously, starting with the first piece of ribbon drawn by the user and continuing to evolve in real-time as long as the user continues sketching. Our method involves minimizing an energy function based on the projections of the ribbon strokes on a proxy surface by taking the controller’s orientations into account. The current implementation considers elevation surfaces. In addition to many examples, we evaluate the time performance of the dynamic shape modeling with respect to an increasing number of input ribbon strokes. Finally, we present images of an artistic creation that combines stylized curve drawings in VR with our surface sketching tool created by a professional artist.

The user experience of distal arm-level vibrotactile feedback for interactions with virtual versus physical displays

Haptic feedback, a natural component of our everyday interactions in the physical world, requires careful design in virtual environments. However, feedback location can vary from the fingertip to the finger, hand, and arm due to heterogeneous input/output technology used for virtual environments, from joysticks to controllers, gloves, armbands, and vests. In this work, we report on the user experience of touch interaction with virtual displays when vibrotactile feedback is delivered on the finger, wrist, and forearm. In a first controlled experiment with fourteen participants and virtual displays rendered through a head-mounted device, we report a user experience characterized by high perceived enjoyment, confidence, efficiency, and integration as well as low perceived distraction, difficulty, and confusion. Moreover, we highlight participants’ preferences for vibrotactile feedback on the finger compared to other locations on the arm or through the VR controller, respectively. In a follow-up experiment with fourteen new participants and physical touchscreens, we report a similar preference for the finger, but also specific nuances of the self-reported experience, not observed in the first experiment with virtual displays. Overall, our results depict an enhanced user experience when distal vibrotactile feedback is available over no vibrations at all during interactions with virtual and physical displays, for which we propose future work opportunities for augmented interactions in virtual worlds.

Smart Pipette: Elevating Laboratory Performance with Tactile Authenticity and Real-Time Feedback.

Mastering the correct use of laboratory equipment is a fundamental skill for undergraduate science students involved in laboratory-based training. However, hands-on laboratory time is often limited, and remote students may struggle as their absence from the physical lab limits their skill development. An air-displacement micropipette was selected for our initial investigation, as accuracy and correct technique are essential in generating reliable assay data. Handling small liquid volumes demands hand dexterity and practice to achieve proficiency. This research assesses the importance of tactile authenticity during training by faithfully replicating the micropipette's key physical and operational characteristics. We developed a custom haptic training approach called 'Smart Pipette' which promotes accurate operation and enhances laboratory dexterity training. A comparative user study with 34 participants evaluated the effectiveness of the Smart Pipette custom haptic device against training with off-the-shelf hardware, specifically the Quest VR hand controller, which was chosen because it is held mid-air similar to a laboratory micropipette. Both training conditions are integrated with the same self-paced virtual simulation displayed on a computer screen, offering clear video instructions and realtime guidance. Results demonstrated that participants trained with the Smart Pipette custom haptic exhibited increased accuracy and precision while making fewer errors than those trained with off-the-shelf hardware. The Smart Pipette and the Quest VR controller had no significant differences in cognitive load and system usability scores. Tactile authentic interaction devices address challenges faced by online learners, while their applicability extends to traditional classrooms, where real-time feedback significantly enhances overall training performance outcomes.

The Development of Malang City Virtual Tourism for Preservation of Traditional Culture using React 360

Malang is one of the tourist destinations in East Java, and always visited by both domestic and foreign tourists. However, Malang isn’t the first destination in tourism, this is due to the lack of intense promotion. Virtual tourism is a virtual reality technology for promoting tourism. Based on these problems, the researchers developed a virtual reality application that can provide immersive experiences to users assembled with tourist destinations in Malang. Application was developed using React 360 framework, because of its ease of development and can be developed for various platforms. The Software Development Life Cycle (SDLC) is the waterfall method. The implementation using VR Headset and VR Controller. In testing, researchers conducted functional testing and non-functional testing. Functional testing is done using black box testing and non-functional testing is done by the System Usability Scale (SUS) method. The results of functional testing using black box are 100% valid. In the usability test, the System Usability Scale (SUS) questionaire obtained a score of 89. From the scores that have been obtained, this application is acceptable, with Grade A and adjective rating Excellent.

Presenting Morphing Shape Illusion: Enhanced Sense of Morphing Virtual Object with Weight Shifting VR Controller by Computational Perception Model.

The haptic sensation is crucial for Virtual Reality, as it gives the presence of objects in a virtual world and thus gives a greater sense of immersion. To provide a sense of the shape of handheld objects, a haptic device that changes weight distribution is proposed. It is known that the visual feedback enhances the haptic sensation of shape, and it is also known that it does for morphing shape as well. Our previous publication presented a perception model for static shape of virtual object. In this paper, we extend the model to produce a plausible sense of morphing shape of handheld objects. Our stochastic model predicts the proper weight actuation for the weight-shifting haptic device, which users can plausibly feel while reducing hardware effort. We evaluated our perception model and resulted in the model accuracy average of 8.1% error. Using this perception model, the amount of weight actuation at 75% probability of plausibility is reduced up to 37%.

MEinVR: Multimodal interaction techniques in immersive exploration

Immersive environments have become increasingly popular for visualizing and exploring large-scale, complex scientific data because of their key features: immersion, engagement, and awareness. Virtual reality offers numerous new interaction possibilities, including tactile and tangible interactions, gestures, and voice commands. However, it is crucial to determine the most effective combination of these techniques for a more natural interaction experience. In this paper, we present MEinVR, a novel multimodal interaction technique for exploring 3D molecular data in virtual reality. MEinVR combines VR controller and voice input to provide a more intuitive way for users to manipulate data in immersive environments. By using the VR controller to select locations and regions of interest and voice commands to perform tasks, users can efficiently perform complex data exploration tasks. Our findings provide suggestions for the design of multimodal interaction techniques in 3D data exploration in virtual reality.

Triggermuscle: Exploring Weight Perception for Virtual Reality Through Adaptive Trigger Resistance in a Haptic VR Controller

It is challenging to provide users with a haptic weight sensation of virtual objects in VR since current consumer VR controllers and software-based approaches such as pseudo-haptics cannot render appropriate haptic stimuli. To overcome these limitations, we developed a haptic VR controller named Triggermuscle that adjusts its trigger resistance according to the weight of a virtual object. Therefore, users need to adapt their index finger force to grab objects of different virtual weights. Dynamic and continuous adjustment is enabled by a spring mechanism inside the casing of an HTC Vive controller. In two user studies, we explored the effect on weight perception and found large differences between participants for sensing change in trigger resistance and thus for discriminating virtual weights. The variations were easily distinguished and associated with weight by some participants while others did not notice them at all. We discuss possible limitations, confounding factors, how to overcome them in future research and the pros and cons of this novel technology.

Virtual Reality for Evaluating Prosthetic Hand Control Strategies: A Preliminary Report.

Improving prosthetic hand functionality is critical in reducing abandonment rates and improving the amputee's quality of life. Techniques such as joint force estimation and gesture recognition using myoelectric signals could enable more realistic control of the prosthetic hand. To accelerate the translation of these advanced control strategies from lab to clinic, We created a virtual prosthetic control environment that enables rich user interactions and dexterity evaluation. The virtual environment is made of two parts, namely the Unity scene for rendering and user interaction, and a Python back-end to support accurate physics simulation and communication with control algorithms. By utilizing the built-in tracking capabilities of a virtual reality headset, the user can visualize and manipulate a virtual hand without additional motion tracking setups. In the virtual environment, we demonstrate actuation of the prosthetic hand through decoded EMG signal streaming, hand tracking, and the use of a VR controller. By providing a flexible platform to investigate different control modalities, we believe that our virtual environment will allow for faster experimentation and further progress in clinical translation.

모멘트 컨트롤러: 가상물체 간의 무게 차이 인식이 가능한 VR 컨트롤러

모멘트 컨트롤러: 가상물체 간의 무게 차이 인식이 가능한 VR 컨트롤러

Immersive sketch-based tree modeling in virtual reality

Tree modeling has been a widely-discussed topic in computer graphics. However, with existing methods, the modeling process is occupied with complex data collection and tedious parameter adjustment, lacking a rich sensory modeling experience. In this paper, we propose an approach to sketch-based tree modeling in an immersive virtual reality environment, aiming to lower the difficulty of modeling and enhance the immersion in the designing process. We first present a sketch sampling and points optimization algorithm to obtain the skeleton of the branch in 3D space. As generating geometry along the skeleton, we apply a vector-projection method to fix the branch polygon twisting. Then, we introduce a bidirectional ray-hit algorithm to determine the branch radius as real-time sketching. To generate random twigs on branches, we introduce a twigs generation algorithm based on Perlin noise and the parent branch direction. Finally, we design a series of interactive methods for users to create tree models in a 3D virtual scene with the VR HMD and controller. Experimental results indicate that our approach can accomplish creating realistic tree models in real-time. The interactive and immersive modeling experience enables users to readily convey their ideas on tree structures in a simple and direct way of sketching. Visualizing and modifying the real-time generated branch results can contribute to provoking the inspiration for creation.

VR 컨트롤러의 모멘트 변화에 따른 최소 무게 차이 인지에 관한 연구

본 연구는 가상 물체의 무게감을 사용자에게 전달하여 몰입감 있는 가상현실 경험을 제공하는 컨트롤러에 관한 것이다. 무게감 전달의 수단으로 컨트롤러 내 무게 중심의 위치를 변화시키면 모멘트의 크기가 변화하는 개념을 활용하였다. 사용자가 무게차를 인지하는 무게 중심의 최소이동거리를 알아보기 위해 무게 중심의 위치를 변경할 수 있는 실험 장치를 제작하여 실험을 수행하였다. 그 결과, 시작 자극의 위치와 무관하게 약 5cm의 무게부 거리 차에서 두 물체 간의 무게 차이를 인지하는 것으로 나타났다. 향후 이러한 실험 결과를 반영하여 모멘트 변화에 따른 무게 인지의 최소이동거리 역치 값을 찾는 실험을 수행할 것이다.

Mixed Reality Interface for Improving Mobile Manipulator Teleoperation in Contamination Critical Applications

This paper presents a mixed reality teleoperation interface for mobile manipulation tasks in contamination critical production environments, where human presence is undesirable. This is achieved by using an intuitive control approach and providing the operator with a sense of depth through various visual feedback modalities. The different visual feeds from a mono-and stereoscopic multi-camera setup are displayed for the operator, in a mixed reality control room developed in Unity. The control interface employs the differentiation of the VR controller’s pose, interpolated into a trajectory for the end-effector. The communication between the operator and the robot is facilitated through ROS for control commands and visual feedback. Speed of operation is typically not crucial in current use cases, while task safety, accuracy, and perception are paramount. The paper presents the latest research developments of a mixed reality interface designed and tested for a mobile manipulator.

The Interaction of Five-Fingered Haptic Controller for Manipulating Object in Virtual Reality

In this paper, we designed a haptic controller interface that can control haptic controller with pressure sensors that collect pressure data from each finger, linear actuators to control different force feedback on each finger, and vibration motor. The haptic controller system consists of haptic controller hardware, haptic controller interface, and game engine module. The haptic controller hardware communicates with haptic controller interface module via serial communication. The game engine interface module performs physics-based interaction between haptic controller and virtual object. We designed the haptic controller interface with previously studied hardware and performed user tests. This paper showed the possibility of more realistic hand operation than the existing VR controller.

VRMouse: Mouse Emulation with the VR Controller for 2D Selection in VR

VRMouse: Mouse Emulation with the VR Controller for 2D Selection in VR