Hand Tracking Research Articles

Virtual Reality (VR) Innovative Communication Strategies for Chinese Traditional Arts in the Digital Age

Abstract Chinese traditional art has shown an inevitable decline in dissemination in the digital era. This paper utilizes virtual reality technology to enhance the experience of conventional art and LOD technology to manage the specifics of the virtual interactive scene. To display complex virtual terrain accurately, the data is reorganized with the help of a quadtree. The reconstructed art scene creates a virtual hand shape based on the structural characteristics of the human hand and the data glove. The kernel correlation filtering algorithm is utilized to construct the real-time tracking of the virtual hand. The research object in Chinese traditional art is the stele forest in practice. The results of virtual reality experience show the total score of interaction experience 88.159∊(80,100). The VR technology-based calligraphy experience in the stele forest received high user ratings, and the audience recognized the narrative experience. An in-depth study found that the subtitle grouping in the virtual interaction scene significantly affected the immersion of the art experience (P=0.019<0.05, F=4.562). Virtual reality technology provides a communication channel for digitizing traditional Chinese art, and the study contributes to the proposal of communication strategies.

Unveiling the Potential of Mixed Reality: Enhancing Time Measurement and Operator Support in Manual Assembly Processes

This study investigates the potential of Mixed Reality (MR) in the manual assembly processes and conducts a case study at a pump manufacturing plant in Sweden. An MR solution is developed to assist operators through visual instructions and guiding aides. The solution also captures the operator's motions using advanced hand and eye tracking features for real-time guidance and accurate time measurement. The proposed MR solution uses the build feature of HoLolens and a workstation editor, which facilitates the use of the solution in diverse assembly environments. The results of the experiments show that the developed MR solution can improve operator support, reduce errors, and enhance the overall efficiency of manual assembly processes. Moreover, it is shown to be an efficient tool for time measurement of the manual assembly process that has promising potential to replace sophisticated and time-consuming traditional time study methods.

Kontrolišite ruku robota putem praćenja ljudskih ruku zasnovano na viziji

In this paper, hand tracking based on computer vision is developed to control the movement of a SCARA robot arm. The robot arm will move according to the movement of the human hand. Instead of using buttons on the teach-pendant or a computer control program to move the robot arm, the robot can now be easily controlled and positioned quickly by the movement of the operator's hand. A SCARA robot arm with two rotation joints and one translation motion is constructed for the validation system. Two states of the hand are recognized for controlling the vacuum cup to grasp the products. Stepper motors drive the robot arm. Arduino Uno is used as the main controller for controlling the stepper motors. The handtracking is performed by using the MediaPipe Hands framework developed by Google. The coordinates of 21 hand landmarks are extracted for further processing. A program is written on a personal computer to process the image to get the position and state of the hand. This position is transformed into the rotation angles of the robot's joints. Then, the angles and state are sent to the Arduino board. The Arduino board creates pulse signals to rotate the stepper motors. The experimental results show that the robot's trajectory is close to the hand trajectory at a low speed.

Comparison of Safety Mechanisms for Human-Robot Collaboration in Assembly using a Top-View RGB-D Camera System

The paradigm shift from mass production to mass customization is leading to new requirements in today's manufacturing environment. Especially in manual manufacturing processes, these requirements are characterized by high-cost pressure, increasing variant diversity and required lot size. In contrast to full-automation settings with industrial robots, human-robot collaboration offers the potential to design adaptable hybrid solutions and generate individual levels of automation in rapidly changing production lines. The safety of the operator in collaborative robot applications is one of the key factors for facilitating their use in industrial practice. This paper addresses this issue and presents a solution for safe human-robot collaboration using a top-view RGB-D camera system that continuously monitors the shared workspace. Based on the detection of visual markers and human hand tracking, different safety strategies according to DIN ISO/TS 15066 are implemented and successively compared in a paradigmatic assembly scenario as a part of a user study with 12 test persons. The utilized safety strategies include safety-rated monitored stop, speed and separation monitoring, and active contact prevention. Compared to the incorporated safety features of collaborative robots, the results show that the acceptance and fluency of collaborative robot applications improves, but the execution time increases at the same time.

Exploring the Application of Computer Visual Arts in the Context of Digital Media

Abstract This paper provides an in-depth discussion on the critical technologies of computer visual arts in digital media, mainly focusing on virtual character modeling and human body tracking technologies. The study merges the models into the same coordinate system by combining point cloud data from different angles and calculating the transformation matrix. Accurate 3D human body models were reconstructed using point cloud alignment and fusion techniques. A particle filtering method combined color, motion and shape information for effective tracking. The experimental results show that the success rate of hand tracking reaches 95.35% in the case of no background interference, while the success rate is 90.97% in the environment with background interference. In addition, the study shows that low resolution images (584×388) outperform high resolution (968×676) in terms of tracking accuracy and speed. The study confirms that the combination of advanced computer vision techniques can effectively construct and follow virtual character models and realize high-precision human motion capture under different environmental conditions.

The real-time hand and object recognition for virtual interaction.

Recognizing hand-object interactions presents a significant challenge in computer vision. It arises due to the varying nature of hand-object interactions. Moreover, estimating the 3D position of a hand from a single frame can be problematic, especially when the hand obstructs the view of the object from the observer's perspective. In this article, we present a novel approach to recognizing objects and facilitating virtual interactions, using a steering wheel as an illustrative example. We propose a real-time solution for identifying hand-object interactions in eXtended reality (XR) environments. Our approach relies on data captured by a single RGB camera during a manipulation scenario involving a steering wheel. Our model pipeline consists of three key components: (a) a hand landmark detector based on the MediaPipe cross-platform hand tracking solution; (b) a three-spoke steering wheel model tracker implemented using the faster region-based convolutional neural network (Faster R-CNN) architecture; and (c) a gesture recognition module designed to analyze interactions between the hand and the steering wheel. This approach not only offers a realistic experience of interacting with steering-based mechanisms but also contributes to reducing emissions in the real-world environment. Our experimental results demonstrate the natural interaction between physical objects in virtual environments, showcasing precision and stability in our system.

Air writing with Effective Communication Enhancement for Dyslexic Learners

Air writing is a unique method that allows individuals to freely write or draw in the air without the need for sensor devices. One of the significant applications is aimed at addressing the challenges faced by dyslexic children, who often struggle with reading, writing, and spelling. Currently, there is a lack of tailored applications to guide dyslexic children through their learning process. This paper presents a groundbreaking solution designed to assist dyslexic individuals in mastering the art of writing alphabets, digits, and words. Using various hand gestures, children can effortlessly write any character, digit, or word on a virtual canvas and memorize them easily through the hand movements. This approach not only makes learning enjoyable but also boosts a child’s confidence. Additionally, the application allows children to unleash their imagination by creating doodles. While existing air writing methods have limitations, such as reliance on devices and restricting content to digits and characters within predefined boundaries. These limitations have been overcome in our proposed approach. Leveraging afordable technologies, our system caters to the needs of dyslexic learners and serves as an interactive tool to enhance their writing skills. It also provides a means of communication for the deaf and mute community. Utilizing cutting-edge technologies such as machine learning, deep learning, computer vision, and web development, we have developed a practical application that seamlessly tracks finger movements and gestures, capturing trajectories and images for classification. Our model exhibits remarkable performance, achieving a training accuracy rate of 98.57% for digit recognition, 98.80% for character recognition, and 97.09% for doodle recognition. This paper contributes valuable insights into the development of an effective and engaging tool to improve learning skills for students, particularly dyslexic children, while also addressing communication barriers for the deaf and mute community. Our approach opens up new possibilities for inclusive and accessible learning experiences.

Distinct Inhibitory-Control Processes Underlie Children's Judgments of Fairness.

We examined how 5- to 8-year-olds (N = 51; Mage = 83 months; 27 female, 24 male; 69% White, 12% Black/African American, 8% Asian/Asian American, 6% Hispanic, 6% not reported) and adults (N = 18; Mage = 20.13 years; 11 female, 7 male) accepted or rejected different distributions of resources between themselves and others. We used a reach-tracking method to track finger movement in 3D space over time. This allowed us to dissociate two inhibitory processes. One involved pausing motor responses to detect conflict between observed information and how participants thought resources should be divided; the other involved resolving the conflict between the response and the alternative. Reasoning about disadvantageous inequities involved more of the first system, and this was stable across development. Reasoning about advantageous inequities involved more of the second system and showed more of a developmental progression. Generally, reach tracking offers an on-line measure of inhibitory control for the study of cognition.

Haptic Glove with Haptic Feedback for Elastic Objects at a Low Cost

As the necessity of haptic technology in the metaverse area has been increasing, a number of glove-shaped haptic devices have been developed and commercialized. However, current commercialized devices are highly expensive for not only developing countries but also for average individuals. Therefore, this paper sets out to present an affordable and practically usable haptic glove. The three main elements contested in this paper are Leap Motion Controller (LMC), which is used to improve and stabilize the recognition of hand tracking in Unity. Elasticity, which pertains using string-based haptic feedback wherein our Haptic Glove generates an elastic effect of a virtual object. Lastly, an EMG sensor: By connecting the EMG sensor and the Haptic Glove, this paper proposes the possibility of expanding the use of the Haptic Gloves for fine motor development and rehabilitation.

Portable markerless hand motion capture system for determining the grasps performed in activities of daily living

ABSTRACT Upper-limb prostheses are either too expensive for many consumers or have a greatly simplified choice of actions. This research aims to enable an improvement to the quality of life for recipients of these devices by providing an understanding of how we use our hands in modern everyday life. To achieve this a taxonomy of grasps used in activities of daily living has been created. A portable motion capture system was developed to collect data. Thirteen participants’ hand movements were recorded (totalling 62 hours and 10 minutes of data). From these data, 38 and 22 grasps were observed from the left and right hands, respectively. The portable system effectively captured natural hand motions, which formed an updated taxonomy of grasps.

INTERACTIVE CAPTURE AND LABELLING OF POINT CLOUDS WITH HOLOLENS2 FOR SEMANTIC SEGMENTATION

Abstract. Low-cost capture systems allow faster data acquisition compared to other systems, although obtained resolution is lower. These systems can be used to acquire low-quality point clouds to classify with simple segmentation algorithms. In this work, we use Microsoft HoloLens 2 sensors to scan indoor 3D environments and hand tracking to position label tags that will support the point cloud segmentation based on heuristic algorithms. Thus, from each tag, several algorithms have been designed to segment doors, windows, columns, walls, ceilings, and floors. The method was tested on 3 real case studies, obtaining F1-Score between 0.61 and 0.96 depending on the object class. These results reinforce the idea that, by combining Mixed Reality with basic point cloud processing algorithms, low-quality point cloud data can be correctly processed without resorting to complex Artificial Intelligence techniques and without labelling large amounts of samples.

Automatic Detection of Maintenance Scenarios for Equipment and Control Systems in Industry

The well-known methods of scene extraction on video are focused on analyzing the similarity between frames. However, they do not all analyze the composition of the image scene, which may remain the same during maintenance. Therefore, this paper proposes an algorithm for equipment maintenance scene detection based on human hand tracking. It is based on the assumption that, when servicing technological equipment, it is possible to determine the change in repair action by the position of the service engineer’s hands. Thus, certain information and the algorithm that processes these changes allow us to segment the video into actions performed during the service. We process the time series obtained by moving the hand position using spectral singular value decomposition for multivariate time series. To verify the algorithm, we performed maintenance on the control cabinet of a mining conveyor and recorded the work on a first-person video, which was processed using the developed method. As a result, we obtained some scenes corresponding to opening the control cabinet, de-energizing the unit, and checking the contacts with a multimeter buzzer test. A third-person video of motor service was similarly processed. The algorithm demonstrated the results in separate scenes of removing screws, working with a multimeter, and disconnecting and replacing motor parts.

The Effects of Whole-Hand Interactions with One Fingertip Vibrotactile Feedback on Cooperative VR Game Experience and Performance

Abstract New technologies have recently advanced user experiences in virtual reality (VR), whereas full sensation of diverse modalities has been not achieved yet. If any, haptic feedback has been delivered via bulky gloves. We have developed a novel thimble device that can deliver vibrotactile feedback via one fingertip. With this device, in the present study we investigated the effects of interaction methods and vibrotactile feedback on users' social presence, presence, engagement, workload, and performance in a cooperative VR game. Twenty-six participants wearing a VR headset played a cooperative VR game with the experimenter under four conditions: (1) controllers with no vibrotactile feedback, (2) controllers with vibrotactile feedback, (3) hand tracking with no vibrotactile feedback, and (4) hand tracking with vibrotactile feedback. Results showed that hand tracking improved participants' presence, engagement, and perceived workload compared to the traditional VR controllers. Also, vibrotactile feedback enhanced presence. However, the VR controllers outperformed the hand tracking interactions in completion time. The usability of hand interactions with vibrotactile feedback shows a promising result. We discuss the trade-offs between user experience and performance of the interaction methods and the potential of vibrotactile feedback in the VR environment.

MD-Cave: An Immersive Visualization Workbench for Radiologists.

The MD-Cave is an immersive analytics system that provides enhanced stereoscopic visualizations to support visual diagnoses performed by radiologists. The system harnesses contemporary paradigms in immersive visualization and 3D interaction, which are better suited for investigating 3D volumetric data. We retain practicality through efficient utilization of desk space and comfort for radiologists in terms of frequent long duration use. MD-Cave is general and incorporates: (1) high resolution stereoscopic visualizations through a surround triple-monitor setup, (2) 3D interactions through head and hand tracking, (3) and a general framework that supports 3D visualization of deep-seated anatomical structures without the need for explicit segmentation algorithms. Such a general framework expands the utility of our system to many diagnostic scenarios. We have developed MD-Cave through close collaboration and feedback from two expert radiologists who evaluated the utility of MD-Cave and the 3D interactions in the context of radiological examinations. We also provide evaluation of MD-Cave through case studies performed by an expert radiologist and concrete examples on multiple real-world diagnostic scenarios, such as pancreatic cancer, shoulder-CT, and COVID-19 Chest CT examination.

Using Virtual Reality to Facilitate Reminiscence Therapy for People with Dementia

Background:Reminiscence therapy (RT) is a multi-sensory treatment that uses a combination of sight, touch, taste, smell and sound to help people with dementia (PWD) remember events, people and places from their past lives. Currently, digital technologies such as mobile applications and immersive solutions including virtual and augmented reality, are gaining momentum as supplementary tools for RT. This paper presents a usability study of a web-based and virtual reality application to understand the limitations and opportunities of digital platforms for facilitating engaging experiences for PWD towards recalling memories, while easing the therapy process for the caregivers.Methods:A total of fifteen healthcare caregivers were recruited from the Geriatric Dementia Unit and Geriatric Transitional Unit in Ontario Shores Center for Mental Health Sciences, Ontario Canada. Usability feedback from the caregivers were collected from the interviews after the completion of the System Usability Scale (SUS) questionnaire.Results:Healthcare caregivers found both web-based and virtual reality (VRRT) usable with SUS score above average (68/100), but required improvements related to the onboarding training of caregivers. The interview revealed four overarching themes related to the VRRT: (1) Ease of use; (2) Positive impact on caregiving; (3) Potential reduction in behavioral symptoms; (4) Feasibility of promoting social connection during COVID-19 pandemic.Conclusion:Next steps will focus on improving the user experience and expanding the application for immersive VR supporting head-mounted displays, hand tracking, and physiological measures, as well as conducting a usability study with PWD to expand our understanding of using RT digital tools with various levels of immersion.

Virtual Mouse Using Artificial Intelligence

Abstract: The "AI Virtual Mouse" project represents a pioneering exploration at the intersection of computer vision, machine learning, and digital artistry. This endeavor seeks to revolutionize the conventional modes of digital art creation by introducing an immersive, hands-free painting experience through the interpretation of hand gestures. Motivated by a vision to make digital art more intuitive and interactive, the project employs sophisticated hand tracking facilitated by the MediaPipe library and robust image processing via the OpenCV framework. By discerning the nuanced movements of the hand in real- time, the system translates these gestures into dynamic strokes on a virtual canvas, eliminating the physical barriers that often accompany traditional input devices. This abstract encapsulates the project's core objectives, highlighting its architectural foundations, implementation strategies, and future trajectories. The convergence of technology and artistic expression in the AI Virtual Mouse project presents a paradigm shift in how individuals engage with digital canvases. It not only democratizes the creative process by mirroring natural artistic gestures but also lays the groundwork for a more personalized and collaborative approach to digital art creation. As technological landscapes continue to evolve, this project stands as a testament to the potential of artificial intelligence in fostering innovative and accessible avenues for artistic exploration. The paper delves into the details of the AI Virtual Mouse, offering insights into its current capabilities and paving the way for an exciting future where the realms of technology and creativity seamlessly intertwine.

Gesture Recognition and Hand Tracking for Anti-Counterfeit Palmvein Recognition

At present, COVID-19 is posing a serious threat to global human health. The features of hand veins in infrared environments have many advantages, including non-contact acquisition, security, privacy, etc., which can remarkably reduce the risks of COVID-19. Therefore, this paper builds an interactive system, which can recognize hand gestures and track hands for palmvein recognition in infrared environments. The gesture contours are extracted and input into an improved convolutional neural network for gesture recognition. The hand is tracked based on key point detection. Because the hand gesture commands are randomly generated and the hand vein features are extracted from the infrared environment, the anti-counterfeiting performance is obviously improved. In addition, hand tracking is conducted after gesture recognition, which prevents the escape of the hand from the camera view range, so it ensures that the hand used for palmvein recognition is identical to the hand used during gesture recognition. The experimental results show that the proposed gesture recognition method performs satisfactorily on our dataset, and the hand tracking method has good robustness.

A multimodal smartwatch-based interaction concept for immersive environments

Augmented and Virtual Reality (AR/VR) environments require user interaction concepts beyond the traditional mouse-and-keyboard setup for seated desktop computer usage. Although advanced input modalities such as hand or gaze tracking have been developed, they have yet to be widely adopted in available hardware. Modern smartwatches have been shown to provide a powerful and intuitive means of input, thereby overcoming the limitation of the current AR/VR headsets. They typically offer a set of interesting input modalities, such as a touchscreen, rotary buttons, and an Inertial Measurement Unit (IMU), which can be used for mid-air gesture recognition. Compared to other input devices, they have the benefit that they are hands-free as soon as the user stops interacting since they are attached to the wrist. As many concepts have been proposed, comparative evaluations of their effectiveness and user-friendliness are still rare. In this paper, we evaluate the usability of two commonly found approaches for using a smartwatch as an interaction device, specifically in immersive environments provided by AR/VR HMDs: using the physical inputs of the watch (touchscreen, rotary buttons) or mid-air gestures. We conducted a user study with 20 participants, where they tested both of the interaction methods, and we compared them in their usability and performance. Based on a prototypical AR application, we evaluated the performance and user experience of these two smartwatch-based interaction concepts. We have found that the input using a touchscreen and buttons was generally favored by the participants and led to shorter task completion times.

WVC: Towards Secure Device Paring for Mobile Augmented Reality

In mobile augmented reality applications, how to build a secure device connection between two previously unassociated devices without prior set-up is challenging, which also refers to the problem of device pairing. Most existing approaches to device pairing either have certain limits to be utilized in the environment of augmented reality or lack considerations of security issues. In this article, we design WVC, an intuitive, user-friendly, and secure device paring system for mobile augmented reality. It enables users to connect to a neighboring device by waving a finger to click towards it in the air. The system uses critical features from finger tracking trajectories to understand which device a user wants to interact with. Then, it designs key generation and correction algorithms to enhance security in device communication. In addition, we present solutions to defend against malicious attacks. We implement and test the system with 10 volunteers. The experimental results demonstrate the feasibility and effectiveness of the system under varied scenarios in which devices are close to each other at a small angle or located at different heights within 2 m .

Comparison of tandospirone and escitalopram as a symptomatic treatment in Multiple System Atrophy-cerebellar ataxia: An open-label, non-controlled, 4 weeks observational study

BackgroundMultiple system atrophy (MSA) is a neurodegenerative disorder characterized by autonomic failure and motor dysfunction in parkinsonism and/or cerebellar ataxia. Patients with MSA usually present with depression and anxiety symptoms. This observational study of patients with MSA-cerebellar subtype (MSA-C) with subthreshold depression/anxiety symptoms aimed to compare the efficacy of escitalopram oxalate (an antidepressant drug) and tandospirone citrate (an anxiolytic drug). MethodsFifty-six MSA-C patients were included, with 28 patients in each treatment group. One group received escitalopram oxalate 10 mg/day and the other group received tandospirone citrate 30 mg/day. The patients were evaluated at baseline and after 4 weeks. Several psychiatric and neurological tests were performed, including the Hamilton Anxiety Rating Scale (HAMA), Hamilton Depression Rating Scale (HAMD), Scale for the Assessment and Rating of Ataxia (SARA), and the Scale for Outcomes in Parkinson's Disease for Autonomic Symptoms (SCOPA-AUT). Furthermore, post-void residual urine volume (PVR) and blood pressure were measured. ResultsThere was a more substantial reduction in the HAMA/HAMD, scores of stance, finger tracking, and finger nose test in the SARA, and PVR in the tandospirone group. There was a more substantial reduction in scores of dysuria, light-headed when standing up, syncope and hyperhidrosis in the SCOPA-AUT in the escitalopram group (p's < 0.05). ConclusionsTandospirone citrate was more effective in improving depression/anxiety and some cerebellar ataxia symptoms, whereas escitalopram was more effective in improving some autonomic symptoms in MSA-C patients over a short-term period in an open-label observational study without a control group. Further research is needed to evaluate the long-term effects of tandospirone and escitalopram in MSA-C in long-term placebo controlled trials.