User's Hand Motion Research Articles

Shape-position perceptive fusion electronic skin with autonomous learning for gesture interaction

Wearable devices, such as data gloves and electronic skins, can perceive human instructions, behaviors and even emotions by tracking a hand's motion, with the help of knowledge learning. The shape or position single-mode sensor in such devices often lacks comprehensive information to perceive interactive gestures. Meanwhile, the limited computing power of wearable applications restricts the multimode fusion of different sensing data and the deployment of deep learning networks. We propose a perceptive fusion electronic skin (PFES) with a bioinspired hierarchical structure that utilizes the magnetization state of a magnetostrictive alloy film to be sensitive to external strain or magnetic field. Installed at the joints of a hand, the PFES realizes perception of curvature (joint shape) and magnetism (joint position) information by mapping corresponding signals to the two-directional continuous distribution such that the two edges represent the contributions of curvature radius and magnetic field, respectively. By autonomously selecting knowledge closer to the user's hand movement characteristics, the reinforced knowledge distillation method is developed to learn and compress a teacher model for rapid deployment on wearable devices. The PFES integrating the autonomous learning algorithm can fuse curvature-magnetism dual information, ultimately achieving human machine interaction with gesture recognition and haptic feedback for cross-space perception and manipulation.

AI Virtual Mouse System Using Computer Vision to avoid COVID-19 spread

In the contemporary digital landscape, enhancing human-computer interaction efficiency and intuitiveness is essential. Traditional input devices like mice and keyboards are being augmented by innovative approaches such as hand gesture recognition, which provides a more natural method of interaction. This paper aims to generate a virtual mouse controlled by hand gestures using computer vision and deep learning techniques. The system employs a webcam to capture live video of the user's hand movements. These movements are analyzed using convolutional neural networks (CNNs) to identify specific gestures, which are then translated into mouse operations like cursor movement, clicking, and scrolling. This solution is hardware-independent, utilizing only the device's camera, making it accessible and straightforward to use. The goal is to create a seamless and efficient interaction method, allowing users to control their computers with simple hand gestures from a distance. Keywords: Convolutional Neural Network, Deep Learning, Hand Gesture Recognition, Virtual Mouse, Computer Vision, OpenCV

Exploring The Techniques and Applications of Hand Gesture Recognition for Human-Computer Interaction

Hand gesture controlled applications are software applications that are controlled by the user's hand movements. Hand gesture controlled applications have the potential to improve user experience by offering more intuitive and immersive ways to interact with system. Hand gesture controlled applications have a huge scope, including gaming, virtual reality, robotics, healthcare, and various other sectors. In this paper, we have combined several disparate end results of subsystems such as virtual mouse, contactless keyboard, finger counting, taking selfies, sign language, game, handling installed applications and volume control using computer vision. For implementation, we used the open-source software library OpenCV, mediapipe and pyautogui. Various hand gestures are assigned to launch and control different applications. These hand gestures are recognized using machine learning trained models, Euclidean distance formula and pixel positions. Overall, it allows humans to interact with computers through hand gestures rather than traditional input/output mechanisms such as the keyboard and mouse, improving accessibility. Key Words: Computer vision, OpenCV, hand gesture, machine learning, TensorFlow, pyautogui, mediapipe, virtual mouse, virtual keyboard, sign language, gaming.

FocalPoint

We propose FocalPoint, a direct manipulation technique in smartphone augmented reality (AR) for selecting small densely-packed objects within reach, a fundamental yet challenging task in AR due to the required accuracy and precision. FocalPoint adaptively and continuously updates a cylindrical geometry for selection disambiguation based on the user's selection history and hand movements. This design is informed by a preliminary study which revealed that participants preferred selecting objects appearing in particular regions of the screen. We evaluate FocalPoint against a baseline direct manipulation technique in a 12-participant study with two tasks: selecting a 3 mm wide target from a pile of cubes and virtually decorating a house with LEGO pieces. FocalPoint was three times as accurate for selecting the correct object and 5.5 seconds faster on average; participants using FocalPoint decorated their houses more and were more satisfied with the result. We further demonstrate the finer control enabled by FocalPoint in example applications of robot repair, 3D modeling, and neural network visualizations.

Use It-No Need to Shake It!

Implicit authentication for traditional objects, such as doors and dumbbells, has rich applications but is rarely studied. An ongoing trend is that traditional objects are retrofitted to smart environments; for instance, a contact sensor is attached to a door to detect door opening (but cannot tell "who is opening the door"). We present the first accurate implicit-authentication system for retrofitted everyday objects, named MoMatch. It makes an authentication decision based on a single natural object use, unlike prior work that requires shaking objects. MoMatch is built on the observation that an object has a motion typically because a human hand moves it; thus, the object's motion and the legitimate user's hand movement should correlate. The main challenge is, given the small amount of data collected during one object use, how to measure the correlation accurately. We convert the correlation measurement problem into an image comparison problem and resolve it using neural networks successfully. MoMatch does not need to profile the user's biometric information and is resilient to mimicry attacks.

Assessing the Cognitive Demand of Hand Controlled Exoskeleton Walking.

Individuals with spinal cord injury have motor and sensory deficits leading to ambulatory problems. Our current research is focused on developing innovative control mechanisms for wearable robotic exoskeletons to provide such users with complete control of their gait while allowing them to perform other activities (such as conversing, etc.). In this study, we evaluated the cognitive load due to using the user's hand movement to control the gait of a robot using a dual-task paradigm. The results show that there was no difference in symmetry and duty cycle between with and without a competing cognitive task, and the number of cognitive responses was similar to healthy controls walking on the treadmill. There was also no difference in obstacle navigation with and without the cognitive task. Results of this study suggest that using our control mechanisms is intuitive, easy to learn, and requires cognitive attention that is similar to normal human walking. Clinical Relevance-Initial evidence to understand the effects of the novel control mechanism on cognitive load over that of typical walking.

Gesture Controlled Robot with Robotic Arm

Abstract: In today's world, industry and daily routine tasks are increasingly attracted to and implemented through robotic automation. Well, this study proposes a method for replacing buttons and joysticks with a more novel technology, namely, commanding the entire Robotic Arm with the user's hand movement, motion, or gesture. The goal is to alter how people think about remote controllers for manually driven robotic arms. Pick and place robots are a type of technology used in the industrial industry to execute pick and place tasks. The system is constructed in such a way that it reduces human error and intervention, resulting in more exact work. The system is designed with a simple, flexible, and minimum control technique in mind. There are numerous domains in which human intervention is problematic, yet the process in question must be operated and regulated, which leads to the application of robots. According to the literature, pick and place robots are conceived and deployed in a variety of industries, including the bottle filling industry, surveillance to identify and eliminate bombs, and so on. The goal of the project is to create a gesture-controlled robot with a robotic arm that can perform any pick-and-place tasks. A radio frequency transmission is used to control the pick and place robot. For the gripper open and close, up and down, forward and backward, base clockwise and counterclockwise, the Robotic Hand has certain independent commands. Four Omni wheels support the robotic arm's displacement on the chassis. The implemented robotic arm has four degrees of freedom. Line follower, wall hugger, obstacle avoider, metal detector, and other capabilities can be added to this robot to increase its versatility. Index Terms: Pick & Place Robot, Robotic arm, Robotic Hand.

A novel hand exoskeleton to enhance fingers motion for tele-operation of a robot gripper with force feedback☆

This research describes a robotic system's design, optimization, and implementation processes for tele-operation with haptic feedback using the Hand Exoskeleton of New technologies research center (HEXON) as a novel haptic mechanism and the HEXON robotic gripper to work collaboratively with the exoskeleton. In particular, a force-sensitive robotic gripper is connected to a hand exoskeleton, mimics the user's hand movements, and transfers haptic feedback through the exoskeleton. The two-finger exoskeleton's design process consists of a multi-criteria optimization procedure to simultaneously maximize applied force to the finger and workspace of the attached exoskeleton, optimizing both the Perpendicular Impact Force (PIF) and the Global Isotropy Index (GII), considering worst-case collision avoidance. Due to the tele-operation application and using a robotic arm as a slave in the remote scenario, the gripper can be used as an end-effector for a palletizing 3-DOF robotic arm, and movement of the gripper can be achieved by using a visual sensor to pinpoint the user's hand location in 3D space. With the aim of the stress analysis software, the exoskeleton and gripper designs were analyzed to evaluate the system strength against applied forces. Finally, based on the achieved results, the gripper and two-fingered exoskeleton prototypes were fabricated. Simulations and experimental results are reported, showing the potential of the HEXON in haptics and tele-operation applications, and the exoskeleton is used to operate a robotic arm and gripper while the user can perceive the objects virtually.

Pseudo‐Hologram with Aerohaptic Feedback for Interactive Volumetric Displays

The ability to display virtual objects midair that can be seen without special headwear is an attractive proposition for the virtual reality systems of the near future. Volumetric display technologies have demonstrated this concept, allowing users to view virtual objects with real dimensions in 3D space while enabling some forms of interaction. Developments have been witnessed in haptic feedback technologies which are used to deliver a sensation of touch in the digital world. As virtual environments find more applications in today's world, haptic feedback devices have potential to integrate with virtual reality systems and contribute to a more immersive and interactive user experience. Herein, an air‐based haptic feedback device, named Aerohaptics, is presented, which delivers mid‐air tactile feedback while the user is manipulating virtual objects within a pseudo‐holographic display. The developed system constitutes a cost‐effective approach with relatively low complexity and various potential applications as no wearable or handheld peripherals are required. The delivered haptic feedback can be accurately directed at specific locations on the user's hand and its intensity can be controlled to suit various interaction scenarios. Through experiments, localized feedback on the user's fingertips as well as varying feedback force according to the user's hand movement is demonstrated.

A simulator for both manual and powered wheelchairs in immersive virtual reality CAVE

A large number of people in the world need to use a wheelchair because of different disabilities. Driving a wheelchair requires complex physical and cognitive abilities which need to be trained. Virtual training helps users acquire driving skills in a safe environment. The aim of this paper is to describe and technically validate simulation models for both manual (MW) and powered wheelchairs (PW) based on immersive virtual reality CAVE (VR). As VR system, the Gait Real-time Analysis Interactive Lab (GRAIL) was used, a CAVE equipped with a motion platform with two degrees of freedom and an optoelectronic motion capture system. A real wheelchair was positioned onto the motion platform with rear wheels free to turn in MW modality, and a commercial joystick was installed on an armrest to simulate the PW modality. Passive markers were used to track the wheel rotation, the joystick and the user hand motion. Custom D-flow applications were developed to manage virtual scene response to user actions. Overground tests, based on single wheel rotation, were performed to verify the simulation model reliability. Quantitative results demonstrated that the MW simulator kinematics was consistent with a real wheelchair overground in the absence of wheel slip and inertia (median error for MW 0.40 °, no systematic bias p = 0.943, high correlation rho > 0.999, p < 0.01). The proposed solution is flexible and adaptable to different wheelchairs, joysticks and optoelectronic systems. The main limitation is the absence of force feedback. Nevertheless, it is a reliable prototype that can be used to validate new virtual scenarios as well as for wheelchair training. The next steps include the system validation with real end users and assessment of the simulator effectiveness as a training tool.

FMG- and RNN-Based Estimation of Motor Intention of Upper-Limb Motion in Human-Robot Collaboration.

Research on human-robot interactions has been driven by the increasing employment of robotic manipulators in manufacturing and production. Toward developing more effective human-robot collaboration during shared tasks, this paper proposes an interaction scheme by employing machine learning algorithms to interpret biosignals acquired from the human user and accordingly planning the robot reaction. More specifically, a force myography (FMG) band was wrapped around the user's forearm and was used to collect information about muscle contractions during a set of collaborative tasks between the user and an industrial robot. A recurrent neural network model was trained to estimate the user's hand movement pattern based on the collected FMG data to determine whether the performed motion was random or intended as part of the predefined collaborative tasks. Experimental evaluation during two practical collaboration scenarios demonstrated that the trained model could successfully estimate the category of hand motion, i.e., intended or random, such that the robot either assisted with performing the task or changed its course of action to avoid collision. Furthermore, proximity sensors were mounted on the robotic arm to investigate if monitoring the distance between the user and the robot had an effect on the outcome of the collaborative effort. While further investigation is required to rigorously establish the safety of the human worker, this study demonstrates the potential of FMG-based wearable technologies to enhance human-robot collaboration in industrial settings.

Securing Health Monitoring via Body-Centric Time-Frequency Signature Authorization

Identity-based attacks serve as the basis of an intruder's attempt to launch security infringements in mobile health monitoring scenarios. Wireless channel perturbations due to the presence of human body are a relative phenomenon depending heavily on the subject's dielectric properties. A new body-centric signature authorization (B-CSAI) approach based on time-frequency domain characteristics was proposed. This method utilizes multiple millimeter wave bands of 27-28 GHz, 29-30 GHz, and 31-32 GHz, thereby enhancing the security in body-centric communications exploiting benefits of subject specific channel signature. The proposed bornprint method is based on the intrinsic identity-related time-frequency domain information, which generated by the user's natural hand motion signature and resulting creeping waves and space waves. It can meet the unconditional keyless authorization requirements. A detailed measurement campaign considering radiation efficiency (ñ(dB) = -25.8, -24.7, -26.4), path-loss exponent, and shadowing factor in three millimeter wave bands, using six human subjects confirm the usability and efficiency of the proposed approach. This also shows that there is a wide space for realizing security from physical mechanisms.

Natural Human–Robot Interface Using Adaptive Tracking System with the Unscented Kalman Filter

Traditional human–robot interfaces usually have limitations in accuracy and/or operational space. This article proposes a natural human–robot interface using an adaptive tracking method, which can effectively expand the operational space while ensuring high accuracy. The natural interface allows the robot to directly reproduce the user's hand movement, making the interaction more intuitive and natural. The leap motion is fixed on the Cartesian platform to capture the movement of the user's hand. Because the Cartesian platform follows the hand and keeps the hand in the center of the detection area, the measurement accuracy is improved and the measurement space can be extended. During the process of acquiring gesture data, the measurement errors were found to increase over time because of the inherent noise of the sensor. To deal with this problem, the unscented Kalman filter is applied to estimate the position of the hand. Moreover, an adaptive velocity control method is proposed to improve the operation accuracy and reduce the task execution time with the consideration of users’ habits and easiness of usage. The effectiveness of this interface is verified by a series of experiments, and the results show that the proposed interface can be used by nonprofessional users for object operation tasks and can provide users with superior interactive experiences.

An IoT-Enabled Stroke Rehabilitation System Based on Smart Wearable Armband and Machine Learning

Surface electromyography signal plays an important role in hand function recovery training. In this paper, an IoT-enabled stroke rehabilitation system was introduced which was based on a smart wearable armband (SWA), machine learning (ML) algorithms, and a 3-D printed dexterous robot hand. User comfort is one of the key issues which should be addressed for wearable devices. The SWA was developed by integrating a low-power and tiny-sized IoT sensing device with textile electrodes, which can measure, pre-process, and wirelessly transmit bio-potential signals. By evenly distributing surface electrodes over user’s forearm, drawbacks of classification accuracy poor performance can be mitigated. A new method was put forward to find the optimal feature set. ML algorithms were leveraged to analyze and discriminate features of different hand movements, and their performances were appraised by classification complexity estimating algorithms and principal components analysis. According to the verification results, all nine gestures can be successfully identified with an average accuracy up to 96.20%. In addition, a 3-D printed five-finger robot hand was implemented for hand rehabilitation training purpose. Correspondingly, user’s hand movement intentions were extracted and converted into a series of commands which were used to drive motors assembled inside the dexterous robot hand. As a result, the dexterous robot hand can mimic the user’s gesture in a real-time manner, which shows the proposed system can be used as a training tool to facilitate rehabilitation process for the patients after stroke.

Creating an interaction interface to improve user engagement in virtual reality systems

Virtual Reality (VR) has come up as a pinnacle of the ground-breaking advances in computing power through developments in fields of electronics, software and mobile computing. VR systems were first introduced to target entertainment and gaming, but finds equal importance in educational and training purposes. The system used in our experiment takes help of Google’s ‘Cardboard’ platform. Although very easy to operate, extremely cost effective and portable, it lacks a sophisticated input system to facilitate interaction between virtual world and the user. We design and implement a wearable device that enables the VR system to recognize user hand movement and gesture to allow manipulation of objects in the virtual world. We measure the effectiveness of above mentioned system by usability and acceptance testing in comparison to the existing interaction techniques. The effects on user engagement is also quantified and analysed. Through our experiments, we intend to establish that such a setup is preferred by users for regular usage and improves their level of interaction and participation.

Recognizing the Gradual Changes in sEMG Characteristics Based on Incremental Learning of Wavelet Neural Network Ensemble

Most myoelectric prosthetic hands use a fixed pattern recognition model to identify the user's hand motion commands. Since surface electromyogram (sEMG) characteristics vary with time, it is difficult to employ the fixed pattern recognition model in identifying hand motion commands stably for a long period of time. In order to adapt to the gradual changes in sEMG characteristics, we utilized incremental learning based on the wavelet neural network (WNN) ensemble, and used negative correlation learning (NCL) to train it. To verify the effect of the proposed method, a group of subjects executed six hand motions in a continual experiment for more than 2 h. Compared with the fixed pattern recognition model, the classification accuracy rate of incremental learning with nonintegration becomes substantially improved. In addition, the results of the WNN ensemble with the fixed-size mode are more stable than those of the WNN ensemble with the growth mode. The experimental results demonstrate that our method can recognize the gradual changes in sEMG characteristics stably. Using the proposed method, the average accuracy rate is found to be 92.17%, even after a long period of time. Moreover, since the update time is short, the proposed method can be successfully applied in myoelectric prosthetic hands.

Handheld free space quantum key distribution with dynamic motion compensation.

Mobile devices have become an inseparable part of our everyday life. They are used to transmit an ever-increasing amount of sensitive health, financial and personal information. This exposes us to the growing scale and sophistication of cyber-attacks. Quantum Key Distribution (QKD) can provide unconditional and future-proof data security but implementing it for handheld mobile devices comes with specific challenges. To establish security, secret keys of sufficient length need to be transmitted during the time of a handheld transaction (~1s) despite device misalignment, ambient light and user's inevitable hand movements. Transmitters and receivers should ideally be compact and low-cost, while avoiding security loopholes. Here we demonstrate the first QKD transmission from a handheld transmitter with a key-rate large enough to overcome finite key effects. Using dynamic beam-steering, reference-frame-independent encoding and fast indistinguishable pulse generation, we obtain a secret key rate above 30kb/s over a distance of 0.5m.

Package Architecture and Component Design for an Implantable Peripheral Nerve Stimulation and Recording System for Advanced Prosthetics

Abstract One of the limitation of current prosthetics is the ability to provide sensory feedback to the human user. Due to this constraint, approximately 60–80 percent of amputees experience a phenomenon known as phantom limb pain, an ongoing painful sensations that to the individual, seems to be coming from the part of the limb that is no longer there. The lack of sensory feedback also limits the intuitive control of the user's hand movement, i.e. sense of grip or position. To address these limitations, we created am implantable system that could provide peripheral nerve stimulation, recording and motor control. The architecture of our Sensory-Stimulation Lead (SSL) system consist of multiple satellites connected to Draper's custom designed nerve electrodes. In this phase of the design, the implanted system is connected to a controller via percutaneous connections. The active electronics of the satellite is enclosed in a hermetic package approximately 14mm in diameter and less than 5mm thick. A custom ceramic feedthrough substrate provides the electrical connections of the internal electronics board to both the nerve electrodes and percutaneous leads. In this paper, we will describe the various packaging components of the system and the design, fabrication, and assembly considerations that drove our technology choices.

Digital Environment for Movement Control in Surgical Skill Training.

Intelligent environments are increasingly becoming useful scenarios for handling computers. Technological devices are practical tools for learning and acquiring clinical skills as part of the medical training process. Within the framework of the advanced user interface, we present a technological application using Leap Motion, to enhance interaction with the user in the process of a laparoscopic surgical intervention and integrate the navigation through augmented reality images using manual gestures. Thus, we intend to achieve a more natural interaction with the objects that participate in a surgical intervention, which are augmented and related to the user's hand movements.

Unobtrusive Tremor Detection While Gesture Controlling a Robotic Arm

&lt;div class=""abs_img""&gt;&lt;img src=""[disp_template_path]/JRM/abst-image/00270001/12.jpg"" width=""300"" /&gt;Gesture based validation&lt;/div&gt; A light weight robotic arm (Jaco) has been interfaced with a novel gesture detection sensor (Leap Motion Controller), substituting complicated conventional input devices, i.e., joysticks and pads. Due to the enhanced precision and high throughput capabilities of the Leap Motion Controller, the unobtrusive measurement of physiological tremor can be extracted. An algorithm was developed to constantly detect and indicate potential user hand tremor patterns in real-time. Additionally a calibration algorithm was developed to allow an optimum mapping between the user hand movement, tracked by the Leap Motion Controller, and the Jaco arm, by filtering unwanted oscillations, allowing for a more natural human-computer interaction. &lt;/span&gt;