Gesture Commands Research Articles

Usability Assessment of Voice-Enabled Technologies for Users with Visual Disabilities

Voice-enabled technologies such as VoiceOver (screen reader) and the Seeing AI app (image recognition) have revolutionized daily tasks for people with visual disabilities, fostering greater independence and information access. However, a gap remains in understanding the user experience (UX) of these technologies. This study investigated how those with visual disabilities interacted with VoiceOver and the Seeing AI app. A convenience sample of eight participants with visual disabilities engaged in direct observations while using these technologies. The study utilized the System Usability Scale (SUS) to assess perceived usability and analyzed findings using descriptive statistics. Results indicated a poorer UX with VoiceOver compared to the Seeing AI app, with challenges identified in graphical user interfaces (GUIs), voice and gesture commands. Relevant recommendations were made to enhance usability. The study emphasizes the need for more intuitive GUIs and optimized voice/gesture interactions for users with visual disabilities.

Diver-robot communication dataset for underwater hand gesture recognition

In this paper, we present a dataset of diving gesture images used for human–robot interaction underwater. By offering this open access dataset, the paper aims at investigating the potential of using visual detection of diving gestures from an autonomous underwater vehicle (AUV) as a form of communication with a human diver. In addition to the image recording, the same dataset was recorded using a smart gesture recognition glove. The glove uses dielectric elastomer sensors and on-board processing to determine the selected gesture and transmit the command associated with the gesture to the AUV via acoustics. Although this method can be used under different visibility conditions and even without line of sight, it introduces a communication delay required for the acoustic transmission of the gesture command. To compare efficiency, the glove was equipped with visual markers proposed in a gesture-based language called CADDIAN and recorded with an underwater camera in parallel to the glove’s onboard recognition process. The dataset contains over 30,000 underwater frames of nearly 900 individual gestures annotated in corresponding snippet folders. The dataset was recorded in a balanced ratio with five different divers in sea and five different divers in pool conditions, with gestures recorded at 1, 2 and 3 metres from the camera. The glove gesture recognition statistics are reported in terms of average diver reaction time, average time taken to perform a gesture, recognition success rate, transmission times and more. The dataset presented should provide a good baseline for comparing the performance of state of the art visual diving gesture recognition techniques under different visibility conditions.

Gradiently Foaming Ultrasoft Hydrogel with Stop Holes for Highly Deformable, Crack-Resistant and Sensitive Conformal Human-Machine Interfaces.

Hydrogels are considered as promising materials for human-machine interfaces (HMIs) owing to their merits of tailorable mechanical and electrical properties; nevertheless, it remains challenging to simultaneously achieve ultrasoftness, good mechanical robustness and high sensitivity, which are the pre-requisite requirements for wearable sensing applications. Herein, for the first time, this work proposes a universal phase-transition-induced bubbling strategy to fabricate ultrasoft gradient foam-shaped hydrogels (FSHs) with stop holes for high deformability, crack-resistance and sensitive conformal HMIs. As a typical system, the FSH based on polyacrylamide/sodium alginate system shows an ultralow Young's modulus (1.68kPa), increased sustainable strain (1411%), enhanced fracture toughness (915.6 Jm-2 ), improved tensile sensitivity (21.77), and compressive sensitivity (65.23 kPa-1 ). The FSHs are used for precisely acquiring and identifying gesture commands of the operator to remotely control a surgical robot for endoscopy and an electric ship in a first-person perspective for cruising, feeding crabs and monitoring the environmental change in real-time.

Designing a 3D gestural interface to support user interaction with time-oriented data as immersive 3D radar charts

The design of intuitive three-dimensional user interfaces is vital for interaction in virtual reality, allowing to effectively close the loop between a human user and the virtual environment. The utilization of 3D gestural input allows for useful hand interaction with virtual content by directly grasping visible objects, or through invisible gestural commands that are associated with corresponding features in the immersive 3D space. The design of such interfaces remains complex and challenging. In this article, we present a design approach for a three-dimensional user interface using 3D gestural input with the aim to facilitate user interaction within the context of Immersive Analytics. Based on a scenario of exploring time-oriented data in immersive virtual reality using 3D Radar Charts, we implemented a rich set of features that is closely aligned with relevant 3D interaction techniques, data analysis tasks, and aspects of hand posture comfort. We conducted an empirical evaluation (n=12)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(n=12)$$\\end{document}, featuring a series of representative tasks to evaluate the developed user interface design prototype. The results, based on questionnaires, observations, and interviews, indicate good usability and an engaging user experience. We are able to reflect on the implemented hand-based grasping and gestural command techniques, identifying aspects for improvement in regard to hand detection and precision as well as emphasizing a prototype’s ability to infer user intent for better prevention of unintentional gestures.

ThumbUp: Secure Smartwatch Controller for Smart Homes Using Simple Hand Gestures

The development of creative applications and intelligent gadgets requires a secure and straightforward interface with human users. We propose, design, and implement ThumbUp, a smartwatch-based two-factor real-time identification and authentication system in which smartwatch users can identify and authenticate themselves using some simple hand and finger movements, such as thumb-up. ThumbUp leverages the signal from the Inertial Measurement Unit (IMU) in in Commercial-Off-The-Shelf (COTS) smart devices to discover the unique pattern generated by each user's simple gestures using a carefully constructed deep learning model. Smart homes provide a comfortable, safe, and efficient living environment, epecially help the sick and aged. We propose strategies for convenient and reliable control in smart homes with gesture command recognition. We build an Auto-Encoder-based filter that reconstructs the raw data to improve the representation of gesture features. Moreover, we adopt the random forest method to analyse the contextual command correlation. And we employ the authentication system based on smartwatch for personalized command feedback and ensure that illegals cannot use the device. We implement our system and undertake rigorous studies to determine its usefulness and efficiency over a three-month period with 65 users. It achieves a 97% accuracy for user classification and an EER of 0.014 for authentication task with a single simple gesture. And our method achieves 91% accuracy for command recognition and 96% command accuracy with contextual informations. Additionally, we conduct a study of user acceptability of our system and explain how gesture proficiency influences authentication accuracy.

Development of Hand Gesture Command Using Leap Motion for Floorplanner Application

Architecture is part of the art of life and works best when it seems to give expression to the life that inhabits it. As technology has developed a lot, the field of architecture has also experienced many developments supported by technology such as making architectural designs. The development experienced in making architectural designs is that it is no longer required to draw design choices with side views on a lot of paper but instead use a simulation application for making architectural designs that are displayed through a monitor layer in 3D so that they can see more realistically. After many innovations emerged regarding simulation applications which generally still use hardware such as a mouse, then these innovations shifted to connecting the simulation application with the virtual world where hardware such as a mouse has been replaced by using gestures or limb movements such as parts of the hand or other parts of the body. While other fields like medic already use virtual tool interaction, architecture fields still just use mouse and keyboard. One of the virtual interaction devices used to be able to detect fingers and palms to interact with 3D simulation applications is Leap Motion. Leap Motion is starting to be widely used to carry out various ways of interacting with hands because it is more interactive and interesting to use even many people think that it can be used for hand rehabilitation or therapy too. However, even though many innovations have developed, there is still no innovation in the use of a hand gesture command system using Leap Motion for simulation applications to design architectural designs in 3D views. Based on these things, the authors designed and developed hand gesture commands using Leap Motion for Architectural Design App. The hand gesture command system used for the Floorplanner simulation application will be tested using a system usability scale by users with architectural, informatics, and unfamiliar backgrounds. Based on the tests carried out, it can be seen that the system is running properly even though it requires getting used to the device first and getting a score from the System Usability Scale of 59,167 out of scale from 0 to 100 which means it is still accepted by users or declared as "Marginally Acceptable" category.

Developing a multifunctional wearable system consisting of a helmet and clothing for two-wheeled vehicle users' safety assistance

Purpose This study aims to develop a safe, wearable clothing system that combines visibility-enhancing and emergency–accident-responding functions for two-wheeled vehicle (TWV) users' safety assistance.Design/methodology/approach First, the wearable system (WS) allowing users to control turn signals, brake lights and emergency flasher only with head movements was developed. Second, multiconnected systems were developed between WSs and a smartphone application (AS), providing accident occurrence recognition, driving photo capture–storage and emergency notification functions. Third, usability testing in each function was performed to assess the operability of the systems.Findings The intuitive interface, which uses head movement as gesture commands, was effectively operated for controlling turn signals, brake lights and emergency flasher when driving, despite differences in user physique and boarding structure among TWVs. In addition, using Bluetooth low energy and Wi-Fi protocols simultaneously can establish automatic accident recognition–notification and driving photo capture–storage–display functions by linking two WSs with one AS.Research limitations/implications This study presents a case using relatively accessible technologies within the fashion industry to improve users' safety and provide fundamental data for convergence education for smart fashion products, highlighting the significance of this study in this convergence era.Originality/value The WSs and the AS of a TWV user visually evoke the attention of other drivers and pedestrians, reducing the risk of accidents; social contribution regarding public safety will be possible by allowing the system to autonomously inform emergencies and receive emergency medical treatment quickly when the accident occurred.

A study on the impact of theater performing arts combined with drama arts on the stage in the context of big data

Abstract In this paper, based on the law of motion, combining the spherical coordinate system representation with the drama art of action feature recognition in the process of theatrical performance, the joint points of wrist, elbow and shoulder and the corresponding feature vectors as well as the school angular velocity are calculated. The affine invariant domain of feature matching is constructed, and the algorithms of dynamic frame difference merging and dynamic detection of human-computer interaction are applied to obtain similar feature components of the recognition of body movements and gesture commands in the process of theatrical performance. The harmony factor, emotion factor and contrast factor in the stage scene are used as evaluation parameters to derive the color evaluation function and keyframes of theatrical stage lighting combined with drama art. For the research purpose, the research on the stage influence of the theatrical performance of combining drama is designed, and the example analysis of the stage influence of the theatrical performance of combining drama is analyzed by using origin software. The results show that the level of the audience’s arousal dimension is significantly higher than its level in CSP, t(1, 29)=3.959, p<0.001, upper threshold (Engagement=0.38, Arousal=0.26), lower threshold (Engagement=0.35, Arousal=0.21), i.e., the specific performance episode for the audience to bring about significant emotional relaxation. Audience ratings of CAC were generally higher than their ratings of CSP, with differences ranging from 0.2 to 1.5. The mean times that subjects were in the “S(n)=1” state in the three modes were 219.86 (SD=60.62), 287.37 (SD=80.68), and 310.7 (SD=45.83), respectively. 45.83). This study greatly improves the impact of stage art and gives a whole new life to the combination of theater performance and drama stage art.

Computer-Vision Based Gesture-Metasurface Interaction System for Beam Manipulation and Wireless Communication.

Hand gesture plays an important role in many circumstances, which is one of the most common interactive methods in daily life, especially for disabled people. Human-machine interaction is another popular research topic to realize direct and efficient control, making machines intelligent and maneuverable. Here, a special human-machine interaction system is proposed and namedas computer-vision (CV) based gesture-metasurface interaction (GMI) system, which can be used for both direct beam manipulations and real-time wireless communications. The GMI system first needs to select its working mode according to the gesture command to determine whether to perform beam manipulations or wireless communications, and then validate the permission for further operation by recognizing unlocking gesture to ensure security. Both beam manipulation and wireless communication functions are validated experimentally, which show that the GMI system can not only realize real-time switching and remote control of different beams through gesture command, but also communicate with a remote computer in real time by translating the gesture language to text message. The proposed non-contact GMI system has the advantages of good interactivity, high flexibility, and multiple functions, which can find potential applications in community security, gesture-command smart home, barrier-free communications, and so on.

Gesture Recognition and Hand Tracking for Anti-Counterfeit Palmvein Recognition

At present, COVID-19 is posing a serious threat to global human health. The features of hand veins in infrared environments have many advantages, including non-contact acquisition, security, privacy, etc., which can remarkably reduce the risks of COVID-19. Therefore, this paper builds an interactive system, which can recognize hand gestures and track hands for palmvein recognition in infrared environments. The gesture contours are extracted and input into an improved convolutional neural network for gesture recognition. The hand is tracked based on key point detection. Because the hand gesture commands are randomly generated and the hand vein features are extracted from the infrared environment, the anti-counterfeiting performance is obviously improved. In addition, hand tracking is conducted after gesture recognition, which prevents the escape of the hand from the camera view range, so it ensures that the hand used for palmvein recognition is identical to the hand used during gesture recognition. The experimental results show that the proposed gesture recognition method performs satisfactorily on our dataset, and the hand tracking method has good robustness.

Design and research of music teaching system based on virtual reality system in the context of education informatization.

Virtual Reality (VR) technology uses computers to simulate the real world comprehensively. VR has been widely used in college teaching and has a huge application prospect. To better apply computer-aided instruction technology in music teaching, a music teaching system based on VR technology is proposed. First, a virtual piano is developed using the HTC Vive kit and the Leap Motion sensor fixed on the helmet as the hardware platform, and using Unity3D, related SteamVR plug-ins, and Leap Motion plug-ins as software platforms. Then, a gesture recognition algorithm is proposed and implemented. Specifically, the Dual Channel Convolutional Neural Network (DCCNN) is adopted to collect the user's gesture command data. The dual-size convolution kernel is applied to extract the feature information in the image and the gesture command in the video, and then the DCCNN recognizes it. After the spatial and temporal information is extracted, Red-Green-Blue (RGB) color pattern images and optical flow images are input into the DCCNN. The prediction results are merged to obtain the final recognition result. The experimental results reveal that the recognition accuracy of DCCNN for the Curwen gesture is as high as 96%, and the recognition accuracy varies with different convolution kernels. By comparison, it is found that the recognition effect of DCCNN is affected by the size of the convolution kernel. Combining convolution kernels of size 5×5 and 7×7 can improve the recognition accuracy to 98%. The research results of this study can be used for music teaching piano and other VR products, with extensive popularization and application value.

Gesture Recognition for Media Interaction: A Streamlit Implementation with OpenCV and MediaPipe.

Abstract: This project aims to create a media player application that responds to hand gestures, using Python and the OpenCV library. The system taps into computer vision methods, like those used in depth-sensing cameras such as Kinect or Intel RealSense, to track and understand hand movements. It processes the depth data to extract hand features and employs machine learning (like CNNs or decision trees) to classify these into gestures. This lets the application accurately interpret user gestures and apply them to media commands—play, pause, volume, and more. All this works through a user-friendly interface that even lets users customize gestures for specific commands. The combo of OpenCV and Python enables an efficient and adaptable media control system. This fusion of computer vision and machine learning offers a seamless, natural way to navigate media playback, making it an immersive experience without needing physical controllers.

Robust and Context-Aware Real-Time Collaborative Robot Handling via Dynamic Gesture Commands

This paper studies real-time collaborative robot (cobot) handling, where the cobot maneuvers an object under human dynamic gesture commands. Enabling dynamic gesture commands is useful when the human needs to avoid direct contact with the robot or the object handled by the robot. However, the key challenge lies in the heterogeneity in human behaviors and the stochasticity in the perception of dynamic gestures, which requires the robot handling policy to be adaptable and robust. To address these challenges, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Conditional Collaborative Handling Process</i> (CCHP) to encode a context-aware cobot handling policy and a procedure to learn such policy from human-human collaboration. We thoroughly evaluate the adaptability and robustness of CCHP and apply our approach to a real-time cobot assembly task with Kinova Gen3 robot arm. Results show that our method leads to significantly less human effort and smoother human-robot collaboration than state-of-the-art rule-based approach even with first-time users.

User-defined mid-air gestures for multiscale GIS interface interaction

ABSTRACT Multisize geographic information system (GIS) has widely used in social life, and the introduction of natural interactive methods such as gesture interaction further broadens the applications of GIS. Gestures, as a natural means of expression, bring a convenient experience to the use of GIS. The user-elicitation method can determine the user’s original operational intention and take advantage of the natural and convenient gesture interaction. However, current research on elicitation methods mainly focuses on defining the function factors of operation commands and rarely considers the influence of environmental factors such as spatial sizes. This paper takes GIS as the object, which is closely related to multisize space. This paper reports a research project on user-defined gestures for interacting with two-dimensional maps in a virtual reality environment, including three types of GIS operation commands and six sizes of maps. The results showed that the user-defined gestures for the same GIS operation commands varied across map sizes. The increase in map size increased the amount of gestural movement. Drawing on our research, we developed a set of gestures for multisize GIS interface interaction. We hope that this research can provide a guiding foundation for mid-air gesture design for all multisize interfaces.

Digital Art Design System Based on Big Data Technology and Interactive Virtual Technology

With the development of modern information technology and the rapid popularization of computers, the society has entered a comprehensive digital era. Digital art design technology with computer technology as an auxiliary tool has developed rapidly and has been applied in many fields of society. At the same time, digital art design education has developed into a new research hotspot in the field of education. It changes the appearance and connotation of modern display design. In order to better develop digital technology and apply modern technology to the field of art design, this article designs a digital art design system based on big data technology and interactive virtual technology. In the method part, this article introduces big data technology, interactive virtual technology, digital art design and 3ds file format, and introduces the related content of SVM support vector machine in the algorithm. In the part of the research results, this article analyzes several aspects of time-consuming, comparison of accuracy of different methods, the relationship between speedup and data volume, average CPU utilization, gesture recognition, and recognition of various functions. From the test results, in the three functions of gesture action, voice command, and keyboard command, the keyboard command has the best effect, reaching 100% accuracy. The gesture command and voice command have a small margin of error, but the accuracy rate also kept at around 90%. It can be seen that the system designed in this article has a good recognition effect in terms of interaction, and has certain significance for the realization of virtualization, informatization, and digital transformation of artistic design.

A brain-inspired intention prediction model and its applications to humanoid robot.

With the development of artificial intelligence and robotic technology in recent years, robots are gradually integrated into human daily life. Most of the human-robot interaction technologies currently applied to home service robots are programmed by the manufacturer first, and then instruct the user to trigger the implementation through voice commands or gesture commands. Although these methods are simple and effective, they lack some flexibility, especially when the programming program is contrary to user habits, which will lead to a significant decline in user experience satisfaction. To make that robots can better serve human beings, adaptable, simple, and flexible human-robot interaction technology is essential. Based on the neural mechanism of reinforcement learning, we propose a brain-inspired intention prediction model to enable the robot to perform actions according to the user's intention. With the spike-timing-dependent plasticity (STDP) mechanisms and the simple feedback of right or wrong, the humanoid robot NAO could successfully predict the user's intentions in Human Intention Prediction Experiment and Trajectory Tracking Experiment. Compared with the traditional Q-learning method, the training times required by the proposed model are reduced by (N 2 - N)/4, where N is the number of intentions.

SMART WHEEL CHAIR USING IOT

According to one study, there are about 6 million people in the world who are paralyzed and need a wheelchair to move. Wheelchairs had to be moved and supported by in the existing system. However, in normal use, these joystick-controlled wheelchairs are difficult to operate. Especially for paralyzed people, the one-way operation of the hard button and joystick made it difficult to operate the joystick. To overcome these problems, the proposed Smart wheelchair using IOT can be moved with a slight tilt of the hand and is also implemented in voice control and Android applications. This project works perfectly with user-specified voice and gesture commands, and attention is also paid to human stability. This technology is also based on wireless technology, which also saves wiring costs. output from the wheelchair's designed gesture and voice controls and obstacle detection.

ASSISTIVE INTERACTION USING GESTURES AND VOICE COMMANDS

To advance is to be human, it's our tendency to explore and move forward and about 50% of the advancements till date have been around the area of communication and accessibility. In the era of keyboards, and keypads, touch screens were a dream, but the human race has gone to those lengths of advancements, such that touch screens are almost a common entity. Now, in the futuristic perspective, voice control and gestures are perceived to be the future! Getting our work done and conveying our messages without even touching. Additionally, while the rest of the world is advancing so much, there ought to be a solution for the disabled community around us, why should they be left behind? With this severe pandemic, we have learnt what goes behind the "touch" and "physical contact", the world has not stopped, no matter what, but temporary solutions where searched and implemented, why not bring out a permanent technology to avoid touch, where it is necessary? Applications of voice and gesture commands are innumerable, from watches to our houses, it has applications everywhere.

Underwater Accompanying Robot Based on SSDLite Gesture Recognition

Underwater robots are often used in marine exploration and development to assist divers in underwater tasks. However, the underwater robots on the market have some problems, such as only a single function of object detection or tracking, the use of traditional algorithms with low accuracy and robustness, and the lack of effective interaction with divers. To this end, we designed a type of gesture recognition based on interaction, using person tracking as an auxiliary means for an underwater accompanying robot (UAR). We train and test the SSDLite detection algorithm using the self-labeled underwater datasets, and combine the kernelized correlation filters (KCF) tracking algorithm with the “Active Control” target tracking rule to continuously track the underwater human body. Our experiments show that the use of underwater datasets and target tracking can effectively improve gesture recognition accuracy by 40–105%. In the outfield experiment, the performance of the algorithm was good. It achieved target tracking and gesture recognition at 29.4 FPS on Jetson Xavier NX, and the UAR made corresponding actions according to the diver gesture command.

Virtual Interface Technology in the Process of Simulation of Complex Functional Modules of Control Systems for Industrial Robots and Multi-Axis Mechatronic Systems

Purpose of research. Development of a tool for debugging intelligent control system algorithms, including the development of a vision system and planning a software trajectory for an industrial robot.Methods. To achieve this goal, a review of existing simulation tools was carried out. A protocol of contactless humanrobot interaction is presented. An algorithm for the recognition of gesture commands based on the difference of three-dimensional binary microblocks and the construction of the skeleton of the human body has been developed. An example of using the ROBOGuid software tool for imitating the motion of an industrial robot in the development and debugging of its own control methods focused on real objects is presented.Results. The use of digital twins of technological equipment to simulate and display real technological processes in a virtual environment, in the context of the formation of a new concept of Industry 4.0 and the sixth technological order, allows improving the main and auxiliary production processes, as well as analyzing, researching and evaluating the economic efficiency of new technological and technical solutions. Simulation allows the development of ergonomic ways of human interaction with mechatronic objects. The solution proposed in the work was tested on the example of working out a complex spatial contour that simulates the milling of a part. Experimental studies of the gesture command recognition algorithm proposed in the work were carried out on the publicly available UCF101 dataset, the results are compared with known approaches to recognizing human actions.Conclusion. The developed interface module was used on the example of working out a complex spatial contour that simulates the milling of a part, and the method of a contactless robot control system has shown its effectiveness and the need to develop this direction.