Hand Detection Research Articles

Detection of Hand Poses with a Single-Channel Optical Fiber Force Myography Sensor: A Proof-of-Concept Study

Force myography (FMG) detects hand gestures based on muscular contractions, featuring as an alternative to surface electromyography. However, typical FMG systems rely on spatially-distributed arrays of force-sensing resistors to resolve ambiguities. The aim of this proof-of-concept study is to develop a method for identifying hand poses from the static and dynamic components of FMG waveforms based on a compact, single-channel optical fiber sensor. As the user performs a gesture, a micro-bending transducer positioned on the belly of the forearm muscles registers the dynamic optical signals resulting from the exerted forces. A Raspberry Pi 3 minicomputer performs data acquisition and processing. Then, convolutional neural networks correlate the FMG waveforms with the target postures, yielding a classification accuracy of (93.98 ± 1.54)% for eight postures, based on the interrogation of a single fiber transducer.

Wearable Pressure Sensor Array with Layer-by-Layer Assembled MXene Nanosheets/Ag Nanoflowers for Motion Monitoring and Human-Machine Interfaces.

Recently, wearable sensors and electronic skin systems have become prevalent, which can be employed to detect the movement status and physiological signals of wearers. Here, a pressure sensor composed of mesh-like micro-convex structure polydimethylsiloxane (PDMS), MXene nanosheet/Ag nanoflower (AgNF) films, and flexible interdigital electrodes was designed by layer-by-layer (LBL) assembly. The unique microstructure of PDMS effectively increases the contact area and improves sensitivity. Moreover, AgNFs were introduced into the MXene as a "bridge," and the synergistic effect of the two further enhanced the performance of the sensor. The pressure sensor has high sensitivity (191.3 kPa-1), good stability (18,000 cycles), fast response/recovery time (80 ms/90 ms), and low detection limit (8 Pa), so it can be used for all-round monitoring of the human body. Sensing arrays were integrated with a wireless transmitter as an intelligent artificial electronic skin for spatial pressure mapping and human-computer interaction sensing. Moreover, we develop a smart glove by a simple method, combining it with a 3D model for wireless accurate detection of hand poses. This provides ideas for hand somatosensory detection technology, leading to health monitoring, intelligent rehabilitation training, and personalized medicine.

Improved 3D-ResNet sign language recognition algorithm with enhanced hand features

In sign language video, the hand region is small, the resolution is low, the motion speed is fast, and there are cross occlusion and blur phenomena, which have a great impact on sign language recognition rate and speed, and are important factors restricting sign language recognition performance. To solve these problems, this paper proposes an improved 3D-ResNet sign language recognition algorithm with enhanced hand features, aiming to highlight the features of both hands, solve the problem of missing more effective information when relying only on global features, and improve the accuracy of sign language recognition. The proposed method has two improvements. Firstly, the algorithm detects the left and right hand regions based on the improved EfficientDet network, uses the improved Bi-FPN module and dual channel and spatial attention module are used to enhance the detection ability of the network for small targets like hand. Secondly, the improved residual module is used to improve the 3D-ResNet18 network to extract sign language features. The global, the left-hand and the right-hand image sequences are divided into three branches for feature extraction and fusion, so as to strengthen the attention to hand features, strengthen the representation ability of sign language features, and achieve the purpose of improving the accuracy of sign language recognition. In order to verify the performance of this algorithm, a series of experiments are carried out on CSL dataset. For example, in the experiments of hand detection algorithm and sign language recognition algorithm, the performance indicators such as Top-N, mAP, FLOPs and Parm are applied to find the optimal algorithm framework. The experimental results show that the Top1 recognition accuracy of this algorithm reaches 91.12%, which is more than 10% higher than that of C3D, P3D and 3D-ResNet basic networks. From the performance indicators of Top-N, mAP, FLOPs, Parm and so on, the performance of the algorithm in this paper is better than several algorithms in recent three years, such as I3D+BLSTM, B3D ResNet, AM-ResC3D+RCNN and so on. The results show that the hand detection network with enhanced hand features and three-dimensional convolutional neural network proposed in this paper can achieve higher accuracy of sign language recognition.

Using Computer Vision and Natural Language Processing Techniques to Curate and Characterize Open Surgical Videos on YouTube

INTRODUCTION: YouTube, the online multimedia platform, remains an untapped resource for surgical procedure content for education and research. Recent efforts to identify this content have lacked the diversity, volume and depth of the expanding corpus of surgical content on the site. We leverage computer vision (CV) and natural language processing (NLP) techniques to index and characterize the vast amount of open, non-minimally invasive surgical content on YouTube. METHODS: Twenty-three surgical procedure terms were used to programmatically query and collect videos, resulting in 9,197 results. Video-level metadata from YouTube were collected. Using a CV model and a NLP tool developed by our group, each video was analyzed for detailed scene understanding (e.g., action, tool, and hand detection) and medical and procedure tags from the Unified Medical Language System (UMLS) were generated. RESULTS: We identified 477, 270, and 556 videos related to head and neck, breast, and gastrointestinal surgery, respectively. 10.1 million frames were analyzed, and 11.9, 3.9, and 5.6 million, hands, surgical tools, and instances of surgical activity, respectively, were detected. 59,776 UMLS tags were also generated from video metadata. UMLS tags and temporal, “surgical signatures” were generated for each type and category of surgical procedures (Fig. 1).Figure 1CONCLUSION: Using novel CV and NLP techniques, we efficiently identified and characterized a large amount of open surgical videos in an automated way that exceeds previous efforts in scope, volume and depth. This technique enables the creation of a continuously expanding and comprehensive, surgical video repository to enhance surgical education and research.

AI Virtual Mouse Using Hand Gesture Recognition

The mouse is one of HCI's (Human Computer Interaction) incredible inventions. A wireless mouse or a Bluetooth mouse still uses devices because a battery is needed for power and a dongle is needed to connect the mouse to the PC.Therefore, they are not totally device-free. The proposed AI virtual mouse system can solve this problem by using a webcam or built-in camera to record hand motions and identify hand tips using computer vision. The system's algorithm makes use of the machine learning algorithm. The computer can be controlled digitally and cando left-click, right-click, scrolling, and computer cursor functions based on hand motions without the need for a physical mouse. Deep learning is the basis for the hand detection method. As a result, by eliminating human contact and the requirement for external devices to operate the computer, the suggested strategy will stop the spread of COVID-19.

Recognize The Alphabet of Fingerspelling Using Statistical Classifiers to Facilitate Communication Between Hearing-Impaired Persons and Others

The predominant method of communication for hearing-impaired and deaf people is still sign language. It is a carefully constructed hand gesture language, and each motion denotes a certain meaning. The purpose of this paper is to create a system for Arabic Sign Language automatic translation. The proposed Arabic sign to Text System consists of five primary stages and serves as a translator for deaf and dumb persons and normal people to improve communication. This system depends on building a two datasets image features for Arabic sign language gestures alphabets from two resources: the Arabic Sign Language dictionary and gestures from different signers' humans. It also uses gesture recognition techniques, which lets the user interact with the outside world. Video and images capture, Video and images processing, Hand Signs construction, classification, and finally text transformation and interpretation. In this paper, we use a set of appropriate features in step hand sign construction and classification of based on different classification algorithms such as (KNN, MLP and C4.5) and compare these results to get better classifier. This system offers a novel technique of hand detection that detects and extracts hand gestures of Arabic Sign from Image or Video..

SMART DUST BIN

In our project “SMART DUSTBIN” the thought comes for up Smart cities. The main aim of our project Smart Dustbin is to keep clean buildings, Colleges, Healing centers and Transport and society. The Smart Dustbin is a way of an improvement of ordinary dustbin by elevating it by utilizing sensors. Smart Dustbin is a thought of implementation which makes a typical dustbin smart and modern way of utilizing ultrasonic sensors for hand detection and to indicate message to the control room over the status for utilizing GSM modem. The LED light is utilized to show the level of garbage collected. The LED light appears when the bin is empty, the bin is partially filled and the bin is completely filled. Hence this “SMART DUSTBIN” makes a big difference to keep the city clean by indicating the level and sending an alert to the garbage management system.

Combination of Deep Cross-Stage Partial Network and Spatial Pyramid Pooling for Automatic Hand Detection

The human hand is involved in many computer vision tasks, such as hand posture estimation, hand movement identification, human activity analysis, and other similar tasks, in which hand detection is an important preprocessing step. It is still difficult to correctly recognize some hands in a cluttered environment because of the complex display variations of agile human hands and the fact that they have a wide range of motion. In this study, we provide a brief assessment of CNN-based object identification algorithms, specifically Densenet Yolo V2, Densenet Yolo V2 CSP, Densenet Yolo V2 CSP SPP, Resnet 50 Yolo V2, Resnet 50 CSP, Resnet 50 CSP SPP, Yolo V4 SPP, Yolo V4 CSP SPP, and Yolo V5. The advantages of CSP and SPP are thoroughly examined and described in detail in each algorithm. We show in our experiments that Yolo V4 CSP SPP provides the best level of precision available. The experimental results show that the CSP and SPP layers help improve the accuracy of CNN model testing performance. Our model leverages the advantages of CSP and SPP. Our proposed method Yolo V4 CSP SPP outperformed previous research results by an average of 8.88%, with an improvement from 87.6% to 96.48%.

DEAF-BSL: Deep lEArning Framework for British Sign Language recognition

The recent development of disability studies in academic bodies has expedited the promotion of investigation on disability. With computer-aided tools, communication between the impaired person and someone who does not understand sign language could be accessible. A large number of people across the world are using sign language (e.g., British Sign Language (BSL) , Asian Sign Language (ASL) , Indian Sign Language (ISL) , etc.) with hand gestures for communication. In BSL recognition, the involvement of both hands overlapping each other becomes the main challenge. Moreover, BSL comprises ambiguous signs concerning viewpoint. However, existing traditional techniques seem in-stable, less accurate, and inefficient. In this work, the BSL fingerspelling alphabet recognition problem explores using a Deep learning framework to address the above-mentioned concerns. Convolutional Neural Network (CNN) is employed to detect and recognize for classification of 26 alphabets after being trained on the BSL corpus dataset. The proposed work outperforms the existing works with better precision (6%), recall (4%), and F-measure (5̃%). It reported better results on the BSL corpus dataset and webcam videos. The model achieved better accuracy (98.0%) for a large lexicon of words than previous models (Goh & Holden [ 6 ]: 69.5%, Rambhau [ 9 ]: 79.2%, and Liwicki et al. [ 8 ]: 92.5%). The 3D CNN-based proposal performs robust hand detection, much more accurate sign recognition, more scalability, and less ambiguity in BSL finger-spelling recognition.

A Survey on Machine Learning Based Techniques for Sign Language Translation System

Abstract: Sign language is the communication method used to bridge the gap among mute and deaf people and the rest of the people. But most people are not familiar with sign language, so communication becomes difficult among mute, deaf and the rest of the people. Improvement in machine learning technology has led to research on its application in sign language recognition and translation system. This paper presents a review of different sign language recognition and translation systems and the various techniques like classification algorithms that were implemented in them. We have found that although many sign language recognition and translation system exist none of them provides efficient and real-time conversation of sign language into audio and many sign language recognition and translation system are hand movement sensor-based which are not affordable to everyone. So, we have also discussed a sign language translation system integrated into a video calling application, in which we will be using a single shot multibox detector for hand detection, for feature extraction Inception V3 will be used, and support vector machines for classification, attention network for generating semantically correct sentences and a text to speech synthesizer for audio output. This system will be easy to access and affordable as it use

A robust context attention network for human hand detection

Current state-of-the-art approaches for human hand detection have achieved great success by making good use of multiscale and contextual information but still remain unsatisfactory for hand detection, especially in complex scenarios. The main reason is that there are some parts similar to human hands, such as wrists, faces and feet. Simply using contextual information makes it difficult to address these problems. In this paper, we propose a Context Attention Feature Pyramid Network (CA-FPN) for human hand detection. In this method, a novel Context Attention Module (CAM) is inserted into the feature pyramid networks. The CAM is designed to capture relative contextual information for hands and build long-range dependencies around hands. Our CA-FPN can achieve state-of-the-art results on two public hand detection datasets: the Oxford and Vision for Intelligent and Applications (VIVA) datasets. Furthermore, the inference time of our CA-FPN is approximately 8.5 FPS on one TITAN X GPU, indicating that it can be used in real-time applications. Besides, the CAM helps improve head detection on the HollywoodHeads dataset, demonstrating its robustness in other detection tasks. The code has been made available at https://github.com/IC-LAB/CA-FPN.

Human Skin Tone Detection

Human skin tone variation can be easily seen from person to person within the same ethnic group or different ethnic group. The skin tone is analysed by melanosome size, significant and progressive variation in size and ethnicity. In face recognition the shape of the face gets identified but background used in real time is usually complex with varying lighting conditions. Automatic skin detection has been intensively studied for human-related recognition systems. The Gaussian model for skin detection is with its generality but they lack in their accuracy. Hence we suggest a new statistical colour model for skin detection called an elliptical boundary model. Detection and tracking of human face and hands are used for gesture recognition and to handle the domain transfer problem. The elliptical boundary model overcomes the limitations of the Gaussian model with better performance based on six chrominance spaces on the face, giving a much higher correct detection ratio with faster speed. For that, we are processing with two main domains that are Full stack development which handles the frontend and backend of the application and Machine Learning algorithms which handles the detection of the skin tone. Here we are dealing with image processing techniques for input and the output will find the skin tone of that particular image that we are processing. The applications that involve skin tone detection are face detection, gesture recognition and image filtering. Hence they overcome the different illumination conditions as different domains of skin pixels. The objective of the project is to detect the skin tone from a human face image with various tone characteristics. We are detecting the basic tone differentiation (Fair, Mild or Dark).

Depth Estimation for Egocentric Rehabilitation Monitoring Using Deep Learning Algorithms

Upper limb impairment is one of the most common problems for people with neurological disabilities, affecting their activity, quality of life (QOL), and independence. Objective assessment of upper limb performance is a promising way to help patients with neurological upper limb disorders. By using wearable sensors, such as an egocentric camera, it is possible to monitor and objectively assess patients’ actual performance in activities of daily life (ADLs). We analyzed the possibility of using Deep Learning models for depth estimation based on a single RGB image to allow the monitoring of patients with 2D (RGB) cameras. We conducted experiments placing objects at different distances from the camera and varying the lighting conditions to evaluate the performance of the depth estimation provided by two deep learning models (MiDaS & Alhashim). Finally, we integrated the best performing model for depth-estimation (MiDaS) with other Deep Learning models for hand (MediaPipe) and object detection (YOLO) and evaluated the system in a task of hand-object interaction. Our tests showed that our final system has a 78% performance in detecting interactions, while the reference performance using a 3D (depth) camera is 84%.

Open surgery tool classification and hand utilization using a multi-camera system.

The goal of this work is to use multi-camera video to classify open surgery tools as well as identify which tool is held in each hand. Multi-camera systems help prevent occlusions in open surgery video data. Furthermore, combining multiple views such as a top-view camera covering the full operative field and a close-up camera focusing on hand motion and anatomy may provide a more comprehensive view of the surgical workflow. However, multi-camera data fusion poses a new challenge: A tool may be visible in one camera and not the other. Thus, we defined the global ground truth as the tools being used regardless their visibility. Therefore, tools that are out of the image should be remembered for extensive periods of time while the system responds quickly to changes visible in the video. Participants (n = 48) performed a simulated open bowel repair. A top-view and a close-up cameras were used. YOLOv5 was used for tool and hand detection. A high-frequency LSTM with a 1-second window at 30 frames per second (fps) and a low-frequency LSTM with a 40-second window at 3 fps were used for spatial, temporal, and multi-camera integration. The accuracy and F1 of the six systems were: top-view (0.88/0.88), close-up (0.81,0.83), both cameras (0.9/0.9), high-fps LSTM (0.92/0.93), low-fps LSTM (0.9/0.91), and our final architecture the multi-camera classifier(0.93/0.94). Since each camera in a multi-camera system may have a partial view of the procedure, we defined a 'global ground truth.' Defining this at the data labeling phase emphasized this requirement at the learning phase, eliminating the need for any heuristic decisions. By combining a system with a high fps and a low fps from the multiple camera array, we improved the classification abilities of the global ground truth.

Rancang Bangun Alat Deteksi Masker, Deteksi Suhu Badan dan Sprayer Disinfektan Sebagai Pendisiplin Protokol Kesehatan Di Lingkungan Sekolah

COVID-19 has swept the world for almost two years. The increase in COVID-19 cases is caused by the lack of public compliance with health protocols. Schools are places where teachers and students gather; some schools in certain zones are permitted by the government to carry out teaching and learning activities face-to-face but still under strict health protocols. Parents are still worried that their children will catch COVID-19 at school because of the school community's neglect of health protocols; robots play an important role in reducing the chain of COVID-19 cases. Through this research, the authors innovate by using cameras to help monitor school residents to comply with health protocols in wearing masks, infrared sensors for hand detection when washing hands and MLX90614 sensors to measure body temperature so that those who do not adhere to the health protocol disciplinary tools are not allowed to enter the school area.

Implementation Details for Controlling Contactless 3D Virtual Endoscopy

In the medical world, with the innovative application of medical informatics, it is possible to enable many aspects of surgeries that were not able to be addressed before. One of these is contactless surgery planning and controlling the visualization of medical data. In our approach to contactless surgery, we adopted a new framework for hand and motion detection based on augmented reality. We developed a contactless interface for a surgeon to control the visualization options in our DICOM (Digital Imaging and Communications in Medicine) viewer platform that uses a stereo camera as a sensor device input that controls hand/finger motions, in contactless mode, and applied it to 3D virtual endoscopy. In this paper, we will present our proposal for defining motion parameters in contactless, incisionless surgeries. We enabled better surgeon’s experience, more precise surgery, real-time feedback, depth motion tracking, and contactless control of visualization, which gives freedom to the surgeon during the surgery. We implemented motion tracking using stereo cameras with depth resolution and precise shutter sensors for depth streaming. Our solution provides contactless control with a range up to 2–3 m that enables the application in the operating room.

The Classification of Abnormal Hand Movement to Aid in Autism Detection: Machine Learning Study

Background A formal autism diagnosis can be an inefficient and lengthy process. Families may wait several months or longer before receiving a diagnosis for their child despite evidence that earlier intervention leads to better treatment outcomes. Digital technologies that detect the presence of behaviors related to autism can scale access to pediatric diagnoses. A strong indicator of the presence of autism is self-stimulatory behaviors such as hand flapping. Objective This study aims to demonstrate the feasibility of deep learning technologies for the detection of hand flapping from unstructured home videos as a first step toward validation of whether statistical models coupled with digital technologies can be leveraged to aid in the automatic behavioral analysis of autism. To support the widespread sharing of such home videos, we explored privacy-preserving modifications to the input space via conversion of each video to hand landmark coordinates and measured the performance of corresponding time series classifiers. Methods We used the Self-Stimulatory Behavior Dataset (SSBD) that contains 75 videos of hand flapping, head banging, and spinning exhibited by children. From this data set, we extracted 100 hand flapping videos and 100 control videos, each between 2 to 5 seconds in duration. We evaluated five separate feature representations: four privacy-preserved subsets of hand landmarks detected by MediaPipe and one feature representation obtained from the output of the penultimate layer of a MobileNetV2 model fine-tuned on the SSBD. We fed these feature vectors into a long short-term memory network that predicted the presence of hand flapping in each video clip. Results The highest-performing model used MobileNetV2 to extract features and achieved a test F1 score of 84 (SD 3.7; precision 89.6, SD 4.3 and recall 80.4, SD 6) using 5-fold cross-validation for 100 random seeds on the SSBD data (500 total distinct folds). Of the models we trained on privacy-preserved data, the model trained with all hand landmarks reached an F1 score of 66.6 (SD 3.35). Another such model trained with a select 6 landmarks reached an F1 score of 68.3 (SD 3.6). A privacy-preserved model trained using a single landmark at the base of the hands and a model trained with the average of the locations of all the hand landmarks reached an F1 score of 64.9 (SD 6.5) and 64.2 (SD 6.8), respectively. Conclusions We created five lightweight neural networks that can detect hand flapping from unstructured videos. Training a long short-term memory network with convolutional feature vectors outperformed training with feature vectors of hand coordinates and used almost 900,000 fewer model parameters. This study provides the first step toward developing precise deep learning methods for activity detection of autism-related behaviors.

Hand hygiene monitoring and compliance system using convolution neural networks.

Hand hygiene monitoring and compliance systems play a significant role in curbing the spread of healthcare associated infections and the COVID-19 virus. In this paper, a model has been developed using convolution neural networks (CNN) and computer vision to detect an individual’s germ level, monitor their hand wash technique and create a database containing all records. The proposed model ensures all individuals entering a public place prevent the spread of healthcare associated infections (HCAI). In our model, the individual’s identity is verified using two-factor authentication, followed by checking the hand germ level. Furthermore, if required the model will request sanitizing/ hand wash for completion of the process. During this time, the hand movements are checked to ensure each hand wash step is completed according to World Health Organization (WHO) guidelines. Upon completion of the process, a database with details of the individual’s germ level is created. The advantage of our model is that it can be implemented in every public place and it is easily integrable. The performance of each segment of the model has been tested on real-time images an validated. The accuracy of the model is 100% for personal identification, 96.87% for hand detection, 93.33% for germ detection and 85.5% for the compliance system respectively.

Using CNN in Surveillance Videos for Recognizing Human Actions Based on Machine Learning in Examination

Video analytics is the process of processing a video, obtaining data, and analyzing the data to derive domain-specific information. In recent years, examining live surveillance films for recognizing events that occur within its coverage area has become increasingly important, in addition to analyzing any video for information retrieval. Face identification from recordings is easy when training models are used. Estimating skin colour aids hand detection. This study project aims to detect suspicious acts such as object exchange, new person admission, peeping into another's answer sheet, and person exchange using footage captured by a security camera during exam. People are paying greater attention to the fairness of exams these days, therefore detecting anomalous conduct to ensure examination order is important. Most of the current methodologies propose a fraudulent model. We extract the optical flow of video data in this system and present a 3D convolutional neural networks model to solve the problem. Face recognition, hand recognition, and detecting contact between the face and hands of the same person and that of distinct people are all required. The automation of suspicious activity detection' will aid in lowering the mistake rate associated with manual monitoring.

Fast and Accurate Hand Visual Detection by Using a Spatial-Channel Attention SSD for Hand-Based Space Robot Teleoperation

Space robot teleoperation is an important technology in the space human-robot interaction and collaboration. Hand-based visual teleoperation can make the operation more natural and convenient. The fast and accuracy hand detection is one of the most difficult and important problem in the hand-based space robot teleoperation. In this work, we propose a fast and accurate hand detection method by using a spatial-channel attention single shot multibox detector (SCA-SSD). The SSD framework is used and improved in our method by introducing spatial-channel attentions with feature fusion. To increase the restricted receptive field in shallow layers, two shallow layers are fused with deep layers by using feature fusion modules. And spatial attention and channel-wise attention are also used to extract more efficient features. This method can not only ease the computational burden but also bring more contextual information. To evaluate the effectiveness of the proposed method, experiments on some public datasets and a custom astronaut hand detection dataset (AHD) are conducted. The results show that our method can improve the hand detection accuracy by 2.7% compared with the original SSD with only 15 fps speed drops. In addition, the space robot teleoperation experiment proves that our hand detection method can be well utilized in the space robot teleoperation system.