Real-time Human-computer Interaction Research Articles

Humidity Adaptive Antifreeze Hydrogel Sensor for Intelligent Control and Human-Computer Interaction.

Conductive hydrogels have emerged as ideal candidate materials for strain sensors due to their signal transduction capability and tissue-like flexibility, resembling human tissues. However, due to the presence of water molecules, hydrogels can experience dehydration and low-temperature freezing, which greatly limits the application scope as sensors. In this study, an ionic co-hybrid hydrogel called PBLL is proposed, which utilizes the amphoteric ion betaine hydrochloride (BH) in conjunction with hydrated lithium chloride (LiCl) thereby achieving the function of humidity adaptive. PBLL hydrogel retains water at low humidity (<50%) and absorbs water from air at high humidity (>50%) over the 17 days of testing. Remarkably, the PBLL hydrogel also exhibits strong anti-freezing properties (-80°C), high conductivity (8.18 Sm-1 at room temperature, 1.9 Sm-1 at -80°C), high gauge factor (GF approaching 5.1). Additionally, PBLL hydrogels exhibit strong inhibitory effects against Escherichia coli (E. coli) and Staphylococcus aureus (S. aureus), as well as biocompatibility. By synergistically integrating PBLL hydrogel with wireless transmission and Internet of Things (IoT) technologies, this study has accomplished real-time human-computer interaction systems for sports training and rehabilitation evaluation. PBLL hydrogel exhibits significant potential in the fields of medical rehabilitation, artificial intelligence (AI), and the Internet of Things (IoT).

A self-powered intelligent integrated sensing system for sports skill monitoring

The use of green intelligent sensing systems which are based on triboelectric nanogenerators have sparked a surge of research in recent years. The development has made significant contributions to the field of promoting human health. However, the integration of an intelligent sensing system with multi-directional triboelectric nanogenerators (TENGs) remains challenges in the field of motion monitoring. To solve this research issue, this study designed a self-powered multifunctional fitness blanket (SF-MFB) which incorporates four TENGs, features multi-sensors and wireless motion monitoring capabilities. It presents a self-powered integrated sensing system which utilizes four TENG sensing units to monitor human motion. Each TENG sensing unit collects the mechanical energy generated during motion. The system is composed of SF-MFB, Bluetooth transmission terminal, and upper computer analysis terminal. Its main purpose is to wirelessly monitor and diagnose human sports skills and enables real-time human–computer interaction. The TENG integrated self-powered sensing system demonstrates practicality in sports skills monitoring, diagnosis, human–computer interaction and entertainment. This research introduces a novel approach for the application of TENG self-powered intelligent integrated sensing system in health promotion.

Dual-silane coupling agents co-grafted on black particles for fast response and high contrast ratio electrophoretic display

As the Internet of Things (IoT) significantly improves people's lifestyles, flexible and low-power displays will become essential to real-time human-computer interaction. Electrophoretic display (EPD) has many advantages, such as low power consumption, flexibility, and high sunlight visibility, which is necessary for IoT terminal display. However, preparing electrophoretic particles with high charge and spatial resistance is difficult. Herein, dual-silane coupling agents, (3-Aminopropyl) trimethoxysilane (APS) and vinyltrimethoxysilane (VTS), were co-grafted on the black particles to optimize the electrophoretic particles' dispersity and charge. To evaluate the charge content and dispersity of particles, we measured the particles with zeta potential, dynamic light scattering, and long-term standing treatment. Fe3O4-APS0.5VTS0.5-PLMA, selected as the black particles for EPD, had an average particle size and zeta potential of 300 nm and 25 mV, respectively. Besides, the concentration of the black and white particle would significantly affect the EPD's performance. When optimizing the white and black particle concentration to 4:1, the contrast ratio of EPD was up to 13.2. In addition, the EPD has a response time of 200 ms and exhibits an invertible black and white state under ±15 V. Finally, to improve the stability of electrophoretic particles, the particles were encapsulated in microcups and driven to the black and white state under 30 V. This work demonstrated that dual-silane coupling agent particles have the potential to achieve the fast response and high contrast ratio EPD.

Real-time human-computer interaction using eye gazes

Eye gaze emerges as a unique channel in human–computer interaction (HCI) that recognizes human intention based on gaze behavior and enables contactless access to control and operate software interfaces on computers. In this paper, we propose a real-time HCI system using eye gaze. First, we capture and track eyes using the Dlib 68-point landmark detector, and design an eye gaze recognition model to recognize four types of eye gazes. Then, we construct an instance segmentation model to recognize and segment tools and parts using the Mask Region-Based Convolutional Neural Network (R-CNN) method. After that, we design an HCI software interface by integrating and visualizing the proposed eye gaze recognition and instance segmentation models. The HCI system captures, tracks, and recognizes the eye gaze through a red–green–blue (RGB) webcam, and provides responses based on the detected eye gaze, including the tool and part segmentation, object selection and interface switching. Experimental results show that the proposed eye gaze recognition method achieves an accuracy of >99 % in a recommended distance between the eyes and the webcam, and the instance segmentation model achieves an accuracy of 99 %. The experimental results of the HCI system operation demonstrate the feasibility and robustness of the proposed real-time HCI system.

Fast Fluid Simulation via Dynamic Multi-Scale Gridding

Recent works on learning-based frameworks for Lagrangian (i.e., particle-based) fluid simulation, though bypassing iterative pressure projection via efficient convolution operators, are still time-consuming due to excessive amount of particles. To address this challenge, we propose a dynamic multi-scale gridding method to reduce the magnitude of elements that have to be processed, by observing repeated particle motion patterns within certain consistent regions. Specifically, we hierarchically generate multi-scale micelles in Euclidean space by grouping particles that share similar motion patterns/characteristics based on super-light motion and scale estimation modules. With little internal motion variation, each micelle is modeled as a single rigid body with convolution only applied to a single representative particle. In addition, a distance-based interpolation is conducted to propagate relative motion message among micelles. With our efficient design, the network produces high visual fidelity fluid simulations with the inference time to be only 4.24 ms/frame (with 6K fluid particles), hence enables real-time human-computer interaction and animation. Experimental results on multiple datasets show that our work achieves great simulation acceleration with negligible prediction error increase.

Realtime Hand Gesture Recognition System for Human Computer Interaction

Humans are using various devices for interacting with the system like mouse, keyboard, joystick etc. We have developed a real time human computer interaction system for virtual mouse based on the hand gestures. The system is designed in 3 modules as detection of hand, recognition of gestures and human computer interaction with control of mouse events to achieve the higher degree of gesture recognition. We first capture the video using the built-in webcam or USB webcam. Each frame of hand is recognized using a media Pipe palm detection model and using opencv fingertips. The user can move the mouse cursor by moving their fingertip and can perform a click by bringing two fingertips to close. So, this system captures frames using a webcam and detects the hand and fingertips and clicks or moves of the cursor. The system does not require a physical device for cursor movement. The developed system can be extended in other scenarios where human-machine interaction is required with more complex command formats rather than just mouse events.

Leveraging deep feature learning for wearable sensors based handwritten character recognition

Despite rapid advancements in technology, handwritten characters still hold significant roles in various fields, including education, communication, biometric signature verification, and health care. These applications often require digitization of the handwritten characters and associated hand movements to facilitate effective analysis and interpretation of the underlying task. Offline and online handwriting recognition are crucial steps involving the digitization of handwritten characters. Most of these existing systems actively use image processing techniques that are highly sensitive to environmental lighting conditions. Surface Electromyography signals (sEMG), being invariant to lighting conditions, are used in online handwriting recognition to facilitate the automatic transcription of handwritten characters. In this article, we have leveraged deep representation learning to build an efficient and robust sEMG-based Handwritten Character Recognition (HCR) pipeline. A Stacked sparse denoising autoencoder network is applied to obtain an effective deep feature representation. These rich low dimensional features obtained are further introduced into basic classifiers, producing state-of-the-art accuracy for the task. Additional experiments were performed to analyze the effect of the fusion of complementary sensing modules (Accelerometer and Gyroscope) on the performance of sEMG based HCR pipeline. Extensive evaluations were performed to ensure the validity of the obtained results. For the experimentation, new datasets consisting of sEMG, Accelerometer, and Gyroscope signals corresponding to 26 handwritten lower English alphabets were collected from 15 subjects. Our proposed pipeline can be used to build real-time Human–Computer Interaction(HCI) applications for smart classrooms facilitating digitization of handwritten notes and clinical applications involving handwriting analysis tasks.

Improved Multi-Stream Convolutional Block Attention Module for sEMG-Based Gesture Recognition.

As a key technology for the non-invasive human-machine interface that has received much attention in the industry and academia, surface EMG (sEMG) signals display great potential and advantages in the field of human-machine collaboration. Currently, gesture recognition based on sEMG signals suffers from inadequate feature extraction, difficulty in distinguishing similar gestures, and low accuracy of multi-gesture recognition. To solve these problems a new sEMG gesture recognition network called Multi-stream Convolutional Block Attention Module-Gate Recurrent Unit (MCBAM-GRU) is proposed, which is based on sEMG signals. The network is a multi-stream attention network formed by embedding a GRU module based on CBAM. Fusing sEMG and ACC signals further improves the accuracy of gesture action recognition. The experimental results show that the proposed method obtains excellent performance on dataset collected in this paper with the recognition accuracies of 94.1%, achieving advanced performance with accuracy of 89.7% on the Ninapro DB1 dataset. The system has high accuracy in classifying 52 kinds of different gestures, and the delay is less than 300 ms, showing excellent performance in terms of real-time human-computer interaction and flexibility of manipulator control.

A Fast and Accurate Spatial Target Snapping Method for 3D Scene Modeling and Mapping in Mobile Augmented Reality

High-performance spatial target snapping is an essential function in 3D scene modeling and mapping that is widely used in mobile augmented reality (MAR). Spatial data snapping in a MAR system must be quick and accurate, while real-time human–computer interaction and drawing smoothness must also be ensured. In this paper, we analyze the advantages and disadvantages of several spatial data snapping algorithms, such as the 2D computational geometry method and the absolute distance calculation method. To address the issues that existing algorithms do not adequately support 3D data snapping and real-time snapping of high data volumes, we present a new adaptive dynamic snapping algorithm based on the spatial and graphical characteristics of augmented reality (AR) data snapping. Finally, the algorithm is experimented with by an AR modeling system, including the evaluation of snapping efficiency and snapping accuracy. Through the experimental comparison, we found that the algorithm proposed in this paper is substantially improved in terms of shortening the snapping time, enhancing the snapping stability, and improving the snapping accuracy of vector points, lines, faces, bodies, etc. The snapping efficiency of the algorithm proposed in this paper is 1.6 times higher than that of the traditional algorithm on average, while the data acquisition accuracy based on the algorithm in this paper is more than 6 times higher than that of the traditional algorithm on average under the same conditions, and its data accuracy is improved from the decimeter level to the centimeter level.

High Inclusiveness and Accuracy Motion Blur Real-Time Gesture Recognition Based on YOLOv4 Model Combined Attention Mechanism and DeblurGanv2

The combination of gesture recognition and aerospace exploration robots can realize the efficient non-contact control of the robots. In the harsh aerospace environment, the captured gesture images are usually blurred and damaged inevitably. The motion blurred images not only cause part of the transmitted information to be lost, but also affect the effect of neural network training in the later stage. To improve the speed and accuracy of motion blurred gestures recognition, the algorithm of YOLOv4 (You Only Look Once, vision 4) is studied from the two aspects of motion blurred image processing and model optimization. The DeblurGanv2 is employed to remove the motion blur of the gestures in YOLOv4 network input pictures. In terms of model structure, the K-means++ algorithm is used to cluster the priori boxes for obtaining the more appropriate size parameters of the priori boxes. The CBAM attention mechanism and SPP (spatial pyramid pooling layer) structure are added to YOLOv4 model to improve the efficiency of network learning. The dataset for network training is designed for the human–computer interaction in the aerospace space. To reduce the redundant features of the captured images and enhance the effect of model training, the Wiener filter and bilateral filter are superimposed on the blurred images in the dataset to simply remove the motion blur. The augmentation of the model is executed by imitating different environments. A YOLOv4-gesture model is built, which collaborates with K-means++ algorithm, the CBAM and SPP mechanism. A DeblurGanv2 model is built to process the input images of the YOLOv4 target recognition. The YOLOv4-motion-blur-gesture model is composed of the YOLOv4-gesture and the DeblurGanv2. The augmented and enhanced gesture data set is used to simulate the model training. The experimental results demonstrate that the YOLOv4-motion-blur-gesture model has relatively better performance. The proposed model has the high inclusiveness and accuracy recognition effect in the real-time interaction of motion blur gestures, it improves the network training speed by 30%, the target detection accuracy by 10%, and the value of mAP by about 10%. The constructed YOLOv4-motion-blur-gesture model has a stable performance. It can not only meet the real-time human–computer interaction in aerospace space under real-time complex conditions, but also can be applied to other application environments under complex backgrounds requiring real-time detection.

INTERACTIVE VIRTUAL OBJECTS (IVOs) FOR NEXT GENERATION OF VIRTUAL MUSEUMS: FROM STATIC TEXTURED PHOTOGRAMMETRIC AND HBIM MODELS TO XR OBJECTS FOR VR-AR ENABLED GAMING EXPERIENCES

Abstract. Virtual museums should not live on the internet through traditional applications sharing collections through simple panoramas or static images and descriptions but reach novel innovative and interactive forms of Virtual Reality (VR) and Augmented Reality (AR), providing a more creative, intimate, personal learning experience. For this reason, the authors propose a method based on advanced Information Technologies (IT) to fully return the tangible and intangible values of different type of works of art. The primary purpose of this study is to create a system of virtual environments through which the digital user, mainly the visitor of an unconventional museum, will be able to physically interact, through visual and tactile methods, with 3D digital models of sculptures, information, and art objects. Thanks to the integration of the latest 3D modelling and digital survey techniques with the Visual Programming Language (VPL) and eXteded Reality (XR) development platforms, authors propose new levels of interactivity between users and Interactive Virtual Objects (IVOs) capable of coming to life, sharing new forms of real-time human-computer interaction for VR-AR enabled gaming experiences and virtual museums, using multiple devices such VR headset, web-based AR platforms, mobile phones, tablets, and PC workstation.

Application of Visual Simulation Technology Based on the Virtools Method in Equipment Virtual Maintenance

In this paper, 3dsmax is used to build three-dimensional model of scene and virtual equipment. Virtools is used to edit various maintenance scenes and build a virtual maintenance system consistent with the actual environment. According to the operator’s action instructions, the system shows the real-time human-computer interaction process, the state change of the model, the scene response process after the operation, which makes the virtual environment and the real installation operation consistent, and realizes the virtual maintenance training of the equipment. The virtual maintenance training system can be applied to the maintenance training in Colleges and universities, so as to reduce the loss of real equipment and improve the efficiency of equipment use.

Research on Virtual Piano Based on Computer Binocular Stereo Vision

The binocular stereo vision technology emerges along with the development of digital technology, which has been widely employed in virtual piano to effectively remedy the user-unfriendliness and unnaturalness of the keyboard and mouse used as the key. This paper discusses how binocular stereo vision technology can be made good use to realize the function of the piano so that a comfortable real-time human-computer interaction can be completed.

Human-computer interaction based on face feature localization

Human-computer interaction is the way in which humans and machines communicate information. With the rapid development of deep learning technology, the technology of human-computer interaction has also made a corresponding breakthrough. In the past, the way human-computer interaction was mostly relied on hardware devices. Through the coordinated work of multiple sensors, people and machines can realize information interaction. However, as theoretical technology continues to mature, algorithms for human-computer interaction are also being enriched. The popularity of convolutional neural networks has made image processing problems easier to solve. Therefore, real-time human-computer interaction can be performed by using image processing, and intelligent of human-computer interaction can be realized. The main idea of this paper is to use the real-time capture of face images and video information to image the face image information. We perform feature point positioning based on the feature points of the face image. We perform expression recognition based on the feature points that are located. At the same time, we perform ray tracing for the identified human eye area. The feature points of the face and the corresponding expressions and implementation movements represent the user's use appeal. Therefore, we can analyze the user's use appeal by locating the face feature area. We define the corresponding action information for specific user face features. We extract the user's corresponding information according to the user's face features, and perform human-computer interaction according to the user's information.

A New Filtering System for Using a Consumer Depth Camera at Close Range.

Using consumer depth cameras at close range yields a higher surface resolution of the object, but this makes more serious noises. This form of noise tends to be located at or on the edge of the realistic surface over a large area, which is an obstacle for real-time applications that do not rely on point cloud post-processing. In order to fill this gap, by analyzing the noise region based on position and shape, we proposed a composite filtering system for using consumer depth cameras at close range. The system consists of three main modules that are used to eliminate different types of noise areas. Taking the human hand depth image as an example, the proposed filtering system can eliminate most of the noise areas. All algorithms in the system are not based on window smoothing and are accelerated by the GPU. By using Kinect v2 and SR300, a large number of contrast experiments show that the system can get good results and has extremely high real-time performance, which can be used as a pre-step for real-time human-computer interaction, real-time 3D reconstruction, and further filtering.

Virtual Reality Based Visualization and Training of a Quadcopter by using RC Remote Control Transmitter

The idea of this research paper is to simulate the flight of a quadcopter-type UAV, which can help to train drone pilots before the actual flight. We designed the quadcopter’s model to help improve the future pilot’s accuracy up to the point of a stable control, because it is necessary to know how to control the parameters of the quadcopter prior to a real-world scenario. Our system is based on a MATLAB Simulink model. The remote sensing input is given by the RC remote control, based on virtual reality, in order to train the data and output in 3D visualization environment. It has been developed and used for the measurement of the simulated system behavior and its performance. The objective of this research paper is to create a training environment of the quadcopter’s flight parameters for a newly trained pilot. The pilot will be able to follow the simulated model and receive visual feedback. Following the simulated model implies stabilizing the moment of input-output. A visual system with real-time human-computer interaction could provide the intended training experience using this model.

Design of Head Movement Controller System (HEMOCS) for Control Mobile Application through Head Pose Movement Detection

Head movement has been found to be a natural way of interaction. It can be used as an alternative control method and provides accessibility for users when used in human computer interface solutions. The combination of Head-mounted displays (HMDs) with mobile devices, provide an innovation of new low cost of human-computer interaction. Such devices are hands-free systems. In this paper, we introduce a new method for recognizing head movement as the controller of mobile application and proposed a new control system using head movement only. The proposed method can determine specific head pose movement and respond it as a controller. The implementation of a music player application on an iOS devices shows that the proposed method is appropriate for a new experience of real-time human-computer interaction with head movement control only.

An HCI paradigm fusing flexible object selection and AOM-based animation

The use of three-dimensional (3D) gesture input devices is important and necessary in 3D systems, but such devices face considerable challenges posed by the high dimensionality of dexterous hand motion. The objective of this study is to achieve real-time interaction in object selection and direct manipulation in 3D application systems by capturing and visualizing the interaction intentions and probing the cognitive behavior models of users. An interactive operation procedure is divided into three stages: object selection, manipulation and reset. Trajectory scene interaction (TSI) is proposed for object selection starting from a fixed position called a forward point (FP). The manipulations exerted on the selected object include grasping and translation. After these manipulations, the gesture is reset to the FP. This work offers four novel contributions. First, flexible object selection and atomic operation model (AOM)-based animations are fused to form a uniform, real-time human-computer interaction (HCI) paradigm. Second, a cognitive behavior model is proposed for recognizing and reacting to hand gestures as captured by a monocular camera. Third, an approach to capturing, expressing, and probing a user's interaction intention is presented. Fourth, a 3D real-time gesture input interface is achieved. The use of the proposed HCI interface, which offers fast speed, satisfactory accuracy and a responsive user experience, is demonstrated in virtual assembly, a game of chess, dialing a cell phone number and menu operation.

Multivariate Output-Associative RVM for Multi-Dimensional Affect Predictions

The current trends in affect recognition research are to consider continuous observations from spontaneous natural interactions in people using multiple feature modalities, and to represent affect in terms of continuous dimensions, incorporate spatio-temporal correlation among affect dimensions, and provide fast affect predictions. These research efforts have been propelled by a growing effort to develop affect recognition system that can be implemented to enable seamless real-time human-computer interaction in a wide variety of applications. Motivated by these desired attributes of an affect recognition system, in this work a multi-dimensional affect prediction approach is proposed by integrating multivariate Relevance Vector Machine (MVRVM) with a recently developed Output-associative Relevance Vector Machine (OARVM) approach. The resulting approach can provide fast continuous affect predictions by jointly modeling the multiple affect dimensions and their correlations. Experiments on the RECOLA database show that the proposed approach performs competitively with the OARVM while providing faster predictions during testing. Keywords—Dimensional affect prediction, Output-associative RVM, Multivariate regression.

Real time gaze estimation with a consumer depth camera

Existing eye-gaze-tracking systems typically require multiple infrared (IR) lights and high-quality cameras to achieve good performance and robustness against head movement. This requirement limits the systems’ potential for broader applications. In this paper, we present a low-cost, non-intrusive, simple-setup gaze estimation system that can estimate the gaze direction under free head movement. In particular, the proposed system only uses a consumer depth camera (Kinect sensor) positioned at a distance from the subject. We develop a simple procedure to calibrate the geometric relationship between the screen and the camera, and subject-specific parameters. A parameterized iris model is then used to locate the center of the iris for gaze feature extraction, which can handle low-quality eye images. Finally, the gaze direction is determined based on a 3D geometric eye model, where the head movement and deviation of the visual axis from the optical axis are taken into consideration. Experimental results indicate that the system can estimate gaze with an accuracy of 1.4–2.7° and is robust against large head movements. Two real-time human–computer interaction (HCI) applications are presented to demonstrate the potential of the proposed system for wide applications.