Vision-based Human-computer Interaction Research Articles

A Visual Human-Computer Interaction System Based on Hybrid Visual Model

The traditional human-computer interaction is mainly through the mouse, keyboard, remote control, and other peripheral equipment electromagnetic signal transmission. This paper aims to build a visual human-computer interaction system through a series of deep learning and machine vision models, so that people can achieve complete human-computer interaction only through the camera and screen. The established visual human-computer interaction system mainly includes the function modes of three basic peripherals in human-computer interaction: keyboard, mouse (X-Y position indicator), and remote control. The convex hull method was used to switch between these three modes. After issuing the mode command, Gaussian mixture was used to quickly identify the moving human body to narrow the scope of our image processing. Subsequently, finger detection in human body was realized based on the Faster-RCNN-ResNET50-FPN model structure, and realized the function of moving mouse and keyboard through the relationship between different fingers. At the same time, the recognition of human body posture was done by using MediaPipe BlazePose, and the action classification models were established through the Angle between body movements so as to realize the control function of remote control. In order to ensure the real-time performance of the interactive system, according to the characteristics of different data processing processes, CPU and GPU computing power resources are used to cross-process images to ensure the real-time performance. The recognition accuracy of the human-computer interaction system is above 0.9 for the key feature points of human body, and above 0.87 for the recognition accuracy of four kinds of command actions. It is hoped that vision-based human-computer interaction will become a widely used interaction mode in the future.

Skin detection in video under uncontrolled illumination

Many vision-based human-computer interaction (HCI) applications require skin detection. However, their performance relies on accuracy in detecting skin regions in video, which is difficult under uncontrolled illumination. The chromatic appearance of skin changes because of shading, often caused by body movement. To address this, we propose a dynamic adaptation method to detect skin regions affected by local color deformations. Static and dynamic skin regions are detected by a corresponding module. The static module includes a facial skin distribution model (FSDM) and a fusion-based background distribution model (FBDM). The FBDM is obtained from a local background distribution model (LBDM) and a global background distribution model (GBDM). The LBDM is obtained by comparing a frame pixel distribution model with the FSDM and GBDM. Next, the FBDM is derived from the LBDM and the GBDM. The dynamic module includes a moving skin distribution model (MSDM), derived from a set of moving skin samples. Initially, moving skin regions are detected using a modified double frame-difference method and then modeled using a Gaussian mixture model. To avoid misidentifying background regions as skin, the final MSDM is obtained by comparing the initial moving skin model to the FSDM and FBDM. Finally, the static and the dynamic models are fused by applying a maximization rule. Experimental results shows that the proposed method can detect skin regions more accurately than state-of-the-art methods.

A Robust Geometric Skin Colour Face Detection Method under Unconstrained Environment of Smartphone Database

Face detection is the primary task in building a vision-based human-computer interaction system and in special applications such as face recognition, face tracking, face identification, expression recognition and also content-based image retrieval. A potent face detection system must be able to detect faces irrespective of illuminations, shadows, cluttered backgrounds, orientation and facial expressions. In previous literature, many approaches for face detection had been proposed. However, face detection in outdoor images with uncontrolled illumination and images with complex background are still a serious problem. Hence, in this paper, we had proposed a Geometric Skin Colour (GSC) method for detecting faces accurately in real world image, under capturing conditions of both indoor and outdoor, and with a variety of illuminations and also in cluttered backgrounds. The selected method was evaluated on two different face video smartphone databases and the obtained results proved the outperformance of the proposed method under the unconstrained environment of these databases.

Computer-Vision based Bare-Hand Augmented Reality Interface for Controlling an AR Object

In this paper, we design and implement a vision-based bare-hand interface which manipulates a virtual object in augmented reality environments in a natural fashion. For years in the researches of developing vision-based human computer interaction, a lot of works have been conducted with the augmented reality technology. Various vision-based interfaces invented are developed by utilising the movement of eyes, hand and body gesture to interact between human and substances to solve the problems such as portability and efficiency of the extensively used interfaces like mouse and keyboard. Many of previous studies for the augmented reality technology are animatedly conducted to make possible to interact with user interface through the information from the image seen with human eyes when people wear the glasses. In this work, we developed a vision-based AR interface which can interact with and control the augmented virtual objects such as architectural or engineering 3D models by recognising simple hand gesture of both human bare-hands in augment reality.

Computer-vision-based bare-hand augmented reality interface for controlling an AR object

In this paper, we design and implement a vision-based bare-hand interface which manipulates a virtual object in augmented reality environments in a natural fashion. For years in the researches of developing vision-based human computer interaction, a lot of works have been conducted with the augmented reality technology. Various vision-based interfaces invented are developed by utilising the movement of eyes, hand and body gesture to interact between human and substances to solve the problems such as portability and efficiency of the extensively used interfaces like mouse and keyboard. Many of previous studies for the augmented reality technology are animatedly conducted to make possible to interact with user interface through the information from the image seen with human eyes when people wear the glasses. In this work, we developed a vision-based AR interface which can interact with and control the augmented virtual objects such as architectural or engineering 3D models by recognising simple hand gesture of both human bare-hands in augment reality.

Vision-based Hand Gesture Interaction Using Particle Filter, Principle Component Analysis and Transition Network

Vision-based human-computer interaction is becoming important nowadays. It offers natural interaction with computers and frees users from mechanical interaction devices, which is favourable especially for wearable computers. This paper presents a human-computer interaction system based on a conventional webcam and hand gesture recognition. This interaction system works in real time and enables users to control a computer cursor with hand motions and gestures instead of a mouse. Five hand gestures are designed on behalf of ve mouse operations: moving, left click, left-double click, right click and no-action. An algorithm based on Particle Filter is used for tracking the hand position. PCA-based feature selection is used for recognizing the hand gestures. A transition network is also employed for improving the accuracy and reliability of the interaction system. This interaction system shows good performance in the recognition and interaction test.

A Priori Laplacian with Hamming Distance: Advanced Dimension Reduction Technique

Images containing faces are essential to intelligent visionbased human computer interaction, and research efforts in face processing include face recognition, face tracking, pose estimation, and expression recognition. The rapidly expanding research in face processing is based on the premise that information about a user’s identity, state, and intent can be extracted from images and that computers can then react accordingly, e.g., by knowing person’s identity, person may be authenticated to utilize a particular service or not. A first step of any face processing system is registering the locations in images where faces are present. However, face registration for whole database is a challenging task because of variability in scale, location, orientation (up-right, rotated), and pose (frontal, profile). Facial expression, occlusion, and lighting conditions also change the overall appearance of face. The Image registration algorithm will register all these images present in the database. The face recognition algorithm which is insensitive to large variation in lighting direction and facial expression is to be implemented. Taking a pattern classification approach, each pixel in an image can be considered as a coordinate in a high-dimensional space. The advantage of this is that the images of a particular face, under varying illumination but fixed pose, lie in a 3D linear subspace of the high dimensional image space—if the face is a Lambertian surface. However, since faces are not truly Lambertian surfaces and do indeed produce self-shadowing images will deviate from this linear subspace. Rather than explicitly modeling this deviation, project the image into a subspace in such a manner which discounts those regions of the face with large deviation. This is achieved by using dimension reduction techniques like Principal component analysis (PCA), Linear Discriminant analysis (LDA), Laplacian faces and other modified approaches like A Priori Laplacian and A Priori Laplacian with Hamming Distance. General Terms Face Recognition, Image Processing, Pattern Recognition

A Multi-Gesture Interaction System Using a 3-D Iris Disk Model for Gaze Estimation and an Active Appearance Model for 3-D Hand Pointing

In this paper, we present a vision-based human-computer interaction system, which integrates control components using multiple gestures, including eye gaze, head pose, hand pointing, and mouth motions. To track head, eye, and mouth movements, we present a two-camera system that detects the face from a fixed, wide-angle camera, estimates a rough location for the eye region using an eye detector based on topographic features, and directs another active pan-tilt-zoom camera to focus in on this eye region. We also propose a novel eye gaze estimation approach for point-of-regard (POR) tracking on a viewing screen. To allow for greater head pose freedom, we developed a new calibration approach to find the 3-D eyeball location, eyeball radius, and fovea position. Moreover, in order to get the optical axis, we create a 3-D iris disk by mapping both the iris center and iris contour points to the eyeball sphere. We then rotate the fovea accordingly and compute the final, visual axis gaze direction. This part of the system permits natural, non-intrusive, pose-invariant POR estimation from a distance without resorting to infrared or complex hardware setups. We also propose and integrate a two-camera hand pointing estimation algorithm for hand gesture tracking in 3-D from a distance. The algorithms of gaze pointing and hand finger pointing are evaluated individually, and the feasibility of the entire system is validated through two interactive information visualization applications.

A strong tracking nonlinear robust filter for eye tracking

Non-intrusive methods for eye tracking are important for many applications of vision-based human computer interaction. However, due to the high nonlinearity of eye motion, how to ensure the robustness of external interference and accuracy of eye tracking pose the primary obstacle to the integration of eye movements into today’s interfaces. In this paper, we present a strong tracking unscented Kalman filter (ST-UKF) algorithm, aiming to overcome the difficulty in nonlinear eye tracking. In the proposed ST-UKF, the Suboptimal fading factor of strong tracking filtering is introduced to improve robustness and accuracy of eye tracking. Compared with the related Kalman filter for eye tracking, the proposed ST-UKF has potential advantages in robustness and tracking accuracy. The last experimental results show the validity of our method for eye tracking under realistic conditions.

A comparison of face and facial feature detectors based on the Viola–Jones general object detection framework

The human face provides useful information during interaction; therefore, any system integrating Vision-Based Human Computer Interaction requires fast and reliable face and facial feature detection. Different approaches have focused on this ability but only open source implementations have been extensively used by researchers. A good example is the Viola–Jones object detection framework that particularly in the context of facial processing has been frequently used. The OpenCV community shares a collection of public domain classifiers for the face detection scenario. However, these classifiers have been trained in different conditions and with different data but rarely tested on the same datasets. In this paper, we try to fill that gap by analyzing the individual performance of all those public classifiers presenting their pros and cons with the aim of defining a baseline for other approaches. Solid comparisons will also help researchers to choose a specific classifier for their particular scenario. The experimental setup also describes some heuristics to increase the facial feature detection rate while reducing the face false detection rate.

A task-driven intelligent workspace system to provide guidance feedback

We introduce a vision-based human–computer interaction system which collaborates with a user by providing feedback during user activities. We design an intelligent workspace system that analyzes the context of a user’s tasks and generates appropriate feedback guiding the user to complete the tasks correctly. While a user is performing the high-level activity, our system analyzes what sub-events the user has already completed and what sub-events are needed next in order for the user to finish the activity. Based on this analysis, the system generates hierarchical feedback messages to assist the user. We test our system on three different types of tasks: tightening nuts on a wheel of a car, assembling a toy spaceship with LEGO blocks, and assembling a laptop computer with its parts. Experimental results demonstrate the accuracy of the vision-based activity analysis, and a comparative user study shows that our system’s feedback enables users to complete assembly tasks with significantly improved efficiency.

A novel strong tracking finite-difference extended Kalman filter for nonlinear eye tracking

Non-intrusive methods for eye tracking are important for many applications of vision-based human computer interaction. However, due to the high nonlinearity of eye motion, how to ensure the robustness of external interference and accuracy of eye tracking poses the primary obstacle to the integration of eye movements into todays’s interfaces. In this paper, we present a strong tracking finite-difference extended Kalman filter algorithm, aiming to overcome the difficulty in modeling nonlinear eye tracking. In filtering calculation, strong tracking factor is introduced to modify a priori covariance matrix and improve the accuracy of the filter. The filter uses finite-difference method to calculate partial derivatives of nonlinear functions for eye tracking. The latest experimental results show the validity of our method for eye tracking under realistic conditions.

An Approach to Tracking Deformable Hand Gesture for Real-Time Interaction

The tracking of deformable hand gesture is a very important task in vision-based HCI (human-computer interaction) research. A novel real-time tracking approach is proposed to capture the motion of deformable hand gesture with single camera. The proposed approach uses a set of 2D hand models in place of high-dimensional 3D model. It achieves auto-initialization by firstly using Bayesian classifier to do posture recognition, and then locating fingers and fingertips to fit image features to recognized posture. It solves the problem of interference among fingers during tracking successfully by the integration of K-means clustering and particle filter. Moreover, a state checking process is embedded into tracking method, and it realizes resumption from tracking failure and update of hand models automatically. Experimental results show that the proposed method can achieve continuous real-time tracking of deformable hand gesture with high precision, and thus it can meet the requirements from real-time vision-based human-computer interaction.

Real-Time Natural Hand Gestures

In human-computer interaction (HCI) applications, traditional devices, such as keyboard and mouse, can become cumbersome and unsuitable. Researchers consider the human hand to be one of the most promising natural HCI media; specifically, using hand gestures to input computer commands. Vision-based HCI hand-gesture analysis and recognition studies require large numbers of a variety of gestures as input and a virtual hand as output to display the results. Creating a virtual hand with natural gestures would improve human hand HCI research by providing data for hand analysis systems and generating visual output for hand simulation systems.

VICs: A modular HCI framework using spatiotemporal dynamics

Many vision-based human-computer interaction systems are based on the tracking of user actions. Examples include gaze tracking, head tracking, finger tracking, etc. In this paper, we present a framework that employs no user tracking; instead, all interface components continuously observe and react to changes within a local neighborhood. More specifically, components expect a predefined sequence of visual events called visual interface cues (VICs). VICs include color, texture, motion, and geometric elements, arranged to maximize the veridicality of the resulting interface element. A component is executed when this stream of cues has been satisfied. We present a general architecture for an interface system operating under the VIC-based HCI paradigm and then focus specifically on an appearance-based system in which a hidden Markov model (HMM) is employed to learn the gesture dynamics. Our implementation of the system successfully recognizes a button push with a 96% success rate.

Nonstationary color tracking for vision-based human-computer interaction

Skin color offers a strong cue for efficient localization and tracking of human body parts in video sequences for vision-based human-computer interaction. Color-based target localization could be achieved by analyzing segmented skin color regions. However, one of the challenges of color-based target tracking is that color distributions would change in different lighting conditions such that fixed color models would be inadequate to capture nonstationary color distributions over time. Meanwhile, using a fixed skin color model trained by the data of a specific person would probably not work well for other people. Although some work has been done on adaptive color models, this problem still needs further studies. We present our investigation of color-based image segmentation and nonstationary color-based target tracking, by studying two different representations for color distributions. We propose the structure adaptive self-organizing map (SASOM) neural network that serves as a new color model. Our experiments show that such a representation is powerful for efficient image segmentation. Then, we formulate the nonstationary color tracking problem as a model transduction problem, the solution of which offers a way to adapt and transduce color classifiers in nonstationary color distributions. To fulfill model transduction, we propose two algorithms, the SASOM transduction and the discriminant expectation-maximization (EM), based on the SASOM color model and the Gaussian mixture color model, respectively. Our extensive experiments on the task of real-time face/hand localization show that these two algorithms can successfully handle some difficulties in nonstationary color tracking. We also implemented a real-time face/hand localization system based on such algorithms for vision-based human-computer interaction.

Detecting faces in images: a survey

Images containing faces are essential to intelligent vision-based human-computer interaction, and research efforts in face processing include face recognition, face tracking, pose estimation and expression recognition. However, many reported methods assume that the faces in an image or an image sequence have been identified and localized. To build fully automated systems that analyze the information contained in face images, robust and efficient face detection algorithms are required. Given a single image, the goal of face detection is to identify all image regions which contain a face, regardless of its 3D position, orientation and lighting conditions. Such a problem is challenging because faces are non-rigid and have a high degree of variability in size, shape, color and texture. Numerous techniques have been developed to detect faces in a single image, and the purpose of this paper is to categorize and evaluate these algorithms. We also discuss relevant issues such as data collection, evaluation metrics and benchmarking. After analyzing these algorithms and identifying their limitations, we conclude with several promising directions for future research.