Face Tracking Research Articles

20‐4: An AI‐Driven Aquarium Guide System for Intelligent Museum

Visitors to aquariums often struggle to access information and differentiate between similar marine species. To address this issue, ITRI has developed an AI‐driven Aquarium Guide System that uses real‐time face tracking and recognition technology, along with fish identification and gesture recognition. When visitors stand in front of a fish tank, the system detects their gaze and facial features, and identifies the species they are looking at. The backend database then displays the fish's name and provides relevant information to visitors.This system has a 98% accuracy rate in recognizing marine life and can adjust the information based on visitor gestures with an accuracy error of less than 1.0 mm. It also uses virtual and augmented reality technology, paving the way for real‐time, immersive mapping technology.The AI‐driven Aquarium Guide System improves the visitor experience at tourist attractions by offering accurate and interactive information about marine life. This increases visitor understanding and appreciation of marine creatures while providing an engaging and enjoyable experience. We are thrilled to announce that this system has won the 2023 CES Innovation Award, recognizing its excellence and innovation. It has already been installed at the National Museum of Marine Science & Technology (NMMST), transforming the way visitors experience and learn about marine life.

Survey On Object Detection, Face Tracking, Digital Mapping and Lane Following For Remotely Piloted Aerial Systems (RPAS)

Survey On Object Detection, Face Tracking, Digital Mapping and Lane Following For Remotely Piloted Aerial Systems (RPAS)

Blockchain as a Services Based Deep Facial Feature Extraction Architecture for Student Attention Evaluation in Online Education

<p>Obtaining a person’s facial features is necessary for processing techniques like face tracking, facial expression, and face recognition. Many factors are involved in locating and detecting facial features, and the most important is eye localization and detection. Recognition of facial expressions is not about catching expressions; it is about determining whether or not students feel an emotional connection to the material or the instructor who presents it. Using blockchain as a service (BaaS) is the third-party creation and management of cloud-based networks for companies which could use for student attention evaluation without spending time and money developing their in-house solutions. Hence to overcome the problem mentioned, this paper is solved by proposing a new technique named deep facial feature extraction system (DFFE), through which the student’s attention is examined. The basic features such as feelings, interest, and attention of students are evaluated by implementing the new Expert Facial Feature Focus Algorithm (EFFF) using deep learning strategies. It is possible that shortly, this algorithm will discover a person’s feelings and thoughts accurately comprehensively assess user’s attention degrees to help people work, study, and live better with greater efficiency achieving 93.2% by analyzing emotions and feelings.</p> <p> </p>

Comparing technologies for conveying emotions through realistic avatars in virtual reality‐based metaverse experiences

AbstractWith the development of metaverse(s), industry and academia are searching for the best ways to represent users' avatars in shared virtual environments (VEs), where real‐time communication between users is required. The expressiveness of avatars is crucial for transmitting emotions that are key for social presence and user experience, and are conveyed via verbal and non‐verbal facial and body signals. In this paper, two real‐time modalities for conveying expressions in virtual reality (VR) via realistic, full‐body avatars are compared by means of a user study. The first modality uses dedicated hardware (i.e., eye and facial trackers) to allow a mapping between the user's facial expressions/eye movements and the avatar model. The second modality relies on an algorithm that, starting from an audio clip, approximates the facial motion by generating plausible lip and eye movements. The participants were requested to observe, for both the modalities, the avatar of an actor performing six scenes involving as many basic emotions. The evaluation considered mainly social presence and emotion conveyance. Results showed a clear superiority of facial tracking when compared to lip sync in conveying sadness and disgust. The same was less evident for happiness and fear. No differences were observed for anger and surprise.

Defining Biometrics With Privacy and Benefits: A Research Agenda

This conceptual paper defines and gives examples of biometrics, explains how biometric tracking is currently used (e.g. to predict IQ), and presents innovative future uses of biometric tracking (e.g. to customize the price of products in real time). Specifically, this paper outlines a novel biometric pricing technology (BPT) which uses facial tracking to set the price of products using a new participatory dynamic pricing (vs. static pricing) system. Based on the privacy paradox, this paper addresses the acceptance, concerns and usage of a new emerging technology by consumers and its potential applications. To explore this, we develop a typology of perceived benefits and perceived privacy to predict consumer reactions to biometric technology. In addition, we present a research agenda to guide future research on biometric pricing technology. This research agenda offers new insights on how biometric tracking and specifically biometric price setting could be explored from multiple angles, including the consumer experience, technology acceptance, online profiling, governance, public policy, regulation, ethical and future usage-based perspectives.

A real time target face tracking algorithm based on saliency detection and Camshift

A real time target face tracking algorithm based on saliency detection and Camshift

Robotic gaze and human views: A systematic exploration of robotic gaze aversion and its effects on human behaviors and attitudes.

Similar to human-human interaction (HHI), gaze is an important modality in conversational human-robot interaction (HRI) settings. Previously, human-inspired gaze parameters have been used to implement gaze behavior for humanoid robots in conversational settings and improve user experience (UX). Other robotic gaze implementations disregard social aspects of gaze behavior and pursue a technical goal (e.g., face tracking). However, it is unclear how deviating from human-inspired gaze parameters affects the UX. In this study, we use eye-tracking, interaction duration, and self-reported attitudinal measures to study the impact of non-human inspired gaze timings on the UX of the participants in a conversational setting. We show the results for systematically varying the gaze aversion ratio (GAR) of a humanoid robot over a broad parameter range from almost always gazing at the human conversation partner to almost always averting the gaze. The main results reveal that on a behavioral level, a low GAR leads to shorter interaction durations and that human participants change their GAR to mimic the robot. However, they do not copy the robotic gaze behavior strictly. Additionally, in the lowest gaze aversion setting, participants do not gaze back as much as expected, which indicates a user aversion to the robot gaze behavior. However, participants do not report different attitudes toward the robot for different GARs during the interaction. In summary, the urge of humans in conversational settings with a humanoid robot to adapt to the perceived GAR is stronger than the urge of intimacy regulation through gaze aversion, and a high mutual gaze is not always a signof high comfort, as suggested earlier. This result can be used as a justification to deviate from human-inspired gaze parameters when necessary for specific robot behavior implementations.

A survey on the pipeline evolution of facial capture and tracking for digital humans

With the introduction of concepts for virtual interaction and digital doubles, a rich scenario has been created for embodied avatars to strive. These avatars, more recently referred to as digital humans, have become a popular area of research, resulting in various techniques and methods that focus on improving the perception of their realism, fidelity, emphatic response, and interactivity. This survey aims to explore the literature and recent advancements on the key processes behind the creation and animation of digital human faces through the view of a general pipeline. The extensive review carried out in this study explores the usual data collection protocols, the main facial codification paradigms and databases, the approaches for digital human asset creation, facial tracking solutions for performance-driven animation, the solving process, and the final rendering delivery. Different quantitative evaluation methods, visual perception tests, and empathetic response evaluations for digital humans are also included in the survey. Additionally, the paper presents an updated summary of public and private frameworks for digital humans that go through the complete general pipeline presented. Finally, the condensed knowledge is discussed, inquiring into the possible direction of future developments in the field.

Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications.

With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user's facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user's face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users' facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user's facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user's eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user's facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.

Improved Adaboost-Camshift Face Tracking System in Complex Background

With the rapid growth of science and technology, people pay more and more attention to pattern recognition and computer interaction. Therefore, in the last few years, face detection and tracking technology in video sequence has become a hot topic for people to study. Face tracking detection has a wide application prospect in human-computer interaction, intelligent monitoring, video conference and other aspects. In this paper, the problem of face tracking in video sequences is divided into two aspects: face detection and moving object tracking algorithm. In the face detection problem, the face detection based on Adaboost algorithm is described in detail, and the three-frame difference method is added to make the algorithm better and enhance the speed of face detection. In terms of moving object tracking algorithm, Camshift face tracking algorithm based on color histogram is adopted, which is not affected by the shape and size of the target and has good real-time performance. However, under the influence of color interference and occlusion, the algorithm will make tracking errors. Therefore, Kalman filter is introduced. The algorithm can directly delineate the candidate areas of face to be detected, so as to ensure the feasibility of face tracking. The simulation video image face tracking system is verified by Matlab software. The experimental results show that the system can accurately detect and track the faces in the video image sequence, not only in the simple background, but also in the complex background and multiple faces can also be well detected and tracked, and the tracking ensures real-time performance.

Analysis of Markerless-Based Tracking Methods of Face Tracker Techniques in Detecting Human Face Movements in 2D And 3D Filter Making

The marker-based tracking method is a method that utilizes markers, while the markerless-based tracking method is a method that does not use markers in making AR. In the markerless-based tracking method, there is a face tracker technique. In previous research, no one has discussed the comparison of effectiveness concerning the success and accuracy of using the face tracker technique. Therefore, this study aims to test the effectiveness of the accuracy and accuracy of success with applying the markerless-based tracking method, the face tracker technique, in detecting facial movements. in 2D and 3D AR with light intensity test parameters of 20 Lux, 40 Lux, and 60 Lux with WRGB light color, Face angle position of 30o and 60o, and face distance from camera 50 cm, 100cm, and 150cm. The results of comparison of superior success accuracy are at a distance of 50 cm; with an accuracy rate for 2D AR of 93.22% and 96.63% for 3D. It was concluded that the face tracker technique's markerless-based tracking method works optimally in 3D compared to 2D. This research finds an attractiveness score of 1.865, a perception score of 1.683, an efficiency score of 1.550, a dependability score of 1.638, a stimulation score of 1.500, and a novelty score of 1.013. Quality with an attractiveness value of 1.68, pragmatic quality of 1.56, and hedonic quality of 1.26. This study concludes that 2D and 3D AR face detection positively evaluates user experience and quality.
 

Quantified analysis of facial movement: A reference for clinical applications.

Most techniques for evaluating unilateral impairments in facial movement yield subjective measurements. The objective of the present study was to define a reference dataset and develop a visualization tool for clinical assessments. In this prospective study, a motion capture system was used to quantify facial movements in 30 healthy adults and 2 patients. We analyzed the displacements of 105 reflective markers placed on the participant's face during five movements (M1-M5). For each marker, the primary endpoint was the maximum amplitude of displacement from the static position (M0) in an analysis of variance. The measurement precision was 0.1 mm. Significant displacements of markers were identified for M1-M5, and displacement patterns were defined. The patients and age-matched healthy participants were compared with regard to the amplitude of displacement. We created a new type of radar plot to visually represent the diagnosis and facilitate effective communication between medical professionals. In proof-of-concept experiments, we collected quantitative data on patients with facial palsy and created a patient-specific radar plot. Our new protocol for clinical facial motion capture ("quantified analysis of facial movement," QAFM) was accurate and should thus facilitate the long-term clinical follow-up of patients with facial palsy. To take account of the limitations affecting the comparison with the healthy side, we created a dataset of healthy facial movements; our method might therefore be applicable to other conditions in which movements on one or both sides of the face are impaired. The patient-specific radar plot enables clinicians to read and understand the results rapidly.

CREBAS: Computer-Based REBA Evaluation System for Wood Manufacturers Using MediaPipe

Recently, musculoskeletal disorders (MSDs) caused by repetitive working postures in industrial sites have emerged as one of the biggest problems in the field of industrial health. The risk of MSDs caused by the repetitive working postures of workers is quantitatively evaluated by using NLE (NIOSH Lifting Equation), OWAS (Ovako Working-posture Analysis System), RULA (Rapid Upper Limb Assessment), REBA (Rapid Entire Body Assessment), etc. Methods used for the working posture analysis include vision-based analysis and motion capture analysis. Vision-based analysis is a method where an expert with ergonomics knowledge watches and manually analyzes recorded working images. Although the analysis is inexpensive, it takes a lot of time to analyze. In addition, the analyst’s subjective opinions or mistakes may be reflected in the results, so it may be somewhat unreliable. On the other hand, motion capture analysis can obtain more accurate and consistent results, but its measurement equipment is very expensive and it requires a large space for measurement. In this paper, we propose a computer-based automated REBA system that can evaluate, automatically and consistently, working postures in order to supplement the shortcomings of these existing methods. The CREBA system uses the body detection learning model of MediaPipe to detect the worker’s area in the recorded images and sets the body area based on the position of the face, detected using the face tracking learning model. In the set area, the positions of joints are tracked using the posture tracking learning model, and the angles of joints are calculated based on the joint positions using the inverse kinematics, and then by automatically calculating the degree of load of the working posture with the REBA evaluation method. In order to verify the accuracy of the evaluation results of the CREBA system, we compared them with the experts’ vision-based REBA evaluation results. The result of the experiment showed a slight difference of about 1.0 points between the evaluation results of the expert group and those of the CREBA system. It is expected that the ergonomic analysis method for the working posture used in this study will reduce workers’ labor intensity and improve their safety and efficiency.

Automated Video-Based Face Detection Using Harris Hawks Optimization with Deep Learning

Face recognition technology automatically identifies an individual from image or video sources. The detection process can be done by attaining facial characteristics from the image of a subject face. Recent developments in deep learning (DL) and computer vision (CV) techniques enable the design of automated face recognition and tracking methods. This study presents a novel Harris Hawks Optimization with deep learning-empowered automated face detection and tracking (HHODL-AFDT) method. The proposed HHODL-AFDT model involves a Faster region based convolution neural network (RCNN)-based face detection model and HHO-based hyperparameter optimization process. The presented optimal Faster RCNN model precisely recognizes the face and is passed into the face-tracking model using a regression network (REGN). The face tracking using the REGN model uses the features from neighboring frames and foresees the location of the target face in succeeding frames. The application of the HHO algorithm for optimal hyperparameter selection shows the novelty of the work. The experimental validation of the presented HHODL-AFDT algorithm is conducted using two datasets and the experiment outcomes highlighted the superior performance of the HHODL-AFDT model over current methodologies with maximum accuracy of 90.60% and 88.08% under PICS and VTB datasets, respectively.

Development of a real-time face tracking system based on the MOSSE correlation filter

The functioning acceleration of the face tracking system can be achieved by using simultaneously a highprecision neural network face detector for initial face detection and a tracking system based on modern high-speed correlation filters. A face tracking system using a MOSSE correlation filter (filter with minimum output sum of squared errors) is proposed. The proposed system allows tracking a human face at high speed in real time, as well as in various face positions. The proposed approach was tested using image processing and deep learning tools from the MATLAB application. Keywords convolutional neural network, YOLO v3 object detection algorithm, deep learning, MOSSE filter, face tracking

Age estimation algorithm based on deep learning and its application in fall detection

<abstract> <p>With the continuous development and progress of society, age estimation based on deep learning has gradually become a key link in human-computer interaction. Widely combined with other fields of application, this paper performs a gradient division of human fall behavior according to the age estimation of the human body, a complete priority detection of the key population, and a phased single aggregation backbone network VoVNetv4 was proposed for feature extraction. At the same time, the regional single aggregation module ROSA module was constructed to encapsulate the feature module regionally. The adaptive stage module was used for feature smoothing. Consistent predictions for each task were made using the CORAL framework as a classifier and tasks were divided in binary. At the same time, a gradient two-node fall detection framework combined with age estimation was designed. The detection was divided into a primary node and a secondary node. In the first-level node, the age estimation algorithm based on VoVNetv4 was used to classify the population of different age groups. A face tracking algorithm was constructed by combining the key point matrices of humans, and the body processed by OpenPose with the central coordinates of the human face. In the secondary node, human age gradient information was used to detect human falls based on the AT-MLP model. The experimental results show that compared with Resnet-34, the MAE value of the proposed method decreased by 0.41. Compared with curriculum learning and the CORAL-CNN method, MAE value decreased by 0.17 relative to the RMSE value. Compared with other methods, the method in this paper was significantly lower, with a biggest drop of 0.51.</p> </abstract>

Introducing a new grading system for Hemifacial spasm severity using facial recognition and facial tracking technology

Introducing a new grading system for Hemifacial spasm severity using facial recognition and facial tracking technology

Assessment of the reproducibility and accuracy of the Visia® Complexion Analysis Camera System for objective skin analysis of facial wrinkles and skin age.

This study aimed to investigate the reproducibility and accuracy of the Visia® Complexion Analysis Camera System by Canfield Scientific for objective skin analysis. Nineteen participants underwent facial capture with the Visia® camera following a standardised protocol. During the first session, the participants sat down and positioned their faces in a capture rig, closed their eyes and had their faces captured from the left, front and right sides, with threefold repetition of the captures from the front side. After 4 weeks, the participants underwent recapture in a similar manner. Based on the frontal views, data for two measurement methods of the Visia® camera system, the absolute scores and the percentiles, were obtained with regard to the skin criterion wrinkles via automated software calculation. Means and standard deviations were evaluated. Based on the side views, the data for the Truskin Ages® were calculated by the Visia® camera system and compared with the calendrical ages, which served as the gold standard for comparison. In the assessment of the reproducibility of the data of the capture system the standard deviation from the frontal captures among all participants was about 3% when the absolute scores of the wrinkles were compared with each other; specifically, the average deviation was 3.36% during the first capture session and 3.4% during the second capture session. Meanwhile, the standard deviation of the measurements was about 9% when the percentiles were compared; specifically, the average deviation was 8.2% during the first capture session and 10.7% during the second capture session. In the assessment of the accuracy the correlation between the calendrical age and the calculated Truskin Age® for both facial sides was very high at a correlation coefficient rho value of >0.8 (right side: r=0.896; left side: r=0.827) and statistically significant at a p-value of <0.001. The average calendrical age and Truskin Age® deviated only slightly from each other and did not differ significantly (right side: p=0.174; left side: p=0.190). The Truskin Age® was slightly higher than the calendrical age by a mean value of 1.37 years for both facial sides. The analysis of the absolute differences revealed that in 50% of the cases, there was a maximum difference of 3 years, and in 75% of the cases, there were maximum differences of 4.5 years for the right side and 5.5 years for the left side. The assessment of the reproducibility and accuracy of the objective measurement method, the Visia® camera system, contributed to the validation of the system. The evaluation of the reproducibility revealed a satisfactory precision of the repeated captures when investigating facial wrinkles. Absolute scores should be preferred over percentiles owing to their better precision. The calculation of the accuracy of the Truskin Age® data from the Visia® camera system revealed only a slight deviation from the true calendrical ages. The correlation between both data groups was highly significant.

Contactless SpO2 with an RGB camera: experimental proof of calibrated SpO2.

Camera-based blood oxygen saturation (SpO2) monitoring allows reliable measurements without touching the skin and is therefore very attractive when there is a risk of cross-infection, in case of fragile skin, and/or to improve the clinical workflow. Despite promising results, productization of the technology is hampered by the unavailability of adequate hardware, especially a camera, which can capture the optimal wavelengths for SpO2 measurements in the red near-infrared region. A regular color (RGB) camera is attractive because of its availability, but also poses several risks and challenges which affect the accuracy of the measurement. To mitigate the most important risks, we propose to add low-cost commercial off-the-shelf (COTS) components to the setup. We executed two studies with this setup: one at a hypoxia lab with SpO2 values in the range 70 - 100% with the purpose to determine the calibration model, and the other study on volunteers to investigate the accuracy for different spot-check scenarios. The proposed processing pipeline includes face tracking and a robust method to estimate the ratio of relative amplitudes of the photoplethysmographic waveforms. Results show that the error is smaller than 4 percent points for realistic screening scenarios where the subject is seated, either with or without head support and/or ambient light.

Video-Driven Neural Physically-Based Facial Asset for Production

Production-level workflows for producing convincing 3D dynamic human faces have long relied on an assortment of labor-intensive tools for geometry and texture generation, motion capture and rigging, and expression synthesis. Recent neural approaches automate individual components but the corresponding latent representations cannot provide artists with explicit controls as in conventional tools. In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets. For data collection, we construct a hybrid multiview-photometric capture stage, coupling with ultra-fast video cameras to obtain raw 3D facial assets. We then set out to model the facial expression, geometry and physically-based textures using separate VAEs where we impose a global MLP based expression mapping across the latent spaces of respective networks, to preserve characteristics across respective attributes. We also model the delta information as wrinkle maps for the physically-based textures, achieving high-quality 4K dynamic textures. We demonstrate our approach in high-fidelity performer-specific facial capture and cross-identity facial motion retargeting. In addition, our multi-VAE-based neural asset, along with the fast adaptation schemes, can also be deployed to handle in-the-wild videos. Besides, we motivate the utility of our explicit facial disentangling strategy by providing various promising physically-based editing results with high realism. Comprehensive experiments show that our technique provides higher accuracy and visual fidelity than previous video-driven facial reconstruction and animation methods.