Facial Analysis Technology Research Articles

Advanced emotion analysis: harnessing facial image processing and speech recognition through deep learning

The human face hides many secrets and is one of the most expressive human features. Human faces even contain hidden information about a person's personality. Considering the fundamental role of the human face, it is necessary to prepare appropriate deep-learning solutions that analyze human face data. This technology is becoming increasingly common in many industries, such as online retail, advertising testing, virtual makeovers, etc. For example, facial analysis technology now allows online shoppers to virtually apply makeup and try on jewelry or new glasses to get an accurate picture of what these products will look like. The human sense of hearing is a treasure trove of information about the current environment and the location and properties of sound-producing objects. For instance, we effortlessly absorb the sounds of birds singing outside the window, traffic passing in the distance, or the lyrics of a song on the radio. The human auditory system can process the intricate mix of sounds reaching our ears and create high-level abstractions of the environment by analyzing and grouping measured sensory signals. The process of obtaining segregation and identifying sources of a received complex acoustic signal, known as sound scene analysis, is a domain where the power of deep learning shines. The machine implementation of this functionality (separation and classification of sound sources) is pivotal in applications such as speech recognition in noise, automatic music transcription, searching and retrieving multimedia data, or recognizing emotions in statements.

Application of Facial Expression Analysis Technology in Violence Risk Assessment of Individuals with Mental Disorders in Supervised Settings.

To explore the association between violent behaviors and emotions in individuals with mental disorders, to evaluate the application value of facial expression analysis technology in violence risk assessment of individuals with mental disorders in supervised settings, and to provide a reference for violence risk assessment. Thirty-nine male individuals with mental disorders in supervised settings were selected, the participant risk of violence, cognitive function, psychiatric symptoms and severity were assessed using the Modified Overt Aggression Scale (MOAS), the Historical, Clinical, Risk Management-Chinese version(HCR-CV), the Positive and Negative Syndrome Scale (PANSS) and the Brief Psychiatric Rating Scale (BPRS). An emotional arousal was performed on the participants and the intensity of their emotions and facial expression action units was recorded before, during and after the arousal. One-way analysis of variance (ANOVA) was used to compare the differences in the intensity of emotions and facial expression action units before, during and after the arousal. Pearson correlation analysis was used to calculate the correlations between the intensity of the seven basic emotional facial expressions and the scores of the assessment scales. The intensity difference of sadness, surprise and fear in different time periods was statistically significant (P<0.05). The intensity of the left medial eyebrow lift action unit was found significantly different before and after the emotional arousal (P<0.05). The intensity of anger was positively correlated with the Modified Overt Aggression Scale score throughout the experiment (P<0.05). Eye action units such as eyebrow lifting, eyelid tightening and upper eyelid lifting can be used as effective action units to identify sadness, anger and other negative emotions associated with violent behaviors. Facial expression analysis technology can be used as an auxiliary tool to assess the potential risk of violence in individuals with mental disorders in supervised settings.

Computer-based facial recognition as an assisting diagnostic tool to identify children with Noonan syndrome

BackgroundNoonan syndrome (NS) is a rare genetic disease, and patients who suffer from it exhibit a facial morphology that is characterized by a high forehead, hypertelorism, ptosis, inner epicanthal folds, down-slanting palpebral fissures, a highly arched palate, a round nasal tip, and posteriorly rotated ears. Facial analysis technology has recently been applied to identify many genetic syndromes (GSs). However, few studies have investigated the identification of NS based on the facial features of the subjects.ObjectivesThis study develops advanced models to enhance the accuracy of diagnosis of NS.MethodsA total of 1,892 people were enrolled in this study, including 233 patients with NS, 863 patients with other GSs, and 796 healthy children. We took one to 10 frontal photos of each subject to build a dataset, and then applied the multi-task convolutional neural network (MTCNN) for data pre-processing to generate standardized outputs with five crucial facial landmarks. The ImageNet dataset was used to pre-train the network so that it could capture generalizable features and minimize data wastage. We subsequently constructed seven models for facial identification based on the VGG16, VGG19, VGG16-BN, VGG19-BN, ResNet50, MobileNet-V2, and squeeze-and-excitation network (SENet) architectures. The identification performance of seven models was evaluated and compared with that of six physicians.ResultsAll models exhibited a high accuracy, precision, and specificity in recognizing NS patients. The VGG19-BN model delivered the best overall performance, with an accuracy of 93.76%, precision of 91.40%, specificity of 98.73%, and F1 score of 78.34%. The VGG16-BN model achieved the highest AUC value of 0.9787, while all models based on VGG architectures were superior to the others on the whole. The highest scores of six physicians in terms of accuracy, precision, specificity, and the F1 score were 74.00%, 75.00%, 88.33%, and 61.76%, respectively. The performance of each model of facial recognition was superior to that of the best physician on all metrics.ConclusionModels of computer-assisted facial recognition can improve the rate of diagnosis of NS. The models based on VGG19-BN and VGG16-BN can play an important role in diagnosing NS in clinical practice.

A megastudy on the predictability of personal information from facial images: Disentangling demographic and non-demographic signals

While prior research has shown that facial images signal personal information, publications in this field tend to assess the predictability of a single variable or a small set of variables at a time, which is problematic. Reported prediction quality is hard to compare and generalize across studies due to different study conditions. Another issue is selection bias: researchers may choose to study variables intuitively expected to be predictable and underreport unpredictable variables (the ‘file drawer’ problem). Policy makers thus have an incomplete picture for a risk-benefit analysis of facial analysis technology. To address these limitations, we perform a megastudy—a survey-based study that reports the predictability of numerous personal attributes (349 binary variables) from 2646 distinct facial images of 969 individuals. Using deep learning, we find 82/349 personal attributes (23%) are predictable better than random from facial image pixels. Adding facial images substantially boosts prediction quality versus demographics-only benchmark model. Our unexpected finding of strong predictability of iPhone versus Galaxy preference variable shows how testing many hypotheses simultaneously can facilitate knowledge discovery. Our proposed L1-regularized image decomposition method and other techniques point to smartphone camera artifacts, BMI, skin properties, and facial hair as top candidate non-demographic signals in facial images.

Lessons Learned from the Point-of-Care Use of a Facial Analysis Technology

Purpose: We aimed to evaluate the utility of facial analysis technology for genetic diagnoses in a typical pediatric genetic clinic. Methods: A retrospective review identified children (aged &lt;18 years) who had not previously received a definitive genetic diagnosis and underwent a comprehensive genetic evaluation. Their photographs and relevant clinical non-facial features were uploaded to the CLINIC application of the Face2Gene web interface, and the resulting analysis was accessed and correlated to the molecular diagnosis. Results: Of the 23 children included, the overall diagnostic yield in this study was 60.9% (14/23). In total, 64.3% of patients had the correct condition suggested in the top 10 differential diagnoses. The gestalt similarity was only 55.6%, but the phenotypic features added by the clinician showed a similarity of more than the medium level in all patients. Conclusion: Our data underscore the usefulness of facial analysis technology as an auxiliary point-of-care tool in pediatric genetic clinics, and we also present some considerations to increase accuracy.

Application of facial analysis Technology in Clinical Genetics: Considerations for diverse populations.

Facial analysis technology in rare diseases has the potential to shorten the diagnostic odyssey by providing physicians with a valuable diagnostic tool. Given that most clinical genetic resources focus on populations of European descent, we compare craniofacial features in genetic syndromes across different populations and review how machine learning algorithms perform on diagnosing genetic syndromes in geographically and ethnically diverse populations. We also discuss the value of populations from ancestrally diverse backgrounds in the training set of machine learning algorithms. Finally, this review demonstrates that across diverse population groups, machine learning models have outstanding accuracy as supported by the area under the curve values greater than 0.9. Artificial intelligence is only in its infancy in the diagnosis of rare disease in diverse populations and will become more accurate as larger and more diverse training sets, including a wider spectrum of ages, particularly infants, are studied.

Individuals with higher suicide risk showed more anger and disgust during rest

Previous research showed that suicide risk was associated with the anger trait and the facial expression of anger when advising on life dilemmas. We investigated if suicide risk was associated with the facial expression of anger during rest, a state when individuals often reflect upon their lives. Participants took a 1-min rest before being assessed for suicide risk. We measured 147 participants’ frontal-view facial expressions during their rest 1475–3694 times using automated facial expression analysis technology. Participants’ suicide risk was significantly positively correlated with their anger and disgust during the rest, which may be related to psychological pain and death-related thoughts among individuals with suicide risk. Therefore, rest for clinical patients should not be seen simply as a “rest” for the mind. Rather, for counselors, rest may open a window to look into patients’ inner thoughts that may be important to their lives.

TEACHING EYE-TRACKING AND FACIAL EXPRESSION ANALYSIS TECHNOLOGY IN AN ONLINE MARKETING RESEARCH CLASS

ABSTRACT Technology continues to change and improve the marketing profession. Marketing research technologies such as eye-tracking and facial expression analysis allow marketers to gain more insights into consumer behaviors. Recently, these technologies have become more affordable and applicable in the classroom learning environment. A very recent addition to these technologies is the ability to collect eye-tracking and facial expression data remotely through a compliant user’s webcam. This paper illustrates how marketing educators can incorporate eye-tracking and facial expression analysis technology into an online marketing research class. The teaching effectiveness results show that this 4-week online project using the remote technology helps students improve their learning and become more ready for future careers. The online experiential project using these technologies allows marketing educators to switch from face-to-face instruction to an online format while still maintaining the same quality of instruction and level of engagement.

CSR is not a panacea: The influence of CSR on disgust and turnover intention

Corporate social responsibility (CSR) is predominantly demonstrated as a panacea for employees. However, this study attempts to challenge common sense and focuses on the conditions under which CSR is related to disgust, subsequently resulting in employee turnover intentions. Drawing on the appraisal theory of disgust, we develop a conditional indirect effects model and propose that employee Machiavellianism and supervisor bottom-line mentality weaken the negative impact of CSR on turnover intention through disgust due to appraisals of violation to moral values. General support for the theoretical model is obtained from three studies, a scenario-based experiment, a laboratory experiment with facial expression analysis technology, and a multi-wave field study. Findings extend relevant literature and provide theoretical evidence for CSR practice.

How corporate social responsibility moderates the relationship between distributive unfairness and organizational revenge: a deontic justice perspective

PurposeThis paper aims to investigate the differential moderating effects of two types (internal/external) of corporate social responsibility (CSR) on the relationship between distributive injustice and organization-directed revenge through the mediating role of negative emotions.Design/methodology/approachThis paper conducts two studies. Study 1 was a vignette study based on a sample of 501 part-time master of business administration students in China aimed at testing the moderating effects of different levels of internal (external) CSR. Study 2 involved a laboratory experiment in which 108 postgraduate students were recruited to scrutinize the contrasting moderating effects of different types of CSR (internal vs external) and test the underlying mechanisms of negative emotions. The latest facial expression analysis technology (FaceReader 5.0 software) was used to detect participants’ emotional state.FindingsStudy 1 demonstrates that internal CSR buffers the relationship between distributive injustice and organizational revenge behavior through negative emotions. However, the moderating effect of external CSR is not significant. Study 2 reveals that compared with external CSR, distributive injustice induces fewer negative emotions in the presence of internal CSR and the mediating role of negative emotions detected by the facial expression analysis software is also verified.Practical implicationsThe authors hope that the findings of this paper can provide theoretical references for enterprise managers to enhance their employee governance, develop more effective intervention policies and formulate corresponding coping mechanisms to prevent and mitigate workplace revenge behaviors.Originality/valueFirst, this paper enriches the literature on the relationship between injustice and organization revenge by introducing CSR as an employee governance tool. Second, this paper reconciles prior inconsistent findings about employee response to CSR in the occurrence of negative events by distinguishing between external and internal CSR and examining the differential moderating effects of two types of CSR. Such distinction is derived from the heterogeneous justice perceptions arising from different CSR actions. In addition, the authors measure participants’ negative emotions through a multi-method approach integrating the latest technology for facial expression analysis and the PANAS scale, which represents a method advancement and provides implications for measuring emotions.

Face Patterns Analysis and Recognition System Based on Quantum Neural Network QNN

The past few years have witnessed a huge increase in the application of facial recognition, detection, and analysis technology. However, face recognition systems remain the most popular among the general public. The facial recognition system can detect the presence of a face when exposed to one. The accuracy and fairness that can be derived from such systems necessitate their use, because humans, particularly security personnel can be tired and target the wrong person as a suspect. However, artificial intelligence systems that are properly trained are capable of efficiently identifying and classifying faces without errors. In this work, the use of Matlab language was employed in building a software system that is capable of recognizing and differentiating different face patterns. The proposed system is equipped with a camera that serves as the practical aspect of the software that captures different shots that are sent to the theoretical part of a special program that is designed to recognize faces by comparing them with a database stored within the program. The practical part of the work involved the use of Quantum Neural Network. In this work, the training dataset is made up of features vectors that were obtained from a well-known set of face images of different people. Here, Principle Component Analysis (PCA) was used for the extraction of feature vectors from images and then prepared for the next training step. The experimental results revealed that efficient face recognition can be achieved through the use of well-trained Quantum Network.

The Hitchhiker’s Guide to Bias and Fairness in Facial Affective Signal Processing: Overview and techniques

Given the increasing prevalence of facial analysis technology, the problem of bias in the tools is now becoming an even greater source of concern. Several studies have highlighted the pervasiveness of such discrimination, and many have sought to address the problem by proposing solutions to mitigate it. Despite this effort, to date, understanding, investigating, and mitigating bias for facial affect analysis remain an understudied problem. In this work we aim to provide a guide by 1) providing an overview of the various definitions of bias and measures of fairness within the field of facial affective signal processing and 2) categorizing the algorithms and techniques that can be used to investigate and mitigate bias in facial affective signal processing. We present the opportunities and limitations within the current body of work, discuss the gathered findings, and propose areas that call for further research.

Facial analysis technology for the detection of Down syndrome in the Democratic Republic of the Congo

Down syndrome is one of the most common chromosomal anomalies affecting the world's population, with an estimated frequency of 1 in 700 live births. Despite its relatively high prevalence, diagnostic rates based on clinical features have remained under 70% for most of the developed world and even lower in countries with limited resources. While genetic and cytogenetic confirmation greatly increases the diagnostic rate, such resources are often non-existent in many low- and middle-income countries, particularly in Sub-Saharan Africa. To address the needs of countries with limited resources, the implementation of mobile, user-friendly and affordable technologies that aid in diagnosis would greatly increase the odds of success for a child born with a genetic condition. Given that the Democratic Republic of the Congo is estimated to have one of the highest rates of birth defects in the world, our team sought to determine if smartphone-based facial analysis technology could accurately detect Down syndrome in individuals of Congolese descent. Prior to technology training, we confirmed the presence of trisomy 21 using low-cost genomic applications that do not need advanced expertise to utilize and are available in many low-resourced countries. Our software technology trained on 132 Congolese subjects had a significantly improved performance (91.67% accuracy, 95.45% sensitivity, 87.88% specificity) when compared to previous technology trained on individuals who are not of Congolese origin (p < 5%). In addition, we provide the list of most discriminative facial features of Down syndrome and their ranges in the Congolese population. Collectively, our technology provides low-cost and accurate diagnosis of Down syndrome in the local population.

Objective differential diagnosis of Noonan and Williams-Beuren syndromes in diverse populations using quantitative facial phenotyping.

IntroductionPatients with Noonan and Williams–Beuren syndrome present similar facial phenotypes modulated by their ethnic background. Although distinctive facial features have been reported, studies show a variable incidence of those characteristics in populations with diverse ancestry. Hence, a differential diagnosis based on reported facial features can be challenging. Although accurate diagnoses are possible with genetic testing, they are not available in developing and remote regions.MethodsWe used a facial analysis technology to identify the most discriminative facial metrics between 286 patients with Noonan and 161 with Williams‐Beuren syndrome with diverse ethnic background. We quantified the most discriminative metrics, and their ranges both globally and in different ethnic groups. We also created population‐based appearance images that are useful not only as clinical references but also for training purposes. Finally, we trained both global and ethnic‐specific machine learning models with previous metrics to distinguish between patients with Noonan and Williams–Beuren syndromes.ResultsWe obtained a classification accuracy of 85.68% in the global population evaluated using cross‐validation, which improved to 90.38% when we adapted the facial metrics to the ethnicity of the patients (p = 0.024).ConclusionOur facial analysis provided for the first time quantitative reference facial metrics for the differential diagnosis Noonan and Williams–Beuren syndromes in diverse populations.

Defining the genotypic and phenotypic spectrum of X-linked MSL3-related disorder

PurposeWe sought to delineate the genotypic and phenotypic spectrum of female and male individuals with X-linked, MSL3-related disorder (Basilicata–Akhtar syndrome). MethodsTwenty-five individuals (15 males, 10 females) with causative variants in MSL3 were ascertained through exome or genome sequencing at ten different sequencing centers. ResultsWe identified multiple variant types in MSL3 (ten nonsense, six frameshift, four splice site, three missense, one in-frame-deletion, one multi-exon deletion), most proven to be de novo, and clustering in the terminal eight exons suggesting that truncating variants in the first five exons might be compensated by an alternative MSL3 transcript. Three-dimensional modeling of missense and splice variants indicated that these have a deleterious effect. The main clinical findings comprised developmental delay and intellectual disability ranging from mild to severe. Autism spectrum disorder, muscle tone abnormalities, and macrocephaly were common as well as hearing impairment and gastrointestinal problems. Hypoplasia of the cerebellar vermis emerged as a consistent magnetic resonance image (MRI) finding. Females and males were equally affected. Using facial analysis technology, a recognizable facial gestalt was determined. ConclusionOur aggregated data illustrate the genotypic and phenotypic spectrum of X-linked, MSL3-related disorder (Basilicata–Akhtar syndrome). Our cohort improves the understanding of disease related morbidity and allows us to propose detailed surveillance guidelines for affected individuals.

Noonan Syndrome in Thai Children

This study describes clinical features of Noonan syndrome and gene mutations, including PTPN11, SOS1, and BRAF in the Thai population.Widely spaced eyes were the most common finding from the digital facial analysis technology used in this study.

Rubinstein–Taybi syndrome in diverse populations

Rubinstein-Taybi syndrome (RSTS) is an autosomal dominant disorder, caused by loss-of-function variants in CREBBP or EP300. Affected individuals present with distinctive craniofacial features, broad thumbs and/or halluces, and intellectual disability. RSTS phenotype has been well characterized in individuals of European descent but not in other populations. In this study, individuals from diverse populations with RSTS were assessed by clinical examination and facial analysis technology. Clinical data of 38 individuals from 14 different countries were analyzed. The median age was 7 years (age range: 7 months to 47 years), and 63% were females. The most common phenotypic features in all population groups included broad thumbs and/or halluces in 97%, convex nasal ridge in 94%, and arched eyebrows in 92%. Face images of 87 individuals with RSTS (age range: 2 months to 47 years) were collected for evaluation using facial analysis technology. We compared images from 82 individuals with RSTS against 82 age- and sex-matched controls and obtained an area under the receiver operating characteristic curve (AUC) of 0.99 (p < .001), demonstrating excellent discrimination efficacy. The discrimination was, however, poor in the African group (AUC: 0.79; p = .145). Individuals with EP300 variants were more effectively discriminated (AUC: 0.95) compared with those with CREBBP variants (AUC: 0.93). This study shows that clinical examination combined with facial analysis technology may enable earlier and improved diagnosis of RSTS in diverse populations.

Development of a deep multimodal hedonic recognition database for oral stimuli

The emotional reactions that certain products evoke (like or dislike) play a key role in the user acceptance or preference toward these products and influence the choice behaviors. Facial expressions constitute a primary channel to express and communicate emotions. In particular, the facial reactions evoked by certain stimuli are generated subconsciously, and they can be easily captured by video recording. However, it is challenging to quantify and objectively describe facial expressions. Until now, most implicit measurements based on facial expression analyses have been implemented considering “basic emotions”, which cannot directly represent like/dislike. How can we identify the affective state of consumers directly by using facial expression analysis technology? In this study, hedonic rating categories are defined based on the hedonic scale, and automatic facial expression analysis technology is applied to evaluate the suitability of the defined categories. Moreover, a visual multimodal facial database (RGB, depth, and infrared images) for the hedonic recognition of Asian individuals is constructed. A hybrid deep learning framework is employed to exploit the spatiotemporal information of the collected data to realize hedonic recognition. Early and late fusion strategies are used to investigate the suitability of the individual modalities and their complementarity. The experimental results indicate that the late fusion of the three modalities yielded the highest recognition accuracy compared to those obtained using individual modalities and the early fusion scheme. These results demonstrate the effectiveness of fusing different visual facial data and indicate that the use of multimodality-based facial expression analyses could enhance the accuracy of hedonic recognition.

A Buffer or a Liability? Double-Edged Sword Effect of Ex Ante CSR Communication

As corporate social responsibility (CSR) matters not only for firms’ competitiveness, but also societies’ sustainability, stakeholders prefer firms engaging in positive CSR. Yet, the effect of ex ante positive CSR communication on relationship between firms’ irresponsibility events occurrence and investment decision is quite overlooked. Drawing on the dual processing theory, we posit that investors feel anger and form moral judgment in the context of moral dilemma. Furthermore, we distinguish between two dimensions of corporate social irresponsibility (CSIR) identified in prior work, CSIR type and CSIR-CSR domain, and propose that the ex ante positive CSR communication exerts a double-edged sword effect (a buffer or a liability) on investor responses depending on different CSIR events. Data from a lab experiment conducted by facial expression analysis technology (Study 1) and a scenario experiment (Study 2) generally support our hypotheses. Our research contributes to the theoretical and practical insights into how and when ex ante CSR communication may, trigger different investor responses, along with directions for future research on business ethics.

Noonan syndrome on the African Continent.

Noonan syndrome is a common genetic syndrome caused by pathogenic variants in genes in the Ras/MAPK signaling pathway. The medical literature has an abundance of studies on Noonan syndrome, but few are from the African continent. The medical literature was searched for studies on Noonan syndrome from the African continent and these reports were added to our experience in Africa. Facial analysis was reviewed from two previous reports from our group using a support vector machine (SVM) algorithm and an analysis using the Face2Gene convolutional neural network technology. Individuals with Noonan syndrome from reports in African populations have the classic phenotype characteristics including typical minor facial anomalies such as widely spaced eyes (31-100%), short stature (71-100%), and congenital heart disease with pulmonary stenosis found in 24-100% of patients. Similarly, the genotypes are similar with the majority of variants occurring in the gene PTPN11 (72%) and 36% of these variants occurred in the amino acid residue Asn308, which is most commonly found in other populations. The two separate facial analysis algorithms successfully discriminated Africans with NS from unaffected matched individuals with area under the curve (AUC) of the receiver operator characteristic of 0.94 (SVM) and 0.979 for the Face2Gene research methodology. Few studies characterizing Noonan syndrome in Africans have been conducted, highlighting the need for more genetic and genomic research in African populations. Available clinical data, genotypes, and facial analysis technology data show that individuals of African descent with NS can be efficiently diagnosed using available standards.