Voice Modalities Research Articles

Feature-level fusion of face and speech based multimodal biometric attendance system with liveness detection

The rapid growth of deep learning and the Internet of Things has spurred the need for touchless biometric systems in areas where cleanliness and non-intrusive user interaction are critical. In order to authenticate seamlessly, traditional biometric methods such as fingerprint, hand, etc. recognition require physical contact and, therefore, risk hygiene issues, hence making face and speaker verification more viable alternatives. A robust Multimodal Biometric Attendance System (MBAS) is needed due to the vulnerabilities and limitations of single modality systems. In this research, we introduce MBAS using feature-level fusion of speech data with face data, combining the best of both worlds. The textural features based on a person’s facial appearance are integrated with dynamic speech information for liveness detection, followed by dimensionality reduction using linear discriminant analysis, and then incorporated into a Bi-LSTM classifier. Therefore, for better security, accuracy, and anti-spoofing attacks, this approach is proposed in addition to increasing accuracy as well as enhancing security against spoofing attacks. Two publicly available datasets, DeepfakeTIMIT and AVSpeech, are extensively explored to evaluate different fusion strategies, classifier types, and standard performance metrics. The proposed system outperformed other cutting-edge biometric based systems by exhibiting a 97.51% high accuracy rate with a precision of 99.10% and an equal error rate of 2.48%. These findings affirm the effectiveness and possible real-world applications of the MBAS concept, along with its enhancement ensuring safety. Furthermore, this study underscores the importance of incorporating advanced liveness detection into secure contactless biometrics solutions for modern attendance management in various industries that encompass both face and voice modalities.

Enhancing Biometric Security with Bimodal Deep Learning and Feature-level Fusion of Facial and Voice Data

Recent research in biometric technologies underscores the benefits of multimodal systems that use multiple traits to enhance security by complicating the replication of samples from genuine users. To address this, we present a bimodal deep learning network (BDLN or BNet) that integrates facial and voice modalities. Voice features are extracted using the SincNet architecture, and facial image features are obtained from convolutional layers. Proposed network fuses these feature vectors using either averaging or concatenation methods. A dense connected layer then processes the combined vector to produce a dual-modal vector that encapsulates distinctive user features. This dual-modal vector, processed through a softmax activation function and another dense connected layer, is used for identification. The presented system achieved an identification accuracy of 99% and a low equal error rate (EER) of 0.13% for verification. These results, derived from the VidTimit and BIOMEX-DB datasets, highlight the effectiveness of the proposed bimodal approach in improving biometric security.

Multimodal Biometric Authentication System Using of Autoencoders and Siamese Networks for Enhanced Security

Ensuring secure and reliable identity verification is crucial, and biometric authentication plays a significant role in achieving this. However, relying on a single biometric trait, unimodal authentication, may have accuracy and attack vulnerability limitations. On the other hand, multimodal authentication, which combines multiple biometric traits, can enhance accuracy and security by leveraging their complementary strengths. In the literature, different biometric modalities, such as face, voice, fingerprint, and iris, have been studied and used extensively for user authentication. Our research introduces a highly effective multimodal biometric authentication system with a deep learning approach. Our study focuses on two of the most user-friendly safety mechanisms: face and voice recognition. We employ a convolutional autoencoder for face images and an LSTM autoencoder for voice data to extract features. These features are then combined through concatenation to form a joint feature representation. A Siamese network carries out the final step of user identification. We evaluated our model’s efficiency using the OMG-Emotion and RAVDESS datasets. We achieved an accuracy of 89.79% and 95% on RAVDESS and OMG-Emotion datasets, respectively. These results are obtained using a combination of face and voice modality.

Understanding is a Two-Way Street: User-Initiated Repair on Agent Responses and Hearing in Conversational Interfaces

Although methods for repairing prior turns in natural conversation are critical for enabling mutual understanding, or successful communication, these methods are seldom built into conversational user interfaces systematically. Chatbots and voice assistants tend to ask users to paraphrase what they said if it was not understood, but users cannot do the same if they encounter trouble in understanding what the agent said. Understanding is a one-way street in most (intent-based) conversation-like interfaces. An exception to this is Moore and Arar (2019), who demonstrate nine types of user-initiated repair on agent responses that are common in natural conversation and who have shown that users will employ these repair features correctly in text-based interfaces if taught. In this small-scale study, we test these user-initiated repairs (in second position) in a voice-based interface. With understanding-oriented repairs, we found that participants employed them much the same way in text and voice. In addition, we examine some hearing- and speaking-oriented repairs that emerged from the use of our novel multi-modal interface. We found that participants used them to manage troubles specific to the voice modality. Analysis of user logs and transcripts suggests that user-initiated repair features are valuable components of conversational interfaces.

Designing an age-friendly conversational AI agent for mobile banking: the effects of voice modality and lip movement

Age factor is a significant barrier in adopting mobile banking mainly due to low competence in using the system and security concerns. Thus, this study presents a conversational artificial intelligence agent-based mobile banking app that is specially designed for seniors considering their needs and physical characteristics. Voice modality and lip-movement were identified as the important factors to improve seniors’ mobile banking service experience and the effect of them were empirically tested by conducting a controlled experiment with between-subject design. In total, 91 participants who were 55 years or older used the app to complete the given tasks of online banking. Results showed that those in the two voice modality conditions, compared to text agent condition, reported higher levels of trust, perceived ease of use, and social presence. The effect of lip movement of the agent also showed significant results in those three dependant variables. The result of mediation analysis revealed that agent's lip movement increased social presence and perceived ease of use, which further increased trust toward the agent. The results are meaningful especially because trust and perceived ease of use were listed as the main reason for not using online banking services among non-users. The implications of the study were discussed.

Design and Feasibility Analysis of a Smartphone-Based Digital Cognitive Assessment Study in the Framingham Heart Study.

Smartphone-based digital technology is increasingly being recognized as a cost-effective, scalable, and noninvasive method of collecting longitudinal cognitive and behavioral data. Accordingly, a state-of-the-art 3-year longitudinal project focused on collecting multimodal digital data for early detection of cognitive impairment was developed. A smartphone application collected 2 modalities of cognitive data, digital voice and screen-based behaviors, from the FHS (Framingham Heart Study) multigenerational Generation 2 (Gen 2) and Generation 3 (Gen 3) cohorts. To understand the feasibility of conducting a smartphone-based study, participants completed a series of questions about their smartphone and app use, as well as sensory and environmental factors that they encountered while completing the tasks on the app. Baseline data collected to date were from 537 participants (mean age=66.6 years, SD=7.0; 58.47% female). Across the younger participants from the Gen 3 cohort (n=455; mean age=60.8 years, SD=8.2; 59.12% female) and older participants from the Gen 2 cohort (n=82; mean age=74.2 years, SD=5.8; 54.88% female), an average of 76% participants agreed or strongly agreed that they felt confident about using the app, 77% on average agreed or strongly agreed that they were able to use the app on their own, and 81% on average rated the app as easy to use. Based on participant ratings, the study findings are promising. At baseline, the majority of participants are able to complete the app-related tasks, follow the instructions, and encounter minimal barriers to completing the tasks independently. These data provide evidence that designing and collecting smartphone application data in an unsupervised, remote, and naturalistic setting in a large, community-based population is feasible.

Drivers' Comprehensive Emotion Recognition Based on HAM.

Negative emotions of drivers may lead to some dangerous driving behaviors, which in turn lead to serious traffic accidents. However, most of the current studies on driver emotions use a single modality, such as EEG, eye trackers, and driving data. In complex situations, a single modality may not be able to fully consider a driver's complete emotional characteristics and provides poor robustness. In recent years, some studies have used multimodal thinking to monitor single emotions such as driver fatigue and anger, but in actual driving environments, negative emotions such as sadness, anger, fear, and fatigue all have a significant impact on driving safety. However, there are very few research cases using multimodal data to accurately predict drivers' comprehensive emotions. Therefore, based on the multi-modal idea, this paper aims to improve drivers' comprehensive emotion recognition. By combining the three modalities of a driver's voice, facial image, and video sequence, the six classification tasks of drivers' emotions are performed as follows: sadness, anger, fear, fatigue, happiness, and emotional neutrality. In order to accurately identify drivers' negative emotions to improve driving safety, this paper proposes a multi-modal fusion framework based on the CNN + Bi-LSTM + HAM to identify driver emotions. The framework fuses feature vectors of driver audio, facial expressions, and video sequences for comprehensive driver emotion recognition. Experiments have proved the effectiveness of the multi-modal data proposed in this paper for driver emotion recognition, and its recognition accuracy has reached 85.52%. At the same time, the validity of this method is verified by comparing experiments and evaluation indicators such as accuracy and F1 score.

A Human-Centered and Adaptive Robotic System Using Deep Learning and Adaptive Predictive Controllers

The rise of single-person households coupled with a drop in social interaction due to the coronavirus disease 2019 (COVID-19) pandemic is triggering a loneliness pandemic. This social issue is producing mental health conditions (e.g., depression and stress) not only in the elderly population but also in young adults. In this context, social robots emerge as human-centered robotics technology that can potentially reduce mental health distress produced by social isolation. However, current robotics systems still do not reach a sufficient communication level to produce an effective coexistence with humans. This paper contributes to the ongoing efforts to produce a more seamless human-robot interaction. For this, we present a novel cognitive architecture that uses (i) deep learning methods for mood recognition from visual and voice modalities, (ii) personality and mood models for adaptation of robot behaviors, and (iii) adaptive generalized predictive controllers (AGPC) to produce suitable robot reactions. Experimental results indicate that our proposed system influenced people’s moods, potentially reducing stress levels during human-robot interaction.

P‐2.25: Research on Virtual Reality Field Based on Multimodal Emotion Recognition

At present, emotion recognition has become a research hotspot in the field of pattern recognition. Considering the problems of incomplete information and strong interference in single‐modal emotion recognition, multimodal emotion recognition has been widely studied. Multimodal data includes, but is not limited to, emoji, text, and voice modality data. There are various ways to express emotion, among which expression, text and voice are the most direct and reliable emotional information carriers. Therefore, it is of great research and practical significance to comprehensively consider the emotion recognition research of expression, text and voice modalities, and to apply its research results to the field of virtual reality (referred to as VR).This paper analyzes the relevant situation of multimodal emotion recognition, extracts features of voice, text and expression, and then fuses them into multimodal for emotional analysis, and applies it to the VR field. The main work content is as follows: the relevant technologies of multimodal emotion recognition research in the field of VR are introduced, including deep learning related technologies, virtual reality technology, and multimodal fusion methods. In terms of deep learning, the focus is on convolutional neural networks and recurrent neural networks and their variants. In terms of virtual reality technology, the characteristics and applications of virtual reality are introduced. In terms of multimodal fusion, three commonly used fusion methods are introduced.

Which components of famous people recognition are lateralized? A study of face, voice and name recognition disorders in patients with neoplastic or degenerative damage of the right or left anterior temporal lobes

We administered to large groups of patients with neoplastic or degenerative damage affecting the right or left ATL, the ‘Famous People Recognition Battery’ (FPRB), in which subjects are required to recognize the same 40 famous people through their faces, voices and names, to clarify which components of famous people recognition are lateralized. At the familiarity level, we found, as expected, a dissociation between a greater impairment of patients with right ATL lesions on the non-verbal (face and voice) recognition modalities and of those with left ATL lesions on name familiarity. Equally expected were results obtained at the naming level, because the worse naming scores for faces and voices were observed in left-sided patients. Less foregone were, for two reasons, results obtained at the semantic level. First, no difference was found between the two hemispheric groups when scores obtained on the verbal (name) and non-verbal (face and voice) recognition modalities were account for. Second, the face and voice recognition modalities showed a different degree of right lateralization. All groups of patients showed, indeed, both at the familiarity and at the semantic level, a greater difficulty in the recognition of voices regarding faces, but this difference reached significance only in patients with right ATL lesions, suggesting a greater right lateralization of the more complex task of voice recognition. A model aiming to explain the greater right lateralization of the more perceptually demanding voice modality of person recognition is proposed.

Yōkobo

As part of a multidisciplinary research and a PhD project to strengthen the connection between retired couples living at home, we imagined and designed Yōkobo. It is a robot at the crossroads of a sensitive approach and a robotic trend that bridges the gap between humans (Human-Robot-Human Interactions field). As a theoretical contribution, Yōkobo is at the intersection of various concepts: behavioral objects, robjects, weak robotics, and slow technology. Yōkobo is a trinket bowl placed in the entrance of homes. Its discreet presence expresses hospitality and celebrates small moments of everyday life, welcoming visitors and inhabitants of the house. The name comes from the contraction of “yōkoso” (welcome in Japanese) and “robot” (with French pronunciation). In addition to these functions, Yōkobo expresses the state of the home using data from connected IoT devices, combining various house parameters (such as temperature, air quality, etc.) to express the home's “mood” through its motions. Finally, Yōkobo used in tandem with house keys, can convey a trace, a message based on motion. And a trace is a memory of the partner's passage. Yōkobo is resolutely innovative and disruptive. It does not sit within the lineage of the general vision of what robots are and what they can do: it is an object intended to be unobtrusive, stemming from ambient computing, while having an ongoing subtle presence. It does not make sounds, unlike voice assistants and the trend for using voice modality interaction. It expresses its environment only through motion and light. to move away from home's companion robots and the biases they can generate through facial representation, Yōkobo has neither an anthropomorphic shape nor can talk. Yōkobo is intended to be made of natural materials such as ceramic, wood, or wool to break with the idea of plastic, disposable, and toy robots, and to improve its integration in everyday home life. as a slow technology product, understanding and integrating Yōkobo into one's life takes time and requires accepting not having a clear, repetitive, and instantaneous response to an action. Its contribution is not measured in terms of efficiency and utility; it is the sum of different experiences with the product over time that creates the object's meaning and value. Getting to know Yōkobo's expressive motions is continuous and progressive. Yōkobo is an object that is understood through perception and touches the poetic sensibility of its users. Yōkobo is a concept that puts people's relationships at the center. It does not impose itself to propose an exclusive Human-Object relationship. It reveals the presence of the other by expressing the last impermanent trace of the other's passage. It is an object of sensitive presence. This work is the result of interdisciplinary research between roboticists, designers, and ergonomists. The navigation (directions and overlay) of this pan.able demonstrates the design and engineering processes, as well as the interaction modalities.

Is voice really persuasive? The influence of modality in virtual assistant interactions and two alternative explanations

PurposeVirtual assistants are increasingly used for persuasive purposes, employing the different modalities of voice and text (or a combination of the two). In this study, the authors compare the persuasiveness of voice-and text-based virtual assistants. The authors argue for perceived human-likeness and cognitive load as underlying mechanisms that can explain why voice- and text-based assistants differ in their persuasive potential by suppressing the activation of consumers' persuasion knowledge.Design/methodology/approachA pre-registered online-experiment (n = 450) implemented a text-based and two voice-based (with and without interaction history displayed in text) virtual assistants.FindingsFindings show that, contrary to expectations, a text-based assistant is perceived as more human-like compared to a voice-based assistant (regardless of whether the interaction history is displayed), which in turn positively influences brand attitudes and purchase intention. The authors also find that voice as a communication modality can increase persuasion knowledge by being cognitively more demanding in comparison to text.Practical implicationsSimply using voice as a presumably human cue might not suffice to give virtual assistants a human-like appeal. For the development of virtual assistants, it might be beneficial to actively engage consumers to increase awareness of persuasion.Originality/valueThe current study adds to the emergent research stream considering virtual assistants in explicitly exploring modality differences between voice and text (and a combination of the two) and provides insights into the effects of persuasion coming from virtual assistants.

Understanding Emerging Media: Voice, Agency, and Precarity in the Post-2011 Arab Mediasphere

The decade following the 2010–2011 Arab uprisings saw a flourishing of emerging media organisations across the region. The most recognisable examples of these new independent media actors include <em>Enab Baladi</em> in Syria, <em>Mada Masr</em> in Egypt, and Inkyfada in Tunisia. However, this phenomenon comprises a much more diverse set of actors from small-scale associative radio stations in Tunisia to numerous exilic Syrian media outlets. Building on previous research as well as recent fieldwork in Tunisia and Turkey, this article is an attempt to make sense of the genesis, development, and relevance of this new class of media actors. We argue that these emerging media organisations can be seen to represent specific interventions into the politics of voice in their various national and local contexts, but ones that share similar logics. To elucidate this argument, we propose a multi-dimensional understanding of these interventions that brings together voices (actors, issues, discourses), modalities of voice (organisational models, values, production value), and the underlying political economy of these emerging media (funding, institutionalisation). However, the article also argues that these interventions, and the logics they share, themselves belie a complex interaction between the political and professional agency and precarity of these media organisations and the individuals, and groups, behind them. We believe that combining these two perspectives is a necessary step for a more nuanced understanding of the nature and practice of these emerging media organisations.

Subverting the Electronic Language Textbook to Make it Relevant

The current article was inspired by two incongruous phenomena: the perception of the lack of gender sensitivity and inclusiveness in a particular language textbook and the coronavirus disease pandemic. As we went completely remote over the past 2.5 semesters, students were encouraged to address those issues via chat and voice modalities. If there is a silver lining to the tragic events of the past 15 months, it was that remote, synchronous discussion was vastly more participatory than in-class discussion with regard to those and other issues. I argue that, in designing our textbook-dependent courses, we can use such deficiencies to our advantage through some of the tools that many of us were introduced to during the pandemic.

The Role of Unimodal Feedback Pathways in Gender Perception During Activation of Voice and Face Areas.

Cross-modal effects provide a model framework for investigating hierarchical inter-areal processing, particularly, under conditions where unimodal cortical areas receive contextual feedback from other modalities. Here, using complementary behavioral and brain imaging techniques, we investigated the functional networks participating in face and voice processing during gender perception, a high-level feature of voice and face perception. Within the framework of a signal detection decision model, Maximum likelihood conjoint measurement (MLCM) was used to estimate the contributions of the face and voice to gender comparisons between pairs of audio-visual stimuli in which the face and voice were independently modulated. Top–down contributions were varied by instructing participants to make judgments based on the gender of either the face, the voice or both modalities (N = 12 for each task). Estimated face and voice contributions to the judgments of the stimulus pairs were not independent; both contributed to all tasks, but their respective weights varied over a 40-fold range due to top–down influences. Models that best described the modal contributions required the inclusion of two different top–down interactions: (i) an interaction that depended on gender congruence across modalities (i.e., difference between face and voice modalities for each stimulus); (ii) an interaction that depended on the within modalities’ gender magnitude. The significance of these interactions was task dependent. Specifically, gender congruence interaction was significant for the face and voice tasks while the gender magnitude interaction was significant for the face and stimulus tasks. Subsequently, we used the same stimuli and related tasks in a functional magnetic resonance imaging (fMRI) paradigm (N = 12) to explore the neural correlates of these perceptual processes, analyzed with Dynamic Causal Modeling (DCM) and Bayesian Model Selection. Results revealed changes in effective connectivity between the unimodal Fusiform Face Area (FFA) and Temporal Voice Area (TVA) in a fashion that paralleled the face and voice behavioral interactions observed in the psychophysical data. These findings explore the role in perception of multiple unimodal parallel feedback pathways.

Hypnosis as a Supplemental Treatment for and a Prevention of Muscle Tension Dysphonia

ABSTRACT Diagnosable only through a healthcare professional, Muscle Tension Dysphonia (MTD) is a compensatory muscle hyperfunction of the laryngeal mechanism. A significant cause of this singing voice disorder is anxiety and is symptomatic of inefficient neuromuscular patterns in the larynx. Numerous noninvasive studio voice modalities treat or reduce the muscular and phonatory symptoms of MTD, but a more systemic treatment of underlying mental causes may be considered. Recent studies have shown positive psychological and neuromuscular implications using hypnosis. This paper argues using hypnosis and autohypnosis as a top-down approach to anxiety-induced MTD.

BIOMEX-DB: A Cognitive Audiovisual Dataset for Unimodal and Multimodal Biometric Systems

Multimodal biometric schemes arise as an interesting solution to the multidimensional reinforcement problem for biometric security systems. Along with the performance dimension, these systems should also comply with required levels for other conditions such as permanence, collectability, and circumvention, among others. In response to the demand for a multimodal and synchronous dataset, we introduce in this paper an open-access database of synchronously recorded electroencephalogram signals (EEG), voice signals, and video feed from 51 volunteers, 25 female, 26 male, captured for, but not limited to, biometric purposes. A total of 140 samples were collected from each user when pronouncing single digits in Spanish, giving a total of 7140 instances. EEG signals were captured using a 14-channel Emotiv™ Epoc headset. The resulting set becomes a valuable resource when working on unimodal biometric systems, but significantly more for the evaluation of multimodal variants. Furthermore, the usefulness of the collected signals extends to being exploited by projects in brain-computer interfaces and face recognition to name just a few. As an initial report on data separability of the related samples, five user recognition experiments are presented: a face recognition identifier with an accuracy of 99%, a speaker identification system with accuracy of 94.2%, a bimodal face-speech verification case with Equal Error Rate around 2.64, an EEG identification example, and a bimodal user identification exercise based on EEG and voice modalities with a registered accuracy of 97.6%.

Neural Correlates of Esophageal Speech: An fMRI Pilot Study

The esophageal speech is one of the possible alaryngeal voices resulting after total laryngectomy. Its production is made by the regurgitation of the air coming from the esophagus, sonorized through the passage from the walls of the upper esophageal sphincter. The neural correlates of this voice have never been investigated, while the neural control of laryngeal voice has been already documented by different studies. Four patients using esophageal speech after total laryngectomy and four healthy controls underwent functional magnetic resonance imaging. The fMRI experiment was carried out using a "Block Design Paradigm." Comparison of the phonation task in the two groups revealed higher brain activities in the cingulate gyrus, the cerebellum and the medulla as well as lower brain activities in the precentral gyrus, the inferior and middle frontal gyrus and the superior temporal gyrus in the laryngectomized group. The findings in this pilot study provide insight into neural phonation control in laryngectomized patients with esophageal speech. The imaging results demonstrated that in patients with esophageal speech, altered brain activities can be observed. The adaptive changes in the brain following laryngectomy reflect the changes in the body and in the voice modality. In addition, this pilot study establishes that a blocked design fMRI is sensitive enough to define a neural network associated with esophageal voice and lays the foundation for further studies in this field.

Security in smart cities: A brief review of digital forensic schemes for biometric data

A smart city is engineered to be a self-sustained ecosystem driven by Internet-of-Things (IoT) devices. Smooth functioning of smart cities is conditioned on seamless communication between users and devices. Smart devices equipped with biometric authentication can offer security as well as personalized experience to the end users. Currently, a number of smart devices employ face, fingerprint, and voice modalities for user verification. However, the biometric data acquired by these devices can be digitally manipulated or tampered with, that can compromise the security of the smart environment. Further, the preponderance of biometric data such as face and voice in social media applications, necessitates the validation of their integrity. In this work, we review state-of-the-art digital forensic schemes for audio-visual biometric data that can be leveraged by applications designed for smart cities.

ДВУХКОМПОНЕНТНАЯ СЕТЕВАЯ МОДЕЛЬ В ТЕХНОЛОГИЯХ ГОЛОСОВОЙ ИДЕНТИФИКАЦИИ ЛИЧНОСТИ

Among the most important parameters of biometric systems with voice modalities that determine their effectiveness, along with reliability and noise immunity, a speed of identification and verification of a person has been accentuated. This parameter is especially sensitive while processing large-scale voice databases in real time regime. Many research studies in this area are aimed at developing new and improving existing algorithms for presentation and processing voice records to ensure high performance of voice biometric systems. Here, it seems promising to apply a modern approach, which is based on complex platform for solving complex massive problems with a large number of elements and taking into account their interrelationships. Thus, there are known some works which while solving problems of analysis and recognition of faces from photographs, transform images into complex networks for their subsequent processing by standard techniques. One of the first applications of complex networks to sound series (musical and speech) analysis are description of frequency characteristics by constructing models - converting the series into networks. On the ontology platform a previously proposed technique of audio information representation aimed on its automatic analysis and speaker recognition has been developed. This implies converting information into the form of associative semantic (cognitive) structure with amplitude and frequency components both. Two speaker exemplars have been recorded and transformed into pertinent networks with consequent comparison of their topological metrics. The set of topological metrics for each of models (amplitude and frequency one ) is a vector, and together those combine a matrix, as a digital network voiceprint. The proposed approach, with its sensitivity to personal conditions-physiological, psychological, emotional, might be useful not only for person identification, but also for determining his /her condition.