Multimodal Voice Research Articles

Voice coding experiences for developers with physical impairments

PurposeThis research explores the current working practices of voice coders with physical impairments and investigates their perceptions of a research prototype utilising a fixed grammar speech coding approach.Design/methodology/approachSemi-structured interviews were initially conducted with seven voice coders with physical impairments to understand their development practices, followed by an exploratory user study with five disabled voice coders to obtain their feedback on a fixed grammar voice coding system.FindingsInterviews provided new insights around the tools voice coders utilise, the need for multimodal coding approaches, as well as experiences in working within mixed ability development teams. Findings from the user evaluation elicited views around the need for distinct monosyllable voice commands, as well as other requirements to support efficient voice coding (e.g. command chaining, intuitive navigation, and custom command definition).Research limitations/implicationsThe research was conducted with experienced voice coders which may have influenced perceptions of the voice coding prototype. The system was also evaluated over a single evaluation session, whereas longitudinal research with novice voice coders could also present additional insights.Practical implicationsFurther research is required into customisable multimodal voice coding approaches for developers with physical impairments who may have varying levels of experience in coding via alternative methods.Originality/valueThe findings present new insights around the working practices and unique requirements of voice coders with physical impairments and highlights important new avenues for future research.

Multimodal Voice Activity Projection for Turn-taking and Effects on Speaker Adaptation

Multimodal Voice Activity Projection for Turn-taking and Effects on Speaker Adaptation

A Review of Recent Machine Learning Approaches for Voice Authentication Systems

Voice authentication systems are a comfortable way of protection since users do not need to remember passwords or carry identification cards. As a unique identifier for all individuals, voice is a practical tool to authenticate people into security services, including online banking and phonebased customer or computer services. Single-model voice authentication systems refer to voice recognition systems that utilize a single voice model to verify the identity of individuals based on their unique vocal characteristics, such as pitch, tone, and other speech patterns. For multi-model voice authentication systems, additional biometric factors like facial recognition or electroencephalogram data are included in the voice authentication process to enhance security. This paper reviews recent single-modal and multimodal voice authentication studies with an explanation of underlying feature extraction and classification methods. This paper also discusses security attacks on voice authentication systems, including random attacks, mimicry attacks, replay attacks, voice synthesizing attacks, counterfeit attacks, and hidden voice command attacks.

Crafting Multi-Modal Interactions on Voice Assistants

Voice assistants are finding adoption because of their ease and intuitiveness of use. While voice has been the dominant mode of interaction of humans with voice assistants, some embodiments of voice assistants also provide alternative modalities for interaction, popularly, via visual or touch interface. In this paper, the end-to-end working of a multi-modal voice assistant is provided, followed by a deep dive into the challenges associated specifically with multimodal voice assistants. This is followed by mitigation strategies for the challenges arising out of multimodal interaction scenarios. Next, architectural and design guidelines are provided that can provide a seamless user experience. Finally, future research areas have been identified

Journal of Engineering and Applied Sciences Technology

Voice assistants are finding adoption because of their ease and intuitiveness of use. While voice has been the dominant mode of interaction of humans with voice assistants, some embodiments of voice assistants also provide alternative modalities for interaction, popularly, via visual or touch interface. In this paper, the end-to-end working of a multi-modal voice assistant is provided, followed by a deep dive into the challenges associated specifically with multimodal voice assistants. This is followed by mitigation strategies for the challenges arising out of multimodal interaction scenarios. Next, architectural and design guidelines are provided that can provide a seamless user experience. Finally, future research areas have been identified

Multi-modal voice pathology detection architecture based on deep and handcrafted feature fusion

Automatic voice pathology detection systems can effectively help clinicians by enabling objective assessment and diagnosis in early stage of voice pathologies. This paper suggests a novel multi-modal architecture utilizing speech and electroglottography (EGG) signals and investigates their effectiveness in automatic detection of voice pathology. The proposed multi-modal framework combines two parallel Convolutional Neural Networks (CNNs), one for voice signals and the other for EGG signals, to obtain deep features. Classical handcrafted features are also obtained in the same manner. These features are then concatenated to obtain a more prominent feature set. In addition, a feature selection method is applied to remove redundant features. Finally, a SVM classifier is utilized to detect the voice pathology. In order to measure the performance of the proposed pathology detection system, various experiments are conducted on Saarbruecken Voice Database (SVD) without excluding any available pathology or sample. The experimental results show that the proposed voice pathology detection method achieves accuracy up to 90.10% using all speech and EGG samples. Also, sensitivity, specificity and F1-score results of 92.9%, 84.6% and 92.57% are obtained, respectively. The proposed method provides better performance than those given in the literature using all SVD samples through cross-validation testing. Hence, it is promising for automatic detection applications of voice pathology.

MMDD-Ensemble: A Multimodal Data-Driven Ensemble Approach for Parkinson's Disease Detection.

Parkinson's disease (PD) is the second most common neurological disease having no specific medical test for its diagnosis. In this study, we consider PD detection based on multimodal voice data that was collected through two channels, i.e., Smart Phone (SP) and Acoustic Cardioid (AC). Four types of data modalities were collected through each channel, namely sustained phonation (P), speech (S), voiced (V), and unvoiced (U) modality. The contributions of this paper are twofold. First, it explores optimal data modality and features having better information about PD. Second, it proposes a MultiModal Data–Driven Ensemble (MMDD-Ensemble) approach for PD detection. The MMDD-Ensemble has two levels. At the first level, different base classifiers are developed that are driven by multimodal voice data. At the second level, the predictions of the base classifiers are fused using blending and voting methods. In order to validate the robustness of the propose method, six evaluation measures, namely accuracy, sensitivity, specificity, Matthews correlation coefficient (MCC), and area under the curve (AUC), are adopted. The proposed method outperformed the best results produced by optimal unimodal framework from both the key evaluation aspects, i.e., accuracy and AUC. Furthermore, the proposed method also outperformed other state-of-the-art ensemble models. Experimental results show that the proposed multimodal approach yields 96% accuracy, 100% sensitivity, 88.88% specificity, 0.914 of MCC, and 0.986 of AUC. These results are promising compared to the recently reported results for PD detection based on multimodal voice data.

Multimodal dialogue in small-group mathematics learning

Abstract In this paper, we combine dialogic and embodied theories of learning to create a unified analytic lens. Embodied cognition is a theoretical approach operating under the premise that thinking and communication are multimodal activities. Under this premise, dialogue between learners needs to be conceptualized using a multimodal lens. We identify multimodal voices as speech and movement bundles situated within a learning context and describe a phenomenon that we call Multimodal Dialogue – multimodal interaction between different multimodal voices. To demonstrate this phenomenon, we analyze a learning sequence by two third-grade students who participated in a mathematics lesson aimed to foster embodied learning of proportion. Our analysis zooms in on the phenomenon of a multimodal voice as a speech-and-movement bundle situated within a learning context. We further show how multimodal dialogic gaps – differences between multimodal voices within and between modalities – drive communication and eventual changes in voices.

$$hf_0$$: A Hybrid Pitch Extraction Method for Multimodal Voice

Pitch or fundamental frequency ( $$f_0$$ ) estimation is a fundamental problem extensively studied for its potential speech and clinical applications. The existing $$f_0$$ estimation methods degrade in performance when applied over real-time audio signals with varying $$f_0$$ modulations and high SNR environment. In this work, a $$f_0$$ estimation method using both signal processing and deep learning approaches is developed. Specifically, we train a convolutional neural network to map the periodicity-rich input representation to pitch classes, such that the number of pitch classes is drastically reduced compared to existing deep learning approaches. Then, the accurate $$f_0$$ is estimated from the nominal pitch classes based on signal processing approaches. The observations from the experimental results showed that the proposed method generalizes to unseen modulations of speech and noisy signals (with various types of noise) for large-scale datasets. Also, the proposed hybrid model significantly reduces the learning parameters required to train the model compared to other methods. Furthermore, the evaluation measures showed that the proposed method performs significantly better than the state-of-the-art signal processing and deep learning approaches.

Perceived Usefulness of Multimodal Voice Assistant Technology

Smart speaker devices are appealing to consumers, but the perceived usefulness of the multimodal voice experience is not fully understood. The purpose of this study was to evaluate the extent to which cognitive load, the relevance of visual information, and personality influence the perceived usefulness of multimodal voice assistant technology in a within-subjects repeated measures design. A multimodal voice prototype was created to answer the question, “What are some extreme weather conditions?” Nine variants, including 3, 5 and 7 system responses with relevant, irrelevant or no information presented on a screen were included. Three tasks were embedded within each condition (Stroop task, sort M&Ms and no task). Perceived usefulness score, recall, personality score, and fluctuations in galvanic skin response (GSR) values were the subjective and objective measures. The findings suggest that when there’s a smaller number of responses/words for the participant to attend to, and subsequently recall, in addition to relevant visual feedback to aid in that recall, they perceive the voice assistant experience to be more useful, while task conducted exhibits marginal significance in determining PU. Scores of conscientiousness, openness to experience, agreeableness, and neuroticism were successful in predicting some variation in the PU responses, while GSR data was not. It is highly recommended that UX designers of the multimodal interface create succinct voice responses with relevant visual feedback to accompany it, and to keep the main use cases of these products in mind to increase the experience’s PU and subsequent behavioral intention to use the product.

An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks

Recently, there has been growing use of deep neural networks in many modern speech-based systems such as speaker recognition, speech enhancement, and emotion recognition. Inspired by this success, we propose to address the task of voice activity detection (VAD) by incorporating auditory and visual modalities into an end-to-end deep neural network. We evaluate our proposed system in challenging acoustic environments including high levels of noise and transients, which are common in real-life scenarios. Our multimodal setting includes a speech signal captured by a microphone and a corresponding video signal capturing the speaker's mouth region. Under such difficult conditions, robust features need to be extracted from both modalities in order for the system to accurately distinguish between speech and noise. For this purpose, we utilize a deep residual network, to extract features from the video signal, while for the audio modality, we employ a variant of WaveNet encoder for feature extraction. The features from both modalities are fused using multimodal compact bilinear pooling to form a joint representation of the speech signal. To further encode the temporal information, we feed the fused signal to a long short-term memory network and the system is then trained in an end-to-end supervised fashion. Experimental results demonstrate the improved performance of the proposed end-to-end multimodal architecture compared to unimodal variants for VAD. Upon the publication of this paper, we will make the implementation of our proposed models publicly available at https://github.com/iariav/End-to-End-VAD and https://israelcohen.com .

Multimodal Voice Conversion Under Adverse Environment Using a Deep Convolutional Neural Network

This paper presents a voice conversion (VC) technique under noisy environments. Typically, VC methods use only audio information for conversion in a noiseless environment. However, existing conversion methods do not always achieve satisfactory results in an adverse acoustic environment. To solve this problem, we propose a multimodal voice conversion model based on a deep convolutional neural network (MDCNN) built by combining two convolutional neural networks (CNN) and a deep neural network (DNN) for VC under noisy environments. In the MDCNN, both the acoustic and visual information are incorporated into the voice conversion to improve its robustness in adverse acoustic conditions. The two CNNs are designed to extract acoustic and visual features, and the DNN is designed to capture the nonlinear mapping relation of source speech and target speech. Experimental results indicate that the proposed MDCNN outperforms two existing approaches in noisy environments.

Distinguishing patterns in drivers’ visual attention allocation using Hidden Markov Models

Driving is an intricate task where different demands compete for the driver’s attention. Current interface designs present novel multi-modal interactions that extend beyond traditional visual-manual modalities. These new interaction paradigms have given rise to additional subtask elements which call upon varying degrees of cognitive, auditory, vocal, visual, and manual resources. The draw on a larger number of resources has made demand assessment and optimization challenging. How these elements impact the driver’s visual behavior may provide insight into the degree to which a vehicle’s user interface influences attentional focus. This report addresses this question by approaching the problem from a computationally predictive perspective. Data were drawn from two studies that captured visual behaviors of drivers during a series of radio tuning tasks using a traditional manual interface and a multi-modal voice enabled interface during highway driving. Manual annotations of glance times and targets were compiled for each task period and then used to train a predictive model. A statistical machine learning approach (Hidden Markov Model) showed that manual radio tuning, voice-based radio tuning, and “just driving” behaviors result in fundamentally and predictably different strategies of visual attention allocation. We report classification accuracies of over 95% for detecting the correct task modality within a 3 class classification framework, extending prior work to show that time series of glance allocations contain highly descriptive information that generalizes well across drivers of different ages, genders, and driving experience. Results suggest that differences in glance allocation strategies serve as an effective evaluator of the visual demand of a vehicle interface, providing an objective methodology for demonstrating that voice-based technologies allow drivers to maintain a broader distribution of visual attention than the traditional manual interface.

Multimodal voice conversion based on non-negative matrix factorization

A multimodal voice conversion (VC) method for noisy environments is proposed. In our previous non-negative matrix factorization (NMF)-based VC method, source and target exemplars are extracted from parallel training data, in which the same texts are uttered by the source and target speakers. The input source signal is then decomposed into source exemplars, noise exemplars, and their weights. Then, the converted speech is constructed from the target exemplars and the weights related to the source exemplars. In this study, we propose multimodal VC that improves the noise robustness of our NMF-based VC method. Furthermore, we introduce the combination weight between audio and visual features and formulate a new cost function to estimate audio-visual exemplars. Using the joint audio-visual features as source features, VC performance is improved compared with that of a previous audio-input exemplar-based VC method. The effectiveness of the proposed method is confirmed by comparing its effectiveness with that of a conventional audio-input NMF-based method and a Gaussian mixture model-based method.

Fault tolerance for highly available internet services: concepts, approaches, and issues

Fault-tolerant frameworks provide highly available services by means of fault detection and fault recovery mechanisms. These frameworks need to meet different constraints related to the fault model strength, performance, and resource consumption. One of the factors that led to this work is the observation that current fault-tolerant frameworks are not always adapted to existing Internet services. In fact, most of the proposed frameworks are not transport-level- or session-level-aware, although the concerned services range from regular services like HTTP and FTP to more recent Internet services such as multimodal conferencing and voice over IP. In this work we give a comprehensive overview of fault tolerance concepts, approaches, and issues. We show how the redundancy of application servers can be invested to ensure efficient failover of Internet services when the legitimate processing server goes down.

Developing a Wireless Speech- and Touch-Based Intelligent Comprehensive Triage Support System

The implementation of voice recognition technology has been expected to occur in mobile healthcare settings, but it is the least studied solution in nursing. The objective of this study is to examine its value to mobile nursing. The study was done at a triage station in an emergency department. The system was developed using VB6.0, Microsoft Speech SDK 5.1 and the Simplified Chinese Language pack, and was installed on touchscreen PCs with wireless headsets. Thirty nurses were enrolled. Accuracy rate and operation time were used to measure the subjects' performance. A "willingness to use" score on a scale of 1 to 10 was used to measure subjects' preference for the system. The results showed that the average accuracy rate was 99%, the average operation time was 108 seconds, and the mean "willingness to use" rating was 8.2. This study demonstrates the value of multimodal voice recognition techniques to mobile nursing.

Speech and gesture share the same communication system

Humans speak and produce symbolic gestures. Do these two forms of communication interact, and how? First, we tested whether the two communication signals influenced each other when emitted simultaneously. Participants either pronounced words, or executed symbolic gestures, or emitted the two communication signals simultaneously. Relative to the unimodal conditions, multimodal voice spectra were enhanced by gestures, whereas multimodal gesture parameters were reduced by words. In other words, gesture reinforced word, whereas word inhibited gesture. In contrast, aimless arm movements and pseudo-words had no comparable effects. Next, we tested whether observing word pronunciation during gesture execution affected verbal responses in the same way as emitting the two signals. Participants responded verbally to either spoken words, or to gestures, or to the simultaneous presentation of the two signals. We observed the same reinforcement in the voice spectra as during simultaneous emission. These results suggest that spoken word and symbolic gesture are coded as single signal by a unique communication system. This signal represents the intention to engage a closer interaction with a hypothetical interlocutor and it may have a meaning different from when word and gesture are encoded singly.

Evaluating the Behavioral Effectiveness of a Multi-Modal Voice Warning Sign in a Visually Cluttered Environment

This research examined the effects of a multi-modal warning sign on compliance behavior. Participants followed a set of printed instructions to perform a chemistry task that involved measuring and mixing disguised (nonhazardous) chemicals. Whether participants wore protective equipment as directed by the warning was measured. The environment around the sign was either visually cluttered or uncluttered. In some conditions, pictorials, a voice warning, and/or a flashing strobe light were added. The results showed that compliance was significantly greater when the warning was presented in an uncluttered environment compared to a cluttered environment. The results also showed that the presence of a voice warning produced a strong and reliable increase in compliance compared to conditions without a voice warning. No statistically reliable effects of pictorials or strobe were found though the results did show a trend of greater compliance when they were present. In addition, compliance was positively related to memory of the warning, perception of hazard, and reported carefulness. The results call attention to the importance of the context in which a warning is placed, and the potential benefits of voice warnings.