Paralinguistic Analysis Research Articles

Disagreeing without tears

Abstract Disagreement, while generally seen as an act threatening interlocutors’ face, is indispensable in particular contexts. Based on an in-depth linguistic and paralinguistic analysis of 12 cases of disagreement from a Chinese venture capital reality TV show We Are the Hero, we found that in this special context disagreement was a ‘dispreferred’ moment, but also an informative, educational, and amusing occurrence with multiple functions. While disagreements were frequently initiated (particularly by the investors), they were mostly indirect, carefully mitigated by participants with linguistic devices such as explanations, private verbs, questions, humor, as well as paralinguistic devices such as friendly facial expressions and movements. Furthermore, disagreements, especially aggravated ones, were often interactively mitigated by both on-stage participants (particularly the host) and the post-production team in an interestingly cooperative way. The interactive process of mitigating disagreements not only saved the interlocutors’ face, but also contributed to the multiple purposes of the show. Our findings suggest that politeness, as an invisible hand behind communicative acts, is still present despite the ‘conflict’ that characterizes this form of TV show.

Unveiling the sound of the cognitive status: Machine Learning-based speech analysis in the Alzheimer's disease spectrum.

Advancement in screening tools accessible to the general population for the early detection of Alzheimer's disease (AD) and prediction of its progression is essential for achieving timely therapeutic interventions and conducting decentralized clinical trials. This study delves into the application of Machine Learning (ML) techniques by leveraging paralinguistic features extracted directly from a brief spontaneous speech (SS) protocol. We aimed to explore the capability of ML techniques to discriminate between different degrees of cognitive impairment based on SS. Furthermore, for the first time, this study investigates the relationship between paralinguistic features from SS and cognitive function within the AD spectrum. Physical-acoustic features were extracted from voice recordings of patients evaluated in a memory unit who underwent a SS protocol. We implemented several ML models evaluated via cross-validation to identify individuals without cognitive impairment (subjective cognitive decline, SCD), with mild cognitive impairment (MCI), and with dementia due to AD (ADD). In addition, we established models capable of predicting cognitive domain performance based on a comprehensive neuropsychological battery from Fundació Ace (NBACE) using SS-derived information. The results of this study showed that, based on a paralinguistic analysis of sound, it is possible to identify individuals with ADD (F1 = 0.92) and MCI (F1 = 0.84). Furthermore, our models, based on physical acoustic information, exhibited correlations greater than 0.5 for predicting the cognitive domains of attention, memory, executive functions, language, and visuospatial ability. In this study, we show the potential of a brief and cost-effective SS protocol in distinguishing between different degrees of cognitive impairment and forecasting performance in cognitive domains commonly affected within the AD spectrum. Our results demonstrate a high correspondence with protocols traditionally used to assess cognitive function. Overall, it opens up novel prospects for developing screening tools and remote disease monitoring.

Non-fluency and language-pair specificity in Chinese-English consecutive interpreting: A corpus-driven study

Language-pair specificity, which refers to linguistic and cultural differences between the language pair, has been hypothesized as one of the variables shaping the interpreting performance and product. The current study adopts a corpus-driven paralinguistic approach to testifying the language-pair specificity hypothesis. The corpus is a bilingual parallel corpus of Chinese-English Interpreting for Premier Press Conferences, which consists of 200,000 words/characters in total. The original and interpreted discourses are aligned at the sentential level and annotated at linguistic, paralinguistic, and extra-linguistic levels. The paralinguistic analysis focuses on non-fluency, specifically the different types of pauses and self-repairs. It is found that a majority of non-fluencies in the interpreted utterances are syntax-driven, which means that most of the pauses and self-repairs in Chinese-English interpreting are related to syntactical structures in the original speeches. The finding implies that language-pair specificity should be considered an important variable in research and training of interpreting between syntactically-contrastive languages.

The Filtering Effect of Face Masks in their Detection from Speech.

Face masks alter the speakers' voice, as their intrinsic properties provide them with acoustic absorption capabilities. Hence, face masks act as filters to the human voice. This work focuses on the automatic detection of face masks from speech signals, emphasising on a previous work claiming that face masks attenuate frequencies above 1 kHz. We compare a paralinguistics-based and a spectrograms-based approach for the task at hand. While the former extracts paralinguistic features from filtered versions of the original speech samples, the latter exploits the spectrogram representations of the speech samples containing specific ranges of frequencies. The machine learning techniques investigated for the paralinguistics-based approach include Support Vector Machines (SVM), and a Multi-Layer Perceptron (MLP). For the spectrograms-based approach, we use a Convolutional Neural Network (CNN). Our experiments are conducted on the Mask Augsburg Speech Corpus (MASC), released for the Interspeech 2020 Computational Paralinguistics Challenge (COMPARE). The best performances on the test set from the paralinguistic analysis are obtained using the high-pass filtered versions of the original speech samples. Nonetheless, the highest Unweighted Average Recall (UAR) on the test set is obtained when exploiting the spectrograms with frequency content below 1 kHz.

Improving the Learning Power of Artificial Intelligence Using Multimodal Deep Learning

Computer paralinguistic analysis is widely used in security systems, biometric research, call centers and banks. Paralinguistic models estimate different physical properties of voice, such as pitch, intensity, formants and harmonics to classify emotions. The main goal is to find such features that would be robust to outliers and will retain variety of human voice properties at the same time. Moreover, the model used must be able to estimate features on a time scale for an effective analysis of voice variability. In this paper a paralinguistic model based on Bidirectional Long Short-Term Memory (BLSTM) neural network is described, which was trained for vocal-based emotion recognition. The main advantage of this network architecture is that each module of the network consists of several interconnected layers, providing the ability to recognize flexible long-term dependencies in data, which is important in context of vocal analysis. We explain the architecture of a bidirectional neural network model, its main advantages over regular neural networks and compare experimental results of BLSTM network with other models.

The sound of silence: Breathing analysis for finding traces of trauma and depression in oral history archives

Abstract Many people experience a traumatic event during their lifetime. In some extraordinary situations, such as natural disasters, war, massacres, terrorism, or mass migration, the traumatic event is shared by a community and the effects go beyond those directly affected. Today, thanks to recorded interviews and testimonials, many archives and collections exist that are open to researchers of trauma studies, holocaust studies, and historians, among others. These archives act as vital testimonials for oral history, politics, and human rights. As such, they are usually either transcribed or meticulously indexed. In this work, we propose to look at the nonverbal signals emitted by victims of various traumatic events when they describe the trauma and we seek to render these for novel representations without taking into account the explicit verbal content. Our preliminary paralinguistic analysis on a manually annotated collection of testimonials from different archives, as well as on a corpus prepared for depression and post-traumatic stress disorder detection indicates a tentative connection between breathing and emotional states of speakers, which opens up new possibilities of exploring oral history archives.

"You sound ill, take the day off": Automatic recognition of speech affected by upper respiratory tract infection.

A combination of passive, non-invasive and nonintrusive smart monitoring technologies is currently transforming healthcare. These technologies will soon be able to provide immediate health related feedback for a range of illnesses and conditions. Such tools would be game changing for serious public health concerns, such as seasonal cold and flu, for which early diagnosis and social isolation play a key role in reducing the spread. In this regard, this paper explores, for the first times, the automated classification of individuals with Upper Respiratory Tract Infections (URTI) using recorded speech samples. Key results presented indicate that our classifiers can achieve similar results to those seen in related health-based detection tasks indicating the promise of using computational paralinguistic analysis for the detection of URTI related illnesses.

Using acoustic paralinguistic information to assess the interaction quality in speech-based systems for elderly users

Abstract At present, there is a growing need for assisting living systems for the elderly. This type of technology must meet certain accessibility and usability requirements to be adopted successfully. Speech-based interfaces represent a very attractive option for improving the usability and acceptability of assistive technologies for elderly users because they provide a very easily accessible interface and also facilitate paralinguistic analysis of the user's voice to provide valuable information about their emotional and mental states while interacting with a computer system. This information can be used to improve the quality of the interaction between the assistive system and its user. In this study, we propose a method based on the automatic recognition of acoustic paralinguistic phenomena for estimating the quality of speech-based interactions. The proposed method automatically recognizes paralinguistic phenomena (e.g., shouting, hyper-articulation, and hesitation) during the interaction between the user and the system. The interaction is then characterized based on the occurrence of these phenomena. A model is trained using this characterization, which can estimate the quality of the interaction. Using this method, we obtained good recognition performance (F-measure around 70%) when classifying paralinguistic phenomena and there was also a strong correlation between the estimated quality of the interaction metrics and the quality of the interaction metrics reported by the users of an assistive system. The proposed method is useful for automatically estimating the quality of interactions perceived by users and it could support the construction of speech-based systems to adapt and personalize content as well as the style of the interaction with the user.

State-of-the-art tasks and achievements of paralinguistic speech analysis systems

We present analytical survey of state-of-the-art actual tasks in the area of computational paralinguistics, as well as the recent achievements of automatic systems for paralinguistic analysis of conversational speech. Paralinguistics studies non-verbal aspects of human communication and speech such as: natural emotions, accents, psycho-physiological states, pronunciation features, speaker’s voice parameters, etc. We describe architecture of a baseline computer system for acoustical paralinguistic analysis, its main components and useful speech processing methods. We present some information on an International contest called Computational Paralinguistics Challenge (ComParE), which is held each year since 2009 in the framework of the International conference INTERSPEECH organized by the International Speech Communication Association. We present sub-challenges (tasks) that were proposed at the ComParE Challenges in 2009-2016, and analyze winning computer systems for each sub-challenge and obtained results. The last completed ComParE-2015 Challenge was organized in September 2015 in Germany and proposed 3 sub-challenges: 1) Degree of Nativeness (DN) sub-challenge, determination of nativeness degree of speakers based on acoustics; 2) Parkinson's Condition (PC) sub-challenge, recognition of a degree of Parkinson’s condition based on speech analysis; 3) Eating Condition (EC) sub-challenge, determination of the eating condition state during speaking or a dialogue, and classification of consumed food type (one of seven classes of food) by the speaker. In the last sub-challenge (EC), the winner was a joint Turkish-Russian team consisting of the authors of the given paper. We have developed the most efficient computer-based system for detection and classification of the corresponding (EC) acoustical paralinguistic events. The paper deals with the architecture of this system, its main modules and methods, as well as the description of used training and evaluation audio data and the best obtained results on machine classification of these acoustic paralinguistic events.

Feature selection methods and their combinations in high-dimensional classification of speaker likability, intelligibility and personality traits

This study focuses on feature selection in paralinguistic analysis and presents recently developed supervised and unsupervised methods for feature subset selection and feature ranking. Using the standard k-nearest-neighbors (kNN) rule as the classification algorithm, the feature selection methods are evaluated individually and in different combinations in seven paralinguistic speaker trait classification tasks. In each analyzed data set, the overall number of features highly exceeds the number of data points available for training and evaluation, making a well-generalizing feature selection process extremely difficult. The performance of feature sets on the feature selection data is observed to be a poor indicator of their performance on unseen data. The studied feature selection methods clearly outperform a standard greedy hill-climbing selection algorithm by being more robust against overfitting. When the selection methods are suitably combined with each other, the performance in the classification task can be further improved. In general, it is shown that the use of automatic feature selection in paralinguistic analysis can be used to reduce the overall number of features to a fraction of the original feature set size while still achieving a comparable or even better performance than baseline support vector machine or random forest classifiers using the full feature set. The most typically selected features for recognition of speaker likability, intelligibility and five personality traits are also reported.

Paralinguistics in speech and language—State-of-the-art and the challenge

Paralinguistic analysis is increasingly turning into a mainstream topic in speech and language processing. This article aims to provide a broad overview of the constantly growing field by defining the field, introducing typical applications, presenting exemplary resources, and sharing a unified view of the chain of processing. It then presents the first broader Paralinguistic Challenge organised at INTERSPEECH 2010 by the authors including a historical overview of the Challenge tasks of recognising age, gender, and affect, a summary of methods used by the participants, and their results. In addition, we present the new benchmark obtained by fusion of participants’ predictions and conclude by discussing ten recent and emerging trends in the analysis of paralinguistics in speech and language.

A New Empirical Method for Assessing Human Attachment Relationship Quality

The theoretical basis of a novel method for assessing attachment is presented along with preliminary data. The method employs an analysis applied to paralinguistic data derived from the tape generated from a newly developed technique based on the Thematic Apperception Test for assessing attachment, the Adult Attachment Projective. The attachment classifications were made blind to the results of the paralinguistic analysis using the traditional Adult Attachment Interview. The paralinguistic analysis method lends itself well to web-based technology and real-time analysis. This method represents an innovation in the ability to use technology-based assessments of Thematic Apperception Test results that bring them more directly into the realm of clinical application.