Conversational Speech Research Articles

The FRA Corpus from the DISease and AGEing Project

The FRA corpus, from the DISAGE project (University of Naples Federico II), is an Italian speech corpus of elderly people with and without neurodegenerative pathologies. The project's main aim is to provide an Italian tool useful to observe language in pathological aging. Therefore, the project enrolled 20 patients diagnosed with Alzheimer's Disease (AD), 20 individuals suffering from Mild Cognitive Impairment (MCI), and 20 Healthy Controls (HC) balanced with respect to sex, age, and education. Participants underwent neuro-psychological testing (MMSE; Raven's Matrices) and only mild AD and amnestic MCI were recruited. The speech corpus is semi-spontaneous and dialogic: speech samples were collected via a three-stage interview designed to encourage the production of descriptive, conversational and narrative speech. Currently, 18 out of the 60 recordings collected have been manually and orthographically transcribed. Data is available on request.

Learning and consolidating the contextualized contour representations of tones from F0 sequences and durational variations via transformers

Many speech characteristics, including conventional acoustic features such as mel frequency cepstrum coefficients and mel-spectrograms, as well as pre-trained contextualized acoustic representations such as wav2vec2.0, are used in a deep neural network or successfully fine-tuned with a connectionist temporal classification for Mandarin tone classification. In this study, the authors propose a transformer-based tone classification architecture, TNet-Full, which uses estimated fundamental frequency (F0) values and aligned boundary information on syllables and words. Key components of the model framework are the contour encoder and rhythm encoder, as well as the cross-attention between contours and rhythms established in the interaction encoder. Using contextual tonal contours as a reference, as well as rhythmic information derived from duration variations to consolidate more on contour representations for tone recognition, TNet-Full achieves absolute performance improvements of 24.4% for read speech (from 71.4% to 95.8%) and 6.3% for conversational speech (from 52.1% to 58.4%) when compared to a naive, simple baseline transformer, TNet-base. The relative improvements are 34.2% and 12.1%. As humans perceive tones, contour abstractions of tones can only be derived from F0 sequences, and tone recognition would be improved if syllable temporal organization was stable and predictable instead of fluctuating as seen in conversations.

“It’s a Bit Tricky, Isn’t It?”—An Acoustic Study of Contextual Variation in /ɪ/ in the Conversational Speech of Young People from Perth

This study presents an acoustic analysis of vowel realisations in contexts where, in Australian English, a historical contrast between unstressed /ɪ/ and /ə/ has largely diminished in favour of a central schwa-like variant. The study is motivated by indications that there is greater complexity in this area of vowel variation than has been conventionally set out in the existing literature, and our goal is to shed new light by studying a dataset of conversational speech produced by 40 young speakers from Perth, WA. In doing so, we also offer some critical thoughts on the use of Wells’ lexical sets as a framework for analysis in work of this kind, in particular with reference to the treatment of items in unstressed position, and of grammatical (or function) words. The acoustic analysis focused on the realisation in F1/F2 space of a range of /ɪ/ and /ə/ variants in both accented and unaccented syllables (thus a broader approach than a focus on stressed kit vowels). For the purposes of comparison, we also analysed tokens of the fleece and happy-tensing lexical sets. Grammatical and non-grammatical words were analysed independently in order to understand the extent to which a high-frequency grammatical word such as it might contribute to the overall pattern of vowel alternation. Our findings are largely consistent with the small amount of previous work that has been carried out in this area, pointing to a continuum of realisations across a range of accented and unaccented contexts. The data suggest that the reduced historical /ɪ/ vowel encountered in unaccented syllables cannot be straightforwardly analysed as a merger with /ə/. We also highlight the way in which the grammatical word it participates in this alternation.

Generative adversarial networks for whispered to voiced speech conversion: a comparative study

AbstractGenerative Adversarial Networks (GANs) have demonstrated promising results as end-to-end models for whispered to voiced speech conversion. Leveraging non-autoregressive systems like GANs capable of performing conditional waveform generation eliminates the need for separate models to estimate voiced speech features, and leads to faster inference compared to autoregressive methods. This study aims to identify the optimal GAN architecture for the whispered to voiced speech conversion task by comparing six state-of-the-art models. Furthermore, we present a method for evaluating the preservation of speaker identity and local accent, using embeddings obtained from speaker- and language identification systems. Our experimental results show that building the speech conversion system based on the HiFi-GAN architecture yields the best objective evaluation scores, outperforming the baseline by $$\sim$$ ∼ 9% relative using frequency-weighted Signal-to-Noise Ratio and Log Likelihood Ratio, as well as by $$\sim$$ ∼ 29% relative using Root Mean Squared Error. In subjective tests, HiFi-GAN yielded a mean opinion score of 2.9, significantly outperforming the baseline with a score of 1.4. Furthermore, HiFi-GAN enhanced ASR performance and preserved speaker identity and accent, with correct language detection rates of up to $$\sim$$ ∼ 98%.

Drivers of hate speech in political conversations on Twitter: the case of the 2022 Italian general election

We study the prevalence and characteristics of toxic speech on Twitter in the run-up to the 2022 Italian General election. We analyzed over 8.5 M tweets shared by 450k unique users and employed a machine learning classifier to estimate the toxicity score of their messages. We found that supporters of different political coalitions exhibit varying levels of toxic speech, sending between 6-8% of toxic messages overall. Notably, Centre-Left politicians received more toxic messages on average, with the largest target of hate receiving over 15,000 toxic replies. We employed Generalized Linear Models to study factors that drive hate speech to political targets, finding that, importantly, politicians employing more abusive and harmful language are also more likely to attract more inflammatory speech. Our findings underscore the critical need for targeted interventions to address hate speech online, fostering healthier dialogue and safeguarding democratic discourse.

What’s so complex about conversational speech? A comparison of HMM-based and transformer-based ASR architectures

Highly performing speech recognition is important for more fluent human-machine interaction (e.g., dialogue systems). Modern ASR architectures achieve human-level recognition performance on read speech but still perform sub-par on conversational speech, which arguably is or, at least, will be instrumental for human-machine interaction. Understanding the factors behind this shortcoming of modern ASR systems may suggest directions for improving them. In this work, we compare the performances of HMM- vs. transformer-based ASR architectures on a corpus of Austrian German conversational speech. Specifically, we investigate how strongly utterance length, prosody, pronunciation, and utterance complexity as measured by perplexity affect different ASR architectures. Among other findings, we observe that single-word utterances – which are characteristic of conversational speech and constitute roughly 30% of the corpus – are recognized more accurately if their F0 contour is flat; for longer utterances, the effects of the F0 contour tend to be weaker. We further find that zero-shot systems require longer utterance lengths and are less robust to pronunciation variation, which indicates that pronunciation lexicons and fine-tuning on the respective corpus are essential ingredients for the successful recognition of conversational speech.

Pragmatic Strategies In The Representation Of Plus-Size Women In Nollywood Films: A Case Study Of Skinny Girl In Transit.

This study investigates the pragmatic strategies employed in Skinny Girl in Transit, a popular Nollywood web series, to examine how language reflects and challenges societal norms about body image. Focusing on conversational implicature, speech acts, and politeness theory, the analysis explores how the protagonist, Tiwa, a plus-size woman, navigates societal pressures concerning her weight and identity. The case study approach is utilized, selecting key scenes from the series where body image is discussed, either implicitly or explicitly. Through the application of pragmatic theories, the research reveals that while language in the series often reinforces societal expectations about slimness, it also provides a platform for resisting stereotypes. Characters use humor, indirect remarks, and polite strategies to address sensitive issues, creating a nuanced portrayal of plus-size women in Nollywood. The findings highlight how Skinny Girl in Transit both mirrors and critiques cultural attitudes toward body size, offering a more complex representation of plus-size women in Nigerian media. This study contributes to the fields of pragmatics, media representation, and African cinema by demonstrating the power of language in shaping cultural perceptions.

Higher Frequency of Stuttered Disfluencies Negatively Affects Communicative Participation in Parkinson's Disease.

Up to 90% of people with Parkinson's disease (PD) develop communication difficulties over the course of the disease. While the negative effect of dysarthria on communicative participation has been well-documented, the impact of the occurrence of acquired stuttered disfluencies on communication in different speech situations is unknown. This study aimed to determine if the frequency of occurrence of stuttered disfluencies affects communicative participation in individuals with PD, and whether such a relationship is mediated by examiner- and self-rated measures of disease severity. Conversational speech samples were collected from 100 people with PD aged 53-91 years to calculate the frequency of occurrence of stuttered disfluencies. Participants completed the Communicative Participation Item Bank to assess participation in communicative situations. Information on overall speech, cognitive, and motor performance was collected using both self-rated and examiner-rated methods. Participants with PD presented with 0.2%-9.9% stuttered disfluencies during conversation. Overall, participants with PD reported their communicative participation to be impacted "a little" (19.5 ± 7.0), but there was considerable interindividual variation. A higher frequency of stuttered disfluencies was associated with significantly lower communicative participation (ρ = -0.32, p < .01). In addition, examiner-rated frequency of stuttered disfluencies (p < .01), speech (p < .01), and motor severity (p = .04) were all significant predictors of communicative participation. Using self-ratings, speech (p < .01) and cognitive (p < .01) measures significantly predicted communicative participation. In people with PD, communicative participation was significantly worse for those with a higher frequency of stuttered disfluencies. Examiner- and self-rated measures of disease severity contributed different information related to communicative constraints. Together, these results highlight the importance of individualized and holistic speech therapy that considers a wide variety of symptoms, including stuttered disfluencies, to ensure positive functional outcomes. https://doi.org/10.23641/asha.26850169.

Investigating vowel differences in North and South Korea: Phonetic analysis of younger speakers in read and conversational speech

Investigating vowel differences in North and South Korea: Phonetic analysis of younger speakers in read and conversational speech

Conversion of Speech to Text: Transcribe and Translate Using Whisper AI on iOS

Abstract: This thesis investigates the implementation and efficiency of Whisper AI for transcribing and translating speech to text on iOS devices. Leveraging a large-scale weakly supervised dataset, Whisper AI demonstrates robust performance across multiple languages and tasks. The study explores its architecture, implementation on iOS, and performance comparisons with existing models. Findings indicate significant potential for real-world applications, despite some computational and accuracy challenges.

Acquisition of a rare variant: ne-realization in the negative utterances of French children and their caregivers

ABSTRACT A growing body of evidence suggests children acquire linguistic variation and follow grammatical constraints on its use from a young age. While prior research predominantly focuses on variables for which both variants are well-attested, stable variation in natural language often includes variants with an unbalanced distribution. In many cases, one variant is extremely rare, which poses a learning challenge for children and may require revision of accounts of children’s acquisition of variation. The negative morpheme ne in French is one such rare variant as it is only rarely realized (<10%) in adult speech. While previous research suggests clear constraints on ne-realization in interadult speech, less is known about ne in French children’s early production and input. In the current study, we analyze ne-realization in the conversational speech of 14 monolingual French-learning children (ages 0.10–8.01) and their caregivers. We found that, for both French children and their caregivers, ne-realization was rare and followed the same linguistic constraint found in interadult speech. Importantly, children showed probabilistic production following the constraint extremely early on – at age 2.05 on average. Further, we found a strong influence of input on variation acquisition: caregivers with higher ne-realization rates had children who used more ne and from a younger age. Taken together, our results suggested that young children can acquire statistically rare but systematically patterned variants, adding further evidence that they are skilled and sophisticated language learners.

Estimates of speech efficiency in monolingual and bilingual speakers of English.

Introduction The Speech Efficiency Score (SES) serves as an acoustic metric for assessing fluency in conversational speech within the temporal domain. This study leverages SES to investigate conversational speech efficiency among native speakers of American English (AE) compared to speakers of Mandarin-accented English (MAE). Methods SES, speaking rate, articulation rate, and vocabulary diversity were measured and compared between two groups: native AE speakers and MAE speakers. The study utilized conversational speech samples collected from both groups to analyze these metrics. Results Findings indicate a disparity in speaking rate and articulation rate between the AE and MAE groups, with the AE group exhibiting significantly faster speech. However, no significant differences were observed in SES and vocabulary diversity between the two groups. Conclusion The results are discussed in the context of the interplay between speaking rate, speech fluency, and vocabulary diversity. These findings shed light on the maintenance of speech efficiency among bilingual speakers, suggesting that despite differences in speaking rate and articulation rate, SES and vocabulary diversity remain comparable between native AE speakers and MAE speakers. .

Conversations in the wild: Data collection, automatic generation and evaluation

The aim of conversational speech processing is to analyze human conversations in natural settings. It finds numerous applications in personality traits identification, speech therapy, speaker identification and verification, speech emotion detection, and speaker diarization. However, large-scale annotated datasets required for feature extraction and conversational model training only exist for a handful of languages (e.g. English, Mandarin, and French) as the gathering, cleaning, and annotation of such datasets is tedious, time-consuming, and expensive. We propose two scalable, language-agnostic algorithms for automatically generating multi-speaker, variable-length, spontaneous conversations. These algorithms synthesize conversations using existing non-conversational speech datasets. We also contribute the resulting datasets (283 hours, 50 speakers). As a comparison, we also gathered the first spontaneous conversational dataset for Urdu (24 hours, 212 speakers) from public talk shows. Using speaker diarization as an example, we evaluate our datasets and report the first baseline diarization error rates (DER) for Urdu (25% for synthetic dataset-based models, and 29% for natural conversations). Our conversational speech generation technique allows training speaker diarization pipelines without the need for preparing huge conversational repositories.

Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?

Abstract Authorship verification is the task of determining if two distinct writing samples share the same author and is typically concerned with the attribution of written text. In this paper, we explore the attribution of transcribed speech, which poses novel challenges. The main challenge is that many stylistic features, such as punctuation and capitalization, are not informative in this setting. On the other hand, transcribed speech exhibits other patterns, such as filler words and backchannels (e.g., um, uh-huh), which may be characteristic of different speakers. We propose a new benchmark for speaker attribution focused on human-transcribed conversational speech transcripts. To limit spurious associations of speakers with topic, we employ both conversation prompts and speakers participating in the same conversation to construct verification trials of varying difficulties. We establish the state of the art on this new benchmark by comparing a suite of neural and non-neural baselines, finding that although written text attribution models achieve surprisingly good performance in certain settings, they perform markedly worse as conversational topic is increasingly controlled. We present analyses of the impact of transcription style on performance as well as the ability of fine-tuning on speech transcripts to improve performance.1

Gendered Subjectivities in English Classrooms: A Critically Queered Examination of Sex-Segregated Education

This study explores the link between gender and language in single-sex schools in Colombia. Using feminist theory, the study analyzes the gendered subjectivities of English-as-a-foreign-language learners in a private school in Bogotá. It examines how heterosexuality is forced upon learners and how their subjectivities are impacted by the heteronormative discourse surrounding sex-gender learning differences. The study utilizes conversation analysis and speech act theory to examine data. The findings reveal that gendered subjectivity is complex and constitutes an ongoing struggle. There is a need for further research on gender and English teaching in educational settings, emphasizing the importance of studying sociolinguistic power relationships.

A novel paradigm for measuring prediction abilities in a rat model using a speech-sound discrimination task

The ability to predict and respond to upcoming stimuli is a critical skill for all animals, including humans. Prediction operates largely below conscious awareness to allow an individual to recall previously encountered stimuli and prepare an appropriate response, especially in language. The ability to predict upcoming words within typical speech patterns aids fluent comprehension, as conversational speech occurs quickly. Individuals with certain neurodevelopmental disorders such as autism and dyslexia have deficits in their ability to generate and use predictions. Rodent models are often used to investigate specific aspects of these disorders, but there is no existing behavioral paradigm that can assess prediction capabilities with complex stimuli like speech sounds. Thus, the present study modified an existing rapid speech sound discrimination paradigm to assess whether rats can form predictions of upcoming speech sound stimuli and utilize them to improve task performance. We replicated prior work showing that rats can discriminate between speech sounds presented at rapid rates. We also saw that rats responded exclusively to the target at slow speeds but began responding to the predictive cue in anticipation of the target as the speed increased, suggesting that they learned the predictive value of the cue and adjusted their behavior accordingly. This prediction task will be useful in assessing prediction deficits in rat models of various neurodevelopmental disorders through the manipulation of both genetic and environmental factors.

Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge

Supervised models for speech enhancement are trained using artificially generated mixtures of clean speech and noise signals. However, the synthetic training conditions may not accurately reflect real-world conditions encountered during testing. This discrepancy can result in poor performance when the test domain significantly differs from the synthetic training domain. To tackle this issue, the UDASE task of the 7th CHiME challenge aimed to leverage real-world noisy speech recordings from the test domain for unsupervised domain adaptation of speech enhancement models. Specifically, this test domain corresponds to the CHiME-5 dataset, characterized by real multi-speaker and conversational speech recordings made in noisy and reverberant domestic environments, for which ground-truth clean speech signals are not available. In this paper, we present the objective and subjective evaluations of the systems that were submitted to the CHiME-7 UDASE task, and we provide an analysis of the results. This analysis reveals a limited correlation between subjective ratings and several supervised nonintrusive performance metrics recently proposed for speech enhancement. Conversely, the results suggest that more traditional intrusive objective metrics can be used for in-domain performance evaluation using the reverberant LibriCHiME-5 dataset developed for the challenge. The subjective evaluation indicates that all systems successfully reduced the background noise, but always at the expense of increased distortion. Out of the four speech enhancement methods evaluated subjectively, only one demonstrated an improvement in overall quality compared to the unprocessed noisy speech, highlighting the difficulty of the task. The tools and audio material created for the CHiME-7 UDASE task are shared with the community.

Automatic Speech Recognition of Conversational Speech in Individuals With Disordered Speech.

This study examines the effectiveness of automatic speech recognition (ASR) for individuals with speech disorders, addressing the gap in performance between read and conversational ASR. We analyze the factors influencing this disparity and the effect of speech mode-specific training on ASR accuracy. Recordings of read and conversational speech from 27 individuals with various speech disorders were analyzed using both (a) one speaker-independent ASR system trained and optimized for typical speech and (b) multiple ASR models that were personalized to the speech of the participants with disordered speech. Word error rates were calculated for each speech model, read versus conversational, and subject. Linear mixed-effects models were used to assess the impact of speech mode and disorder severity on ASR accuracy. We investigated nine variables, classified as technical, linguistic, or speech impairment factors, for their potential influence on the performance gap. We found a significant performance gap between read and conversational speech in both personalized and unadapted ASR models. Speech impairment severity notably impacted recognition accuracy in unadapted models for both speech modes and in personalized models for read speech. Linguistic attributes of utterances were the most influential on accuracy, though atypical speech characteristics also played a role. Including conversational speech samples in model training notably improved recognition accuracy. We observed a significant performance gap in ASR accuracy between read and conversational speech for individuals with speech disorders. This gap was largely due to the linguistic complexity and unique characteristics of speech disorders in conversational speech. Training personalized ASR models using conversational speech significantly improved recognition accuracy, demonstrating the importance of domain-specific training and highlighting the need for further research into ASR systems capable of handling disordered conversational speech effectively.

Implementation of Finite State Automata Method in Text to Speech Conversion System

Sound processing uses a variety of methods. One of them is converting writing into sound or commonly known as Text To Speech.Text To Speech is a tool for converting writing input into sound by changing the phoneme of writing.Text To Speech Text To Speech consists of two sub-systems namely word to phoneme converters and phoneme to speech converter. The phoneme itself consists of word fragments that have different speech from one another with each other. To perform the phoneme separation process, the Finite State Automata method. Finite State Automata can recognise the simplest language and is an abstract machine whose input and output are discrete. The input system that is processed by the system only consists of two words. If there are more than two word inputs, the programme will not be able to perform word separation properly. After the word separation is done, then the system performs phoneme to speech converter.The resulting speech is processed by the Indo_TTS database which was database developed using the diphone prosody method concatenation prosody methodThe word that has been separated will be stored into the MySQL database as a result of words that have been tested by the system. Later this system will be intended for general public, students and people with speech impairment. The percentage of successful syllable separation and homograph sound output is 98.7% in the programme's success rate.

The Effort of Repairing a Misperceived Word Can Impair Perception of Following Words, Especially for Listeners With Cochlear Implants.

In clinical and laboratory settings, speech recognition is typically assessed in a way that cannot distinguish accurate auditory perception from misperception that was mentally repaired or inferred from context. Previous work showed that the process of repairing misperceptions elicits greater listening effort, and that this elevated effort lingers well after the sentence is heard. That result suggests that cognitive repair strategies might appear successful when testing a single utterance but fail for everyday continuous conversational speech. The present study tested the hypothesis that the effort of repairing misperceptions has the consequence of carrying over to interfere with perception of later words after the sentence. Stimuli were open-set coherent sentences that were presented intact or with a word early in the sentence replaced with noise, forcing the listener to use later context to mentally repair the missing word. Sentences were immediately followed by digit triplets, which served to probe carryover effort from the sentence. Control conditions allowed for the comparison to intact sentences that did not demand mental repair, as well as to listening conditions that removed the need to attend to the post-sentence stimuli, or removed the post-sentence digits altogether. Intelligibility scores for the sentences and digits were accompanied by time-series measurements of pupil dilation to assess cognitive load during the task, as well as subjective rating of effort. Participants included adults with cochlear implants (CIs), as well as an age-matched group and a younger group of listeners with typical hearing for comparison. For the CI group, needing to repair a missing word during a sentence resulted in more errors on the digits after the sentence, especially when the repair process did not result in a coherent sensible perception. Sentences that needed repair also contained more errors on the words that were unmasked. All groups showed substantial increase of pupil dilation when sentences required repair, even when the repair was successful. Younger typical hearing listeners showed clear differences in moment-to-moment allocation of effort in the different conditions, while the other groups did not. For CI listeners, the effort of needing to repair misperceptions in a sentence can last long enough to interfere with words that follow the sentence. This pattern could pose a serious problem for regular communication but would go overlooked in typical testing with single utterances, where a listener has a chance to repair misperceptions before responding. Carryover effort was not predictable by basic intelligibility scores, but can be revealed in behavioral data when sentences are followed immediately by extra probe words such as digits.