Speech Recognition Research Articles

The function of ASR-generated live transcription in simultaneous interpreting: trainee interpreters’ perceptions from post-task interviews

This study explores whether live transcription generated with the technology of automatic-speech-recognition (ASR) can be used to facilitate simultaneous interpreting. This article reports an analysis of trainee interpreters’ perceptions based on post-task structured interviews after an eye-tracked interpreting task without live transcription in the first half and with live transcription in the second half, which was done by a group of trainee interpreters from a postgraduate professional interpreting programme. The interviews were analysed in triangulation with the eye-tracking data about their interpreting behaviours. The results show that most participants perceived live transcription beneficial, with data indicating improved performance and lowered error rates in terms of terminologies, numbers, and proper names. It is also found that while some interpreters reported that they can adeptly manage multimodal inputs, others reported challenges in optimizing their focus of attention when live transcription was provided. The overall interference score in interpreting with live transcription spikes from 9 to 13.2, suggesting fluctuating cognitive demand. Eye-tracking data further corroborate these attentional dynamics, echoing participants’ self-reported behaviours. The study points to the need for training programmes to equip interpreters with capabilities to utilize technological tools such as live transcription, ensuring optimal attention management and overall performance.

Wearable Photonic Artificial Throat for Silent Communication and Speech Recognition.

Advocating for the voices of the disabled, particularly through wearable artificial throats, has garnered significant attention recently. Such devices necessitate sensors with stretchability, high sensitivity, and excellent skin conformability. In this study, an intelligent photonic artificial throat has been developed. It features a sandwich-structured optical fiber sensor encapsulated in Dragon Skin 20, which has an elastic modulus similar to human tissue and is integrated with sensitivity-enhancing rings and fabric for enhanced wearability. With ultrafast response (response time: 10 ms, recovery time: 32 ms) and high sensitivity (1.92 μW/mN), it detects throat area vibrations and muscle contractions, accurately identifying tones in Mandarin, vowels, words, and sentences in English, achieving accurate bilingual detection. It also distinguishes animal sounds (horse neighing and cuckoo's call) and pop songs when mounted on speakers. Furthermore, the artificial throat can accurately detect subtle movements of the head and neck, and by combining nodding actions with the MORSE code, silent communication between individuals has been successfully achieved. Integrated with an advanced artificial intelligence (AI) algorithm, it recognizes tones (97.50%), vowel letters (97.00%), common words (98.00%) and sentences (96.52%), opening prospects for biomedical applications, language education, speech recognition, motion monitoring, and more.

JLMS25 and Jiao-Liao Mandarin Speech Recognition Based on Multi-Dialect Knowledge Transfer

Jiao-Liao Mandarin, a distinguished dialect in China, reflects the linguistic features and cultural heritage of the Jiao-Liao region. However, the labor-intensive and costly nature of manual transcription limits the scale of transcribed corpora, posing challenges for speech recognition. We present JLMS25, a transcribed corpus for Jiao-Liao Mandarin, alongside a novel multi-dialect knowledge transfer (MDKT) framework for low-resource speech recognition. By leveraging phonetic and linguistic knowledge from neighboring dialects, the MDKT framework improves recognition in resource-constrained settings. It comprises an acoustic feature extractor, a dialect feature extractor, and two modules—WFAdapter (weight decomposition adapter) and AttAdapter (attention-based adapter)—to enhance adaptability and mitigate overfitting. The training involves a three-phase strategy: multi-dialect AID-ASR multi-task learning in phase one, freezing the dialect feature extractor in phase two, and fine-tuning only the adapters in phase three. Experiments on the Jiao-Liao Mandarin subset of the KeSpeech dataset and JLMS25 dataset show that MDKT outperforms full-parameter fine-tuning, reducing Character Error Rate (CER) by 5.4% and 7.7% and Word Error Rate (WER) by 6.1% and 10.8%, respectively.

Triboelectric Nanogenerator-Based Flexible Acoustic Sensor for Speech Recognition.

The way people interact with machines through flexible acoustic sensors is revolutionizing the way we live. However, developing a human-machine interaction acoustic sensor that simultaneously offers low cost, high stability, high fidelity, and high sensitivity remains a significant challenge. In this study, a sensor based on a sound-driven triboelectric nanogenerator was proposed. A poly(vinylidene fluoride) (PVDF)/graphene oxide (GO) composite nanofiber film was obtained as the dielectric layer through electrospinning, and copper-nickel alloy conductive fabric was used as the electrode. An imitation embroidery shed structure was designed in the shape of a ring to secure the upper and lower electrodes and the dielectric layer as a whole. Due to the porosity of the electrode, the large contact area of the dielectric layer, and the high stability of the imitation embroidery shed structure, the sensor achieves a sensitivity of 4.76 V·Pa-1 and a frequency response range of 20-2000 Hz. A multilayer attention convolutional network (MLACN) was designed for speech recognition. The designed speech recognition system achieved a 99.5% accuracy rate in recognizing common word pronunciations. The integration of sound-driven triboelectric nanogenerator-based flexible acoustic sensors with deep learning techniques holds great promise in the field of human-machine interaction.

A speech recognition method with enhanced transformer decoder

Addressing the issue that the Transformer decoder struggles to capture local features for monotonic alignment in speech recognition, and simultaneously incorporating language model cold fusion training into the decoder, an enhanced decoder-based speech recognition model is investigated. The enhanced decoder separates and combines the two attention mechanisms in the Transformer decoder into cross-attention layers and a self-attention language model module. The cross-attention layers are utilized to capture local features more efficiently from the encoder output, and the self-attention language model module is used to pre-train with additional domain-related text, followed by cold fusion training. Experimental results on the Mandarin Aishell-1 dataset demonstrate that when the encoder is a Conformer, the enhanced decoder achieves a 16.1% reduction in character error rate compared to the Transformer decoder. Furthermore, when the language model is pre-trained with suitable text data, the performance of the cold fusion-trained model is further enhanced.

Developing a Dataset of Audio Features to Classify Emotions in Speech

Emotion recognition in speech has gained increasing relevance in recent years, enabling more personalized interactions between users and automated systems. This paper presents the development of a dataset of features obtained from RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) to classify emotions in speech. The paper highlights audio processing techniques such as silence removal and framing to extract features from the recordings. The features are extracted from the audio signals using spectral techniques, time-domain analysis, and the discrete wavelet transform. The resulting dataset is used to train a neural network and the support vector machine learning algorithm. Cross-validation is employed for model training. The developed models were optimized using a software package that performs hyperparameter tuning to improve results. Finally, the emotional classification outcomes were compared. The results showed an emotion classification accuracy of 0.654 for the perceptron neural network and 0.724 for the support vector machine algorithm, demonstrating satisfactory performance in emotion classification.

Request and complaint recognition in call-center speech using a pointwise-convolution recurrent network

Request and complaint recognition in call-center speech using a pointwise-convolution recurrent network

Comparison of TextBlob and Custom Spelling Correctors for Grammar Autocorrection

The current report is devoted to 'AutoBot Assistant', which represents an automated means purpose multi function assistant application for helping users to manage files and folders, voices, information, etc. It has been developed into a Python-applied project using GUI through PyQt5 and APIs like OpenWeatherMap and NewsAPI. AutoBot would use both text to speech and speech recognition to make it a friendly and interactive user experience. The study presents an overview of the entire research and findings based on the development process, key features, usability, and applications with respect to personal and professional dimensions. The results reveal the versatility of this assistant to simplify user interaction with computer systems.

Multi-source sparse broad transfer learning for parkinson's disease diagnosis via speech.

Diagnosing Parkinson's disease (PD) via speech is crucial for its non-invasive and convenient data collection. However, the small sample size of PD speech data impedes accurate recognition of PD speech. Therefore, we propose a novel multi-source sparse broad transfer learning (SBTL) method, inspired by incremental broad learning, which balances model learning capability and the overfitting associated with limited sample size of PD speech data. Specifically, SBTL initially leverages a sparse network to preprocess highly overlapping PD speech data, facilitating the identification of intrinsic invariant features between the multi-source auxiliary domain and the target data, which contributes to reducing model complexity. Subsequently, SBTL evaluate transfer effectiveness by virtue of the incremental learning mechanism, adaptively adjusting model structure to ensure the positive transfer of knowledge from the multi-source auxiliary domains to the target domain. Numerous experimental results show that, compared to transfer learning methods for PD diagnosis via speech, SBTL consistently demonstrates significant advantages with a smaller standard deviation, particularly leading by at least 2.58%, 5.71%, 12%, and 14.81% in accuracy, precision, sensitivity, and F1-score, respectively. Even when compared to some well-known transfer learning methods, SBTL still exhibits significant advantages in most cases while maintaining comparable sensitivity. These demonstrate that SBTL is an effective, efficient, and stable multi-source transfer learning method for PD speech recognition, giving more accurate assistance information for clinicians on decision-making for PD in practice.

How I Fell in Love with Language, Linguistics, and “Activist Sociolinguistics”: An Autobiographical Essay

My autobiographical essay begins with a brief section on my high school experience, then goes into more substantive detail about my research and publications over the past 55 years at the universities I attended (University of California, Santa Cruz; University of Pennsylvania) or at which I worked (University of Guyana, Stanford University) and since I retired in 2019. I mention my key mentors and influencers, including Roger Keesing, J. Herman Blake, Robert Le Page, and William Labov. And I identify some of the foci of my research over the years, including vowel laxing in Guyanese personal pronouns, prior creolization in the history of African American Vernacular English (AAVE), the Ebonics controversy, stylistic variation in sociolinguistics, quotative all, and racial disparities in automated speech recognition. Finally, I focus on “Activist Sociolinguistics,” including fighting for increased success for AAVE and other vernacular speakers in schools and for increased justice for them in the courtroom.

Predicting hearing aids outcomes using machine learning.

Introduction The aim of this study is to measure the effectiveness of hearing aid fitting in improving understanding in quiet and in noise, and to investigate the factors that significantly influence these results. This study will be carried out through a retrospective analysis of the results obtained from patients fitted with hearing aids at hearing-aid centers between 2018 and 2021. This study explores and classifies the predictive factors of patient outcomes, looking at the impact of the choice of hearing aid technology (category level), the personalized adjustments made by the hearing care professional (amplification level, binaural loudness balancing) and the patient follow-up and daily use (data logging). Methods Hearing impaired people were fitted in hearing-aid centers from 2018 to 2021. This included 77,661 patients. In the first part of the study, the data are statistically analyzed, and various correlations are studied. Then, machine learning and feature explanatory algorithms are used for prediction, in particular, a neural network based on PyTorch and the eXtreme Gradient Boosting (XGBoost). For explanatory power, the SHapley Additive exPlanation (SHAP) method was used to evaluate the individual contribution of each variable. Results Using predictive models (SHAP value and XGBoost), the effect of technology level on Speech Perception In Noise (SPIN) and Speech Perception In Quiet (SPIQ) scores is significant, and binaural loudness compensation is significant for improving test results by the same amount. Finally, linear relationships were found between the initial Speech Noise Ratio (SNR) and the adjusted SNR, as well as for the Speech Recognition Threshold (SRT). Finally, the effect of wearing hearing aids for more than 9 hours per day was analyzed and resulted in better recovery. This is a retrospective study. This bias is compensated by the large amount of data. Conclusion Big data analysis is a new method to evaluate predictive factors for hearing aid outcomes.

A Model to Validate Different Inflections of Assamese Verbs

: A verb in a sentence defines the action to be performed and keeps the subject of the sentence in motion. Assamese is a language spoken by millions of people in Assam, a North Eastern state of India. In the Assamese language, different inflections of verbs express the tense of the action, whether it is present, past or future. Verbs are inflected according to person, number, gender, tense, and voice, thus showing their existence in different situations. In this research work, we have built a tool to check whether an Assamese verb form is valid or invalid. After checking the validity, it also displays the meaning of the inflected form of the root verb in English. The root verbs are conjugated with the inflectional morphemes, and inflected verb forms are recorded. Background: This work is completely new and done by the authors for the Assamese language. To build this tool, a few research papers on this language were studied, and a summary of them is written in the Related Work section. Methods: In this experiment, 72 root verbs and 132 inflections were used. GUI was designed by using Tkinter in Python. The inflected verb form passed through three modules consisting of different searching techniques. Assamese corpus, Google search, Assamese Wikipedia, and Manual evaluation, respectively. Result: Finally, 662 Assamese valid verb forms were found that have meaning and are used in a different context. Corpus search, Google search, Assamese Wikipedia and Manual Evaluation gave 232, 733, 550 and 112 numbers of overall search results accordingly. Novelty: This paper presents a novel validation framework for Assamese language verb inflections, which could help the linguistic community better comprehend verb inflection patterns and linguistic diversity. The manuscript explores the inflections of a lesser-known low-resource language that would be beneficial. This study could be considered as innovative and important because it has applications in areas like speech recognition, machine translation, and natural language processing. Social Implication: The linguistic community would benefit greatly from the creation of a new, freely accessible, user-friendly interface intended exclusively for validating Assamese verb inflections. The approaches used here to recognize correct inflections of Assamese verbs have been performed with in-depth analysis. The approach could have implications for language learning and education, improving machine translation and natural language processing systems. Validating the verb inflections may help in raising the awareness and recognition of research focused on underdeveloped or marginalized languages. As a result, speakers of these languages might have more equitable access to information, resources, and opportunities, thus resulting in a decrease in linguistic prejudice.

Flexible capacitive pressure sensor with high sensitivity and wide detection range applying solvent-exchanged porous lignin-cellulose hydrogel as the dielectric layer.

Recently, the application of flexible sensors for wearable devices has attracted much attention from researchers. This study reports a capacitive pressure sensor with a solvent-exchanged porous lignin-cellulose hydrogel as the dielectric layer (SPLCH). The detection range and sensitivity of the sensor were modulated by adjusting the ratio of alkaline lignin(AL) and sodium lignosulfonate(SL). The existence of the porous structure and the ester bonds generated by cross-linking were demonstrated by material characterization experiments. The compression modulus of SPLCH is in the range of 310-320 kPa. The capacitive pressure sensing prepared in this way has a sensitivity of 5.18kPa-1 and 2.87kPa-1, a wide detection range (0-185.22 kPa), a short response-recovery time (25-50 ms), a low detection limit (1.875 Pa), and a good stability of repetitive loading (20,000 cycles) within two phases, respectively, which is finally applied practically in the detection of breathing, swallowing, human motion signals, speech recognition, pedometers, and Morse code information transmission scenarios. It provides a new idea for future porous structure fabrication processes and utilization of lignin to prepare flexible capacitive pressure sensors with high detection range and high sensitivity.

Individual Differences in the Recognition of Spectrally Degraded Speech: Associations With Neurocognitive Functions in Adult Cochlear Implant Users and With Noise-Vocoded Simulations.

When listening to speech under adverse conditions, listeners compensate using neurocognitive resources. A clinically relevant form of adverse listening is listening through a cochlear implant (CI), which provides a spectrally degraded signal. CI listening is often simulated through noise-vocoding. This study investigated the neurocognitive mechanisms supporting recognition of spectrally degraded speech in adult CI users and normal-hearing (NH) peers listening to noise-vocoded speech, with the hypothesis that an overlapping set of neurocognitive functions would contribute to speech recognition in both groups. Ninety-seven adults with either a CI (54 CI individuals, mean age 66.6 years, range 45-87 years) or age-normal hearing (43 NH individuals, mean age 66.8 years, range 50-81 years) participated. Listeners heard materials varying in linguistic complexity consisting of isolated words, meaningful sentences, anomalous sentences, high-variability sentences, and audiovisually (AV) presented sentences. Participants were also tested for vocabulary knowledge, nonverbal reasoning, working memory capacity, inhibition-concentration, and speed of lexical and phonological access. Linear regression analyses with robust standard errors were performed for speech recognition tasks on neurocognitive functions. Nonverbal reasoning contributed to meaningful sentence recognition in NH peers and anomalous sentence recognition in CI users. Speed of lexical access contributed to performance on most speech tasks for CI users but not for NH peers. Finally, inhibition-concentration and vocabulary knowledge contributed to AV sentence recognition in NH listeners alone. Findings suggest that the complexity of speech materials may determine the particular contributions of neurocognitive skills, and that NH processing of noise-vocoded speech may not represent how CI listeners process speech.

Ghadeer-speech-crowd-corpus: Speech dataset.

The availability of raw data is a considerable challenge across most branches of science. In the absence of data, neither experiments can be conducted nor development can be undertaken. Despite their importance, raw data are still lacking across many scientific fields. A literature survey conducted at the beginning of our study indicated a significant lack of Arabic speech datasets. Therefore, this study aims to address this problem by proposing a new Arabic and English dataset called Ghadeer-Speech-Crowd-Corpus. This dataset was designed to target more than one branch of speech-processing applications, such as crowd speaker identification, speech synthesis (text-to-speech), and speech recognition (speech-to-text). Speech samples were recorded over three months from 210 Iraqi Arab citizens living in different parts of Iraq and included more than one accent. The proposed dataset was fully balanced with respect to sex and recordings (same number of Arabic and English recordings). Additionally, it is a mono dataset and contains 15,626 audio samples recorded at a sampling rate of 44,100 Hz, 16-bit depth, and bit rate of 705.6 kb/s. The recordings were conducted at the Academy for Media Training of the College of Media, University of Baghdad.

Blind Source Separation Using Time-Delayed Dynamic Mode Decomposition

Blind Source Separation (BSS) is a significant field of study in signal processing, with many applications in various fields such as audio processing, speech recognition, biomedical signal analysis, image processing and communication systems. Traditional methods, such as Independent Component Analysis (ICA), often rely on statistical independence assumptions, which may limit their performance in systems with significant temporal dynamics. This paper introduces an extension of the dynamic mode decomposition (DMD) approach by using time-delayed coordinates to implement BSS. Time-delay embedding enhances the capability of the method to handle complex, nonstationary signals by incorporating their temporal dependencies. We validate the approach through numerical experiments and applications, including audio signal separation, image separation and EEG artifact removal. The results demonstrate that modification achieves superior performance compared to conventional techniques, particularly in scenarios where sources exhibit dynamic coupling or non-stationary behavior.

Android Based Visual Product Identification System for the Blind

The proposed system, "Android-Based Visual Product Identification for the Blind," aims to enhance the independence of visually impaired individuals by enabling them to identify everyday products using an Android smartphone. The system leverages computer vision techniques integrated with Android’s mobile platform to capture images of products, analyse them using a trained machine learning model, and provide audio feedback to the user. The application focuses on product packaging, barcodes, and distinct visual features, ensuring accurate identification in real-time. The solution is designed to be user-friendly with voice commands, accessible navigation, and low latency, making it practical for daily use. By offering a portable and reliable means of product identification, this tool addresses the challenges faced by the blind in interacting with their surroundings, improving their quality of life and fostering independence.

Software Review

D-ID Creative Reality Studio (D-ID) is a platform for creating Artificial Intelligence (AI) presenter (digital human) videos, translating videos, and designing conversational agents. D-ID seamlessly integrates deep-learning face animation technology, large language models (LLMs), natural language processing (NLP), and speech synthesis and recognition (SSR), offering new possibilities for immersive language teaching and learning. As highlighted in the slogan, “Amaze your audience with your art” (D-ID, 2024), both language learners and teachers can utilise D-ID to create personalised multimodal digital learning resources. Aligning with the interaction hypothesis (Long, 1996), a conversational agent can serve as a language partner, offering accurate linguistic input and facilitating conversational practice anytime and anywhere. Additionally, the incorporation of a digital human is consistent with the Cognitive Theory of Multimedia Learning (Mayer & Moreno, 2003), which asserts that dual-channel input—integrating visual and auditory modalities—enhances memory retention and overall learning effectiveness. This software review centres on the web-based version of D-ID and provides a detailed analysis of the main features that can contribute to second language acquisition.

Emotional stimulated speech-based assisted early diagnosis of depressive disorders using personality-enhanced deep learning.

Early diagnosis of depression is crucial, and speech-based early diagnosis of depression is promising, but insufficient data and lack of theoretical support make it difficult to be applied. Therefore, it is valuable to combine psychiatric theories, collect speech recognition data for depression, and develop a practicable recognition method for depression. In this study, 24 patients with major depressive disorders (MDDs) and 36 healthy controls (HCs) were recruited to participate in a multi-task speech experiment. Descriptive statistics and tests of variance were used to analyze subjects' personality and speech changes. Subsequently, the speech task with the most depressive cues was explored using the Bidirectional Long - Short Term Memory (Bi-LSTM) algorithm, on which a personality-assisted multitasking deep model, i.e., multi-task attentional temporal convolutional network model (TCN-MTA). Statistical analyses of speech duration showed that the fable reading, neutral stimulus, and negative stimulus tasks had significant differences on subjects' speech duration, and the negative stimulus task had significant differences between the depressed and control groups (p < 0.001, 0.03, 0.04). Notably, the Big Five personality emotional stability scores were significantly different between the depressed and control groups (0.03). Depression was best identified using Bi-LSTM in negative (Youden index = 0.44) and positive stimulus speech (Youden index = 0.42). Further, the specificity of 0.72 and sensitivity of 0.87 for recognizing depression in negative stimulus speech using our proposed TCN-MTA outperforms existing methods. The sample size enrolled in this study is higher than the minimum sample size calculated through G-Power 3.1, but the sample size in this study is still small. The proposed deep learning-based personality-assisted multitasking method could accurately recognize major depression, which demonstrated the potential of the method based on the fusion of specialized theories and artificial intelligence.

A preliminary simulator study on exploring responses of drivers to driving system reminders on four stimuli in vehicles

In the realm of autonomous vehicles, society is undergoing a transition from conventional human-driven vehicles to autonomous driving systems. Therefore, there is an increasing demand for vehicles integrated with assistive driving systems. This pilot study designed to explore which type of driving system reminders, namely Text display, Image display, alarm notification, or humanoid voice command, provokes stronger preferences and higher rates of cooperation from drivers. A high-fidelity driving simulator mainly consisting of a Logitech PlayStation driving system, a reminder playing system and an emotion-detecting model was developed in a lab-setting environment. A cohort of participants (N = 6) was recruited to participate in the experiment, where they were tasked with completing assignments across four driving sessions, followed by a subsequent questionnaire. During each driving session, the participants were exposed to six reminders designed for different driving conditions, including seatbelt check, fuel level check, rear mirror check, over speed reminder, obstacles reminder and drowsy driving reminder. Concurrently, the participants’ driving performance was observed by the researcher, while changes in their emotional states were detected by the model. Subsequent to the driving sessions, participants were invited to complete a questionnaire for assessing the various formats of driving reminders presented by the four stimuli, utilizing a 5-level Likert scale. The results revealed that driving reminders with sounds (alarm notification and humanoid voice command) exhibited higher recognition and cooperation rates among drivers than the silent reminders (text display and image display). Participants demonstrated stronger preferences for Voice-based driving reminders, which aligns with the observed behaviours of drivers. Despite the limitations of a small sample size of participants, this within-subject study which collected data from 24 individual driving sessions (6 participants x 4 driving sessions) provides insights on enhancing communication between human drivers and computer-assisted driving systems by developing improved alert systems for drivers. It also seeks to enhance the field of automotive user interface design by developing more intuitive and responsive interactions between humans and humanoid-assistant in future autonomous vehicles.