Speech Recognition Research Articles

Biomimetic hair-assisted GaN optical devices for bidirectional airflow detection

Airflow sensing plays a pivotal role in numerous fields, including medicine, industry, and environmental monitoring. However, detecting bidirectional airflow using a single sensing unit poses significant challenges. In this work, a miniature airflow sensing device is introduced, utilizing a GaN optical chip integrated with a biomimetic hair structure. The sensing device comprises a monolithic GaN chip that handles both light emission and detection. The biomimetic hairs, constructed from nylon fibers and PDMS film, undergo structural bending in converting airflow signals into optical changes, modulating the light captured by the on-chip detector. The intensity of the airflow directly correlates with the bending extent of the biomimetic hair, facilitating the precise detection of airflow rates through changes in the photocurrent. The integrated device can measure a wide range of airflow rates from −23.87 ms−1 to 21.29 ms−1, and exhibit a rapid response time of 13 ms and a detection limit of 0.1 ms−1. Characterized by its compact size, fast response time, and bidirectional detection ability, the developed device holds immense potential for applications in breath detection, speech recognition, encoding information, and the realization of logic operations.

Polish Speech and Text Emotion Recognition in a Multimodal Emotion Analysis System

Emotion recognition by social robots is a serious challenge because sometimes people also do not cope with it. It is important to use information about emotions from all possible sources: facial expression, speech, or reactions occurring in the body. Therefore, a multimodal emotion recognition system was introduced, which includes the indicated sources of information and deep learning algorithms for emotion recognition. An important part of this system includes the speech analysis module, which was decided to be divided into two tracks: speech and text. An additional condition is the target language of communication, Polish, for which the number of datasets and methods is very limited. The work shows that emotion recognition using a single source—text or speech—can lead to low accuracy of the recognized emotion. It was therefore decided to compare English and Polish datasets and the latest deep learning methods in speech emotion recognition using Mel spectrograms. The most accurate LSTM models were evaluated on the English set and the Polish nEMO set, demonstrating high efficiency of emotion recognition in the case of Polish data. The conducted research is a key element in the development of a decision-making algorithm for several emotion recognition modules in a multimodal system.

New Proposed Solution for Speech Recognition Without Labeled Data: Tutoring System for Children with Autism Spectrum Disorder

New Proposed Solution for Speech Recognition Without Labeled Data: Tutoring System for Children with Autism Spectrum Disorder

Adversarial Attack and Defense for Commercial Black-box Chinese-English Speech Recognition Systems

The attacker can generate adversarial examples (AEs) to stealthily mislead automatic speech recognition (ASR) models, raising significant concerns about the security of intelligent voice control (IVC) devices. Existing adversarial attacks mainly generate AEs to mislead ASR models to output specific target English commands (e.g., open the door). However, it remains unknown whether AEs can be used to issue commands in other languages to attack commercial black-box ASR models. In this paper, taking Chinese phrases (e.g., 支付宝付款) and “Chinese-English code-switching” phrases (e.g., 关闭GPS) as the target commands, we propose adversarial attacks for commercial multilingual ASR models. In particular, if a multilingual speech recognition model can recognize Chinese and English, we call it a Chinese-English speech recognition model. In English, the meaning of “支付宝付款” and “关闭GPS” are “Alipay payment” and “turn off GPS”, respectively. In detail, we generate transferable AEs based on the open-sourced conventional DataTang Mandarin ASR model. Given 55 target commands, the success rate for generating AEs of them is up to 96% and 80% for Aliyun ASR API and Tencentyun ASR API, respectively. Our AEs can trigger actual attack actions on voice assistants (e.g., Apple Siri, Xiaomi Xiaoaitongxue) or spread malicious messages through ASR API services, while the target commands in the AEs are inaudible to human beings. Finally, by analyzing the spectrum differences between benign audio clips and AEs, we propose a general defense against adversarial audio attacks.

Phoneme-by-Phoneme Speech Recognition as a Classification of Series on a Set of Sequences of Elements of Complex Objects Using an Improved Trie-Tree

Sequences, including vector sequences, are applicable in any subject domains. Sequences of scalar values or vectors (series) can be produced by higher-order sequences, for example: a series of states, or elements of complex objects. This academic paper is devoted to the application of an improved trie-tree in the classification of series on a set of sequences of elements of complex objects using the dynamic programming method. The implementation areas of dynamic programming have been considered. It has been shown that dynamic programming is adapted to multi-step operations of calculating additive (multiplicative) similarity/difference measures. It is argued that the improved trie-tree is applicable in the problem of classifying a series on a set of sequences of elements of complex objects using such similarity/difference measures. An analysis of hierarchical representations of sets of sequences has been performed. The advantages of the improved trie-tree over traditional representations of other highly branching trees have been described. A formal description of the improved trie-tree has been developed. An explanation has been given to the previously obtained data on a significant speed gain for operations of adding and deleting sequences in the improved trie-tree relative to the use of an array with an index table (24 and 380 times, respectively). The problem of phoneme-by-phoneme recognition of speech commands has been formulated as a problem of classifying series on a set of sequences of elements of complex objects and a method for its solving has been presented. A method for classifying a series on a set of sequences of elements of complex objects using the improved trie-tree is developed. The method has been studied using the example of phoneme-by-phoneme recognition with a hierarchical representation of the dictionary of speech command classes. In this method, recognition of speech commands is executed traversing the improved trie-tree that stores a set of transcriptions of speech commands – sequences of transcription symbols that denote classes of sounds. Numerical studies have shown that classifying a series as sequences of elements of complex objects increases the frequency of correct classification compared to classifying a series on a set of series, and using the improved trie-tree reduces the time spent on classification.

Speech recognition using an english multimodal corpus with integrated image and depth information

Traditional English corpora mainly collect information from a single modality, but lack information from multimodal information, resulting in low quality of corpus information and certain problems with recognition accuracy. To solve the above problems, this paper proposes to introduce depth information into multimodal corpora, and studies the construction method of English multimodal corpora that integrates electronic images and depth information, as well as the speech recognition method of the corpus. The multimodal fusion strategy adopted integrates speech signals and image information, including key visual information such as the speaker’s lip movements and facial expressions, and uses deep learning technology to mine acoustic and visual features. The acoustic model in the Kaldi toolkit is used for experimental research.Through experimental research, the following conclusions were drawn: Under 15-dimensional lip features, the accuracy of corpus A under monophone model was 2.4% higher than that of corpus B under monophone model when the SNR (signal-to-noise ratio) was 10dB, and the accuracy of corpus A under the triphone model at the signal-to-noise ratio of 10dB was 1.7% higher than that of corpus B under the triphone model at the signal-to-noise ratio of 10dB. Under the 32-dimensional lip features, the speech recognition effect of corpus A under the monophone model at the SNR of 10dB was 1.4% higher than that of corpus B under the monophone model at the SNR of 10dB, and the accuracy of corpus A under the triphone model at the SNR of 10dB was 2.6% higher than that of corpus B under the triphone model at the SNR of 10dB. The English multimodal corpus with image and depth information has a high accuracy, and the depth information helps to improve the accuracy of the corpus.

Few‐Shot Contrastive Learning‐Based Multi‐Round Dialogue Intent Classification Method

ABSTRACTTraditional text classification models face challenges in handling long texts and understanding topic transitions in dialogue scenarios, leading to suboptimal performance in automatic speech recognition (ASR)‐based multi‐round dialogue intent classification. In this article, we propose a few‐shot contrastive learning‐based multi‐round dialogue intent classification method. First, the ASR texts are partitioned, and role‐based features are extracted using a Transformer encoder. Second, refined sample pairs are forward‐propagated, adversarial samples are generated by perturbing word embedding matrices and contrastive loss is applied to positive sample pairs. Then, positive sample pairs are input into a multi‐round reasoning module to learn semantic clues from the entire scenario through multiple dialogues, obtain reasoning features, input them into a classifier to obtain classification results, and calculate multi‐task loss. Finally, a prototype update module (PUM) is introduced to rectify the biased prototypes by using gated recurrent unit (GRU) to update the prototypes stored in the memory bank and few‐shot learning (FSL) task. Experimental evaluations demonstrate that the proposed method outperforms state‐of‐the‐art methods on two public datasets (DailyDialog and CM) and a private real‐world dataset.

Multilingual speech recognition initiative for African languages

Multilingual speech recognition initiative for African languages

The Effect of Ambient Artificial Intelligence Notes on Provider Burnout.

Healthcare provider burnout is a critical issue with significant implications for individual well-being, patient care, and healthcare system efficiency. Addressing burnout is essential for improving both provider well-being and the quality of patient care. Ambient artificial intelligence (AI) offers a novel approach to mitigating burnout by reducing the documentation burden through advanced speech recognition and natural language processing technologies that summarize the patient encounter into a clinical note to be reviewed by clinicians. To assess provider burnout and professional fulfilment associated with Ambient AI technology during a pilot study, assessed using the Stanford Professional Fulfillment Index (PFI). A pre-post observational study was conducted at University of Iowa Health Care with 38 volunteer physicians and advanced practice providers. Participants used a commercial ambient AI tool, over a 5-week trial in ambulatory environments. The AI tool transcribed patient-clinician conversations and generated preliminary clinical notes for review and entry into the electronic medical record. Burnout and professional fulfillment were assessed using the Stanford PFI at baseline and post-intervention. Pre-test and post-test surveys were completed by 35/38 participants (92% survey completion rate). Results showed a significant reduction in burnout scores, with the median burnout score improving from 4.16 to 3.16 (p=0.005), with validated Stanford PFI cutoff for overall burnout 3.33. Burnout rates decreased from 69% to 43%. There was a notable improvement in interpersonal disengagement scores (3.6 vs. 2.5, p<0.001), although work exhaustion scores did not significantly change. Professional fulfillment showed a modest, non-significant upward trend (6.1 vs. 6.5, p=0.10). Ambient AI significantly reduces healthcare provider burnout and may enhance professional fulfillment. By alleviating documentation burdens, ambient AI can improve operational efficiency and provider well-being. These findings suggest that broader implementation of ambient AI could be a strategic intervention to combat burnout in healthcare settings.

Breathy Vocal Quality, Background Noise, and Hearing Loss: How Do These Adverse Conditions Affect Speech Perception by Older Adults?

Although breathy vocal quality and hearing loss are both prevalent age-related changes, their combined impact on speech communication is poorly understood. This study investigated whether breathy vocal quality affected speech perception and listening effort by older listeners. Furthermore, the study examined how this effect was modulated by the adverse listening environment of background noise and the listener's level of hearing loss. Nineteen older adults participated in the study. Their hearing ranged from near-normal to mild-moderate sensorineural hearing loss. Participants heard speech material of low-context sentences, with stimuli resynthesized to simulate original, mild-moderately breathy, and severely breathy conditions. Speech intelligibility was measured using a speech recognition in noise paradigm, with pupillometry data collected simultaneously to measure listening effort. Simulated severely breathy vocal quality was found to reduce intelligibility and increase listening effort. Breathiness and background noise level independently modulated listening effort. The impact of hearing loss was not observed in this dataset, which can be due to the use of individualized signal to noise ratios and a small sample size. Results from this study demonstrate the challenges of listening to speech with a breathy vocal quality. Theoretically, the findings highlight the importance of periodicity cues in speech perception in noise by older listeners. Breathy voice could be challenging to separate from the noise when the noise also lacks periodicity. Clinically, it suggests the need to address both listener- and talker-related factors in speech communication by older adults.

ESTIMATION OF FORMANT INFORMATION USING AUTOCORRELATION FUNCTION OF VOICE SIGNAL

Context. The current scientific problem of extracting biometric characteristics of a user of a voice authentication system, which can significantly increase its reliability, is considered. There has been performed estimation of formant information from the voice signal, which is a part of the user template in the voice authentication system and is widely used in the processing of speech signals in other applications, including in the presence of interfering noise components. The work is distinguished by the investigation of a polyharmonic signal. Objective. The purpose of the work is to develop procedures for generating formant information based on the results of calculating the autocorrelation function of the analyzed fragment of the voice signal and their subsequent spectral analysis. Method. The procedures for generating formant information in the process of digital processing of voice signal are proposed. Initially, the autocorrelation function of the analyzed fragment of the voice signal is calculated. Based on the results of the autocorrelation function estimation, the amplitude-frequency spectrum is calculated, from which the formant information is extracted, for example, by means of threshold processing. When the signal-to-noise ratio of the analyzed voice signal fragment is low, it is advisable to iteratively calculate the autocorrelation function. The latter allows increasing the signal-to-noise ratio and the efficiency of formant information extraction. However, each subsequent iteration of the autocorrelation function calculation is associated with an increase in the required computational resource. The latter is conditioned by the doubling of the amount of processed data at each iteration. Results. The developed procedures for generating formant information were investigated both in the processing of model and experimental voice signals. The model signals had a low signal-to-noise ratio. The proposed procedures allow to determine more precisely the width of the spectrum of extracted formant frequencies, significantly increase the number of extracted formants, including cases at low signal-to-noise ratio. Conclusions. The conducted model experiments have confirmed the performance and reliability of the proposed procedures for extracting formant information both in the processing of model and experimental voice signals. The results of the research allow to recommend their use in practice for solving problems of voice authentication, speaker differentiation, speech and gender recognition, intelligence, counterintelligence, forensics and forensic examination, medicine (diseases of the speech tract and hearing). Prospects for further research may include the creation of procedures for evaluating formant information based on phase data of the processed voice signal.

Release from same-talker speech-in-speech masking: Effects of masker intelligibility and other contributing factorsa).

Human speech perception declines in the presence of masking speech, particularly when the masker is intelligible and acoustically similar to the target. A prior investigation demonstrated a substantial reduction in masking when the intelligibility of competing speech was reduced by corrupting voiced segments with noise [Huo, Sun, Fogerty, and Tang (2023), "Quantifying informational masking due to masker intelligibility in same-talker speech-in-speech perception," in Interspeech 2023, pp. 1783-1787]. As this processing also reduced the prominence of voiced segments, it was unclear whether the unmasking was due to reduced linguistic content, acoustic similarity, or both. The current study compared the masking of original competing speech (high intelligibility) to competing speech with time reversal of voiced segments (VS-reversed, low intelligibility) at various target-to-masker ratios. Modeling results demonstrated similar energetic masking between the two maskers. However, intelligibility of the target speech was considerably better with the VS-reversed masker compared to the original masker, likely due to the reduced linguistic content. Further corrupting the masker's voiced segments resulted in additional release from masking. Acoustic analyses showed that the portion of target voiced segments overlapping with masker voiced segments and the similarity between target and masker overlapped voiced segments impacted listeners' speech recognition. Evidence also suggested modulation masking in the spectro-temporal domain interferes with listeners' ability to glimpse the target.

Maxillectomy patients' speech and performance of contemporary speaker-independent automatic speech recognition platforms in Japanese.

Automatic speech recognition (ASR) can potentially help older adults and people with disabilities reduce their dependence on others and increase their participation in society. However, maxillectomy patients with reduced speech intelligibility may encounter some problems using such technologies. To investigate the accuracy of three commonly used ASR platforms when used by Japanese maxillectomy patients with and without their obturator placed. Speech samples were obtained from 29 maxillectomy patients with and without their obturator and 17 healthy volunteers. The samples were input into three speaker-independent speech recognition platforms and the transcribed text was compared with the original text to calculate the syllable error rate (SER). All participants also completed a conventional speech intelligibility test to grade their speech using Taguchi's method. A comprehensive articulation assessment of patients without their obturator was also performed. Significant differences in SER were observed between healthy and maxillectomy groups. Maxillectomy patients with an obturator showed a significant negative correlation between speech intelligibility scores and SER. However, for those without an obturator, no significant correlations were observed. Furthermore, for maxillectomy patients without an obturator, significant differences were found between syllables grouped by vowels. Syllables containing /i/, /u/ and /e/ exhibited higher error rates compared to those containing /a/ and /o/. Additionally, significant differences were observed when syllables were grouped by consonant place of articulation and manner of articulation. The three platforms performed well for healthy volunteers and maxillectomy patients with their obturator, but the SER for maxillectomy patients without their obturator was high, rendering the platforms unusable. System improvement is needed to increase accuracy for maxillectomy patients.

From One-dimensional to Multidimensional Map Neural Networks

In the contemporary landscape, artificial neural networks transcend their traditional role of function approximation and have found diverse applications in fields such as image classification, machine translation, speech recognition, and natural language processing. In some datasets, traditional architectures exhibit low test and training accuracy, coupled with high loss and prolonged training times. This study aims to introduce innovative neural network architectures that outperform conventional models. The research presents a novel framework integrating one-dimensional and multidimensional map neural networks, consisting of three distinct architectures: 1D-Map, 2D-Map, and 3D-Map. A systematic performance comparison with traditional models is conducted following the implementation of these architectures. The evaluation spans four datasets, encompassing the domains of heat treatment of electroless Ni-P nano coatings, letter recognition, combined cycle power plants, and Seoul bike sharing demand. In the heat treatment dataset, the proposed 3D-Map architecture reached 0.055 higher average test accuracy than traditional MLP architecture. In the letter recognition dataset, the test accuracy of 3D-Map architecture was 0.0523 higher than the test accuracy of the LSTM architecture. In the combined cycle power plant dataset, the test accuracy of the 3D-Map architecture was 0.0612 more than the test accuracy of the MLP architecture. In the Seoul bike-sharing demand dataset, the test accuracy of the 2D-Map architecture was 0.0696 higher than the test accuracy of the LSTM architecture. The study's findings underscore the consistently superior performance of the proposed architectures compared to their traditional counterparts.

An Intelligent Recording Method for Field Geological Survey Data in Hydraulic Engineering Based on Speech Recognition

An Intelligent Recording Method for Field Geological Survey Data in Hydraulic Engineering Based on Speech Recognition

Speech recognition in adverse conditions by humans and machines.

In the development of automatic speech recognition systems, achieving human-like performance has been a long-held goal. Recent releases of large spoken language models have claimed to achieve such performance, although direct comparison to humans has been severely limited. The present study tested L1 British English listeners against two automatic speech recognition systems (wav2vec 2.0 and Whisper, base and large sizes) in adverse listening conditions: speech-shaped noise and pub noise, at different signal-to-noise ratios, and recordings produced with or without face masks. Humans maintained the advantage against all systems, except for Whisper large, which outperformed humans in every condition but pub noise.

Augmented Reality Control based Energy Management System for Residence

Introduction: The roots of virtual reality (VR) go back to the work of Ivan Sutherland in the 1960s. At the time, virtual reality showed promise in universities and labs, peaking in the early 1990s at a stable level. The latest form of IoT is merging with the Internet of Everything (IOE). Using this technology, users can access any device, device, and machine in the environment through any middleware application or system. In recent years, smart speakers have become a common commercial method for controlling household appliances, solving some of the problems of remote control. For example, speech recognition is difficult in noisy environments and is minimized using noise cancellation. It also takes a long time to determine the current state of the device as the user need to request or issue a command to the smart speaker and need to wait for a response. The evaluation of prototype system found, using a device with gestures and interacting with a virtual 3D device instead of a real device was acceptable in terms of usability. In this article, the Unity and Vuforia are used to create an AR-based IoT virtual switch that can control any device connected to a LAN controller via HTTP requests. Methods: The system is made experimental with the Augmented Reality (AR), the virtual communication system is designed instead of physical mode. It provide the real time based image for active control of Energy management. It consist of virtual switches, it supports the AR based power control system. Results: The proposed system are made evaluated and tested under various load conditions. The virtual image consist of virtual switches to make real time control of power equipment. The proposed system achieve the latency of about 0.06 sec. Conclusion: The advancement in day today activities required with AR based energy management system with proper control measures. The designed system is supportive for such mechanism. The further system is proposed to make more energy saving concept for enhancement in proposed work.

Direct articulatory observation reveals phoneme recognition performance characteristics of a self-supervised speech model

Variability in speech pronunciation is widely observed across different linguistic backgrounds, which impacts modern automatic speech recognition performance. Here, we evaluate the performance of a self-supervised speech model in phoneme recognition using direct articulatory evidence. Findings indicate significant differences in phoneme recognition, especially in front vowels, between American English and Indian English speakers. To gain a deeper understanding of these differences, we conduct real-time MRI-based articulatory analysis, revealing distinct velar region patterns during the production of specific front vowels. This underscores the need to deepen the scientific understanding of self-supervised speech model variances to advance robust and inclusive speech technology.

EMPOWERING INDEPENDENT LEARNING THROUGH ASSISTIVE TECHNOLOGY: A QUANTITATIVE STUDY ON ITS IMPACT FOR STUDENTS WITH DISABILITIES

BACKGROUND The integration of assistive technologies in educational settings offers a promising avenue to enhance the learning experience for students with disabilities. These technologies, including voice recognition software, screen readers, and customizable learning management systems (LMS), have the potential to promote autonomy and engagement by accommodating diverse learning needs. This research will evaluate the effectiveness of these assistive technologies in supporting independent learning among these students. RESEARCH AIM The research aims to assess the usage and accessibility of assistive technologies in GBS, quantify their impact on students with disabilities, and identify the most effective technologies for enhancing learning outcomes. METHODS The study will involve surveys and performance metrics analysis among students with disabilities across various campuses of GBS. Statistical methods, including regression analysis and t-tests, will be employed to analyse the data. RESULTS Expected findings include detailed insights into assistive technology usage patterns and quantitative evidence of their impact on independent learning and academic performance. Recommendations for effective assistive technologies will be provided. CONCLUSION This research is expected to offer empirical evidence on the role of assistive technologies in supporting independent learning for students with disabilities, guiding educational institutions towards creating more inclusive environments.

Research on the prognosis and efficacy criteria of sudden sensorineural hearing loss

Sudden sensorineural hearing loss is a common and frequently-occurring disease. Looking at the efficacy evaluation of the diagnosis and treatment of sudden sensorineural hearing loss at home and abroad, each country has its own characteristics in the indicators and criteria of the treatment efficacy evaluation, and the common indicators of efficacy evaluation include: average pure-tone hearing threshold, the absolute value of hearing improvement of impaired frequency, the percentage of average pure-tone hearing threshold increase, speech discrimination score, speech recognition threshold, etc. The evaluation criteria mostly take the efficacy grade or the improvement level of average hearing threshold as the criterion of effective treatment, and the lack of unified standards is not conducive to the homogenization of global research related to sudden sensorineural hearing loss. In order to provide reference for the clinical evaluation of sudden sensorineural hearing loss in China, this study systematically reviewed the clinical practice guidelines and the evaluation indexes and criteria of clinical efficacy in high quality clinical multicenter studies of the disease in many countries.