Speech Recognition Processing Research Articles

Elevating Performance Through AI-Driven Mock Interviews

Abstract: This paper proposes an innovative AI-based mock interview platform designed to enhance interview preparedness by assessing candidates across three key dimensions: emotions, confidence, and knowledge. Utilizing deep learning convolutional neural networks, the system analyzes facial expressions to gauge emotional responses, while speech recognition and natural language processing evaluate the candidate's confidence levels. Additionally, semantic analysis and keyword mapping assess the candidate's knowledge by comparing responses with relevant online resources. This comprehensive approach aims to reduce pre-interview anxiety, boost confidence, and refine interview skills, providing a more effective preparation tool compared to traditional methods.

Advancing Accessibility through Automatic Speech Recognition and NLP Integration

Abstract: The integration of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) technologies has the potential to revolutionize accessibility and inclusive communication. This article explores the fundamentals of ASR, including acoustic modeling, language modeling, and speech signal processing techniques, and discusses the challenges posed by language diversity in developing accurate ASR systems. The advent of ASR technology has opened up numerous possibilities for enhancing accessibility across various domains, such as assistive technology, education, healthcare, and media. The integration of ASR with NLP techniques enables the processing and analysis of spoken language data, leading to the development of voice-enabled virtual assistants, conversational AI systems, and cross-lingual communication tools. However, several challenges remain, including the need for robust and accurate ASR systems, privacy and security concerns, and ethical considerations in the development and deployment of these technologies. The article also presents future directions, such as the integration of ASR and NLP with emotion recognition and sentiment analysis, advances in deep learning techniques, and the application of these technologies in healthcare and accessibility. Overall, the integration of ASR and NLP holds immense promise for creating more natural, empathetic, and inclusive communication systems, but their development and deployment must be approached with care to ensure fairness, transparency, and user privacy.

DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. In particular, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness. In this article, we propose DeepGD , a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high fault-revealing power from large unlabeled datasets. DeepGD not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs. The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability, (1) white-box, coverage-based approaches fare poorly, (2) DeepGD outperforms existing black-box test selection approaches in terms of fault detection, and (3) DeepGD also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.

Effectiveness Analysis of Entrepreneurial Legal Risk Prevention Based on Multi-Modal Deep Learning Model

With the emergence of the upsurge of entrepreneurship, entrepreneurs are increasingly concerned about legal risks. In the process of entrepreneurship, legal risk is the biggest hidden danger of entrepreneurial enterprises. The prevention and avoidance of legal risks is a long and arduous process, and not all risks can be identified and avoided in time. The deep learning method has brought great changes to the fields of speech recognition, image recognition and natural language processing. The tasks in these fields only involve single-mode input, but more recent applications need to involve multi-mode intelligence. Multimodal deep learning mainly includes three aspects: multimodal learning representation, multimodal signal fusion and multimodal application. Through the research on the legal risks in recent years, this paper believed that there are many legal problems in the development of entrepreneurial enterprises, including contract disputes, trade secret disputes, trademark infringement disputes, copyright infringement disputes, and so on. This paper aimed to study the problem of Legal Risks of Entrepreneurship (LRE) that entrepreneurs are concerned about, and proposed a new solution from the perspective of image recognition technology. 486 entrepreneurs were investigated by questionnaire. In the process of entrepreneurship, 87.04% of entrepreneurs encountered legal risks, and they would turn to lawyers for help, which is a better way. However, in most cases, entrepreneurs are exposed to LRE because of their insufficient understanding of economic law, so they solve it through lawyers. However, if a lawyer is hired, the cost would be very high, which would bring great economic pressure to the enterprise. Only 17.90% of entrepreneurs would safeguard their legitimate rights and interests through their own knowledge and legal weapons without resorting to lawyers. It can be seen that entrepreneurs have relatively low practical ability in the use of LRE, and their legal practical ability is obviously insufficient.

Towards automated transcribing and coding of embodied teamwork communication through multimodal learning analytics

AbstractEffective collaboration and teamwork skills are critical in high‐risk sectors, as deficiencies in these areas can result in injuries and risk of death. To foster the growth of these vital skills, immersive learning spaces have been created to simulate real‐world scenarios, enabling students to safely improve their teamwork abilities. In such learning environments, multiple dialogue segments can occur concurrently as students independently organise themselves to tackle tasks in parallel across diverse spatial locations. This complex situation creates challenges for educators in assessing teamwork and for students in reflecting on their performance, especially considering the importance of effective communication in embodied teamwork. To address this, we propose an automated approach for generating teamwork analytics based on spatial and speech data. We illustrate this approach within a dynamic, immersive healthcare learning environment centred on embodied teamwork. Moreover, we evaluated whether the automated approach can produce transcriptions and epistemic networks of spatially distributed dialogue segments with a quality comparable to those generated manually for research objectives. This paper makes two key contributions: (1) it proposes an approach that integrates automated speech recognition and natural language processing techniques to automate the transcription and coding of team communication and generate analytics; and (2) it provides analyses of the errors in outputs generated by those techniques, offering insights for researchers and practitioners involved in the design of similar systems.Practitioner notesWhat is currently known about this topic Immersive learning environments simulate real‐world situations, helping students improve their teamwork skills. In these settings, students can have multiple simultaneous conversations while working together on tasks at different physical locations. The dynamic nature of these interactions makes it hard for teachers to assess teamwork and communication and for students to reflect on their performance. What this paper adds We propose a method that employs multimodal learning analytics for automatically generating teamwork‐related insights into the content of student conversations. This data processing method allows for automatically transcribing and coding spatially distributed dialogue segments generated from students working in teams in an immersive learning environment and enables downstream analysis. This approach uses spatial analytics, natural language processing and automated speech recognition techniques. Implications for practitioners Automated coding of dialogue segments among team members can help create analytical tools to assist in evaluating and reflecting on teamwork. By analysing spatial and speech data, it is possible to apply learning analytics advancements to support teaching and learning in fast‐paced physical learning spaces where students can freely engage with one another.

Talking Hands: Speech to Sign Language Converter

Deaf people face communication challenges in many aspects of their lives. This project proposes a speech-to-sign language converter that can help to bridge the communication gap between deaf and hearing people. The converter would use speech recognition technology to transcribe spoken language into text. This text would then be translated into sign language images or GIFs. The images or GIFs would be displayed on a screen, allowing deaf people to easily understand the message. The converter would be designed to be as accurate and user-friendly as possible. It would also be customizable to support different sign languages. This project has the potential to make a significant impact on the lives of deaf people. It could help them to communicate more effectively in school, at work, and in social settings. It could also help to break down communication barriers and foster greater understanding between deaf and hearing people. The converter would be developed using a variety of technologies, including speech recognition, machine translation, and image processing. The objective of our project is to reduce the communication gap between normal people and disabled people by providing a virtual animation translator for sign language.

Houses classification using vision transformer with shifted patch tokenization

Deep learning has recently made great progress leading to revolutionizing image recognition, speech recognition, and natural language processing tasks that were previously challenging to make using traditional techniques. Image classification offers a lot of potential for architectural design, even though it is rarely used to uncover new techniques. It can be used to determine the client’s preferences and design a building that satisfies those preferences. The different architectural styles based on culture, region, and time are one of the main challenges for image classification in architecture. Hence, it can be challenging for untrained clients to recognize an architectural style, and sometimes some buildings are made up of various types that are difficult to classify as a single style. This paper investigates the potential of employing state-of-art cutting-edge image classification algorithms in houses classification. In addition, the paper proposes the uses of Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA) in order to enhance the performance of Vision transformer (ViT) when trained to classify house images with a small dataset, opposed to the regular ViT which requires huge dataset in order to converge. Experimentally, these techniques proved to have a positive impact on the performance of the ViT, which reached 96.85% accuracy when SPT and LSA are employed.

Speech to Text Transcription

Abstract: This paper aims to enhance speech recognition and audio processing, converting spoken sentences into text and taking input from various input sources like microphones, audio, and video files. Notably, it offers robust audio conversion capabilities, supporting MP3 to WAV and other formats. To enhance the scalability and user experience, the system is implemented as a Flask-powered web application, providing users with a seamless interface accessible through a Flask-powered web browser serves to facilitate intuitive and user-friendly communication, making the application versatile, different users -and adaptable to features here the first. This comprehensive design meets the needs of users looking for efficient speech recognition and audio processing solutions, especially for web-based applications

Mobile/VR/Robotics/IoT-Based Chatbots and Intelligent Personal Assistants for Social Inclusion

Chatbots are dialogue systems that utilize computational linguistics (CL), including automatic speech recognition (ASR) and natural language processing (NLP). Artificially intelligent conversational agents combine personalization, interoperability, and scalability with the aim of promoting safe information monitoring, management, and retrieval. AI chatbots offer user-friendly engagement for innovative e-learning, care assistance, and multilingual digital content creation, promoting inclusiveness and AI-enhanced communication. They are a breakthrough for future societies and economies as they can offer cost-effective, tailor-made, and instant exchange of knowledge and skills, including on-site assistance, health monitoring, and e-consultation. AI chatbots are constantly improving to the benefit of their users. Their economic and social impact is expected to rise as individuals, especially vulnerable populations, are trained to use them for lifelong learning, decision-making, and problem-solving.

Let’s Talk Games: An Expert Exploration of Speech Interaction with NPCs

Recent years have witnessed significant advances in speech recognition and language processing technologies, enabling natural language conversations with computers. Concurrently, the gaming industry seeks to heighten immersion as one of the leading mediums for entertainment. This work investigates the potential and challenges of using speech interaction in single-player video games, particularly for interactions with NPCs. We conducted an online survey with video game experts ( N = 20 ) alongside in-depth interviews with researchers specializing in conversational user interfaces and game user research ( N = 16 ). Our findings emphasize experts’ recognition of the considerable potential of speech interaction in games, fostering increased immersion, engagement, and entertainment. Additionally, experts address pertinent concerns like privacy issues and play environment limitations. Drawing from our findings, we provide practical recommendations for integrating speech interaction in single-player games. These encompass potential benefits, challenges, accessibility, and social implications. We further address potential regulatory requirements and offer implementation tips to enhance player experience.

Integrated Intelligent Computing Models for Cognitive-Based Neurological Disease Interpretation in Children: A Survey

INTRODUCTION: This piece of work provides the description of integrated intelligent computing models for the interpretation of cognitive-based neurological diseases in children. These diseases can have a significant impact on children's cognitive and developmental functioning. 
 OBJECTIVES: The research work review the current diagnosis and treatment methods for cognitive based neurological diseases and discusses the potential of machine learning, deep learning, Natural language processing, speech recognition, brain imaging, and signal processing techniques in interpreting the diseases.
 METHODS: A survey of recent research on integrated intelligent computing models for cognitive-based neurological disease interpretation in children is presented, highlighting the benefits and limitations of these models.
 RESULTS: The significant of this work provide important implications for healthcare practice and policy, with strengthen diagnosis and treatment of cognitive-based neurological diseases in children.
 CONCLUSION: This research paper concludes with a discussion of the ethical and legal considerations surrounding the use of intelligent computing models in healthcare, as well as future research directions in this area.

Artificial intelligence in the diagnosis and treatment of acute appendicitis: a narrative review.

Artificial intelligence is transforming healthcare. Artificial intelligence can improve patient care by analyzing large amounts of data to help make more informed decisions regarding treatments and enhance medical research through analyzing and interpreting data from clinical trials and research projects to identify subtle but meaningful trends beyond ordinary perception. Artificial intelligence refers to the simulation of human intelligence in computers, where systems of artificial intelligence can perform tasks that require human-like intelligence like speech recognition, visual perception, pattern-recognition, decision-making, and language processing. Artificial intelligence has several subdivisions, including machine learning, natural language processing, computer vision, and robotics. By automating specific routine tasks, artificial intelligence can improve healthcare efficiency. By leveraging machine learning algorithms, the systems of artificial intelligence can offer new opportunities for enhancing both the efficiency and effectiveness of surgical procedures, particularly regarding training of minimally invasive surgery. As artificial intelligence continues to advance, it is likely to play an increasingly significant role in the field of surgical learning. Physicians have assisted to a spreading role of artificial intelligence in the last decade. This involved different medical specialties such as ophthalmology, cardiology, urology, but also abdominal surgery. In addition to improvements in diagnosis, ascertainment of efficacy of treatment and autonomous actions, artificial intelligence has the potential to improve surgeons' ability to better decide if acute surgery is indicated or not. The role of artificial intelligence in the emergency departments has also been investigated. We considered one of the most common condition the emergency surgeons have to face, acute appendicitis, to assess the state of the art of artificial intelligence in this frequent acute disease. The role of artificial intelligence in diagnosis and treatment of acute appendicitis will be discussed in this narrative review.

A Practical Approach for Employing Tensor Train Decomposition in Edge Devices

Deep Neural Networks (DNN) have made significant advances in various fields including speech recognition and image processing. Typically, modern DNNs are both compute and memory intensive, therefore their deployment in low-end devices is a challenging task. A well-known technique to address this problem is Low-Rank Factorization (LRF), where a weight tensor is approximated by one or more lower-rank tensors, reducing both the memory size and the number of executed tensor operations. However, the employment of LRF is a multi-parametric optimization process involving a huge design space where different design points represent different solutions trading-off the number of FLOPs, the memory size, and the prediction accuracy of the DNN models. As a result, extracting an efficient solution is a complex and time-consuming process. In this work, a new methodology is presented that formulates the LRF problem as a (FLOPs vs. memory vs. prediction accuracy) Design Space Exploration (DSE) problem. Then, the DSE space is drastically pruned by removing inefficient solutions. Our experimental results prove that the design space can be efficiently pruned, therefore extract only a limited set of solutions with improved accuracy, memory, and FLOPs compared to the original (non-factorized) model. Our methodology has been developed as a stand-alone, parameterized module integrated into T3F library of TensorFlow 2.X.

Multilingual Framework for Risk Assessment and Symptom Tracking (MRAST).

The importance and value of real-world data in healthcare cannot be overstated because it offers a valuable source of insights into patient experiences. Traditional patient-reported experience and outcomes measures (PREMs/PROMs) often fall short in addressing the complexities of these experiences due to subjectivity and their inability to precisely target the questions asked. In contrast, diary recordings offer a promising solution. They can provide a comprehensive picture of psychological well-being, encompassing both psychological and physiological symptoms. This study explores how using advanced digital technologies, i.e., automatic speech recognition and natural language processing, can efficiently capture patient insights in oncology settings. We introduce the MRAST framework, a simplified way to collect, structure, and understand patient data using questionnaires and diary recordings. The framework was validated in a prospective study with 81 colorectal and 85 breast cancer survivors, of whom 37 were male and 129 were female. Overall, the patients evaluated the solution as well made; they found it easy to use and integrate into their daily routine. The majority (75.3%) of the cancer survivors participating in the study were willing to engage in health monitoring activities using digital wearable devices daily for an extended period. Throughout the study, there was a noticeable increase in the number of participants who perceived the system as having excellent usability. Despite some negative feedback, 44.44% of patients still rated the app's usability as above satisfactory (i.e., 7.9 on 1-10 scale) and the experience with diary recording as above satisfactory (i.e., 7.0 on 1-10 scale). Overall, these findings also underscore the significance of user testing and continuous improvement in enhancing the usability and user acceptance of solutions like the MRAST framework. Overall, the automated extraction of information from diaries represents a pivotal step toward a more patient-centered approach, where healthcare decisions are based on real-world experiences and tailored to individual needs. The potential usefulness of such data is enormous, as it enables better measurement of everyday experiences and opens new avenues for patient-centered care.

Speech Recognition Enhancement and Compression Perception in Russian Translation Teaching Cooperative System Application

Abstract In the digital age, characterized by the ubiquity of big data and artificial intelligence, Russian translation education is poised for transformative change. This study explores the integration of advanced speech recognition and compressed perception technologies into Russian translation instruction, highlighting their potential to elevate educational quality and translator training. Utilizing the Spring Boot framework, we developed a comprehensive Russian translation teaching system, incorporating a Russian-Chinese speech translation model that leverages the RNN Transducer model and an N-gram language model, alongside compressed perception theory for efficient speech signal processing. Our findings reveal a notable enhancement in Russian speech recognition accuracy and processing speed, with a 4.73% improvement in endto-end speech detection accuracy and a significant reduction in word error rate, outperforming CTC and CTC-LM models by 14.79% to 24.85%. Furthermore, speech quality and intelligibility scores experienced marked improvements. The application of speech recognition enhancement and compressed perception technologies emerges as a vital strategy for enriching Russian translation pedagogy and fostering the development of proficient translators.

Robust object detection for autonomous driving based on semi-supervised learning

Deep learning based on labeled data has brought massive success in computer vision, speech recognition, and natural language processing. Nevertheless, labeled data is just a drop in the ocean compared with unlabeled data. How can people utilize the unlabeled data effectively? Research has focused on unsupervised and semi-supervised learning to solve such a problem. Some theoretical and empirical studies have proved that unlabeled data can help boost the generalization ability and robustness under adversarial attacks. However, current theoretical research on the relationship between robustness and unlabeled data limits its scope to toy datasets. Meanwhile, the visual models in autonomous driving need a significant improvement in robustness to guarantee security and safety. This paper proposes a semi-supervised learning framework for object detection in autonomous vehicles, improving the robustness with unlabeled data. Firstly, we build a baseline with the transfer learning of an unsupervised contrastive learning method—Momentum Contrast (MoCo). Secondly, we propose a semi-supervised co-training method to label the unlabeled data for retraining, which improves generalization on the autonomous driving dataset. Thirdly, we apply the unsupervised Bounding Box data augmentation (BBAug) method based on a search algorithm, which uses reinforcement learning to improve the robustness of object detection for autonomous driving. We present an empirical study on the KITTI dataset with diverse adversarial attack methods. Our proposed method realizes the state-of-the-art generalization and robustness under white-box attacks (DPatch and Contextual Patch) and black-box attacks (Gaussian noise, Rain, Fog, and so on). Our proposed method and empirical study show that using more unlabeled data benefits the robustness of perception systems in autonomous driving.

OAVA: the open audio-visual archives aggregator

AbstractThe purpose of the current article is to provide an overview of an open-access audiovisual aggregation and search service platform developed for Greek audiovisual content during the OAVA (Open Access AudioVisual Archive) project. The platform allows the search of audiovisual resources utilizing metadata descriptions, as well as full-text search utilizing content generated from automatic speech recognition (ASR) processes through deep learning models. A dataset containing reliable Greek audiovisual content providers and their resources (1710 in total) is created. Both providers and resources are reviewed according to specific criteria already established and used for content aggregation purposes, to ensure the quality of the content and to avoid copyright infringements. Well-known aggregation services and well-established schemas for audiovisual resources have been studied and considered regarding both aggregated content and metadata. Most Greek audiovisual content providers do not use established metadata schemas when publishing their content, nor technical cooperation with them is guaranteed. Thus, a model is developed for reconciliation and aggregation. To utilize audiovisual resources the OAVA platform makes use of the latest state-of-the-art ASR approaches. OAVA platform supports Greek and English speech-to-text models. Specifically for Greek, to mitigate the scarcity of available datasets, a large-scale ASR dataset is annotated to train and evaluate deep learning architectures. The result of the above-mentioned efforts, namely selection of content, metadata, development of appropriate ASR techniques, and aggregation and enrichment of content and metadata, is the OAVA platform. This unified search mechanism for Greek audiovisual content will serve teaching, research, and cultural activities. OAVA platform is available at: https://openvideoarchives.gr/.

Research on the Reform of the Teaching Mode of Rural English Education Assistance Based on the Technical Support of Network Technology

Abstract Under the background of the development of network technology, this paper aims to promote rural English teaching and constructs an English teaching model that combines English recognition technology and rural teaching. The main process of speech recognition is examined by analyzing different speech recognition technologies. Using a deep learning network, an English speech recognition model has been established. Combined with the English acoustic features in the network data, fluency of English speech is evaluated. Data embedding is performed on the English sequences in the network, combined with the sequence probability in the English data, so as to determine whether the English speech is correct or not. The Eval value for the English recognition model based on deep learning is 5.49%, while the test value is 5.89%, as per the results. As the English dataset increases, so does the English recognition technique proposed in this paper, and the accuracy remains above 0.6, and when the dataset is 500, the speech recognition accuracy is 0.8. The teaching model that combines speech recognition techniques with English teaching improves students’ English to a certain extent.

Predicting Transitions from Mild Cognitive Impairment to Dementia within 5‐years

AbstractBackgroundEarly detection of individuals at high risk of future progression toward dementia is in great need, particularly in the era of emerging treatments for Alzheimer’s disease. Given the easier administration of neuropsychological (NP) tests over invasive and expensive biomarker acquisition, we seek to predict the future cognitive status of an individual based on digital voice recordings of NP exams.MethodWe utilized automatic speech recognition and natural language processing techniques to create an automated tool that predicts if an individual with mild cognitive impairment (MCI) will develop dementia in a fixed follow‐up window (5 years). We characterize the predictive accuracy of this approach using data from the Framingham Heart Study. First, the automated transcription of the digital voice recordings were classified into 8 main NP sub‐tests including memory assessment, naming and language skill, verbal fluency, general questions, etc. Using a large language model, only the participants’ sentences were encoded into quantitative data. These data and the participants’ characteristics, such as age, sex, ApoE4 (+/‐) genotype, and education were employed to train and test a binary classifier, predicting dementia withing 5 years after a positive MCI diagnosis. We also provide a comparison between our approach and alternative models using demographics and ApoE genotype or the Mini‐Mental State Examination (MMSE) score.ResultWe evaluated the performance of the classification task on the data containing 111 stable MCI and 103 progressive MCI subjects. Our method achieved an average accuracy of 75.5% on the held‐out test data while a baseline model using age, sex, education, and ApoE has an average accuracy of 71.8%. The sensitivity and specificity of the proposed method versus the baseline model are 78.2% vs. 81.8% and 72.7% vs. 61.8%, respectively. Our approach also significantly outperforms the model based on the MMSE score by a margin of 14% in accuracy.ConclusionWe have analyzed digital voice recordings of NP exams for participants with MCI to predict their cognitive status in 5 years. The proposed approach offers a fully automated procedure with higher accuracy, providing an opportunity to develop an accessible and easy‐to‐administer screening tool that could be easily adapted to any language.

Noise estimation based on optimal smoothing and minimum controlled through recursive averaging for speech enhancement

One of the most significant challenges in the real time speech processing applications is the elimination of noise in the corrupted speech data. This noise can significantly impact the efficacy/performance of the applications of speech processing. However, developing a robust noise reduction algorithm is crucial for improving the accuracy of automatic speech recognition (ASR) and other speech processing systems under uncontrolled conditions. In this paper, we propose an algorithm to reduce the background noise in the degraded speech data under highly non-stationary conditions. The proposed optimal smoothing and minima controlled (OSMC) technique uses recursive averaging to enhance degraded speech data. Initially, a smoothed periodogram and local minima of the degraded speech data are computed and determined the time-frequency dependent threshold factor. The ratio of smoothed periodogram to local minima is used to find the active regions of speech in the degraded speech data by adapting the Bayesian minimum cost decision rule. To calculate the estimated noise spectrum for each frequency bin, a time-frequency smoothing factors are used. The perceptual evaluation of speech quality (PESQ) and normalized covariance metric (NCM) are used to evaluate the speech quality and intelligibility of the proposed technique over competing algorithms after speech enhancement. The experimental results demonstrated that the proposed algorithm has given a significant improvement in terms of average values of PESQ by 15.03% and 16.71% and NCM by 3.45% and 5.73% for NOIZEUS and Kannada speech databases at 5 dB and 10 dB respectively, over unprocessed speech under highly non-stationary noisy environments.