Voice Input Research Articles

SOVAR: System of Visual Assistance and Recognition

Abstract. Around 295 million individuals globally suffer from moderate to severe vision impairment. They struggle with daily activities and depend heavily on others for assistance. Leveraging augmented reality (AR) and artificial intelligence (AI), we have developed SOVAR, a mobile application that enables greater independence for visually impaired individuals in their daily lives. SOVAR includes two modules: navigation and scene understanding. Navigation involves two phases: mapping and guidance. During mapping, SOVAR builds and optimizes the maps with key locations labeled by users via voice input. During guidance, SOVAR plans the path and guides users to requested key locations, with visual and audio assistance and realtime obstacle avoidance. The scene understanding module includes a Large Vision Language Model (LVLM) to help the users through image captioning and visual question answering. For navigation, our user study shows that participants successfully navigated to the key locations from three separate locations in 86.67% of trials without intervention. The success rate improves with increased user familiarity with the application. For scene understanding, our study shows that leveraging LVLMs to help the visually impaired allowed the participants to answer the visual-related questions with an accuracy of 100%. In this work, we developed SOVAR, a mobile application leveraging AR and AI to assist the visually impaired in navigation and scene understanding. The promising results from the user study on SOVAR demonstrate the effectiveness of AR and AI for visual assistance and indicate their potential impact on general assistive technologies.

Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework

AbstractDuring aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.

Prototype of a Virtual Assistant System Integrated with AI

This research focuses on the development of a prototype virtual assistant system integrated with artificial intelligence (AI) using the Rational Unified Process (RUP) method. The system is designed to improve user flexibility and efficiency by allowing interaction through voice commands, removing the need for traditional input devices. Python was selected as the primary programming language due to its robust capabilities in handling AI-driven applications. The system utilizes Whisper API for speech recognition, enabling the virtual assistant to accurately interpret voice inputs. Additionally, the integration of Chat GPT API allows the assistant to process and generate responses in a natural, context-aware manner. The combination of these technologies is expected to enhance user experience by making the system more intuitive and seamless, applicable to both daily tasks and complex business environments. The RUP method, structured into phases such as inception, elaboration, construction, and transition, was applied to ensure that the development process was iterative, flexible, and aligned with user needs. The results indicate that the integration of Whisper API with Chat GPT API significantly improves the quality and accuracy of voice-based interaction, streamlining system operation while minimizing the need for complex graphical interfaces. This research demonstrates the potential of voice-driven AI systems in increasing overall operational efficiency.

Data augmentation using a 1D-CNN model with MFCC/MFMC features for speech emotion recognition

Speech emotion recognition (SER) is attractive in several domains, such as automated translation, call centres, intelligent healthcare, and human–computer interaction. Deep learning models for emotion identification need considerable labelled data, which is only sometimes available in the SER industry. A database needs enough speech samples, good features, and a better classifier to identify emotions efficiently. This study uses data augmentation to enhance the amount of input voice samples and address the data shortage issue. The database capacity increases by adding white noise to the speech signals by data augmentation. In this work, the Mel-frequency Cepstral Coefficient (MFCC) and Mel-frequency Magnitude Coefficient (MFMC) features, along with a one-dimensional convolutional neural network (1D-CNN), are used to classify speech emotions. The datasets utilized to estimate the model's enactment were AESDD, CAFE, EmoDB, IEMOCAP, and MESD. The data augmentation with the 1D-CNN (MFMC) model performed best, with an average accuracy of 99.2% for AESDD, 99.5% for CAFE, 97.5% for EmoDB, 92.4% for IEMOCAP and 96.9% for the MESD database. The proposed 1D-CNN (MFMC) with data augmentation outperforms the 1D-CNN (MFCC) without data augmentation in emotion recognition.

VoiceStyle: Voice-Based Face Generation via Cross-Modal Prototype Contrastive Learning

Can we predict a person’s appearance solely based on their voice? This article explores this question by focusing on generating a face from an unheard voice segment. Our proposed method, VoiceStyle, combines cross-modal representation learning with generation modeling, enabling us to incorporate voice semantic cues into the generated face. In the first stage, we introduce cross-modal prototype contrastive (CMPC) learning to establish the association between voice and face. Recognizing the presence of false negative and deviate positive instances in real-world unlabeled data, we not only use voice–face pairs in the same video but also construct additional semantic positive pairs through unsupervised clustering, enhancing the learning process. Moreover, we recalibrate instances based on their similarity to cluster centers in the other modality. In the second stage, we harness the powerful generative capabilities of StyleGAN to produce faces. We optimize the latent code in StyleGAN’s latent space, guided by the learned voice–face alignment. To address the importance of selecting an appropriate starting point for optimization, we aim to automatically find an optimal starting point by utilizing the face prototype derived from the voice input. The entire pipeline can be implemented in a self-supervised manner, eliminating the need for manually labeled annotations. Through extensive experiments, we demonstrate the effectiveness and performance of our VoiceStyle method in both cross-modal representation learning and voice-based face generation.

Pick and Place Control of a 3-DOF Robot Manipulator Based on Image and Pattern Recognition

Board games like chess serve as an excellent testbed for human–robot interactions, where advancements can lead to broader human–robot cooperation systems. This paper presents a chess-playing robotic system to demonstrate controlled pick and place operations using a 3-DoF manipulator with image and speech recognition. The system identifies chessboard square coordinates through image processing and centroid detection before mapping them onto the physical board. User voice input is processed and transcribed into a string from which the system extracts the current and destination locations of a chess piece with a word error rate of 8.64%. Using an inverse-kinematics algorithm, the system calculates the joint angles needed to position the end effector at the desired coordinates actuating the robot. The developed system was evaluated experimentally on the 3-DoF manipulator with a voice command used to direct the robot movement in grasping a chess piece. Consideration was made involving both the own pieces as well as capturing the opponent’s pieces and moving the captured piece outside the board workspace.

Smart Assistant for Essential Voice Interaction (EVI Assistant)

Voice assistants have emerged as a significant trend in contemporary society, revolutionizing human-computer interaction by processing and responding to verbal commands. By utilizing artificial intelligence, these programs efficiently interpret voice input to perform a wide range of tasks. In today's fast-paced world, voice assistants are essential tools that enhance productivity, accessibility, and convenience. For instance, Google's smartphone assistant, which has become familiar to children due to increased smartphone usage during the pandemic, and Amazon's Alexa, which controls household devices through the Internet of Things, showcase their broad utility. The research provides a comprehensive overview of voice assistants, with a focus on the various Python packages used in their development. Emphasizing the accessibility benefits for individuals with physical disabilities, voice assistants can perform tasks such as answering questions, setting reminders, and controlling smart devices.

An Efficient Voice-Based Emotion Recognition Using Hybrid Capsule Slime Mould Dense Deep Learning Framework

Emotion recognition is an acceptable task of understanding the other’s emotions and thoughts. Modern technology allows machines to recognize objects without the need for human intervention. The existing emotion recognition system faces more difficulties in making an accurate result with limited audio files. To address this problem, a Bag of audio terms-based hybrid deep learning models will be introduced it is known as the pioneering deep learning model. Input voice data is considered from a large dataset and pre-processed using a Data normalization and adaptive bilinear filtering approach. Afterward, acoustic features are taken out from the voice signals to capture related information for emotion recognition. These features can include linear prediction coefficients (LPC), three-dimensional (3D) log-mel spectrum, mel-frequency cepstral coefficients (MFCCs), and Prosodic features. Subsequently, feature selection is performed using an improved wild horse optimization (WHO) approach. Finally, a hybrid capsule slime mould dense deep learning framework (HCSDN) is used for voice-based emotion recognition. IEMOCAP and EMODB datasets are used to calculate system performance. The performance metrics denote the proposed system achieves 96.78% accuracy, 96.45% specificity, 95.81% precision, 4.256% error rate, and 94.256% sensitivity, 0.75% false positive rate in terms of the IEMOCAP dataset. Similarly, the proposed system achieves 96.85% accuracy, 95.74% specificity, 96.12% precision, 3.432% error rate, 95.25% sensitivity, and 0.62% false positive rate in terms of the EMODB dataset.

EarSpeech: Exploring In-Ear Occlusion Effect on Earphones for Data-efficient Airborne Speech Enhancement

Earphones have become a popular voice input and interaction device. However, airborne speech is susceptible to ambient noise, making it necessary to improve the quality and intelligibility of speech on earphones in noisy conditions. As the dual-microphone structure (i.e., outer and in-ear microphones) has been widely adopted in earphones (especially ANC earphones), we design EarSpeech which exploits in-ear acoustic sensory as the complementary modality to enable airborne speech enhancement. The key idea of EarSpeech is that in-ear speech is less sensitive to ambient noise and exhibits a correlation with airborne speech. However, due to the occlusion effect, in-ear speech has limited bandwidth, making it challenging to directly correlate with full-band airborne speech. Therefore, we exploit the occlusion effect to carry out theoretical modeling and quantitative analysis of this cross-channel correlation and study how to leverage such cross-channel correlation for speech enhancement. Specifically, we design a series of methodologies including data augmentation, deep learning-based fusion, and noise mixture scheme, to improve the generalization, effectiveness, and robustness of EarSpeech, respectively. Lastly, we conduct real-world experiments to evaluate the performance of our system. Specifically, EarSpeech achieves an average improvement ratio of 27.23% and 13.92% in terms of PESQ and STOI, respectively, and significantly improves SI-SDR by 8.91 dB. Benefiting from data augmentation, EarSpeech can achieve comparable performance with a small-scale dataset that is 40 times less than the original dataset. In addition, we validate the generalization of different users, speech content, and language types, respectively, as well as robustness in the real world via comprehensive experiments. The audio demo of EarSpeech is available on https://github.com/EarSpeech/earspeech.github.io/.

Performance Optimization of Voice-Assisted File Management Systems

In this paper, we present a novel approach for managing the file system in Linux using a voice assistant. Our system allows users to perform file system operations such as creating directories, renaming files, and deleting files by issuing voice commands. We develop a voice assistant using Python libraries and integrate it with the file system in Linux. The voice assistant is capable of understanding natural language and executing commands based on the user’s voice inputs. We conduct experiments to evaluate the performance of the system and demonstrate that our approach is effective and efficient in managing the file system using voice commands. Our system can enhance the accessibility and usability of the file system in Linux for individuals with disabilities or those who prefer a hands-free approach to file management.

Design of an AI Chatbot Using Generative AI to Access Local Database

The "College Chatbot" project aims to create an advanced conversational agent tailored for college-related inquiries. This innovative chatbot incorporates a versatile interface, capable of seamlessly processing both text and voice inputs, enhancing user accessibility. Its distinctive feature includes multilingual output capabilities, enabling responses in local languages to cater to a diverse user base. The primary function of the chatbot is to comprehensively address a wide array of college-related queries, providing students and users with accurate and timely information on various aspects of college life, academic affairs, and related topics. By combining a user-friendly interface with linguistic flexibility, the College Chatbot seeks to revolutionize the way students interact with and retrieve information from their educational institutions. Keywords- AI chatbot, Generative AI, Local database, Natural Language processing, Data Integrarion,User Intraction, Query mechanism, Model training.

Lipwatch: Enabling Silent Speech Recognition on Smartwatches using Acoustic Sensing

Silent Speech Interfaces (SSI) on mobile devices offer a privacy-friendly alternative to conventional voice input methods. Previous research has primarily focused on smartphones. In this paper, we introduce Lipwatch, a novel system that utilizes acoustic sensing techniques to enable SSI on smartwatches. Lipwatch leverages the inaudible waves emitted by the watch's speaker to capture lip movements and then analyzes the echo to enable SSI. In contrast to acoustic sensing-based SSI on smartphones, our development of Lipwatch takes into full consideration the specific scenarios and requirements associated with smartwatches. Firstly, we elaborate a wake-up-free mechanism, allowing users to interact without the need for a wake-up phrase or button presses. The mechanism utilizes the inertial sensors on the smartwatch to detect gestures, in combination with acoustic signals that detecting lip movements to determine whether SSI should be activated. Secondly, we design a flexible silent speech recognition mechanism that explores limited vocabulary recognition to comprehend a broader range of user commands, even those not present in the training dataset, relieving users from strict adherence to predefined commands. We evaluate Lipwatch on 15 individuals using a set of the 80 most common interaction commands on smartwatches. The system achieves a Word Error Rate (WER) of 13.7% in user-independent test. Even when users utter commands containing words absent in the training set, Lipwatch still demonstrates a remarkable 88.7% top-3 accuracy. We implement a real-time version of Lipwatch on a commercial smartwatch. The user study shows that Lipwatch can be a practical and promising option to enable SSI on smartwatches.

Integrating NLP Chatbot into Career Guidance Web Application

Artificial Intelligence (AI) is the development of computer systems that can perform tasks that typically require human intelligence. The development of an advanced web-based chatbot system involves the integration of various technologies and methodologies to create a seamless conversational interface. Leveraging HTML, CSS, and JavaScript for frontend development, alongside Python Flask framework for backend functionality, the project aims to deliver a robust and intuitive user experience. Key to the system's operation is the incorporation of PyTorch for machine learning, allowing for the implementation of sophisticated natural language processing (NLP) techniques. Through tokenization and stemming using NLTK (Natural Language Toolkit), the system enhances its understanding of user queries, enabling it to provide relevant and contextually appropriate responses. At the core of the system lies a supervised learning model trained on a custom dataset compiled from user interactions. This model utilizes bag-of-words representations and neural network architectures to predict responses based on input text. By continuously refining its predictions through feedback mechanisms, the system improves its accuracy and responsiveness over time, ensuring a more tailored and effective user experience. Voice input functionality, facilitated by speech recognition technology, further enhances user interaction by allowing users to communicate with the chatbot using spoken commands. This feature expands the accessibility and convenience of the system, catering to a broader range of users with varying preferences and abilities. In addition to its conversational capabilities, the system prioritizes security and privacy, implementing robust user authentication mechanisms and secure data storage practices. By adhering to industry best practices and standards, the project aims to instill trust and confidence in users regarding the confidentiality and integrity of their data.User interface design and usability are also key considerations in the development process, with careful attention given to creating an intuitive and visually appealing interface. Through thoughtful design choices and user feedback, the system seeks to optimize the user experience and streamline the interaction flow, ensuring that users can easily navigate and utilize its features. Overall, the project endeavors to create an intelligent and user-friendly chatbot system that enhances communication and interaction between users and technology. By leveraging advanced technologies and methodologies, the system aims to provide personalized assistance and support to users, ultimately improving productivity, efficiency, and satisfaction in various domains and applications

Eye-Hand Typing: Eye Gaze Assisted Finger Typing via Bayesian Processes in AR.

Nowadays, AR HMDs are widely used in scenarios such as intelligent manufacturing and digital factories. In a factory environment, fast and accurate text input is crucial for operators' efficiency and task completion quality. However, the traditional AR keyboard may not meet this requirement, and the noisy environment is unsuitable for voice input. In this article, we introduce Eye-Hand Typing, an intelligent AR keyboard. We leverage the speed advantage of eye gaze and use a Bayesian process based on the information of gaze points to infer users' text input intentions. We improve the underlying keyboard algorithm without changing user input habits, thereby improving factory users' text input speed and accuracy. In real-time applications, when the user's gaze point is on the keyboard, the Bayesian process can predict the most likely characters, vocabulary, or commands that the user will input based on the position and duration of the gaze point and input history. The system can enlarge and highlight recommended text input options based on the predicted results, thereby improving user input efficiency. A user study showed that compared with the current HoloLens 2 system keyboard, Eye-Hand Typing could reduce input error rates by 28.31 % and improve text input speed by 14.5%. It also outperformed a gaze-only technique, being 43.05% more accurate and 39.55% faster. And it was no significant compromise in eye fatigue. Users also showed positive preferences.

Voice to Text Conversion using Deep Learning

Speech recognition is one of the quick developing engineering innovation. It has numerous applications in different areas, and offers numerous potential benefits. Numerous individuals might not communicate due to the dialect obstruction. Our objective is to diminish this boundary with our program planned and created to get to the framework in particular cases, giving crucial help in empowering individuals to share data by working the framework utilizing voice input May. This venture takes that calculate under consideration and endeavors to guarantee that our program recognizes discourse and changes over the input sound to content; This empowers the client to perform record operations such as spare, open, or select out of voice-only input. We plan a framework that recognizes human voices and sound clips and interprets between English and English. The yield is in content arrange and we offer choices to change over the sound from one dialect to another. Following, we trust to include a work that gives word reference implications for English words. Neural machine interpretation is the essential strategy utilized to perform machine interpretation within the industry. This work on discourse acknowledgment starts with an presentation to the innovation and its applications in different areas. Portion of the report is based on computer program enhancements in speech recognition.

IoT Based Smart E-Mirror using OpenCV with Raspberry PI

This paper describes the design, construction and working of an IoT based smart mirror. Every morning our day begins by watching ourselves at least once in mirror before leaving our homes. We interact with it psychological to find out how we look and how our attire is. Smart Mirror or Magic Mirror is one of the applications of Raspberry Pie. A computer screen embedded in mirror looks very futuristic. The Raspberry Pie stays at back scenes and controls the data displayed on mirror. While looking at mirror you can look at various notifications from social sites as well news, weather forecast and more things. Such mirrors can be programmed to work as AI and control home appliances by voice input or touch screen. The Raspberry Pi is connected to monitor via HDMI as well as it also has inbuilt Wi-Fi and Bluetooth interfaces so we can just swipe music and videos to mirror

Vanderbilt Electronic Health Record Voice Assistant Supports Clinicians.

Electronic health records (EHRs) present navigation challenges due to time-consuming searches across segmented data. Voice assistants can improve clinical workflows by allowing natural language queries and contextually aware navigation of the EHR. To develop a voice-mediated EHR assistant and interview providers to inform its future refinement. The Vanderbilt EHR Voice Assistant (VEVA) was developed as a responsive web application and designed to accept voice inputs and execute the appropriate EHR commands. Fourteen providers from Vanderbilt Medical Center were recruited to participate in interactions with VEVA and to share their experience with the technology. The purpose was to evaluate VEVA's overall usability, gather qualitative feedback, and detail suggestions for enhancing its performance. VEVA's mean system usability scale score was 81 based on the 14 providers' evaluations, which was above the standard 50th percentile score of 68. For all five summaries evaluated (overview summary, A1C results, blood pressure, weight, and health maintenance), most providers offered a positive review of VEVA. Several providers suggested modifications to make the technology more useful in their practice, ranging from summarizing current medications to changing VEVA's speech rate. Eight of the providers (64%) reported they would be willing to use VEVA in its current form. Our EHR voice assistant technology was deemed usable by most providers. With further improvements, voice assistant tools such as VEVA have the potential to improve workflows and serve as a useful adjunct tool in health care.

An NLP-based system for modulating virtual experiences using speech instructions

The application of virtual reality (VR) technology in rehabilitation is a research area that has been attracting remarkable attention recently, especially as an intervention with learning and developmental disorders such as dyslexia and autism. Multi-modal VR-based rehabilitative systems can provide a rich experience to patients by responding to various types of bio-signals. This paper proposes a framework for incorporating voice input in rehabilitative games. The proposed framework equips collaborative learning environments with voice based interaction which allows for a more dynamic flow of therapy sessions between physically impaired patients and specialists in a rehabilitative VR environment. The proposed framework’s pipeline begins with a speech recognition module to parse utterances into sentences. Next, comes the feature extraction module to identify the intent of the user and entities as well. Finally, we modulate the 3D environment in Unity. The framework was designed in a way that guarantees smooth integration with any game or 3D environment. Several technologies were tested in terms of accuracy and speed to increase the overall performance of the pipeline. To the best of our knowledge, this research is the first to do the following: 1) build a dataset of instructions, inspired by a product already used in the healthcare market, 2) based on which we could realize a multi-modal system, with voice as one of the modalities, for the context of VR-based rehabilitation. We evaluate every stage in the system pipeline, and we show that we can achieve considerable results on each of these stages including the speech recognition module, with a 5.64% word error rate, and on intent classification where the BiLSTM model outperforms BERT in the Named Entity Recognition by 11% in terms of the F1-score. We envisage the proposed system to lend itself well to emerging healthcare frameworks in the Metaverse.

Voice-based age, gender, and language recognition based on ResNet deep model and transfer learning in spectro-temporal domain

In personal identity recognition systems, detecting a person's age, gender, and language using voice signal characteristics is a crucial issue, especially because of the importance of security considerations. Age, gender, and language classification problems are important in signal processing because they are used to analyze and understand human behavior, interactions, and preferences. This can be especially useful in the fields of human-computer interaction, psychology, and social science research. In this paper, a new system for detecting a speaker's age, gender, and language based on deep learning models is presented. Deep learning models have shown great efficacy in various fields of signal processing. For this paper, a range of deep models were tested, including convolutional neural networks (CNNs), recurrent neural network (RNN), and a fine-tuning ResNet34 architecture. Additionally, techniques such as transfer learning were applied to improve the efficiency of the proposed system. The input voice signals are preprocessed by applying the spectro-temporal transform to obtain additional features that can be fed to the ResNet34 model, which is designed specifically for the task of voice signal processing. The dataset used in this paper was sourced from the Mozilla common voice initiative, which is dedicated to advancing speech recognition and language identification technologies. The performance of the proposed algorithm was evaluated in the presence of Gaussian noise to determine its robustness. The experimental results demonstrated that the proposed algorithm significantly outperformed basic algorithms and other deep neural networks in terms of age and gender recognition from voice signals.

The effects of tones and time pressure on the user experience of voice input on mobile devices for Mandarin Chinese: an experimental study

ABSTRACT This study experimented with mobile devices to investigate the effects of tones and time pressure on the voice input experience for Mandarin Chinese. Voice input for Mandarin Chinese is gaining popularity, and its implementation on mobile devices has recently developed very rapidly. Previous studies have focused mainly on technical considerations for designing a voice input system and have not investigated the user experience of voice input. Two experiments were conducted to test single-character and two-character phrases. Four measurements, the number of errors, user satisfaction, mental workload, and skin conductance, were gathered from sixty-four participants for each experiment. The results indicated that Mandarin tones had a significant influence on the number of errors and user satisfaction, and time pressure significantly influenced all measurements. Specifically, the falling-rising tone produced the highest number of errors, and the rising tone led to the lowest satisfaction. A high level of time pressure resulted in low satisfaction, a high mental workload, and high skin conductance. The results indicated that (a) Mandarin tones significantly affected error rates and user satisfaction, and (b) time pressure had a significant effect on user experience. The findings have design implications for voice input systems in terms of user experience.