Audio Output Research Articles

Sign Language to Speech Conversion

Human communication relies heavily on verbal and non-verbal cues, with sign language serving as a crucial method of interaction for individuals with hearing impairments. However, the communication barrier between sign language users and those unfamiliar with it remains a significant challenge. This paper presents an innovative approach to bridging this gap through an Arduino-based system that converts sign language gestures directly into speech without the use of a camera. By utilizing flex sensors and an accelerometer, this system detects hand gestures and movements, processes this data, and produces corresponding audio output. This approach offers a more accessible and portable solution compared to traditional camera-based systems, potentially revolutionizing real-time communication for the deaf community. The experimental results demonstrate a gesture recognition accuracy of 92% and a response time of under 500 milliseconds, indicating the system's viability for practical applications

Implementation of Software for Learning Message Transmission Using Morse Code

The article presents the implementation of software for training message transmission using Morse code. This is a method of sign coding when letters of the alphabet, numbers, punctuation marks and other symbols are represented as sequences of short and long signals called dots and dashes. It is intended for transmission over serial communication channels. A unique feature of Morse code is the ability to be encoded and decoded by humans without the use of special terminal devices. The paper shows a software implementation of processing input characters using a telegraphic key, it includes: various mathematical requirements and conditions that allow you adjusting to each user, provided that his input speed will either fall or remain unchanged. However, it is possible to adjust the minimum duration threshold in such a way that the system will slowly adapt to a slow increase in input speed. At the same time, the higher the value of the minimum threshold, the faster the adaptation is; however, it will fall within the limit of the maximum input speed that the device can recognize. During the implementation of the program, a flow chart of message transliteration, signal reception and transmission was developed. The whole program implements the following functions: a simple and intuitive translator, with audio output, a simulator for consolidating the skill of entering Morse code using a key; consolidating the skills of recognition and decoding by ear. To create the final application for training, the Processing toolkit was chosen, since its key parameter when choosing a development tool has a simple writing syntax to increase the speed of writing code, as well as the ability of the environment to work with the communication channel of the microcontroller.

A Novel Online Position Estimation Method and Movement Sonification System: The Soniccup.

Existing methods to obtain position from inertial sensors typically use a combination of multiple sensors and orientation modeling; thus, obtaining position from a single inertial sensor is highly desirable given the decreased setup time and reduced complexity. The dead reckoning method is commonly chosen to obtain position from acceleration; however, when applied to upper limb tracking, the accuracy of position estimates are questionable, which limits feasibility. A new method of obtaining position estimates through the use of zero velocity updates is reported, using a commercial IMU, a push-to-make momentary switch, and a 3D printed object to house the sensors. The generated position estimates can subsequently be converted into sound through sonification to provide audio feedback on reaching movements for rehabilitation applications. An evaluation of the performance of the generated position estimates from a system labeled 'Soniccup' is presented through a comparison with the outputs from a Vicon Nexus system. The results indicate that for reaching movements below one second in duration, the Soniccup produces positional estimates with high similarity to the same movements captured through the Vicon system, corresponding to comparable audio output from the two systems. However, future work to improve the performance of longer-duration movements and reduce the system latency to produce real-time audio feedback is required to improve the acceptability of the system.

Thai Currency Classification Application for Visually Impaired Persons

This research presents an application designed to assist visually impaired individuals in identifying Thai banknotes and coins. The application utilizes image processing and machine learning techniques, specifically a Convolutional Neural Network with ResNet101 architecture, to accurately classify 11 types of Thai currency. It is designed for offline use on smartphones, providing real-time audio and text output to enhance accessibility and understanding for users with visual impairments. The dataset includes 2,593 images of Thai banknotes and coins, split into 80% for training and 20% for testing. The application employs the trained model to conduct real-world tests using a smartphone camera, testing with actual banknotes and coins, achieving an average accuracy of 92.73%.

A Lightweight Visual Understanding System for Enhanced Assistance to the Visually Impaired Using an Embedded Platform

Visually impaired individuals often face significant challenges in navigating their environments due to limited access to visual information. To address this issue, a portable, cost-effective assistive tool is proposed to operate on a low-power embedded system such as the Jetson Nano. The novelty of this research lies in developing an efficient, lightweight video captioning model within constrained resources to ensure its compatibility with embedded platforms. This research aims to enhance the autonomy and accessibility of visually impaired people by providing audio descriptions of their surroundings through the processing of live-streaming videos. The proposed system utilizes two distinct lightweight deep learning modules: an object detection module based on the state-of-the-art YOLOv7 model, and a video captioning module that utilizes both the Video Swin Transformer and 2D-CNN for feature extraction, along with the Transformer network for caption generation. The goal of the object detection module is for providing real-time multiple object identification in the surrounding environment of the blind while the video captioning module is to provide detailed descriptions of the entire visual scenes and activities including objects, actions, and relationships between them. The user interacts via a headphone with the proposed system using a specific audio command to trigger the corresponding module even object detection or video captioning and receiving an audio description output for the visual contents. The system demonstrates satisfactory results, achieving inference speeds between 0.11 to 1.1 seconds for object detection and 0.91 to 1.85 seconds for video captioning, evaluated through both quantitative metrics and subjective assessments.

AudioScene: Enhancing Visual Independence Through Scene Recognition

Visually challenged individuals face numerous challenges in navigating and understanding their surroundings due to their reliance on visual information. This initiative aims to empower them by utilizing cutting-edge technology to enhance their accessibility and independence. By employing deep learning algorithms captions are generated for the image, providing users with information about their surroundings. Leveraging advanced image captioning techniques and datasets like MS COCO, key features are highlighted. The system then delivers this information as audio output to the user, enabling them to navigate with confidence. Ultimately, this innovative solution, AudioScene, offers valuable support to visually impaired individuals, facilitating safer and more informed travel experiences.

Real-Time View Assistance for the Blind using Image Processing

Visually impaired people usually have difficulty in doing daily activities. Imagine a future where visually impaired people can seamlessly and independently identify objects and people in their environment. The aim of this research is to increase the independence and mobility of visually impaired people by developing a real-time object and person recognition system. This system uses the power of machine learning and uses computer vision techniques to accurately identify and classify objects and people in the user's environment. Through the integration of speakers or headphones, the system provides auditory feedback to the user and conveys important information about the detected object or person. By combining advanced image processing algorithms with audio output, this solution serves as a valuable tool for visually impaired people, allowing them to effectively perceive and understand their surroundings. This innovative approach demonstrates the potential of technology to bridge access gaps and empower people with visual impairments in their daily lives.

American Sign Language Detection with Multi-Modal and Multi-Linguistic Output

Abstract: Our cutting-edge project introduces an advanced Sign Language Detection System, emphasizing American Sign Language (ASL) and fundamental prepositions. Utilizing sophis-ticated computer vision techniques for precise gesture recognition, the system delivers real-time video, audio, and text outputs, providing a multifaceted communication experience. A distinctive feature allowing users to select their preferred language for both audio and text outputs enhances inclusivity and customization. The system’s adaptability extends to include additional languages, ensuring its versatility in catering to diverse linguistic prefer-ences. This groundbreaking initiative addresses the pervasive communication challenges faced by the deaf and hard-of-hearing community, offering a comprehensive and highly customizable solution to bridge language gaps and foster seamless, meaningful interactions for individuals using sign language. Through the fusion of advanced technology and usercentric design, our Sign Language Detection System aims to significantly improve accessibility and communication inclusivity.

Development of Sign Language Interpreter Glove for Speech-Impaired Deaf Individuals with Levenshtein Algorithm as an Autotext Correction System.

Assistive technology is technology developed in the form of aids, adaptive tools, and rehabilitation for individuals with disabilities to compensate for their lacking abilities. With assistive technology available to people with special needs, it can help them enhance their independent living skills and reduce their dependence on others, including in communication activities. This research aims to assess the feasibility of developing assistive technology for the speech-impaired, specifically a glove that can translate the SIBI letter sequence into words displayed on a 6 x 12 LCD screen and also produce sound output from the sign language hand movements of speech-impaired individuals when using the glove. The research method used in this study is Research and Development (R&D). The calibration test phase indicated that the prototype was in good condition, as evidenced by an accuracy rate above 90% for voltage at 0◦ and 90◦ angles produced by each finger. Consequently, the next calibration phase, which involves translating sensor readings into SIBI letters through digital data values, can be carried out by taking the ADC values of each finger. Subsequently, the glove was tested to read 7 out of 20 alphabets and achieved a success rate of ≥ 90% for 5 alphabets. The lowest success rate was 70% for the letter E. The average success rate for the 7 alphabet experiments was 91.4%. In the field test phase, the glove was tested on a deaf-mute student to form several words, and the output text displayed on the LCD and audio output matched the readings corrected by the auto-text correction system.

YOLOv5 driven smart glasses for visually impaired

This project introduces an innovative AI-based interactive shopping assistance system tailored specifically for visually impaired individuals, aiming to enhance their shopping experience and promote inclusivity in retail environments. Leveraging advanced artificial intelligence algorithms, the system provides personalized assistance, navigation support, and seamless interaction for users facing visual challenges. Through intuitive interfaces and adaptive technologies, disabled individuals are empowered to navigate retail spaces with greater independence and confidence. The system utilizes a camera module and YOLO-based deep learning algorithms on a Raspberry Pi 4 for real-time object classification, converting processed information into accessible audio output. In addition to these features, the system incorporates a language customization capability using OCR and gTTS technology, enabling conversion of audio output into multiple languages based on user preference. By prioritizing the needs of visually impaired individuals, this groundbreaking system aims to enhance accessibility and promote an inclusive shopping experience for all.

Audio Enabled Object Detection System with NLP based Interaction

The project aims to develop an innovative object detection system that uses audio input and output to enhance user engagement and provide detailed product information. The system employs deep learning techniques for real-time object identification, generating an audio-based output with the object's name. In order to enable users to interact with the system verbally, it also incorporates Natural Language Processing (NLP), which interprets and processes speech in order to identify the object. The system also includes an online search module, which provides users with descriptions of the object's attributes and potential applications. The system undergoes rigorous testing and optimization to ensure accuracy and responsiveness. The project employs user-friendly technology that can assist people with low vision or disabilities, providing a practical and time-saving alternative for seeking instant information about objects. In order to make object recognition and information retrieval a smooth and inclusive experience for all users, this audio-enabled system is a first step towards bridging the gap between artificial intelligence and human-technology interaction. Key Words: Object Detection, Natural Language Processing, YOLOv4, Google Gemini, MeaningCloud, Wikipedia library

Facial Emotion Recognition for Visually Impaired People using Transfer Learning

Individuals with visual impairment often face chal- lenges in social interactions, specifically at recognizing emotional cues. The proposed framework tackles this issue head-on by de- vising a Facial Emotion Recognition(FER) system, by employing an advanced Transfer Learning approach within Convolutional Neural Networks (CNNs). By leveraging the dataset FER-2013 [13], the proposed system aims to transcend the limitationsof traditional emotion recognition methods. Transfer learningallows the model to benefit from pre-trained knowledge on vast datasets, making it more efficient and effective in capturing complex facial features associated with different emotions. This approach is designed to offer better accuracy and generalization capabilities than other conventional methods. During training, the system will be designed to comprehensively capture the intricacies of facial expressions, enabling it to not only identify individuals but also interpret subtle changes in their emotional states throughout conversations. An innovative audio output system will be integrated into the FER system to provide a smoothand accessible experience for visually impaired users, allowing for a better understanding of social dynamics. By emphasizing transfer learning, this framework is designed to be efficient and robust, potentially revolutionizing emotional understanding for visually impaired individuals and setting a new standard in the field by showcasing the superior performance achievable throughadvanced machine learning techniques. Ultimately, this research aims to bridge the social gap for the visually impaired by fosteringinclusivity, independence, and safety in their daily life.

Representation of internal speech by single neurons in human supramarginal gyrus

Speech brain–machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.

BLINDS EYE: Advancing Navigation for the Visually Impaired with Arduino Uno

Blinds Eye Advancing Navigation for the Visually Impaired with Arduino Uno. Navigating their surroundings independently, visually impaired individuals encounter challenges with traditional aids. This advanced blind navigation system, integrating an Arduino Uno for processing, a GPS module for real-time tracking, and a panic button for emergencies, presents a groundbreaking solution. Ultrasonic and IR sensors detect obstacles, issuing audio alerts and adjusting routes. An RF module aids in locating a misplaced smart stick. Overcoming limitations of existing solutions, the system's audio output provides crucial information, ensuring safer and more efficient independent travel for the visually impaired." Key Words: Arduino Uno, Blind navigation, Visually impaired, GPS module, Ultrasonic sensors, Infrared sensors, RF module, Assistive technology

MediBot For Symptoms-Driven Disease Prediction and Treatment Suggestions

To promote awareness regarding healthcare for underprivileged and inactive individuals globally, post-pandemic scenarios such as COVID-19 have led to hesitancy in seeking medical attention for minor ailments like fever and cold. This study presents a groundbreaking solution: the Medibot, which offers disease prediction based on symptoms and provides treatment recommendations through text and audio input and output. The Medibot assists users anytime, anywhere with its 24/7 services, saving time. Utilizing Deep Learning, it is capable of making decisions akin to humans and is also cost-effective for users. Keywords— “Health care”, “MediBot”, “Diseases prediction”, “Treatment recommendations”, “Deep learning”, “Cost-efficient”

Sign Recognition and Voice Conversion Device for Dumb

This project proposes a comprehensive hardware setup for home automation and environmental monitoring. The system includes components such as a Bluetooth module for remote control, a voice module (APR9600) for audio feedback, an LCD for display purposes, relay modules for controlling electrical devices, bulbs with holders for lighting, a CPU fan for temperature regulation, speakers for audio output, and a 5V power supply for operation. The core functionality of the system revolves around the integration of flux sensors for environmental monitoring. When flux sensors detect specific conditions, predefined audio signals are played through the voice module. These conditions correspond to six distinct signs, facilitating real-time alerts and notifications. Moreover, the system offers remote control capabilities through Bluetooth connectivity, enabling users to manage lighting and fan operations conveniently. Additionally, temperature monitoring is implemented using a DHT11 sensor, triggering the activation of the CPU fan when temperature thresholds are exceeded. Furthermore, an LCD display enhances user interaction by providing visual feedback and status updates. The project utilizes Arduino UNO as the central processing unit and incorporates 12V 1A adapters for powering specific components. Connectors and power cables are utilized for seamless integration and electrical connectivity. Overall, this project presents an integrated solution for home automation and environmental monitoring, offering flexibility, convenience, and enhanced user experience.

Development of Dynamic Image Recognition System for Hand Sign Language into Audio and Visual Output using Artificial Neural Networks

Abstract: This system provides a simple solution to a complex problem faced by deaf and dumb community. This paper visualizes the development web application of a real-time image recognition system whose main aim is to convert or translate the hand signs used in sign language in textual or visual and audio form output by using artificial neural networks. This system used computer vision to see signs as an input for processing and makes accurate predictions by using deep learning methods in realtime. By using artificial neural networks (ANN), the system achieves better performance in predicting or recognizing hand signs. The system makes communication between people using sign language and people who don’t know sign language effective by providing output as an audio and texted / visual, which even helps people in both ways. The project aims only to provide the society a solution by helping people who use sign language to express themselves to make it more effective and give them an ability to speak by using the technologies.

The Making of Object Recognition Eyeglasses for the Visually Impaired using Image AI

People with visual impairment may face struggles in their daily activities, as these may affect them socially, physically, and psychologically. This study aims to address this problem by utilizing quantitative experimental research to come up with Object Recognition Eyeglasses out of ImageAI. This device aims to assist the visually impaired person by recognizing the object in front of them and giving an audio output of the name of the object. Throughout the testing process, the Object Recognition Eyeglasses showed accuracy in recognizing different objects and their different varieties. It also showed its capability to recognize objects from far distances, with a maximum distance of 45 meters, and its efficiency in delivering a timely recognition and audio output with an average time interval of 1.61 and 0.63 seconds respectively. Based on these results, the Object Recognition Eyeglasses stands as an accurate, efficient, and capable assistive device that can help visually impaired people in their day-to-day lives. However, this device still needs improvement in terms of convenience by using a phone instead and modifying it to not require any internet connection.

Translation of English Videos to Indian Regional Languages

In spite of many languages being spoken in India, it is difficult for the people to understand Indian regional languages like English, Gujrati, Kannada, Tamil, Telugu, Punjabi, Malayalam, etc. The recognition and synthesis of speech are prominent emerging technologies in natural language processing and communication domains. This paper aims to leverage the open-source applications of these technologies, machine translation, text-to-speech system (TTS), and speech-to-text system (STT) to convert available online resources to Indian languages. This application takes an English language video as an input and separates the audio from video. It then divides the audio file into several smaller chunks based on the timestamps. These audio chunks are then individually converted into text using WhisperAI speech recognition model.And facebooks M2M100 model for translation. After this translation, a TTS system is required to convert the text into the desired audio output. Not many open source TTS systems are available for Indian regional languages. This application is beneficial to visually impaired people as well as individuals who are not capable of reading text to acquire knowledge in their native language. In future, this application aims to achieve ubiquitous communication enabling people of different regions to communicate with each other breaking the language barriers. Index Terms- Translation of English to Regional indian languages, ,translation and transcription using WhisperAI and FacebookM2M100, Jupiter Notebook

Visionary Audio Hub using Machine Learning

Image-to-text-to-speech conversion, powered by machine learning, is an emerging field to transform how we access and engage with information. By integrating optical character recognition (OCR) and text-to-speech (TTS) technologies, machine learning enables the extraction of text from images and its conversion into speech with greater accuracy and efficiency than ever before. This technology holds immense potential to enhance accessibility for a broad spectrum of users, including those with visual impairments, students, tourists, researchers, and musicians. For instance, individuals with visual impairments can use image-to-text-to-speech conversion to access scanned textbooks and course materials in speech format, facilitating easier comprehension and study. Similarly, tourists can leverage this technology to translate foreign language signs and text into speech, aiding navigation in unfamiliar environments. Researchers can benefit from image-to-text-to-speech conversion by extracting data from scientific papers and documents, simplifying analysis and synthesis processes. Moreover, musicians can explore new creative avenues by converting text to speech and manipulating the audio output to innovate musical compositions. Machine learning algorithms play a crucial role in improving the quality and naturalness of synthesized speech in these systems. By considering factors such as language, accent, and prosody, machine learning algorithms generate speech that closely resembles human speech patterns, enhancing clarity and understanding. Keywords— Image-to-text-to-speech conversion, Machine learning, Optical character recognition (OCR), Text-to-speech (TTS) technologies, Accessibility, Naturalness of synthesized speech.