Speech Commands Research Articles

Application of Hidden Markov Models in Speech Command Recognition

In this study, vector quantization and hidden Markov models were used to achieve speech command recognition. Pre-emphasis, a hamming window, and Mel-frequency cepstral coefficients were first adopted to obtain feature values. Subsequently, vector quantization and HMMs (hidden Markov models) were employed to achieve speech command recognition. The recorded speech length was three Chinese characters, which were used to test the method. Five phrases pronounced mixing various human voices were recorded and used to test the models. The recorded phrases were then used for speech command recognition to demonstrate whether the experiment results were satisfactory.

Incorporating Noise Robustness in Speech Command Recognition by Noise Augmentation of Training Data.

The advent of new devices, technology, machine learning techniques, and the availability of free large speech corpora results in rapid and accurate speech recognition. In the last two decades, extensive research has been initiated by researchers and different organizations to experiment with new techniques and their applications in speech processing systems. There are several speech command based applications in the area of robotics, IoT, ubiquitous computing, and different human-computer interfaces. Various researchers have worked on enhancing the efficiency of speech command based systems and used the speech command dataset. However, none of them catered to noise in the same. Noise is one of the major challenges in any speech recognition system, as real-time noise is a very versatile and unavoidable factor that affects the performance of speech recognition systems, particularly those that have not learned the noise efficiently. We thoroughly analyse the latest trends in speech recognition and evaluate the speech command dataset on different machine learning based and deep learning based techniques. A novel technique is proposed for noise robustness by augmenting noise in training data. Our proposed technique is tested on clean and noisy data along with locally generated data and achieves much better results than existing state-of-the-art techniques, thus setting a new benchmark.

Development of Smart Healthcare System Based on Speech Recognition Using Support Vector Machine and Dynamic Time Warping

This paper presents an effective solution based on speech recognition to provide elderly people, patients and disabled people with an easy control system. The goal is to build a low-cost system based on speech recognition to easily access Internet of Things (IoT) devices installed in smart homes and hospitals without relying on a centralized supervisory system. The proposed system used a Raspberry Pi board to control home appliances through wireless with smartphones. The main purpose of this system is to facilitate interactions between the user and home appliances through IoT communications based on speech commands. The proposed framework contribution uses a hybrid Support Vector Machine (SVM) with a Dynamic Time Warping (DTW) algorithm to enhance the speech recognition process. The proposed solution is a machine learning-based system for controlling smart devices through speech commands with an accuracy of 97%. The results helped patients and elderly people to access and control IoT devices that are compatible with our system using speech recognition. The proposed speech recognition system is flexible with scalability and availability in adapting to existing smart IoT devices, and it provides privacy in managing patient devices. The research provides an effective method to integrate our systems among medical institutions to help elderly people and patients.

Robust Understanding of Robot-Directed Speech Commands Using Sequence to Sequence With Noise Injection.

This paper describes a new method that enables a service robot to understand spoken commands in a robust manner using off-the-shelf automatic speech recognition (ASR) systems and an encoder-decoder neural network with noise injection. In numerous instances, the understanding of spoken commands in the area of service robotics is modeled as a mapping of speech signals to a sequence of commands that can be understood and performed by a robot. In a conventional approach, speech signals are recognized, and semantic parsing is applied to infer the command sequence from the utterance. However, if errors occur during the process of speech recognition, a conventional semantic parsing method cannot be appropriately applied because most natural language processing methods do not recognize such errors. We propose the use of encoder-decoder neural networks, e.g., sequence to sequence, with noise injection. The noise is injected into phoneme sequences during the training phase of encoder-decoder neural network-based semantic parsing systems. We demonstrate that the use of neural networks with a noise injection can mitigate the negative effects of speech recognition errors in understanding robot-directed speech commands i.e., increase the performance of semantic parsing. We implemented the method and evaluated it using the commands given during a general purpose service robot (GPSR) task, such as a task applied in RoboCup@Home, which is a standard service robot competition for the testing of service robots. The results of the experiment show that the proposed method, namely, sequence to sequence with noise injection (Seq2Seq-NI), outperforms the baseline methods. In addition, Seq2Seq-NI enables a robot to understand a spoken command even when the speech recognition by an off-the-shelf ASR system contains recognition errors. Moreover, in this paper we describe an experiment conducted to evaluate the influence of the injected noise and provide a discussion of the results.

Multimodal Fusion of Speech and Gesture Recognition based on Deep Learning

This paper proposes a multimodal fusion architecture based on deep learning. The architecture consists of two forms: speech command and hand gesture. First, the speech and gesture commands input by users are recognized by CNN for speech command recognition and LSTM for hand gesture recognition respectively. Secondly, the obtained results are searched by keywords and compared by similarity degree to obtain recognition results. Finally, the two results are fused to output the final instructions. Experiments show that the proposed multi-mode fusion model is superior to the single-mode fusion model.

EdgeRNN: A Compact Speech Recognition Network With Spatio-Temporal Features for Edge Computing

Driven by the vision of Internet of Things, some research efforts have already focused on designing a network of efficient speech recognition for the development of edge computing. Other researches (such as tpool2) do not make full use of spatial and temporal information in the acoustic features of speech. In this paper, we propose a compact speech recognition network with spatio-temporal features for edge computing, named EdgeRNN. Alternatively, EdgeRNN uses 1-Dimensional Convolutional Neural Network (1-D CNN) to process the overall spatial information of each frequency domain of the acoustic features. A Recurrent Neural Network (RNN) is used to process the temporal information of each frequency domain of the acoustic features. In addition, we propose a simplified attention mechanism to enhance the portion of the network that contributes to the final identification. The overall performance of EdgeRNN has been verified on speech emotion and keywords recognition. The IEMOCAP dataset is used in speech emotion recognition, and the unweighted average recall (UAR) reaches 63.98%. Speech keywords recognition uses Google's Speech Commands Datasets V1 with a weighted average recall (WAR) of 96.82%. Compared with the experimental results of the related efficient networks on Raspberry Pi 3B+, the accuracies of EdgeRNN have been improved on both of speech emotion and keywords recognition.

Design and Analysis of an Automatic UI Adaptation Framework from Multimodal Emotion Recognition using an RGB-D Sensor

Abstract This work presents the design and analysis of an Automatic adaptive user interface (AUI) that uses a novel solution for the recognition of the emotional state of a user through both facial expressions and body posture from an RGB-D sensor. Six basic emotions are recognized through facial expressions in addition to the physiological state recognized through the body posture. The facial expressions and body posture are acquired in real-time from a Kinect sensor. A scoring system is used to improve the recognition by minimizing the confusion between the different emotions. The implemented solution achieves an accuracy rate of above 90%. The recognized emotion is then used to derive an Automatic AUI where the user is provided the help automatically and can use speech commands to modify the User Interface (UI). A comprehensive user study is performed to compare the usability of the Automatic AUI with a manual system. Results show that even though the Automatic AUI is quantitatively slower and thus not as efficient, it results in a similar effectiveness and error safety compared to a manual system. In addition, the Automatic AUI results in a significantly positive user experience compared to a manual system.

QCNN Inspired Reconfigurable Keyword Spotting Processor With Hybrid Data-Weight Reuse Methods

The keyword spotting (KWS) system is one of the most important interfaces between humans and machines since it is usually the start of automatic speech recognition and natural language processing techniques. However, for KWS hardware, it is still a problem to make one specified chip both low power and high performed under multiple scenarios, such as in meeting rooms, on different traffic or in parks and so on, for different scenarios own wide range signal-noise-ratios (SNRs). The problem leads to the requirements of balanced design between KWS system accuracy and the hardware cost under various noise types and levels. To overcome the balanced design and tradeoff problems, a complete KWS processor including an Mel-Frequency Cepstrum Coefficients (MFCC) feature extractor and a quantized Convolutional Neural Network (QCNN) accelerator is proposed for wide SNR range and low-power KWS in this paper. Firstly, the approach to quantize CNNs into QCNNs with high accuracy is proposed with considerations of hardware-software tradeoff. With the tradeoff of KWS system accuracy and hardware cost, the 4bit/8bit dual-working-mode strategy is proposed to keep low hardware cost and high accuracy under different scenarios. To be specific, the training, tuning and validating of the CNNs and QCNNs are taken with the dataset of 10 keywords chosen from the Google Command Speech Dataset (GCSD). Secondly, a serial FFT based MFCC extractor is implemented with low power and small footprint. Finally, with a novel hybrid reuse strategy of input data and network weight, a reconfigurable and approximate computing based QCNN accelerator is designed. Implemented and verified under TSMC 22nm ULL technology, with the area of 1.42mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , the QCNN accelerator can achieve 5.26μW/9.08μW power consumption in 4bit/8bit work mode with accuracy of 88% and 93% respectively, which is superior to the state-of-the-art processors.

Design and Control of Dual-Arm Cooperative Manipulator using Speech Commands

Cooperative manipulators are among the subject of interest in the scientific community for the last few years. Here an overview of the design and control of such cooperative manipulators using Speech Commands in English, Hindi, and Tamil is discussed. Here we choose two identical Robot arms from lynxmotion, and both manipulators move in conjunction with one another to achieve more payload while grasping or handling the object by the end effector. The simultaneous control of identical robot manipulators could be performed by pronouncing simple speech commands by the end user using a smartphone, which then is converted into text format using a speech recognition engine and this text fed to servo controller helps in actuating the joints of identical robot arms. Cooperative manipulators are used for handling radioactive elements and also in the field of medicine as rehabilitation aid and also in surgeries. An Android app specifically built for this purpose communicates through Bluetooth technology makes the interface for end-user simple to control both identical robot arms simultaneously.

Classifying Emotional traits from Speech file using Machine Learning

With the proliferated usage of speech as a voice command over various devices in existing times, it offers hands-free as well as comfortable experience for operating the current trends of devices. However, the devices are operated just on the basis of recognition of the speech commands and not its context. It is now well known that context factor plays a contributory role in mechanism an artificial intelligence in such devices where a comprehensive analytical modeling can be carried out. In the direction of extraction contextual information from speech, it is first necessary to recognize the speech in it logical form which can be disrupted because of various external causes. Apart from this, even if the recognition is carried out well than the next challenging part will be to extract the emotional factor within the speech. The existing review shows that research-based approaches are not sufficient enough to extract meaning full information which causes a greater degree of impediment in performing classification. Therefore, the proposed system introduces a novel and cost effective framework where machine learning is utilized in a very unique manner for performing this task. The paper discusses about an analytica approach which perform feature extraction followed by applying machine learning to show that proposed system offers faster response time with higher matching found during the testing operation phase.

Home Automation by Speech Recognition

The speech control is now most important feature of a smart home. In this paper, we projected voice command module that is used to enable the user for a hands-free interaction between smart home and himself. We presented mainly three components that is required for a simple and an efficient control of smart home device(s). The wake-up-word parts allows the actual speech command processing. The voice recognition part maps the spoken voice command to text and then Voice Control Interface passes that text into an appropriate JSON format for the home automation. We evaluate every possibility of using a voice control module in the smart home by distinctly analyzing each and every component of module

Realisasi Bentuk Tindak Tutur Direktif Guru Bahasa Indonesia dalam Proses Pembelajaran

The purpose of this study was to determine the form of directive speech acts of Indonesian language teachers in the learning process. In this study primary data were collected directly by the researcher in the form of utterances and their context in the Indonesian language learning process that contained directive speech acts. There are three stages in this research, namely the provision of data, data analysis and presentation of data analysis results. The results of the study, in its realization, the use of utterances are still classified as directive utterances even though they are delivered using different modes and are not in accordance with what is said, namely the imperative, declarative, and interrogative modes. The use of command speech is a form of directive speech that is predominantly used by instructors during learning. In conclusion, the directive speech acts of teachers in the Indonesian learning process generally have varied forms, speech containing directive speech tends to be found when the teacher explains the material, and when the teacher gives an assignment.&#x0D; Keywords: Speech Acts, Directive, Indonesian Language Learning

Designing and Evaluating a Customizable Head-mounted Vision Enhancement System for People with Low Vision

Recent advances in head-mounted displays (HMDs) present an opportunity to design vision enhancement systems for people with low vision , whose vision cannot be corrected with glasses or contact lenses. We aim to understand whether and how HMDs can aid low vision people in their daily lives. We designed ForeSee , an HMD prototype that enhances people’s view of the world with image processing techniques such as magnification and edge enhancement. We evaluated these vision enhancements with 20 low vision participants who performed four viewing tasks: image recognition and reading tasks from near- and far-distance. We found that participants needed to combine and adjust the enhancements to comfortably complete the viewing tasks. We then designed two input modes to enable fast and easy customization: speech commands and smartwatch-based gestures. While speech commands are commonly used for eyes-free input, our novel set of onscreen gestures on a smartwatch can be used in scenarios where speech is not appropriate or desired. We evaluated both input modes with 11 low vision participants and found that both modes effectively enabled low vision users to customize their visual experience on the HMD. We distill design insights for HMD applications for low vision and spur new research directions.

A neurophysiological model of speech production deficits in fragile X syndrome.

Fragile X syndrome is the most common inherited intellectual disability and monogenic cause of autism spectrum disorder. Expressive language deficits, especially in speech production, are nearly ubiquitous among individuals with fragile X, but understanding of the neurological bases for these deficits remains limited. Speech production depends on feedforward control and the synchronization of neural oscillations between speech-related areas of frontal cortex and auditory areas of temporal cortex. Interaction in this circuitry allows the corollary discharge of intended speech generated from an efference copy of speech commands to be compared against actual speech sounds, which is critical for making adaptive adjustments to optimize future speech. We aimed to determine whether alterations in coherence between frontal and temporal cortices prior to speech production are present in individuals with fragile X and whether they relate to expressive language dysfunction. Twenty-one participants with full-mutation fragile X syndrome (aged 7–55 years, eight females) and 20 healthy controls (matched on age and sex) completed a talk/listen paradigm during high-density EEG recordings. During the talk task, participants repeated pronounced short vocalizations of ‘Ah’ every 1–2 s for a total of 180 s. During the listen task, participants passively listened to their recordings from the talk task. We compared pre-speech event-related potential activity, N1 suppression to speech sounds, single trial gamma power and fronto-temporal coherence between groups during these tasks and examined their relation to performance during a naturalistic language task. Prior to speech production, fragile X participants showed reduced pre-speech negativity, reduced fronto-temporal connectivity and greater frontal gamma power compared to controls. N1 suppression during self-generated speech did not differ between groups. Reduced pre-speech activity and increased frontal gamma power prior to speech production were related to less intelligible speech as well as broader social communication deficits in fragile X syndrome. Our findings indicate that coordinated pre-speech activity between frontal and temporal cortices is disrupted in individuals with fragile X in a clinically relevant way and represents a mechanism contributing to prominent speech production problems in the disorder.

An Automatic Data Construction Approach for Korean Speech Command Recognition

An Automatic Data Construction Approach for Korean Speech Command Recognition

Designing User Interaction using Gesture and Speech for Mixed Reality Interface

Mixed Reality (MR) is the next evolution of humans interacting with computer as MR can combine the physical environment and digital environment and making them coexist with each other [1]. Interaction is still a valid research area in MR, and this paper focuses on interaction rather than other research areas such as tracking, calibration, and display [2] because the current interaction technique still not intuitive enough to let the user interact with the computer. This paper explores the user interaction using gesture and speech interaction for 3D object manipulation in mixed reality environment. The paper explains the design stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, MR prototype is proposed to integrate the interaction technique using gesture and speech. The paper concludes with results and discussion.

РОЗВИТОК МОВЛЕННЄВОЇ АКТИВНОСТІ В ДІТЕЙ З РАННІМ ДИТЯЧИМ АУТИЗМОМ

У статті обґрунтовано актуальність проблеми розвитку мовленнєвої активності у дітей з раннім дитячим аутизмом (РДА). Відмічається, що діти-аутисти страждають від власної нездатності налагодити соціальну взаємодію, підтримувати контакт з чужою людиною, труднощі сприймання інформації та розуміння ситуації спілкування. Розкрито суть поняття «мовленнєва активність». Проаналізовано сучасні дослідження з проблеми. Представлено результати власного експериментального дослідження. Було встановлено, що мовленнєва активність у дітей з РДА сформована на низькому рівні. Для дошкільників характерно мовленнєвий негативізм; нестійкий інтерес до ситуації обстеження, вибіркове виконання завдань; відсутність інтересу до завдань; наявність ехолалій, фраз-штампів, автономного мовлення, відсутність власного ініціативного мовлення; відсутність потреби відповідати на поставлені запитання, звертатися із запитаннями, виправляти допущені помилки. У статті представлено корекційно-розвивальну програму, завданнями якої є розвиток мовленнєвої активності у дітей-аутистів. На заняттях приділяється увага розвитку фонематичного слуху; загальної і тонкої моторики, кінестетичних відчуттів, міміки, пантоміміки, просторових організацій рухів; мовної моторики для формування артикуляційної бази звуків, фізіологічного і фонаційного дихання; розвитку навичок правильного вживання звуків у різних формах і видах мовлення, у всіх ситуаціях спілкування. Показано, що серія логоритмічних занять сприяє успішній адаптації, реабілітації, особистісному становленню особи, її інтеграції в соціум. За результатами корекційно-розвивальної роботи у дітей з РДА підвищився рівень мовленнєвої активності.

Image and command hybrid model for vehicle control using Internet of Vehicles

AbstractThe growing volume of traffic generates the ultimate challenge on the research for traffic control. Vehicle automation is getting closer to reality due to the massive mobile communication capacity provided by fifth‐generation (5G) wireless sensor networks (WSNs). Nowadays, manually controlling and monitoring vehicles at traffic rush hours is becoming difficult for safety control. Thus, the present condition requires providing control services and devices, which are functioning in real‐time worldwide mode through soft Internet of Vehicles (IoV) services over a software‐defined network (SDN). The SDN makes software integration and progress monitoring much easier and convenient. This article focuses on WSN‐based vehicle automation following controlling and observation through speech recognition and image processing. The article also demonstrates the working prototype of the proposed mechanism to automatically drive a car and to detect obstacles through voice and image‐based deep learning, respectively. Moreover, obstacle detection on the road is a major concern in vehicle controlling systems. The system detects every obstacle on roads using cameras mounted to cover the view for driver assistance. Once the obstacle found, it generates alert for the driver to stop a collision. All these experiments require a high‐speed communication platform for evaluations. A case study is used on vehicle observation and controlling using speech commands and image processing with real‐world on‐road vehicle data sets.

InstructableCrowd: Creating IF-THEN Rules for Smartphones via Conversations with the Crowd

Natural language interfaces have become a common part of modern digital life. Chatbots utilize text-based conversations to communicate with users; personal assistants on smartphones such as Google Assistant take direct speech commands from their users; and speech-controlled devices such as Amazon Echo use voice as their only input mode. In this paper, we introduce InstructableCrowd, a crowd-powered system that allows users to program their devices via conversation. The user verbally expresses a problem to the system, in which a group of crowd workers collectively respond and program relevant multi-part IF-THEN rules to help the user. The IF-THEN rules generated by InstructableCrowd connect relevant sensor combinations (e.g., location, weather, device acceleration, etc.) to useful effectors (e.g., text messages, device alarms, etc.). Our study showed that non-programmers can use the conversational interface of InstructableCrowd to create IF-THEN rules that have similar quality compared with the rules created manually. InstructableCrowd generally illustrates how users may converse with their devices, not only to trigger simple voice commands, but also to personalize their increasingly powerful and complicated devices.

InstructableCrowd: Creating IF-THEN Rules for Smartphones via Conversations with the Crowd

Natural language interfaces have become a common part of modern digital life. Chatbots utilize text-based conversations to communicate with users; personal assistants on smartphones such as Google Assistant take direct speech commands from their users; and speech-controlled devices such as Amazon Echo use voice as their only input mode. In this paper, we introduce InstructableCrowd, a crowd-powered system that allows users to program their devices via conversation. The user verbally expresses a problem to the system, in which a group of crowd workers collectively respond and program relevant multi-part IF-THEN rules to help the user. The IF-THEN rules generated by InstructableCrowd connect relevant sensor combinations (e.g., location, weather, device acceleration, etc.) to useful effectors (e.g., text messages, device alarms, etc.). Our study showed that non-programmers can use the conversational interface of InstructableCrowd to create IF-THEN rules that have similar quality compared with the rules created manually. InstructableCrowd generally illustrates how users may converse with their devices, not only to trigger simple voice commands, but also to personalize their increasingly powerful and complicated devices.