Speech Commands Research Articles

Phantom in the opera: adversarial music attack for robot dialogue system

This study explores the vulnerability of robot dialogue systems' automatic speech recognition (ASR) module to adversarial music attacks. Specifically, we explore music as a natural camouflage for such attacks. We propose a novel method to hide ghost speech commands in a music clip by slightly perturbing its raw waveform. We apply our attack on an industry-popular ASR model, namely the time-delay neural network (TDNN), widely used for speech and speaker recognition. Our experiment demonstrates that adversarial music crafted by our attack can easily mislead industry-level TDNN models into picking up ghost commands with high success rates. However, it sounds no different from the original music to the human ear. This reveals a serious threat by adversarial music to robot dialogue systems, calling for effective defenses against such stealthy attacks.

HoloGCS: mixed reality-based ground control station for unmanned aerial vehicle

Human–robot interaction (HRI), which studies the interaction between robots and humans, appears as a promising research idea for the future of smart factories. In this study, HoloLens as ground control station (HoloGCS) is implemented, and its performance is discussed. HoloGCS is a mixed reality-based system for controlling and monitoring unmanned aerial vehicles (UAV). The system incorporates HRI through speech commands and video streaming, enabling UAV teleoperation. HoloGCS provides a user interface that allows operators to monitor and control the UAV easily. To demonstrate the feasibility of the proposed systems, a user case study (user testing and SUS-based questionnaire) was performed to gather qualitative results. In addition, throughput, RTT, latency, and speech accuracy were also gathered and analyzed to evaluate quantitative results.

Interacting with Smart Virtual Assistants for Individuals with Dysarthria: A Comparative Study on Usability and User Preferences

This study explores the effectiveness and user experience of different interaction methods used by individuals with dysarthria when engaging with Smart Virtual Assistants (SVAs). It focuses on three primary modalities: direct speech commands through Alexa, non-verbal voice cues via the Daria system, and eye gaze control. The objective is to assess the usability, workload, and user preferences associated with each method, catering to the varying communication capabilities of individuals with dysarthria. While Alexa and Daria facilitate voice-based interactions, eye gaze control offers an alternative for those unable to use voice commands, including users with severe dysarthria. This comparative approach aims to determine how the usability of each interaction method varies, conducted with eight participants with dysarthria. The results indicated that non-verbal voice interactions, particularly with the Daria system, were favored because of their lower workload and ease of use. The eye gaze technology, while viable, presented challenges in terms of the higher workload and usability. These findings highlight the necessity of diversifying interaction methods with SVAs to accommodate the unique needs of individuals with dysarthria.

ArabAlg: A new Dataset for Arabic Speech Command Recognition for Machine Learning Applications

ArabAlg: A new Dataset for Arabic Speech Command Recognition for Machine Learning Applications

A Novel Single-Word Speech Recognition on Embedded Systems Using a Convolution Neuron Network with Improved Out-of-Distribution Detection

Advancements in AI have elevated speech recognition, with convolutional neural networks (CNNs) proving effective in processing spectrogram-transformed speech signals. CNNs, with lower parameters and higher accuracy compared to traditional models, are particularly efficient for deployment on storage-limited embedded devices. Artificial neural networks excel in predicting inputs within their expected output range but struggle with anomalies. This is usually harmful to a speech recognition system. In this paper, the neural network classifier for speech recognition is trained with a “negative branch” method, incorporating directional regularization with out-of-distribution training data, allowing it to maintain a high confidence score to the input within distribution while expressing a low confidence score to the anomaly input. It can enhance the performance of anomaly detection of the classifier, addressing issues like misclassifying the speech command that is out of the distribution. The result of the experiment suggests that the accuracy of the CNN model will not be affected by the regularization of the “negative branch”, and the performance of abnormal detection will be improved as the number of kernels of the convolutional layer increases.

Impact of Industrial Noise on Speech Interaction Performance and User Acceptance when Using the MS HoloLens 2

Even though assistance systems offer more potential due to the increasing maturity of the inherent technologies, Automatic Speech Recognition faces distinctive challenges in the industrial context. Speech recognition enables immersive assistance systems to handle inputs and commands hands-free during two-handed operative jobs. The results of the conducted study (with n = 22 participants) based on the counterbalanced within-subject design demonstrated the performance (word error rate and information transfer rate) of the HMD HoloLens 2 as a function of the sound pressure level of industrial noise. The negative influence of industrial noise was higher on the word error rate of dictation than on the information transfer rate of the speech command. Contrary to expectations, no statistically significant difference in performance was found between the stationary and non-stationary noise. Furthermore, this study confirmed the hypothesis that user acceptance was negatively influenced by erroneous speech interactions. Furthermore, the erroneous speech interaction had no statistically significant influence on the workload or physiological parameters (skin conductance level and heart rate). It can be summarized that Automatic Speech Recognition is not yet a capable interaction paradigm in an industrial context.

End-to-end Multi-modal Low-resourced Speech Keywords Recognition Using Sequential Conv2D Nets

Advanced Neural Networks are widely used to recognize multi-modal conversational speech with significant improvements in accuracy automatically. Significantly, Convolutional Neural sheets have retreated cutting-edge performance in Automatic Voice Recognition (AVR) recently more appropriately in English; however, the Hindi language has not been explored and examined well on AVR systems. The work in this article has exposed a three-layered two-dimensional Sequential Convolutional neural architecture. The Sequential Conv2D is an end-to-end system that can instantaneously exploit speech signal spectral and temporal structures. The network has been trained and tested on different cepstral features such as Frequency and Time variant-Mel-Filters, Gamma-tone Filter Cepstral Quantities, Bark-Filter band Coefficients, and Spectrogram features of speech structures. The experiment was performed on two low-resourced speech command datasets; Hindi with 27,145 Speech Keywords developed by TIFR and 23,664 (1-s utterances) of English speech commands by Google TensorFlow and AIY English Speech Commands. The experimental outcome showed that the model achieves significant performance of Convolutional layers trained on spectrograms with 91.60% accuracy, compared to that achieved in other cepstral feature labels for English speech. However, the model achieved an accuracy of 69.65% for Hindi audio words in which bark-frequency cepstral coefficients features outperformed spectrogram features.

Analisis Tindak Tutur Lokusi dalam Daftar Putar “Zental Health” pada Saluran Youtube Zenius

Locutionary speech acts are a type of speech act that focuses on the meaning of a sentence. This research is based on speakers whose pronunciation does not emphasize actual speech. The aim of this research is to describe the form and intent of locutionary speech acts in the "Zental Health"playlist on the Zenius YouTube channel. The research approach used is pragmatic, with qualitative methods in the form of descriptive research. This research uses a data collection method using free listening and note-taking techniques. At the data analysis stage, we used the divide and match method. Meanwhile, the analysis presentation technique used is informal. This research produced several locutionary speech acts contained in 4 videos on the "Zental Health" playlist on the Zenius YouTube channel, which contains 40 data points, including 16 news speech acts, 19 question speech acts, and 5 command speech acts. With this research, the author hopes that it can provide benefits for readers and other researchers as a reference source regarding locutionary speech acts.

Talk to the Wall: The Role of Speech Interaction in Collaborative Visual Analytics.

We present the results of an exploratory study on how pairs interact with speech commands and touch gestures on a wall-sized display during a collaborative sensemaking task. Previous work has shown that speech commands, alone or in combination with other input modalities, can support visual data exploration by individuals. However, it is still unknown whether and how speech commands can be used in collaboration, and for what tasks. To answer these questions, we developed a functioning prototype that we used as a technology probe. We conducted an in-depth exploratory study with 10 participant pairs to analyze their interaction choices, the interplay between the input modalities, and their collaboration. While touch was the most used modality, we found that participants preferred speech commands for global operations, used them for distant interaction, and that speech interaction contributed to the awareness of the partner's actions. Furthermore, the likelihood of using speech commands during collaboration was related to the personality trait of agreeableness. Regarding collaboration styles, participants interacted with speech equally often whether they were in loosely or closely coupled collaboration. While the partners stood closer to each other during close collaboration, they did not distance themselves to use speech commands. From our fndings, we derive and contribute a set of design considerations for collaborative and multimodal interactive data analysis systems. All supplemental materials are available at https://osf.io/8gpv2.

AUDIO SPECTROGRAM TRANSFORMER (AST): ADVANTAGES OVER TRADITIONAL ALGORITHMS IN SPEECH-TO-TEXT (STT)

Automatic Speech Recognition (ASR) has seen significant advancements in recent years, largely due to the development of deep learning models. One of the most notable advancements is the Spectrogram Transformer, a variant of the Transformer architecture tailored for audio processing tasks. In this paper, we review the Spectrogram Transformer and compare it with other traditional ASR algorithms. We discuss its benefits, such as improved performance on noisy audio and better modeling of long-range dependencies. Additionally, we explore its applications in various domains, including voice assistants, transcription services, and audio indexing. Through experiments on benchmark datasets like LibriSpeech and the Speech Commands Dataset, we demonstrate the effectiveness of the Spectrogram Transformer in achieving state-of-the-art performance. Our findings suggest that the Spectrogram Transformer offers a promising direction for future advancements in ASR technology.

Virtual Speech System Based on Sensing Technology and Teaching Management in Universities

Abstract In this paper, digital speech is compressed using discrete Fourier transform, discrete cosine transform, and improved discrete cosine transform, and compressed sensing technology is proposed. Based on the compressed sensing technology, the frequency domain parameter compression algorithm and the speech coding and decoding algorithm are designed, and the interactive 3D virtual speech system design is completed through the pre-processing of the speech system, the extraction of speech features and the design of speech control commands. The virtual voice system designed in this paper is introduced in the teaching management mode of colleges and universities, and the main functions of the system include four major sections: notification management, online Q&A, virtual voice system interaction, and teaching resource management. The virtual voice system built using sensing technology is simulated and tested, and the practical application effect of the system is studied through empirical analysis. The experimental results show that the amplitude of the sound recorded by the compression sensor in the voice sensing experiment is more concentrated, the range is concentrated between [-0.025,0.025], and the detected voice is smaller and more effective than the amplitude recorded by the cell phone. Students were mainly satisfied and very satisfied with the four system functions designed in this paper, and in terms of the online Q&A function, only one student expressed great dissatisfaction and the total number of satisfied people was 119, and the students were highly satisfied with the teaching management of the system designed in this study.

Speak Acts of Commanding in a Collection of Short Stories by Annie M.G. Schmidt

The speech act commanding, which is part of the speech act directive, is one of the grammatical materials commonly taught in foreign language teaching to develop the form of scientific sentences. The research aims to look at command speech acts in a collection of short stories by Annie M.G. Schmidt's aim is for readers to understand and increase their knowledge about command speech acts in Dutch contained in short stories as we can produce examples of speech acts. This research uses qualitative enhancement with data collection techniques. This research uses listening and note-taking techniques by listening to the use of Dutch and noting the speech forms of the characters in the short story collection by Annie M.G. Schmidt. The results of the analysis show that the speech acts used in this research are illocutionary because the collection of short stories by Annie M.G. Schmidt contains various forms of speech acts such as assertive, directive, expressive, commissive, and declaration. Based on 5 types of short stories by Annie M.G. Schmidt, namely the short story De Reus en De Draak, the short story Het Meisje dat haar naam kwijt was, the short story De Kikvorst, the short story Het neurievolkje and the short story Het beest met achternaam, there are speech acts carried out directly or indirectly, but there are more direct speech acts than acts said indirectly.

Effect of water-waves on recognition of speech signals transmitted over a wireless optical communication channel

In many sea-exploration scenarios, a wireless optical communication link between an onshore station and a floating marine structure or between a floating buoy and an underwater vehicle is required. In order to establish high-quality wireless optical communication, the effect of sea-waves must be taken into consideration. In this work, we investigate the influence of surface water waves having various amplitudes, frequencies and waveforms on the quality of an analog speech signal transmitted over a wireless optical communication link. Such investigation is also carried out on an encrypted speech signal. The quality of the speech signals is determined by utilizing a deep learning neural network that is trained to recognize ten speech commands and produces classification probabilities. In our experiments, either the detector or the light source is non-stationary as a result of the water-waves, thus distorting the received analog signal. We also show a method of deriving the frequency of the water waves by using a remote tracking unit based on micro-electro-mechanical systems (MEMS) mirrors.

Homonymy Resolution During Interpretation of Speech Commands by a Mobile Robot

Homonymy Resolution During Interpretation of Speech Commands by a Mobile Robot

Raspberry Pi-based robust speech command recognition for normal and hearing-impaired (HI)

Raspberry Pi-based robust speech command recognition for normal and hearing-impaired (HI)

Improving speech command recognition through decision-level fusion of deep filtered speech cues

Improving speech command recognition through decision-level fusion of deep filtered speech cues

Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 Months.

Brain-computer interfaces (BCIs) can be used to control assistive devices by patients with neurological disorders like amyotrophic lateral sclerosis (ALS) that limit speech and movement. For assistive control, it is desirable for BCI systems to be accurate and reliable, preferably with minimal setup time. In this study, a participant with severe dysarthria due to ALS operates computer applications with six intuitive speech commands via a chronic electrocorticographic (ECoG) implant over the ventral sensorimotor cortex. Speech commands are accurately detected and decoded (median accuracy: 90.59%) throughout a 3-month study period without model retraining or recalibration. Use of the BCI does not require exogenous timing cues, enabling the participant to issue self-paced commands at will. These results demonstrate that a chronically implanted ECoG-based speech BCI can reliably control assistive devices over long time periods with only initial model training and calibration, supporting the feasibility of unassisted home use.

The Effectiveness of the Implementation of Speech Command Recognition Algorithms in Embedded Systems

Speech is the basis of human-computer interfaces, smart house, IoT and control systems. Implementing a real time voice control system through speech commands recognition in different environments requires simple algorithms and special purpose systems. The article analyzes the existing speech commands recognition algorithms in embedded systems. A simple and efficient algorithm for voice control systems through limited speech commands has been proposed.

Bidirectional Representations for Low-Resource Spoken Language Understanding

Speech representation models lack the ability to efficiently store semantic information and require fine tuning to deliver decent performance. In this research, we introduce a transformer encoder–decoder framework with a multiobjective training strategy, incorporating connectionist temporal classification (CTC) and masked language modeling (MLM) objectives. This approach enables the model to learn contextual bidirectional representations. We evaluate the representations in a challenging low-resource scenario, where training data is limited, necessitating expressive speech embeddings to compensate for the scarcity of examples. Notably, we demonstrate that our model’s initial embeddings outperform comparable models on multiple datasets before fine tuning. Fine tuning the top layers of the representation model further enhances performance, particularly on the Fluent Speech Command dataset, even under low-resource conditions. Additionally, we introduce the concept of class attention as an efficient module for spoken language understanding, characterized by its speed and minimal parameter requirements. Class attention not only aids in explaining model predictions but also enhances our understanding of the underlying decision-making processes. Our experiments cover both English and Dutch languages, offering a comprehensive evaluation of our proposed approach.

Adversarial Deep Learning based Dampster–Shafer data fusion model for intelligent transportation system

Intelligent Transportation Systems (ITS) have revolutionized transportation by incorporating advanced technologies for efficient and safe mobility. However, these systems face challenges ensuring security and resilience against adversarial attacks. This research addresses these challenges and introduces a novel Dampster–Shafer data fusion-based Adversarial Deep Learning (DS-ADL) Model for ITS in fog cloud environments. Our proposed model focuses on three levels of adversarial attacks: original image level, feature level, and decision level. Adversarial examples are generated at each level to evaluate the system’s vulnerability comprehensively. To enhance the system’s capabilities, we leverage the power of several vital components. Firstly, we employ Dempster–Shafer-based Multimodal Sensor Fusion, enabling the fusion of information from multiple sensors for improved scene understanding. This fusion approach enhances the system’s perception and decision-making abilities. For feature extraction and classification, we utilize ResNet 101, a deep learning architecture known for its effectiveness in computer vision tasks. We introduced a novel Monomodal Multidimensional Gaussian Model (MMGM-DD) based Adversarial Detection approach to detect adversarial examples. This detection mechanism enhances the system’s ability to identify and mitigate adversarial attacks in real-time. Additionally, we incorporate the Defensive Distillation method for adversarial training, which trains the model to be robust against attacks by exposing it to adversarial examples during the training process. To evaluate the performance of our proposed model, we utilize two datasets: Google Speech Command version 0.01 and the German Traffic Sign Recognition Benchmark (GTSRB). Evaluation metrics include latency delay and computation time (fog–cloud), accuracy, MSE, loss, and F-score for attack detection and defense. The results and discussions demonstrate the effectiveness of our Dampster–Shafer data fusion-based Adversarial Deep Learning Model in enhancing the robustness and security of ITS in fog–cloud environments. The model’s ability to detect and defend against adversarial attacks while maintaining low-latency fog–cloud operations highlights its potential for real-world deployment in ITS.