Audio Signal Research Articles

Multimodal GRU with directed pairwise cross-modal attention for sentiment analysis

Multimodal sentiment analysis combines text, audio, and visual signals to understand human emotions. However, current methods often face challenges in handling asynchronous signals and capturing long-term dependencies between different modalities. Early techniques that merge multiple modalities often introduce unnecessary complexity, while newer methods that treat each modality separately may miss important relationships between the signals. Transformer-based models are effective, but they are typically too resource-heavy for practical use. To overcome these issues, we introduce the multimodal GRU model (MulG), which uses a cross-modal attention mechanism to better synchronize the different signals and capture their dependencies. MulG also employs GRU layers, which are efficient in handling sequential data, making it both accurate and computationally efficient. Extensive experiments on datasets such as CMU-MOSI, CMU-MOSEI, and IEMOCAP demonstrate that MulG outperforms existing methods in accuracy, F1 score, and correlation. Specifically, MulG achieves 82.2% accuracy on CMU-MOSI’s 7-class task, 82.1% on CMU-MOSEI, and 90.6% on IEMOCAP’s emotion classification. Further ablation studies show that each component of the model contributes significantly to its overall performance. By addressing the limitations of previous approaches, MulG offers a practical and scalable solution for applications like analyzing user-generated content and improving human-computer interaction.

Tactile, Audio, and Visual Dataset During Bare Finger Interaction with Textured Surfaces

This paper presents a comprehensive multi-modal dataset capturing concurrent haptic, audio, and visual signals recorded from ten participants as they interacted with ten different textured surfaces using their bare fingers. The dataset includes stereoscopic images of the textures, and fingertip position, speed, applied load, emitted sound, and friction-induced vibrations, providing an unprecedented insight into the complex dynamics underlying human tactile perception. Our approach utilizes a human finger (while most previous studies relied on rigid sensorized probes), enabling the naturalistic acquisition of haptic data and addressing a significant gap in resources for studies of human tactile exploration, perceptual mechanisms, and artificial tactile perception. Additionally, fifteen participants completed a questionnaire to evaluate their subjective perception of the surfaces. Through carefully designed data collection protocols, encompassing both controlled and free exploration scenarios, this dataset offers a rich resource for studying human multi-sensory integration and supports the development of algorithms for texture recognition based on multi-modal inputs. A preliminary analysis demonstrates the dataset’s potential, as classifiers trained on different combinations of data modalities show promising accuracy in surface identification, highlighting its value for advancing research in multi-sensory perception and the development of human-machine interfaces.

Speech Recognized by Cavity Magnon Polaritons

AbstractReservoir computing, distinguished by its reduced training overhead compared to traditional recurrent neural networks, emerges as a proficient architecture for neural networks. Here, an innovative reservoir computing (RC) paradigm based on cavity magnonics for a speech recognition task is demonstrated, which leverages the high tunability, low energy consumption, and fast response of cavity magnon polaritons (CMPs). The audio signal is filtered into a pulse train and then input into the CMP device to rapidly alter the nonlinear dynamics of the CMPs. Due to the short‐term memory effect and nonlinear response of the CMP device, virtual nodes are generated in the time domain, forming a recurrent neural network. By maximizing the nonlinear coefficient of the CMP device, exceptional speech recognition accuracy is achieved, with an error rate of less than 0.8%. This work heralds a new era for magnon‐based reservoir computing, promising enhanced capabilities for addressing complex temporal tasks in the future.

Audible enclaves crafted by nonlinear self-bending ultrasonic beams

Delivering audible content to a targeted listener without disturbing others is paramount in audio engineering. However, achieving this goal has long been challenging due to the diffraction of low-frequency (long-wavelength) audio waves in linear acoustics. Here, we introduce an approach for creating remote audio spots, dubbed audible enclaves, by harnessing the local nonlinear interaction of two self-bending ultrasonic beams with distinct spectra. The self-bending ultrasonic beams created by acoustic metasurfaces, though inaudible, can bypass obstacles such as human heads. At their intersection behind obstacles, highly localized audible enclaves are formed due to the local nonlinear interactions. Additionally, we demonstrate the ultrabroadband capabilities of our metasurface-based implementation both numerically and experimentally, spanning from 125 Hz to 4 kHz (6 octave bands), covering the majority of the audible frequency range. The practicality of our proposed technique is underscored by its compact implementation size (0.16 m, equivalent to 0.06 wavelengths at 125 Hz), as well as its robust performance under wideband transient audio signal excitation and in a common room with reverberations. Our proposed audible enclaves hold significant potential for various applications in advanced audio engineering, including private speech communications, immersive spatial audio reproduction, and high-resolution sound/quiet zone control.

Continuous and discrete decoding of overt speech with scalp electroencephalography (EEG)

Objective. Neurological disorders affecting speech production adversely impact quality of life for over 7 million individuals in the US. Traditional speech interfaces like eye-tracking devices and P300 spellers are slow and unnatural for these patients. An alternative solution, speech brain-computer interfaces (BCIs), directly decodes speech characteristics, offering a more natural communication mechanism. This research explores the feasibility of decoding speech features using non-invasive EEG.Approach. Nine neurologically intact participants were equipped with a 63-channel EEG system with additional sensors to eliminate eye artifacts. Participants read aloud sentences selected for phonetic similarity to the English language. Deep learning models, including Convolutional Neural Networks and Recurrent Neural Networks with and without attention modules, were optimized with a focus on minimizing trainable parameters and utilizing small input window sizes for real-time application. These models were employed for discrete and continuous speech decoding tasks.Main results. Statistically significant participant-independent decoding performance was achieved for discrete classes and continuous characteristics of the produced audio signal. A frequency sub-band analysis highlighted the significance of certain frequency bands (delta, theta, gamma) for decoding performance, and a perturbation analysis was used to identify crucial channels. Assessed channel selection methods did not significantly improve performance, suggesting a distributed representation of speech information encoded in the EEG signals. Leave-One-Out training demonstrated the feasibility of utilizing common speech neural correlates, reducing data collection requirements from individual participants.Significance. These findings contribute significantly to the development of EEG-enabled speech synthesis by demonstrating the feasibility of decoding both discrete and continuous speech features from EEG signals, even in the presence of EMG artifacts. By addressing the challenges of EMG interference and optimizing deep learning models for speech decoding, this study lays a strong foundation for EEG-based speech BCIs.

VIP SpaceNav – smart buildings for people with visual impairments

People with visual impairments have difficulties in navigating indoor spaces and identifying elements of interest in the environment. There are a plethora of assistive systems that address this problem, but most of them are either expensive, depend on specialized hardware or lack intuitiveness. VIP SpaceNav is a platform for low-cost, intuitive configuration and safe navigation, intended for people with visual impairments. The system’s content management system stores a 3D model of the building, together with localization information and elements of interest. The mobile application dedicated to users with visual impairments runs a localization component, as well as an obstacle detection module, providing navigation cues and information about points of interest in the vicinity, with the help of text-to-speech and audio signals. This paper presents a concept of indoor guidance for people with visual impairments and a first materialization of the prototype, based on state-of-the-art localization and spatial sounds. It also describes the piloting of the system in complex scenarios in a real building, demonstrating its potential of deployment in any indoor space, with minimal financial and time resources.

Deep learning and robotics enabled approach for audio based emotional pragmatics deficits identification in social communication disorders.

The aim of this study is to develop Deep Learning (DL) enabled robotic systems to identify audio-based emotional pragmatics deficits in individuals with social pragmatic communication deficits. The novelty of the work stems from its integration of deep learning with a robotics platform for identifying emotional pragmatics deficits. In this study, the proposed methodology utilizes the implementation of machine and DL-based classification techniques, which have been applied to a collection of open-source datasets to identify audio emotions. The application of pre-processing and converting audio signals of different emotions utilizing Mel-Frequency Cepstral Coefficients (MFCC) resulted in improved emotion classification. The data generated using MFCC were used for the training of machine or DL models. The trained models were then tested on a randomly selected dataset. DL has been proven to be more effective in the identification of emotions using robotic structure. As the data generated by MFCC is of a single dimension, therefore, one-dimensional DL algorithms, such as 1D-Convolution Neural Network, Long Short-Term Memory, and Bidirectional-Long Short-Term Memory, were utilized. In comparison to other algorithms, bidirectional Long Short-Term Memory model has resulted in higher accuracy (96.24%), loss (0.2524 in value), precision (92.87%), and recall (92.87%) in comparison to other machine and DL algorithms. Further, the proposed model was deployed on the robotic structure for real-time detection for improvement of social-emotional pragmatic responses in individuals with deficits. The approach can serve as a potential tool for the individuals with pragmatic communication deficits.

Advancements in Noise Removal: A Review of Traditional and AI-Driven Audio Signal Processing Techniques

Noise removal is one of the more prominent research areas of signal processing. Its diversified application scope from diverse falls into the list under speech enhancements, restorations of music, telecommunications aids, and a hearing aid, which reduced unwanted noises and developed through minimal deterioration of its original methods over time. There is a review of the traditional techniques as well as some of the modern techniques that include spectral subtraction, Wiener filtering, and adaptive filtering. Deep learning-based methods in recent years among such approaches are convolutional neural networks and recurrent neural networks. These discussions include capabilities and limitations related to computation efficiency, real-time processing, and effectiveness in the presence of noisy environments. In conclusion, there is a presentation of future directions for research through the use of hybrid models as well as the artificial intelligence-driven approach to eliminate noise for better and adaptive applications.

Method for Resonant Frequency Attenuation in Dynamic Audio Equalizer

The attenuation of resonant frequencies across the entire spectrum of an audio signal is important because it helps to eliminate the harshness, sibilance, clear muddiness, boominess, and proximity effect of any sound source. This paper presents a method for the attenuation of resonant frequencies across the entire spectrum of an audio signal. A spectrum obtained by the Fast Fourier Transform is segmented into bands—one-third octave bands and Equivalent Rectangular Bandwidth-scale bands—in order to obtain the maximum value per band. Additionally, a curve representing the general shape of the spectrum is generated using the standard deviation to create a threshold curve for detecting resonant frequencies. The array with maximum values per bands and the array with the threshold curve are used to detect the resonant frequencies and calculate the attenuation for each filter. Subsequently, the coefficients of a second-order section of IIR-Peak filters are calculated for processing the input signal. Twenty audio files from different sources are utilized to test the algorithm. The output produced is then compared to that produced by the commercially available Soothe2 and RESO plug-ins. The Root Mean Square Level and the Loudness Units Full Scale integrated metrics are reported. The proposed plug-in output is more attenuated than the output from commercial plug-ins under factory conditions. The average RMS attenuation is −2.32 dBFS, while Soothe2 and RESO exhibit −1.27 dBFS and −1.10 dBFS, respectively. The attenuation per octave band over time is calculated using the Wavelet Transform. Finally, an annotator agreement used as a subjective result is made with 40 people related to audio and music in order to verify if the attenuation generated by the present work at resonant frequencies agrees with subjective opinion. The octave band analysis and annotator agreement show that the proposed plug-in performs better on audio from vocal, percussion, and guitar ensembles.

Automatic Stutter Speech Recognition and Classification Using Hyper-Heuristic Search Algorithm

Introduction: The normal flow of speech is disrupted by the repetition or delay of sounds or phrases in stuttering or stammering, a speech impairment. Speech flow becomes difficult when someone stutters. Recent advances in deep learning have changed the speech domain, while stuttering recognition has received very little attention. Problem: The detection of stuttering is a fascinating area of research that encompasses pathology, psychology, and signal processing, making it challenging to detect. Stress, delays in early development, and other anomalies are the causes of stuttering. This too seems to be complicated and perplexing. Methods: To overcome these problems, a feature of automatic Stutter speech recognition and the Stutter Audio classification with Stutter audio by deep learning using Hyper-Heuristic Search Algorithm for Stutter Audio signal analysis is employed. The study performs the sound digitalization in which the Analog signals to a digital signal by using sampling and quantization, the Mel Spectrogram algorithm is employed in the Audio classification, the Automated driving to medical devices is done in the Audio deep learning, the Hyper-parameter tuning is performed in the Feature Optimization, the Automatic Speech Recognition is implemented, in which here we use the Weighted Finite-State Transducer framework, the Perceptual Linear Prediction(PLP) and Viterbi search, the Discrimination training by deep neural networks and we implement the Hyper-heuristic Search Algorithm. Results: Using the suggested system flow, the results analyse Automatic Speech Recognition and reaction time with the Classification accuracies of Stutter Audio signal analysis. In summary: The model shows a predetermined order of heuristics that enhances the process of fixing stuttering issues. This study evaluates Automatic Speech Recognition (ASR) and Response Time with a higher accuracy rate and more efficacy.

Initial findings creating a temperature prediction model using vibroacoustic signals originating from tissue needle interactions

This research explores the acquisition and analysis of vibroacoustic signals generated during tissue-tool interactions, using a conventional aspiration needle enhanced with a proximally mounted MEMS audio sensor, to extract temperature information. Minimally invasive temperature monitoring is critical in thermotherapy applications, but current methods often rely on additional sensors or simulations of typical tissue behavior. In this study, a commercially available needle was inserted into water-saturated foams with temperatures ranging from 25 to 55 °C, varied in 5° increments. Given that temperature affects the speed of sound, water’s heat capacity, and the mechanical properties of most tissues, it was hypothesized that the vibroacoustic signals recorded during needle insertion would carry temperature-dependent information. The acquired signals were segmented, processed, and analyzed using signal processing techniques and a deep learning algorithm. Results demonstrated that the audio signals contained distinct temperature-dependent features, enabling temperature prediction with a root mean squared error of approximately 3 °C. We present these initial laboratory findings, highlighting significant potential for refinement. This novel approach could pave the way for a real-time, minimally invasive method for thermal monitoring in medical applications.

Analysis of Doppler Audio Signals from the Carotid Artery

Analysis of Doppler Audio Signals from the Carotid Artery

Enhancing audio signal segmentation for musical instruments with refined MFCC-based features

Enhancing audio signal segmentation for musical instruments with refined MFCC-based features

Diode Amplitude Modulation and Demodulation

Diode amplitude modulation and demodulation have been investigated and the modulation index and modulation percentage were calculated. Of the several amplitude modulation methods, the diode modulation has been chosen to execute the amplitude modulation process because it was the first method used and it is the simplest method. A 50 Hz sine wave is used as a modulating signal to amplitude modulate a 1 kHz sine wave used as a carrier wave. During the modulation process, the amplitude of the modulating signal was kept constant at a value of 6 V while the amplitude of the carrier wave was given values between 6 V to 12 V increasing the value in steps of 1 V. The shapes of the modulation stages and of the demodulation processes were displayed on the oscilloscope screen using the simulation programmed named NI Multisim 14.2. The experiments were also carried out practically in the advanced electronics laboratory in the Physics department/Faculty of Science/University of Benghazi using real electronic equipment and parts. The recovered audio signal in the demodulation process was the same as that used as a modulating signal.

Spatial audio signal processing for augmented telepresence applications

Spatial audio signal processing for augmented telepresence applications

COVID-19 detection from optimized features of breathing audio signals using explainable ensemble machine learning

COVID-19 detection from optimized features of breathing audio signals using explainable ensemble machine learning

Trajectory of Fifths Based on Chroma Subbands Extraction - A New Approach to Music Representation, Analysis, and Classification.

In this article, we propose a new method of representing and analyzing music audio records. The method is based on the concept of the trajectory of fifths, which was initially developed for the analysis of music represented in MIDI format. To adapt this concept to the needs of audio signal processing, we implement a short-term spectral analysis of a musical piece, followed by a mapping of its subsequent spectral timeframes onto signatures of fifths reflecting relative intensities of sounds associated with each of the 12 pitch classes. Subsequently, the calculation of the characteristic points of the consecutive signatures of fifths enables the creation of the trajectory of fifths. The results of the experiments and statistical analysis conducted in a set of 8996 audio music pieces belonging to 10 genres indicate that this kind of trajectory, just as its MIDI-compliant precursor, is a source of valuable information (i.e., feature coefficients) concerning the harmonic structure of music, which may find use in audio music classification processes.

Separation of Simultaneous Speakers with Acoustic Vector Sensor.

This paper presents a method of sound source separation in live audio signals, based on sound intensity analysis. Sound pressure signals recorded with an acoustic vector sensor are analyzed, and the spectral distribution of sound intensity in two dimensions is calculated. Spectral components of the analyzed signal are selected based on the calculated source direction, which leads to a spatial filtration of the sound. The experiments were performed with test signals convolved with impulse responses of a real sensor, recorded for a varying sound source position. The experiments evaluated the proposed method's ability to separate sound sources, depending on their position, spectral content, and signal-to-noise ratio, especially when multiple sources are active at the same time. The obtained results are presented and discussed. The proposed algorithm provided signal-to-distortion ratio (SDR) values 10-12 dB, and Short-Time Objective Intelligibility Measure (STOI) values in the range 0.86-0.94, an increase by 0.15-0.30 compared with the unprocessed speech signal. The proposed method is intended for applications in automated speech recognition systems, speaker diarization, and separation in the concurrent speech scenarios, using a small acoustic sensor.

MELAUDIS: A Large-Scale Benchmark Acoustic Dataset For Intelligent Transportation Systems Research

Acoustic traffic sensors provide valuable information about road traffic at an affordable cost, gaining significant attention in recent years. However, the field of audio signal processing for Intelligent Transportation Systems (ITS) lacks real-world complex datasets. We introduce MELAUDIS, the first comprehensive real-world dataset designed for vehicle detection, traffic status monitoring, and vehicle type classification. In MELAUDIS audio recordings from multi-lane roads with two-way traffic are provided that encompass various traffic conditions, ambient noises, and weather settings, including urban environments and rainy weather. The dataset includes six vehicle types, bicycles, motorcycles, cars, buses, trucks, and trams, in both single-vehicle and multi-vehicle contexts. It is comprised of 5,792 background noise recordings, 7,345 vehicle sound samples, and 2,955 idling sound recordings, making it the largest urban acoustic dataset. Labeling and data cleansing required over 1,200 man-hours, improving classification accuracy from 65.1% to 82.84% using log-mel-spectrograms and CNNs. By offering a diverse range of labeled audio recordings, MELAUDIS serves as a benchmark to advance research in ITS.

Self-supervised learning for intelligent disease diagnosis using audio signals: beyond copd to a spectrum of diseases

Self-supervised learning for intelligent disease diagnosis using audio signals: beyond copd to a spectrum of diseases