Audio Analysis Research Articles

Predicting Engagement of Older People’s Virtual Teams from Video Call Analysis

This study examines seniors’ creative engagement in group activities using synchronous communication tools and explores automatic assessment methods through behavioral and psychophysiological measurements. Working with a small senior group on collaborative creative tasks, we implemented a comprehensive data collection approach using audio-visual and physiological measurements. Machine learning models were used to evaluate group creative engagement levels using various data subsets. Results show that engagement assessment can be effective with different feature combinations, allowing flexibility across contexts and constraints. The multimodal approach, combining facial, audio, and body analysis, achieved optimal performance and is recommended when conditions permit. Our research provides insights into seniors’ online creative participation and presents an automated system for detecting creative engagement in virtual teams, supporting active participation strategies.

Detection of tuberculosis using cough audio analysis: a deep learning approach with capsule networks

PurposeTuberculosis (TB) is a widespread infectious disease that requires early detection for effective treatment and control. This study aims to improve TB detection using cough audio analysis, comparing the performance of capsule networks to other deep learning models.MethodsWe used cough audio recordings from 1105 individuals with a new or worsening cough for at least two weeks, totaling 9772 recordings. These recordings were processed into spectral images, and HOG features were extracted. Various models, including Capsule Networks + FCNN, CNN, VGG16, and ResNet50 were trained and evaluated.ResultsCapsule Networks + FCNN achieved the best performance with an accuracy of 0.97, sensitivity of 0.98, specificity of 0.96, F1 score of 0.97, and precision of 0.97, outperforming other models. This attribute is due to the model’s ability to learn complex features from spectral images.ConclusionsThis study concludes that Capsule Networks are more effective than typical CNN-based models in diagnosing TB from cough audio. This suggests that advanced deep learning frameworks could significantly enhance TB screening accuracy, especially in resource-limited areas.

Employing deep learning model to evaluate speech information in acoustic simulations of Cochlear implants

Acoustic vocoders play a key role in simulating the speech information available to cochlear implant (CI) users. Traditionally, the intelligibility of vocoder CI simulations is assessed through speech recognition experiments with normally-hearing subjects, a process that can be time-consuming, costly, and subject to individual variability. As an alternative approach, we utilized an advanced deep learning speech recognition model to investigate the intelligibility of CI simulations. We evaluated model’s performance on vocoder-processed words and sentences with varying vocoder parameters. The number of vocoder bands, frequency range, and envelope dynamic range were adjusted to simulate sound processing settings in CI devices. Additionally, we manipulated the low-cutoff frequency and intensity quantization of vocoder envelopes to simulate psychophysical temporal and intensity resolutions in CI patients. The results were evaluated within the context of the audio analysis performed in the model. Interestingly, the deep learning model, despite not being originally designed to mimic human speech processing, exhibited a human-like response to alterations in vocoder parameters, resembling existing human subject results. This approach offers significant time and cost savings compared to testing human subjects, and eliminates learning and fatigue effects during testing. Our findings demonstrate the potential of speech recognition models in facilitating auditory research.

MelCochleaGram-DeepCNN: Sequentially Fused Spectrogram and the DeepCNN Classifiers-based Audio Spoof Detection System

Automatic Speaker Verification (ASV) systems are crucial in various fields, enabling speaker identification for authentication, fraud detection, and forensic applications. While the simplicity and effectiveness of speech biometrics are driving the demand for ASV systems, their increasing popularity raises concerns about vulnerability to speech attacks. To enhance the security of these systems, the work in this paper proposes a spectrogram-based solution that leverages the robustness of spectrograms in audio analysis and feature extraction. The proposed model consists of two main components: frontend and backend. In the frontend, it introduces a novel spectrogram MelCochleaGram (MCG) by fusing Mel Spectrogram and Cochleagram, sequentially. For the backend implementation, pre-trained deep learning models, including ResNet50, ResNet50V2, and InceptionV3, are employed using the Keras framework. These models are individually paired with MCG to detect deepfake and replay attacks. To validate the effectiveness of the proposed system, thorough experimentation is conducted on two datasets: the DEepfake CROss-lingual (DECRO) evaluation dataset and the Voice Spoofing Detection Corpus (VSDC). The proposed combination of MCG with ResNet50 has achieved an Equal Error Rate (EER) of 0.2%, and 1.2% for deepfake detection over DECRO English and Chinese datasets, respectively. Also, for replay attack detection, the proposed combination has produced an EER of 1.4%.

A Diffusion Modeling-Based System for Teaching Dance to Digital Human

The introduction of artificial intelligence (AI) has triggered changes in modern dance education. This study investigates the application of diffusion-based modeling and virtual digital humans in dance instruction. Utilizing AI and digital technologies, the proposed system innovatively merges music-driven dance generation with virtual human-based teaching. It achieves this by extracting rhythmic and emotional information from music through audio analysis to generate corresponding dance sequences. The virtual human, functioning as a digital tutor, demonstrates dance movements in real time, enabling students to accurately learn and execute dance postures and rhythms. Analysis of the teaching outcomes, including effectiveness, naturalness, and fluidity, indicates that learning through the digital human results in enhanced user engagement and improved learning outcomes. Additionally, the diversity of dance movements is increased. This system enhances students’ motivation and learning efficacy, offering a novel approach to innovating dance education.

Unattended noise measurements: use of new technologies to automatically qualify noise events for greater efficiency, precision and time savings

For more than 40 years, the noise assessment in the environment has consisted of measuring and storing the time history typically every second, then marking the events using post-processing software or automatically or manually by coding. Technological advances now make it possible to enrich the simple measurement of the time history thanks to additional data allowing a better evaluation of the measured phenomena: audio recording and AI analysis of the type of noise, analysis of the predominant direction during a noise event. We propose to present an innovative solution based on a 3D sound intensity probe allowing the determination of the azimuth and the elevation every time a noise event occurs. The results are expressed by frequency bands and A weighted filter. The influence of the type of field (active and reactive intensity) will be developed. Particular attention will be paid to the accuracy and the validity of the measurement as well as the limits of the principle.

Gender and Family Stereotypes in a Photograph: Research Using the Eye-Tracking Method

Abstract The article presents the results of a 2022 study involving students training to be civic studies teachers. Three research goals were set: to determine which gender and family stereotypes burden the interpretation of a photograph; to identify qualitative differences in the perception, analysis, and interpretation of the photograph depending on the form of the questions asked; and to gain respondents’ feedback on the test. The test used a computer and was based on a family photograph and a series of questions. Three methods of data collection and analysis were used (audio recording analysis, eye-tracking, interview). The study found that future teachers are substantially influenced by gender stereotypes in Czech society. It also confirmed the importance of didactic structuring when analysing iconographic material. The findings show the importance of gender-sensitive education and the development of didactic skills for interpreting visual texts as part of undergraduate training for future civic studies teachers.

Computational elucidation of nonverbal behavior and body language in music therapy

Abstract Music therapy has shown efficacy in serious and chronic conditions, mental disorders, and disabilities. However, there is still much to explore regarding the mechanisms through which music interventions exert their effects. A typical session involves interactions between the therapist, the client, and the musical work itself, and to help address the challenges of capturing and comprehending its dynamics, we extend our general computational paradigm (CP) for analyzing the expressive and social behavioral processes in arts therapies. The extension includes bodily and nonverbal aspects of the behavior, offering additional insights into the client's emotional states and engagement. We have used this version of the CP, which employs AI pose estimation technology, image processing, and audio analysis, to capture therapy-related psychometrics and their intra- and inter-session analysis. The CP is applied in a real-world proof-of-concept study, and the results enable us to pinpoint meaningful events and emergent properties not captured by the human eye, complementing the therapist's interpretations. The resulting data may also be useful in other scientific and clinical areas.

Smartphone automated motor and speech analysis for early detection of Alzheimer's disease and Parkinson's disease: Validation of TapTalk across 20 different devices.

Smartphones are proving useful in assessing movement and speech function in Alzheimer's disease and other neurodegenerative conditions. Valid outcomes across different smartphones are needed before population-level tests are deployed. This study introduces the TapTalk protocol, a novel app designed to capture hand and speech function and validate it in smartphones against gold-standard measures. Twenty different smartphones collected video data from motor tests and audio data from speech tests. Features were extracted using Google Mediapipe (movement) and Python audio analysis packages (speech). Electromagnetic sensors (60Hz) and a microphone acquired simultaneous movement and voice data, respectively. TapTalk video and audio outcomes were comparable to gold-standard data: 90.3% of video, and 98.3% of audio, data recorded tapping/speech frequencies within±1Hz of the gold-standard measures. Validation of TapTalk across a range of devices is an important step in the development of smartphone-based telemedicine and was achieved in this study. TapTalk evaluates hand motor and speechfunctions across a wide range of smartphones.Data showed 90.3% motor and 98.3% speech accuracy within +/-1Hz of gold standards.Validation advances smartphone-based telemedicine for neurodegenerative diseases.

Mitigating Complex Cyber Threats: An Integrated Multimodal Deep Learning Framework for Enhanced Security

Abstract: The rapidly evolving landscape of cyber threats poses significant challenges to traditional cybersecurity methods, particularly in detecting and mitigating complex attacks that combine multiple data modalities. This article introduces a novel approach utilizing Multimodal Deep Learning (MDL) to enhance cybersecurity detection and prevention capabilities. By fusing Natural Language Processing (NLP) and computer vision techniques, our proposed MDL framework demonstrates superior performance in analyzing both textual and visual data associated with potential threats. Through a series of experiments and case studies, we show that this integrated approach significantly improves detection accuracy, reduces false positives, and exhibits remarkable adaptability to emerging attack vectors. Our results indicate a 37% increase in threat detection efficiency and a 42% reduction in false positives compared to traditional unimodal methods. Furthermore, we explore the potential applications of this technology in email security, social media monitoring, and advanced malware detection. This article not only contributes to the growing body of knowledge in AI-driven cybersecurity but also provides a roadmap for future developments, including the integration of audio analysis and the application of Explainable AI (XAI) to enhance trust and transparency in MDL-based security systems.

Extracting Features from Text Flows based on Semantic Similarity for Text Classification: an Approach Inspired by Audio Analysis

Text classification is a mainly investigated challenge in Natural Language Processing (NLP) research. The higher performance of a classification model depends on a representation that can extract valuable information about the texts. Aiming not to lose crucial local text information, a way to represent texts is through flows, sequences of information collected from texts. This paper proposes an approach that combines various techniques to represent texts: the representation by flows, the benefit of the word embeddings text representation associated with lexicon information via semantic similarity distances, and the extraction of features inspired by well-established audio analysis features.In order to perform text classification, this approach splits the text into sentences and calculates a semantic similarity metric to a lexicon on an embedding vector space. The sequence of semantic similarity metrics composes the text flow. Then, the method performs the extraction of twenty-five features inspired by audio analysis (named Audio-Like Features). The features adaptation from audio analysis comes from a similitude between a text flow and a digital signal, in addition to the existing relationship between text, speech, and audio. We evaluated the method in three NLP classification tasks: Fake News Detection in English, Fake News Detection in Portuguese, and Newspaper Columns versus News Classification. The approach efficacy is compared to baselines that embed semantics in text representation: the Paragraph Vector and the BERT. The objective of the experiments was to investigate if the proposed approach could compete with the baselines methods improve their efficacy when associated with them. The experimental evaluation demonstrates that the association between the proposed and the baseline methods can enhance the baseline classification efficacy in all three scenarios. In the Fake News Detection in Portuguese task, our approach surpassed the baselines and obtained the best effectiveness (PR-AUC = 0.98).

Semiautomated generation of species-specific training data from large, unlabeled acoustic datasets for deep supervised birdsong isolation.

Bioacoustic monitoring is an effective and minimally invasive method to study wildlife ecology. However, even the state-of-the-art techniques for analyzing birdsongs decrease in accuracy in the presence of extraneous signals such as anthropogenic noise and vocalizations of non-target species. Deep supervised source separation (DSSS) algorithms have been shown to effectively separate mixtures of animal vocalizations. However, in practice, recording sites also have site-specific variations and unique background audio that need to be removed, warranting the need for site-specific data. Here, we test the potential of training DSSS models on site-specific bird vocalizations and background audio. We used a semiautomated workflow using deep supervised classification and statistical cleaning to label and generate a site-specific source separation dataset by mixing birdsongs and background audio segments. Then, we trained a deep supervised source separation (DSSS) model with this generated dataset. Because most data is passively-recorded and consequently noisy, the true isolated birdsongs are unavailable which makes evaluation challenging. Therefore, in addition to using traditional source separation (SS) metrics, we also show the effectiveness of our site-specific approach using metrics commonly used in ornithological analyses such as automated feature labeling and species-specific trilateration accuracy. Our approach of training on site-specific data boosts the source-to-distortion, source-to-interference, and source-to-artifact ratios (SDR, SIR, and SAR) by 9.33 dB, 24.07 dB, and 3.60 dB respectively. We also find our approach allows for automated feature labeling with single-digit mean absolute percent error and birdsong trilateration accuracy with a mean simulated trilateration error of 2.58m. Overall, we show that site-specific DSSS is a promising upstream solution for wildlife audio analysis tools that break down in the presence of background noise. By training on site-specific data, our method is robust to unique, site-specific interference that caused previous methods to fail.

Optimizing poultry audio signal classification with deep learning and burn layer fusion

This study introduces a novel deep learning-based approach for classifying poultry audio signals, incorporating a custom Burn Layer to enhance model robustness. The methodology integrates digital audio signal processing, convolutional neural networks (CNNs), and the innovative Burn Layer, which injects controlled random noise during training to reinforce the model's resilience to input signal variations. The proposed architecture is streamlined, with convolutional blocks, densely connected layers, dropout, and an additional Burn Layer to fortify robustness. The model demonstrates efficiency by reducing trainable parameters to 191,235, compared to traditional architectures with over 1.7 million parameters. The proposed model utilizes a Burn Layer with burn intensity as a parameter and an Adamax optimizer to optimize and address the overfitting problem. Thorough evaluation using six standard classification metrics showcases the model's superior performance, achieving exceptional sensitivity (96.77%), specificity (100.00%), precision (100.00%), negative predictive value (NPV) (95.00%), accuracy (98.55%), F1 score (98.36%), and Matthew’s correlation coefficient (MCC) (95.88%). This research contributes valuable insights into the fields of audio signal processing, animal health monitoring, and robust deep-learning classification systems. The proposed model presents a systematic approach for developing and evaluating a deep learning-based poultry audio classification system. It processes raw audio data and labels to generate digital representations, utilizes a Burn Layer for training variability, and constructs a CNN model with convolutional blocks, pooling, and dense layers. The model is optimized using the Adamax algorithm and trained with data augmentation and early-stopping techniques. Rigorous assessment on a test dataset using standard metrics demonstrates the model's robustness and efficiency, with the potential to significantly advance animal health monitoring and disease detection through audio signal analysis.

Detection and Segmentation of Mouth Region in Stereo Stream Using YOLOv6 and DeepLab v3+ Models for Computer-Aided Speech Diagnosis in Children

This paper describes a multistage framework for face image analysis in computer-aided speech diagnosis and therapy. Multimodal data processing frameworks have become a significant factor in supporting speech disorders’ treatment. Synchronous and asynchronous remote speech therapy approaches can use audio and video analysis of articulation to deliver robust indicators of disordered speech. Accurate segmentation of articulators in video frames is a vital step in this agenda. We use a dedicated data acquisition system to capture the stereovision stream during speech therapy examination in children. Our goal is to detect and accurately segment four objects in the mouth area (lips, teeth, tongue, and whole mouth) during relaxed speech and speech therapy exercises. Our database contains 17,913 frames from 76 preschool children. We apply a sequence of procedures employing artificial intelligence. For detection, we train the YOLOv6 (you only look once) model to catch each of the three objects under consideration. Then, we prepare the DeepLab v3+ segmentation model in a semi-supervised training mode. As preparation of reliable expert annotations is exhausting in video labeling, we first train the network using weak labels produced by initial segmentation based on the distance-regularized level set evolution over fuzzified images. Next, we fine-tune the model using a portion of manual ground-truth delineations. Each stage is thoroughly assessed using the independent test subset. The lips are detected almost perfectly (average precision and F1 score of 0.999), whereas the segmentation Dice index exceeds 0.83 in each articulator, with a top result of 0.95 in the whole mouth.

Music recommendation systems in music information retrieval: Leveraging machine learning and data mining techniques

Music Information Retrieval (MIR) has become a pivotal area of research with the rise of digital music platforms, enabling personalized music recommendations to enhance user experience. This paper explores the integration of machine learning and data mining techniques in music recommendation systems. We discuss user-based and item-based collaborative filtering, matrix factorization methods like Singular Value Decomposition (SVD) and Alternating Least Squares (ALS), and content-based filtering that incorporates audio feature analysis, metadata, and lyrics analysis. Additionally, we delve into hybrid recommendation systems, combining collaborative and content-based approaches using advanced models such as neural networks and hybrid autoencoders. Our finding show that, hybrid systems provide the most accurate and personalize recommendations, albeit requiring significant computational resources. Practical applications from platforms likes Spotify and Pandora illustrate the effectiveness of these approaches in real-world settings.

Quality of Life of Patients Using Esophageal Speech after Total Laryngectomy: A Systematic Review Study.

(1) Background: The present systematic study aimed to assess whether using esophageal speech (ES) as a method of vocal rehabilitation in patients after total laryngectomy enhances their quality of life (QoL) and vocal functionality based on patients' reports. (2) Methods: Data collection was conducted from PubMed, Google Scholar, and Speech Bite, and the PRISMA Flow Diagram tool was used to record different stages of the literature search process. In the review, nine studies were included, while a bias check was carried out using the Critical Appraisal Skills Programme (CASP) checklists. Survey analysis incorporated quantitative and qualitative data, including standardized questionnaires and audio analyses. (3) Results: A technique's effectiveness depends on the method's functionality and the patient's abilities. Furthermore, the findings revealed that ES use unexpectedly affects quality of life regarding patients. While statistical analysis of the studies showed that some patients reported improvement in quality of life and vocal functionality, others faced challenges such as difficulty in learning the technique, long-term intervention, and unsatisfactory phonetic performance. Some studies observed quantitative measures, such as improved Voice Impairment Index (VHI) scores and Voice-Related Quality of Life (V-RQOL) scores. However, results were not uniformly positive across studies, with a subset of patients reporting minimal improvement. (4) Conclusions: The limited literature on the effect of ES on patients' QoL appears to influence the results in different ways. However, research data support that patients' communication and psychological state seem to improve significantly compared to patients who have not been rehabilitated. The final assessment of the technique's effectiveness on quality of life must depend on many factors.

Landscape in the Twentieth Century Composing in the Aspect of Performing Interpretation

The aim of the study is to cover the topic of landscape in the composer’s practice of the 20th century from the perspective of performing interpretation, as a complex artistic phenomenon consisting of a number of elements. The following research methods were chosen: monitoring the processes of performance of works; systematization of video sources; audio analysis; comparison and synthesis. The conclusions of the study were theses that emphasize the importance of: a deep sense of style, which includes works dedicated to the landscape; perfect performance technique; a creative approach to the pace and nature of the interpretation; the ability of musicians to create a special atmosphere of natural pictures in the sound fabric. The academic novelty of the article is researching the landscape in academic music of the 20th century from the perspective of performance interpretation as a single complex multi-component system. Its practical significance is emphasized by the understanding of the key components of the process of performing such works. Keywords: miniature, components of interpretation, timbre, choral style, dynamic shades

Exploring the potential of combining Mel spectrograms with neural networks optimized by the modified crayfish optimization algorithm for acoustic speed violation identification

Enforcing vehicle speed limits is paramount for road safety. This paper pioneers an innovative approach by synergizing signal processing and Convolutional Neural Networks (CNNs) to detect speeding violations, addressing a critical aspect of traffic management. While traditional methods have shown efficacy, the potential synergy of signal processing and AI techniques remains largely unexplored. We bridge this gap by harnessing Mel spectrograms extracted from vehicle recordings, representing intricate audio features. These spectrograms serve as inputs to a tailored CNN architecture, meticulously designed for pattern recognition in speeding-related audio cues. An altered variant of the crayfish optimization algorithm (COA) was employed to tune the CNN model. Our methodology aims to discriminate between normal driving sounds and instances of speed limit breaches. Notably absent from previous literature, our fusion method yields promising initial results, demonstrating its potential to accurately identify speeding violations. This contribution not only enhances traffic safety and management but also provides a pioneering framework for integrating signal processing and AI techniques in innovative ways, with implications extending to broader audio analysis domains.

Analysis Of Personal Relationships From A Sentimental Standpoint And Support For Mental Health

Mental complaint, which is primarily brought on by depressions, and commending causes of early death moment. Suicidal studies brought on by depression seriously hamper daily performing. Sentiment analysis, which aims to discover the character of textbook and categorizes into positive, negative, and neutral, is a trendy issue that has been on exploration for decades. In moment’s digital terrain, sentiment analysis has access to a lot of data. So, our thing is to concentrate on developing a system for textbook, videotape, and audio analysis- grounded depression identification. This system will be created using sentiment analysis and natural language processing ways. Depending on the feelings deduced from stoner input, the application will classify textbook, audio, and visual signals as positive or negative.

Bimodal deep learning model for subjectively enhanced emotion classification in films

This research delves into the concept of color grading in film, focusing on how color influences the emotional response of the audience. The study commenced by recalling state-of-the-art works that process audio–video signals and associated emotions by machine learning. Then, assumptions of subjective tests for refining and validating an emotion model for assigning specific emotional labels to selected film excerpts were presented. The insights gained from these subjective evaluations facilitated the creation of a comprehensive database of movie excerpts. This database was subsequently employed to both train and evaluate the efficacy of deep learning models. The latter half of the study shifts focus to the intelligent analysis of audio and video signals that form film excerpts. This involved exploring diverse methodologies for parameterizing these signals. Models that demonstrated the highest accuracy on the test dataset on audio/video only were amalgamated to forge a bimodal model, which integrates both audio and video signals for emotion classification. The bimodal model exhibited superior accuracy in tests compared to a model that solely relied on video signal classification. This enhancement in performance was achieved with only a marginal increase in the complexity and the number of parameters within the model.