Audio Sets Research Articles

АДАПТИВНИЙ ПІДХІД ДО ВИЗНАЧЕННЯ СТАНУ ВУЛИКА ЗА ДОПОМОГОЮ НЕЙРОННИХ МЕРЕЖ ТА АНАЛІЗУ АУДІО

Problem statement. Monitoring the queen bee is crucial for the health and produc-tivity of a bee colony. The queen plays a vital role in reproduction and maintaining the colony's population. Utilizing neural networks, such as CNNs along with sound analysis, can be a valuable tool for monitoring queen bees and assessing their behavior and health within the hive. Purpose. Finding the best way to preprocess audio data and review it is an important task that, if performed well, will help to track the bee hive population and its health state in general. Related work. With the development of deep learning, several studies were done on their application along with sound analysis in bee hive state identi-fication, such as swarming or bee queen detection. Materials and methods. A public la-beled dataset “To bee or not to bee” with bee hive sounds was used for training. It was split by source and then cut into pieces 4 seconds each. Then, CNN models were trained using different audio feature extraction methods, such as MFCCs and STFT. First, it was trained on a first audio set, and then trained models were utilized to analyze their per-formance on the evaluation set of the second audio set. Results and discussion. According to the training and evaluation results, MFCCs-based models have given constant good results, and when used on a limited audio dataset, pre-trained model showed better per-formance than the one trained from zero. Conclusions. The experiment showed that MFCCs is a better performant feature extraction method for a task of the bee hive sounds analysis and bee queen presence identification. Also, even though training a model on a full audio set results in better performance, pre-trained models can detect a bee queen absence in another hive even after an additional training on a limited audio dataset.

Performance Analysis of Deep Learning Model-Compression Techniques for Audio Classification on Edge Devices

Audio classification using deep learning models, which is essential for applications like voice assistants and music analysis, faces challenges when deployed on edge devices due to their limited computational resources and memory. Achieving a balance between performance, efficiency, and accuracy is a significant obstacle to optimizing these models for such constrained environments. In this investigation, we evaluate diverse deep learning architectures, including Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM), for audio classification tasks on the ESC 50, UrbanSound8k, and Audio Set datasets. Our empirical findings indicate that Mel spectrograms outperform raw audio data, attributing this enhancement to their synergistic alignment with advanced image classification algorithms and their congruence with human auditory perception. To address the constraints of model size, we apply model-compression techniques, notably magnitude pruning, Taylor pruning, and 8-bit quantization. The research demonstrates that a hybrid pruned model achieves a commendable accuracy rate of 89 percent, which, although marginally lower than the 92 percent accuracy of the uncompressed CNN, strikingly illustrates an equilibrium between efficiency and performance. Subsequently, we deploy the optimized model on the Raspberry Pi 4 and NVIDIA Jetson Nano platforms for audio classification tasks. These findings highlight the significant potential of model-compression strategies in enabling effective deep learning applications on resource-limited devices, with minimal compromise on accuracy.

Segment beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation

Augmented Reality (AR) devices, emerging as prominent mobile interaction platforms, face challenges in user safety, particularly concerning oncoming vehicles. While some solutions leverage onboard camera arrays, these cameras often have limited field-of-view (FoV) with front or downward perspectives. Addressing this, we propose a new out-of-view semantic segmentation task and Segment Beyond View (SBV), a novel audio-visual semantic segmentation method. SBV supplements the visual modality, which miss the information beyond FoV, with the auditory information using a teacher-student distillation model (Omni2Ego). The model consists of a vision teacher utilising panoramic information, an auditory teacher with 8-channel audio, and an audio-visual student that takes views with limited FoV and binaural audio as input and produce semantic segmentation for objects outside FoV. SBV outperforms existing models in comparative evaluations and shows a consistent performance across varying FoV ranges and in monaural audio settings.

TalkingStyle: Personalized Speech-Driven 3D Facial Animation with Style Preservation.

It is a challenging task to create realistic 3D avatars that accurately replicate individuals' speech and unique talking styles for speech-driven facial animation. Existing techniques have made remarkable progress but still struggle to achieve lifelike mimicry. This paper proposes "TalkingStyle", a novel method to generate personalized talking avatars while retaining the talking style of the person. Our approach uses a set of audio and animation samples from an individual to create new facial animations that closely resemble their specific talking style, synchronized with speech. We disentangle the style codes from the motion patterns, allowing our method to associate a distinct identifier with each person. To manage each aspect effectively, we employ three separate encoders for style, speech, and motion, ensuring the preservation of the original style while maintaining consistent motion in our stylized talking avatars. Additionally, we propose a new style-conditioned transformer decoder, offering greater flexibility and control over the facial avatar styles. We comprehensively evaluate TalkingStyle through qualitative and quantitative assessments, as well as user studies demonstrating its superior realism and lip synchronization accuracy compared to current state-of-the-art methods. To promote transparency and further advancements in the field, we also make the source code publicly available at https://github.com/wangxuanx/TalkingStyle.

Self-Disclosure and Management of Feminine Male Personas on The Tiktok Platform

Social media platforms are growing every time. The development of social media already has various kinds, such as Facebook, Twitter, Youtube, and others. One of the new social media that is currently very popular is Tiktok. As one of the popular social media, Tiktok is a platform for users to be able to carry out the self-disclosure process, especially male users who tend to be feminine. Self-Disclosure as a Disclosure Process: The process of self-disclosure on TikTok is an act in which male individuals express their feminine side through the videos and content they share. This reflects their openness to identity and self-expression which may differ from the gender stereotypes generally expected of men. There are 2 (two) processes that occur during actual self-disclosure on the Tiktok platform, namely on the front stage and back stage and the management of feminine male personas on TikTok involves an understanding of how these individuals build, maintain and manage their image and identity in those platforms. They use elements such as visual effects, background music, and editing to amplify the message they want to convey and create an impact on their audience. TikTok as a platform provides a "stage" for feminine male individuals to perform and interact with followers and other users. They can upload videos with the style and content they choose, take advantage of TikTok's creative features, and manipulate visual and audio settings to create engaging experiences. This study uses identity theory in the process of self-disclosure, and focuses on male research sources. feminist man. This research uses qualitative methods, constructivist paradigms, sociocultural traditions, and digital phenomena approaches

Acoustic analysis of video conferencing platforms: Evaluating communication effectiveness for individuals with hearing loss

AIM: This research investigates the impact of video conferencing platforms on the transmitted sound to evaluate their effectiveness in facilitating communication for individuals with hearing loss. The preliminary analysis includes comparing the transmitted sound deviations from source recordings in Teams, Zoom, and Google Meet and exploring differences in audio quality across different settings within each platform (not focused on DSP limitations or audio transmission fidelity). METHODS: In this study, we employed babble noise, BKB speech, and a combination of babble noise and BKB speech (SNR: 9dB) as stimuli. We compared different acoustic features of the received signal, including automatic audio gain adjustments, frequency range, and frequency response, with the source sound. RESULTS: Zoom applies a smooth high-frequency roll-off for speech in a noisy signal, beginning at approximately 14.5 kHz and removing all frequency content above 18 kHz. The roll-off for Teams begins at a relatively low frequency around 7 kHz. Although Google Meet starts the roll-off around 20 kHz, the gain was inconsistent and had a sharp roll-off. CONCLUSION: Each platform exhibited distinct characteristics and introduced noticeable modifications to the stimuli. Results indicated that Zoom with optional audio settings enabled demonstrates the highest fidelity to dynamic range and frequency spectrum.

Enhancing Public Health Communication Regarding Vaccine Trials: Design and Development of the Pan-European VACCELERATE Toolkit.

The pan-European VACCELERATE network aims to implement the first transnational harmonized and sustainable vaccine trial Volunteer Registry, being a single entry point for potential volunteers of large-scale vaccine trials across Europe. This work exhibits a set of harmonized vaccine trial-related educational and promotional tools for the general public, designed and disseminated by the pan-European VACCELERATE network. This study primarily aimed to design and develop a standard toolkit to increase positive attitudes and access to trustworthy information for better access and increased recruitment to vaccine trials for the public. More specifically, the produced tools are focused on inclusiveness and equity, and are targeting different population groups, including underserved ones, as potential volunteers for the VACCELERATE Volunteer Registry (older individuals, migrants, children, and adolescents). The promotional and educational material is aligned with the main objectives of the Volunteer Registry to increase public literacy and awareness regarding vaccine-related clinical research or trials and trial participation, including informed consent and legal issues, side effects, and frequently asked questions regarding vaccine trial design. Tools were developed per the aims and principles of the VACCELERATE project, focusing on trial inclusiveness and equity, and are adjusted to local country-wise requirements to improve public health communication. The produced tools are selected based on the cognitive theory, inclusiveness, and equity of differently aged and underrepresented groups, and standardized material from several official trustworthy sources (eg, COVID-19 Vaccines Global Access; the European Centre for Disease Prevention and Control; the European Patients' Academy on Therapeutic Innovation; Gavi, the Vaccine Alliance; and the World Health Organization). A team of multidisciplinary specialists (infectious diseases, vaccine research, medicine, and education) edited and reviewed the subtitles and scripts of the educational videos, extended brochures, interactive cards, and puzzles. Graphic designers selected the color palette, audio settings, and dubbing for the video story-tales and implemented QR codes. This study presents the first set of harmonized promotional and educational materials and tools (ie, educational cards, educational and promotional videos, extended brochures, flyers, posters, and puzzles) for vaccine clinical research (eg, COVID-19 vaccines). These tools inform the public about possible benefits and disadvantages of trial participation and build confidence among participants about the safety and efficacy of COVID-19 vaccines and the health care system. This material has been translated into several languages and is intended to be freely and easily accessible to facilitate dissemination among VACCELERATE network participant countries and the European and global scientific, industrial, and public community. The produced material could help fill knowledge gaps of health care personnel, providing the appropriate future patient education for vaccine trials, and tackling vaccine hesitancy and parents' concerns for potential participation of children in vaccine trials.

Synthetic speech attribution using self supervised audio spectrogram transformer

The ability to synthesize convincing human speech has become easier due to the availability of speech generation tools. This necessitates the development of forensics methods that can authenticate and attribute speech signals. In this paper, we examine a speech attribution task, which identifies the origin of a speech signal. Our proposed method known as Synthetic Speech Attribution Transformer (SSAT) converts speech signals into mel spectrograms and uses a self-supervised pretrained transformer for attribution. This transformer is pretrained on two large publicly available audio datasets: Audio Set and LibriSpeech. We finetune the pretrained transformer on three speech attribution datasets: the DARPA SemaFor Audio Attribution dataset, the ASVspoof2019 dataset, and the 2022 IEEE SP Cup dataset. SSAT achieves high closed-set accuracy on all datasets (99.8% on ASVspoof2019 dataset, 96.3% on SP Cup dataset, and 93.4% on DARPA SemaFor Audio Attribution dataset). We also investigate the method’s ability to generalize to unknown speech generation methods (open-set scenario). SSAT has high performance, achieving an open-set accuracy of 90.2% on the ASVspoof2019 dataset and 88.45% on DARPA SemaFor Audio Attribution dataset. Finally, we show that our approach is robust to typical compression rates used by YouTube for speech signals.

Predicting Presentation Skill of a Speaker Using Automatic Speaker and Audience Measurement

Effective presentation skills are an important ability for students and professionals to possess. Automatic analysis of presentation skills can help provide feedback to a speaker, and a complete analysis is possible only with both speaker and audience measurement. In this article, we propose a methodology to predict presentation skills on a small dataset using a model learned on a larger and reasonably closer dataset, which we call the source dataset. A classroom-type multimodal dataset with videos of both the speaker and the audience was collected. A rich set of audio, visual, and textual cues were extracted that relate to the presentation skill of a speaker. The presentation skills and audience engagement are inferred using models learned on the source dataset, which is close to our data. The analysis shows that adaptation of the source model to the target domain data is required for better performance. Also, automatic prediction of audience engagement is done to analyze the relationship between audience engagement and the presentation skill of the speaker. The analysis shows that a weak correlation exists between the automatic audience engagement prediction and the audience rating given for the presentation skill of the speaker.

Multi-Label Extreme Learning Machine (MLELMs) for Bangla Regional Speech Recognition

Extensive research has been conducted in the past to determine age, gender, and words spoken in Bangla speech, but no work has been conducted to identify the regional language spoken by the speaker in Bangla speech. Hence, in this study, we create a dataset containing 30 h of Bangla speech of seven regional Bangla dialects with the goal of detecting synthesized Bangla speech and categorizing it. To categorize the regional language spoken by the speaker in the Bangla speech and determine its authenticity, the proposed model was created; a Stacked Convolutional Autoencoder (SCAE) and a Sequence of Multi-Label Extreme Learning machines (MLELM). SCAE creates a detailed feature map by identifying the spatial and temporal salient qualities from MFEC input data. The feature map is then sent to MLELM networks to generate soft labels and then hard labels. As aging generates physiological changes in the brain that alter the processing of aural information, the model took age class into account while generating dialect class labels, increasing classification accuracy from 85% to 95% without and with age class consideration, respectively. The classification accuracy for synthesized Bangla speech labels is 95%. The proposed methodology works well with English speaking audio sets as well.

Classifying Autism From Crowdsourced Semistructured Speech Recordings: Machine Learning Model Comparison Study.

BackgroundAutism spectrum disorder (ASD) is a neurodevelopmental disorder that results in altered behavior, social development, and communication patterns. In recent years, autism prevalence has tripled, with 1 in 44 children now affected. Given that traditional diagnosis is a lengthy, labor-intensive process that requires the work of trained physicians, significant attention has been given to developing systems that automatically detect autism. We work toward this goal by analyzing audio data, as prosody abnormalities are a signal of autism, with affected children displaying speech idiosyncrasies such as echolalia, monotonous intonation, atypical pitch, and irregular linguistic stress patterns.ObjectiveWe aimed to test the ability for machine learning approaches to aid in detection of autism in self-recorded speech audio captured from children with ASD and neurotypical (NT) children in their home environments.MethodsWe considered three methods to detect autism in child speech: (1) random forests trained on extracted audio features (including Mel-frequency cepstral coefficients); (2) convolutional neural networks trained on spectrograms; and (3) fine-tuned wav2vec 2.0—a state-of-the-art transformer-based speech recognition model. We trained our classifiers on our novel data set of cellphone-recorded child speech audio curated from the Guess What? mobile game, an app designed to crowdsource videos of children with ASD and NT children in a natural home environment.ResultsThe random forest classifier achieved 70% accuracy, the fine-tuned wav2vec 2.0 model achieved 77% accuracy, and the convolutional neural network achieved 79% accuracy when classifying children’s audio as either ASD or NT. We used 5-fold cross-validation to evaluate model performance.ConclusionsOur models were able to predict autism status when trained on a varied selection of home audio clips with inconsistent recording qualities, which may be more representative of real-world conditions. The results demonstrate that machine learning methods offer promise in detecting autism automatically from speech without specialized equipment.

Method of selection and assessment of hearing aids efficiency based on comparative speech audiometry

Objectives to improve the quality of hearing aids (HA) selection for patients with sensorineural hearing loss using a comparative free sound field speech audiometry according to our modified method "Delta Test".
 Material and methods. The study involved 56 patients aged from 18 to 62 years with bilateral chronic sensorineural hearing loss of 2-4 degrees. The study group included 32 patients, a test for speech recognition in a free sound field (speech audiometry) was conducted using the method proposed by us. The control group consisted of 24 patients who were aided without the use of comparative speech audiometry.
 The "Delta Test" included the use of audio files sets containing a speech material in pure form and mixed with speech noise with different signal-to-noise ratios, supplied through a speaker system connected to a personal computer. The percentage of correctly repeated words was measured without HA and with several HA having different settings. The effectiveness of using HA was defined as the difference in the percentage of speech recognition when using HA in relation to the "ear without HA".
 Results. According to "The International Outcome Inventory for Hearing Aids" the average score was 4.13 0.10 in the group where the HA were selected using the "Delta Test", which is statistically significantly higher than in the control group, where the average score was 3.720.15(p 0.05). "Delta Test" allows optimally select the HA parameters. This method is easy to perform and does not require expensive equipment.

Predicting human perception of the urban environment in a spatiotemporal urban setting using locally acquired street view images and audio clips

This study investigates people's perception of visual and auditory landscapes in a mixed-use urban environment. A set of audio and visual data is collected at different intervals during the day in local streets with the help of an audio recorder and camera setup. The High and Low-level features from the collected audio and visual datasets are captured with the help of custom Deep Learning (DL) models and other standard algorithms. The collected data is used in the perception survey, which included human subjects (n = 73). The evaluation of the individual perception is done with the help of eight and six auditory and visual perceptual attributes, respectively. The results from the survey are then studied in relation to the features extracted from algorithms. Finally, a street of 10 km length is chosen within the study area where a spatiotemporal street-level visual and auditory data is collected. Statistical analysis and Machine Learning modeling are performed in the surveyed dataset to predict the human perception of audio and visual scenes in the chosen street. The results helped in understanding specific audio and visual features that are related to individual perceptions. Further, these relationships are utilized to create prediction models, which helped in creating spatiotemporal visual and auditory perception maps.

“We are a fly on the wall listening”: Constructions of Musical Intimacy in The Beatles' Anthology

Abstract Between 1995 and 1996, The Beatles released the three volume, six-disc Anthology, a sprawling audio set issued to coincide with a televised documentary of the same name. One of the first major releases of its kind, the Anthology albums curated and compiled previously unreleased demos, studio recordings, and live performances, under the auspices of giving listeners insight into the band's creative process. Beyond the appeal of new Beatles product however, the Anthology albums are unique in exploiting particular manifestations of musical intimacy that ultimately serve to further cultivate the group's well-established pop cultural mythos.

Active Object Discovery and Localization Using Sound-Induced Attention

Industrial intelligent devices are usually equipped with both microphones and cameras to perceive and understand the physical world. Though visual object detection technology has achieved a great success, its combination with other sensing modalities remains unsolved. In this article, we establish a novel sound-induced attention framework for the visual object detection, and develop a two-stream weakly supervised deep learning architecture to combine the visual and audio modalities for localizing the sounding object. A dataset is constructed from the Audio Set to validate the proposed method and some realistic experiments are conducted to demonstrate the effectiveness of the proposed system.

Audiovisual accessibility: Evaluating workflows for transcribing and captioning digital archival content

Accessibility of content should be a major focus for all digital library projects. This article describes how the University of Utah’s J. Willard Marriott Library embarked on a grant-funded project to examine different workflows for creating captions and transcripts for archival audiovisual content. Although many institutions have experimented with various methods for creating captioning and transcripts, there is no perfect standard for effectively and inexpensively generating these for archival collections. While working with a diverse set of archival audio and audiovisual files, the J. Willard Marriott Library compared different captioning workflows based on the amount of time to transcribe, the cost of each workflow, and the quality of the final results. This article presents the findings from this study, with information about the different processes that were used to generate the captions for archival content.

Can Children of Different Ages Recognize Dog Communication Signals in Different Situations?

The presented study examines the ability of 265 children aged 4–12 years to correctly assign contextual cues and inner state values to a set of audio and audio-visual recordings of dog vocalizations and behaviors in different situations. Participants were asked to mark which situation each recording captured, what inner state of the dog it showed, and what inner state a human would feel in the same situation. Recognition of the inner state of dogs was affected by the age of the child when evaluating the audio recordings (p < 0.001), and such a tendency was revealed in evaluating the audiovisual materials (p = 0.08). The inner state of dog evaluation was associated with both the situation assessment (p < 0.01) and human inner state (p < 0.001) in the case of audio recordings, but it was only correlated with situation assessment in audio-visual recordings (p < 0.01). The contextual situations were recognized by the participants only in the audio materials, with “stranger” being the best recognized situation, while “play” was the least recognized. Overall, children aged 4–5 years showed a limited ability to understand dog signals compared to children aged 6–12 years, who were successful in recognizing the dogs’ stimuli more than 80% of the time. Therefore, children younger than 6 years of age require increased supervision when interacting with dogs.

СКРАЙБІНГ-ПРЕЗЕНТАЦІЯ ЯК ЗАСІБ ПІДВИЩЕННЯ ЕФЕКТИВНОСТІ ОСВІТНЬОГО ПРОЦЕСУ В ЗАКЛАДАХ ЗАГАЛЬНОЇ СЕРЕДНЬОЇ ОСВІТИ

У статті здійснено аналіз основних підходів до визначення сутності поняття «скрайбінг-презентація» та її значення в освітньому процесі. Виокремлено основні переваги скрайбінг-презентацій: оригінальність, можливість багаторазового використання, можливість спілкування з учнями протягом перегляду, ефект паралельного проходження, мінімальна кількість витрат, універсальність візуалізації, ефективність, стислість і образність та ін. Акцентовано увагу на недоліках використання скрайбінг-презентацій в освітньому процесі. З’ясовано, що використання скрайбінг-презентацій дає змогу мотивувати та стимулювати навчально-пізнавальну діяльність учнів, активізувати увагу на базових поняттях, візуалізувати навчальний матеріал у зручну для сприймання та засвоєння форму, що сприяє підвищенню ефективності освітнього процесу загалом. Наголошено, що скрайбінг дозволяє перетворювати тези лекції в слова й образи, описувати зв’язки і робити змістові акценти на базових поняттях навчального матеріалу. Визначено, що головне завдання означеного способу подачі інформації – динамічно, швидко і якісно донести її до учнівської аудиторії Узагальнено основні види скрайбінгу в залежності від способу розроблення: ручний та комп’ютерний. Зокрема встановлено, що ручний скрайбінг поділяється на такі підвиди: скрайбінг-фасилітацію, аплікаційний, магнітний, фланелеграфний скрайбінг. До видів комп’ютерного скрайбінгу зараховано відео-скрайбінг, мальований, онлайн-скрайбінг. Відзначено, що визначення виду скрайбінгу залежить від дидактичної мети, завдань уроку, технічного забезпечення та рівня підготовленості вчителя до розроблення скрайбінг-презентацій. Встановлено, що, в переважній більшості розробок скрайбінгу, вчителі не дотримуються вимог, що значно знижує ефективність використання скрайбінг-презентацій. Обґрунтовано необхідність визначення чітких вимог до скрайбінгу, які сприятимуть підвищенню ефективності освітнього процесу в закладах загальної середньої освіти. Виокремлено й схарактеризовано основні групи вимог до скрайбінг-презентації: до змісту; до об’єктів (звукового ряду, зображення, тексту); до дизайну й до її використання.

Special issue on “Multi‐modal information learning and analytics of cross‐media big data”

Special issue on “Multi‐modal information learning and analytics of cross‐media big data”

Lung Sound Recognition Algorithm Based on VGGish-BiGRU

Pulmonary breathing sound plays a key role in the prevention and diagnosis of the lung diseases. Its correlation with pathology and physiology has become an important research topic in the pulmonary acoustics and the clinical medicine. However, it is difficult to fully describe lung sound information with the traditional features because lung sounds are complex and nonstationary signals. And the traditional convolutional neural network cannot also extract the temporal features of the lung sounds. To solve the problem, a lung sound recognition algorithm based on VGGish-BiGRU is proposed on the basis of transfer learning, which combines VGGish network with the bidirectional gated recurrent unit neural network (BiGRU). In the proposed algorithm, VGGish network is pretrained using audio set, and the parameters are transferred to VGGish network layer of the target network. The temporal features of the lung sounds are extracted through retraining BiGRU network with the lung sound data. During retraining BiGRU network, the parameters in VGGish layers are frozen, and the parameters of BiGRU network are fine-tuned. The experimental results show that the proposed algorithm effectively improves the recognition accuracy of the lung sounds in contrast with the state-of-the-art algorithms, especially the recognition accuracy of asthma.