Audio Generation Research Articles

Enhancing Text Summarization and Audio Generation Using Hybrid Model

Abstract In today's fast-paced world, efficiently processing the vast amount of information in lengthy documents poses significant challenges. Traditional methods of reading and understanding multi-page documents, often spanning six to seven pages, are time-consuming and cumbersome for students, scholars, and professionals who face constant time constraints. This project introduces a novel approach by integrating advanced text summarization with real-time audio generation. Unlike existing solutions, which either focus on summarization or audio conversion separately, this dual-system solution uniquely combines both, allowing users to quickly grasp key content and consume it in audio form while on the move. The summarization component effectively condenses large documents into concise summaries, preserving the critical information and context. Meanwhile, the audio generation component transforms these summaries into speech, enhancing accessibility for auditory learners and users on the go. This innovative combination not only saves time but also offers flexibility, catering to diverse user preferences and learning styles.

REAL-TIME ACCENT TRANSLATOR

This paper introduces the "Real-Time Accent Translator," a lightweight and accessible web-based application that bridges the gap between multilingual communication and accent adaptation. Built on Flask, the system integrates Google Translate and Google Text-to-Speech (gTTS) APIs to provide seamless translation and speech synthesis. Users can input text in a source language, specify the target language, and optionally adjust accents for languages such as English. The application translates the text, synthesizes speech with the desired accent, and provides an audio output in real time. The architecture is designed for simplicity, leveraging third-party APIs to ensure rapid deployment and scalability without the need for extensive computational resources. This paper discusses the technical implementation, including API integration, RESTful communication, and real-time audio generation. Additionally, the potential use cases of the system are highlighted, including cross-cultural communication, language learning, and accessibility for non-native speakers.

Harmonizing Emotion and Sound: A Novel Framework for Procedural Sound Generation Based on Emotional Dynamics

The present work proposes a novel framework for emotion-driven procedural sound generation, termed SONEEG. The framework merges emotional recognition with dynamic sound synthesis to enhance user schooling in interactive digital environments. The framework uses physiological and emotional data to generate emotion-adaptive sound, leveraging datasets like DREAMER and EMOPIA. The primary innovation of this framework is the ability to capture emotions dynamically since we can map them onto a circumplex model of valence and arousal for precise classification. The framework adopts a Transformer-based architecture to synthesize associated sound sequences conditioned on the emotional information. In addition, the framework incorporates a procedural audio generation module employing machine learning approaches: granular and wavetable synthesis and physical modeling to generate adaptive and personalized soundscapes. A user study with 64 subjects evaluated the framework through subjective ratings of sound quality and emotional fidelity. Analysis revealed differences among samples in sound quality, with some samples getting consistently high scores and some getting mixed reviews. While the emotion recognition model reached 70.3% overall accuracy, it performed better at distinguishing between high-arousal emotions but struggled to distinguish between emotions of similar arousal. This framework can be utilized in different fields such as healthcare, education, entertainment, and marketing; real-time emotion recognition can be applied to deliver personalized adaptive experiences. This step includes acquiring multimodal emotion recognition in the future and utilizing physiological data to understand people's emotions better.

Student use of generative AI as a composing process supplement: Concerns for intellectual property and academic honesty

This article discusses the nuanced challenges of using Generative Artificial Intelligence in multimodal compositions while maintaining an ethical adherence to ideas of academic honesty and intellectual property. Through examining hypothetical scenarios, we can see that multimodality complicates the concept of “fair use” in academic contexts, since image or audio generation via AI functions differently than text generated by a Large Language Model. In thinking through the case studies, the article presents an argument for how educators can still use Generative AI in their multimodal composition assignments, through teaching students to us it as a process supplement and to always be critically aware of their citational responsibilities. This understanding of Generative AI use is placed in conversation with our understanding of intellectual property law as relates to both the classroom and broader digital composing environments, to better prepare students to create texts in their future careers.

Programmatic Pitch

Diffusion models have firmly established their excellence for image generation tasks, as evidenced by the success of renowned models such as DALL-E, Midjourney, and Stable Diffusion. This advanced development of diffusion models for visual content raises an interesting question: can diffusion models be adapted for audio generation tasks? In this study, we introduce a novel diffusion model architecture designed to generate mel spectrograms, visual representations of sound which can subsequently be converted into audible music. Given the exceptional capability of diffusion models to produce high- quality images, their application to mel spectrogram generation is particularly promising. Our proposed diffusion model deviates minimally from the conventional architectures employed for visual content, making this research especially useful for examining the potential for cross-domain application between image and audio generation. The proposed model has been trained on a dataset consisting of over 186 hours of Lofi audio, offering the model diverse samples for generalized learning. However, a combination of research limitations led to subpar results, paving the way for further studies to build on this one.

Talking Face Generation With Audio-Deduced Emotional Landmarks.

The goal of talking face generation is to synthesize a sequence of face images of the specified identity, ensuring the mouth movements are synchronized with the given audio. Recently, image-based talking face generation has emerged as a popular approach. It could generate talking face images synchronized with the audio merely depending on a facial image of arbitrary identity and an audio clip. Despite the accessible input, it forgoes the exploitation of the audio emotion, inducing the generated faces to suffer from emotion unsynchronization, mouth inaccuracy, and image quality deficiency. In this article, we build a bistage audio emotion-aware talking face generation (AMIGO) framework, to generate high-quality talking face videos with cross-modally synced emotion. Specifically, we propose a sequence-to-sequence (seq2seq) cross-modal emotional landmark generation network to generate vivid landmarks, whose lip and emotion are both synchronized with input audio. Meantime, we utilize a coordinated visual emotion representation to improve the extraction of the audio one. In stage two, a feature-adaptive visual translation network is designed to translate the synthesized landmarks into facial images. Concretely, we proposed a feature-adaptive transformation module to fuse the high-level representations of landmarks and images, resulting in significant improvement in image quality. We perform extensive experiments on the multi-view emotional audio-visual dataset (MEAD) and crowd-sourced emotional multimodal actors dataset (CREMA-D) benchmark datasets, demonstrating that our model outperforms state-of-the-art benchmarks.

DEVELOPMENT AND REALIZATION OF AN AUDIOMETER WITH AUTOMATIC TESTS AND INTELLIGENT DIAGNOSTICS USING NEURAL NETWORKS

An audiometer is a medical device that measures hearing state; most audiometers are for manual and physical use. The audio signal generator generates all the frequencies between 20 Hz and 22 kHz. This project will produce a tone-based audiometer (software and hardware) based on the Arduino board with GUI MATLAB. This work allowed us to do the test automatically with intelligent diagnosis. The smart part is based on the Neural Network Model. Our work comprises an electronic card and three parts or interfaces: Interface 1: subject management, Interface 2: hearing test, and Interface 3: audiogram. The tests and results of the prototype are very satisfactory; this is a positive sign that the prototype is working as expected and is a viable solution.

Dance2MIDI: Dance-driven multi-instrument music generation

Dance-driven music generation aims to generate musical pieces conditioned on dance videos. Previous works focus on monophonic or raw audio generation, while the multi-instrument scenario is under-explored. The challenges associated with dance-driven multi-instrument music (MIDI) generation are twofold: (i) lack of a publicly available multi-instrument MIDI and video paired dataset and (ii) the weak correlation between music and video. To tackle these challenges, we have built the first multi-instrument MIDI and dance paired dataset (D2MIDI). Based on this dataset, we introduce a multi-instrument MIDI generation framework (Dance2MIDI) conditioned on dance video. Firstly, to capture the relationship between dance and music, we employ a graph convolutional network to encode the dance motion. This allows us to extract features related to dance movement and dance style. Secondly, to generate a harmonious rhythm, we utilize a transformer model to decode the drum track sequence, leveraging a cross-attention mechanism. Thirdly, we model the task of generating the remaining tracks based on the drum track as a sequence understanding and completion task. A BERT-like model is employed to comprehend the context of the entire music piece through self-supervised learning. We evaluate the music generated by our framework trained on the D2MIDI dataset and demonstrate that our method achieves state-of-the-art performance.

Video and Audio Deepfake Datasets and Open Issues in Deepfake Technology: Being Ahead of the Curve

The revolutionary breakthroughs in Machine Learning (ML) and Artificial Intelligence (AI) are extensively being harnessed across a diverse range of domains, e.g., forensic science, healthcare, virtual assistants, cybersecurity, and robotics. On the flip side, they can also be exploited for negative purposes, like producing authentic-looking fake news that propagates misinformation and diminishes public trust. Deepfakes pertain to audio or visual multimedia contents that have been artificially synthesized or digitally modified through the application of deep neural networks. Deepfakes can be employed for benign purposes (e.g., refinement of face pictures for optimal magazine cover quality) or malicious intentions (e.g., superimposing faces onto explicit image/video to harm individuals producing fake audio recordings of public figures making inflammatory statements to damage their reputation). With mobile devices and user-friendly audio and visual editing tools at hand, even non-experts can effortlessly craft intricate deepfakes and digitally altered audio and facial features. This presents challenges to contemporary computer forensic tools and human examiners, including common individuals and digital forensic investigators. There is a perpetual battle between attackers armed with deepfake generators and defenders utilizing deepfake detectors. This paper first comprehensively reviews existing image, video, and audio deepfake databases with the aim of propelling next-generation deepfake detectors for enhanced accuracy, generalization, robustness, and explainability. Then, the paper delves deeply into open challenges and potential avenues for research in the audio and video deepfake generation and mitigation field. The aspiration for this article is to complement prior studies and assist newcomers, researchers, engineers, and practitioners in gaining a deeper understanding and in the development of innovative deepfake technologies.

Automatic Audio and Image Caption Generation with Deep Learning

A novel approach to image caption generation tailored specifically for visually impaired individuals. The proposed system employs advanced computer vision algorithms to analyze images and generate descriptive textual captions. Furthermore, it integrates seamless text-to-speech conversion functionality, allowing for the automatic transformation of these captions into spoken audio, thereby enabling access to visual content for individuals with visual impairments. The goal of this project is to generate descriptive captions for a given photograph or image. We achieve this by employing Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) models, both of which are advanced deep learning techniques. Using computer vision, the system identifies the content of the image and generates a relevant caption. This caption is then converted into audio using Natural Language Processing (NLP).

Central Kurdish Text-to-Speech Synthesis with Novel End-to-End Transformer Training

Recent advancements in text-to-speech (TTS) models have aimed to streamline the two-stage process into a single-stage training approach. However, many single-stage models still lag behind in audio quality, particularly when handling Kurdish text and speech. There is a critical need to enhance text-to-speech conversion for the Kurdish language, particularly for the Sorani dialect, which has been relatively neglected and is underrepresented in recent text-to-speech advancements. This study introduces an end-to-end TTS model for efficiently generating high-quality Kurdish audio. The proposed method leverages a variational autoencoder (VAE) that is pre-trained for audio waveform reconstruction and is augmented by adversarial training. This involves aligning the prior distribution established by the pre-trained encoder with the posterior distribution of the text encoder within latent variables. Additionally, a stochastic duration predictor is incorporated to imbue synthesized Kurdish speech with diverse rhythms. By aligning latent distributions and integrating the stochastic duration predictor, the proposed method facilitates the real-time generation of natural Kurdish speech audio, offering flexibility in pitches and rhythms. Empirical evaluation via the mean opinion score (MOS) on a custom dataset confirms the superior performance of our approach (MOS of 3.94) compared with that of a one-stage system and other two-staged systems as assessed through a subjective human evaluation.

Cross-modal generative model for visual-guided binaural stereo generation

Binaural stereo audio is recorded by imitating the way the human ear receives sound, which provides people with an immersive listening experience. Existing approaches leverage autoencoders and directly exploit visual spatial information to synthesize binaural stereo, resulting in a limited representation of visual guidance. For the first time, we propose a visually guided generative adversarial approach for generating binaural stereo audio from mono audio. Specifically, we develop a Stereo Audio Generation Model (SAGM), which utilizes shared spatio-temporal visual information to guide the generator and the discriminator to work separately. The shared visual information is updated alternately in the generative adversarial stage, allowing the generator and discriminator to deliver their respective guided knowledge while visually sharing. The proposed method learns bidirectional complementary visual information, which facilitates the expression of visual guidance in generation. In addition, spatial perception is a crucial attribute of binaural stereo audio, and thus the evaluation of stereo spatial perception is essential. However, previous metrics failed to measure the spatial perception of audio. To this end, a metric to measure the spatial perception of audio is proposed for the first time. The proposed metric is capable of measuring the magnitude and direction of spatial perception in the temporal dimension. Further, considering its function, it is feasible to utilize it instead of demanding user studies to some extent. The proposed method achieves state-of-the-art performance on 2 datasets and 5 evaluation metrics. Qualitative experiments and user studies demonstrate that the method generates space-realistic stereo audio.

Pdf to Voice by Using Deep Learning

Audio books are extraordinary for people who, like most people, want to listen to themselves read. These can't be bought and stored within the library at domestic. Audiobooks are a notable manner to rest your eyes and take a damage from the steady stimulation of virtual gadgets. Others as a time shop. For instance, hold studying books even as doing exceptional responsibilities on the equal time. Not only will this lessen the issues of millennials, but it's going to also be a valuable tool for lowering human being’s visibility. The ability to transform any content material into an audiobook is a true present of humanity. Our technology may be used to develop such gadgets. Text-to-speech and other recitation applications are extensively used to assist college students broaden studying comprehension abilities. The PDF to Audio System is a screen reader designed and developed for powerful audio verbal exchange. The International Organization for Standardization (ISO) has unique PDF documents as an open file format. Document layout is one of the handiest formats for digital conversation and facts change. This is very essential if we want to improve the accessibility of our readers' display screen by adding audio to our content material. Features from PDF files encompass hyperlinks and buttons, as well as audio and video documents. Multiple languages may be supported the use of PDF-to- audio generation, which lets in customers to hear text definitely (spoken).

Visual-guided scene-aware audio generation method based on hierarchical feature codec and rendering decision

Visually guided spatial sound generation (VGSSG) is a well-suited multimodal learning method for dealing with recorded videos. However, existing methods are difficult to be directly applied to spatial sound generation for movie clips. This is mainly due to (1) the existence of Cinematic Audiovisual Language (CAL) in movies, which makes it difficult to construct spatial sound mapping models directly through data-driven based methods. (2) The problem of the inadequate model performance, which is caused by the excessive heterogeneous gap between audiovisual modal information. To solve the aforementioned problems, we propose a VGSSG method based on CAL decision-making and hierarchical feature coding and decoding, which effectively accomplishes spatial sound generation based on the CAL of movies. Specifically, to solve the problem of CAL modeling, a multimodal information-guided movie audio rendering decision maker is established, which can decide the rendering strategy based on the CAL of the current clip. To narrow the heterogeneous gap that hinders the fusion between audiovisual modal data, we propose a codec structure based on hierarchical fusion of audiovisual features and full-scale skip-connections, which improves the efficiency of the comprehensive utilization of audiovisual modal data, and demonstrates the effectiveness of adopting shallow features in VGSSG task. We integrate both 2-channel and 6-channel spatial audio generation into a unified framework. In addition, we establish a movie audiovisual bimodal dataset with hand-crafted CAL annotations. Experimentally, we demonstrate that compared with the existing methods, our method has higher performance in terms of reducing generation distortion.

Audio Generation with Multiple Conditional Diffusion Model

Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing datasets into a new dataset comprising the audio and corresponding conditions and use a series of evaluation metrics to evaluate the controllability performance. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation.

The Dark Side of Language Models: Exploring the Potential of LLMs in Multimedia Disinformation Generation and Dissemination

Disinformation - the deliberate spread of false or misleading information poses a significant threat to our society by undermining trust, exacerbating polarization, and manipulating public opinion. With the rapid advancement of artificial intelligence and the growing prominence of large language models (LLMs) such as ChatGPT, new avenues for the dissemination of disinformation are emerging. This review paper explores the potential of LLMs to initiate the generation of multi-media disinformation, encompassing text, images, audio, and video. We begin by examining the capabilities of LLMs, highlighting their potential to create compelling, context-aware content that can be weaponized for malicious purposes. Subsequently, we examine the nature of disinformation and the various mechanisms through which it spreads in the digital landscape. Utilizing these advanced models, malicious actors can automate and scale up disinformation effectively. We describe a theoretical pipeline for creating and disseminating disinformation on social media. Existing interventions to combat disinformation are also reviewed. While these efforts have shown success, we argue that they need to be strengthened to effectively counter the escalating threat posed by LLMs. Digital platforms have, unfortunately, enabled malicious actors to extend the reach of disinformation. The advent of LLMs poses an additional concern as they can be harnessed to significantly amplify the velocity, variety, and volume of disinformation. Thus, this review proposes augmenting current interventions with AI tools like LLMs, capable of assessing information more swiftly and comprehensively than human fact-checkers. This paper illuminates the dark side of LLMs and highlights their potential to be exploited as disinformation dissemination tools.

Using deep learning for recreating binaural audio

In recent years, there has been an increase in research around generating spatialized audio using a mono audio signal. Methods like using neural networks which combine image segmentation with object location for adding back the spatial qualities are often developed. Instead, our project focuses on taking arbitrary mono input sound and an input angle, and outputting spatial stereo audio of the input sound with the directionality of the angle. This is different from current implementations as it is a simpler approach to what spatial audio generation is, and it allows for the use in the model in new areas. Using a binaural microphone and a custom-made anechoic chamber, 120 hours of labelled binaural audio was recorded for use in our model. The audio consists of frequency sweeps, pink noise, and phonetically balanced speech. Our method is to predict a complex short-time Fourier transform mask which will contain the phase and amplitude within it. The model is an autoencoder based on the U-NET model, which is applied to the mono input before being compared to the labelled data for training. With this simpler approach, we hope to make spatial audio more accessible for a variety of applications.

Generativni modeli veštačke inteligencije - zauzdati još jednu tehnološku revoluciju

Generative AI promises to revolutionize many industries (entertainment, marketing, healthcare, finance, and research) by empowering machines to create new data content inspired by existing data. It experienced exponential growth in recent years. In 2023 breakout year Gen AI impact reached 2.6-4.4 trillion USD (2.5-4.2% of global GDP). The development of modern LLM-based models has been facilitated by improvements in computing power, data availability, and algorithms. These models have diverse applications in text, visual, audio, and code generation across various domains. Leading companies are rapidly deploying Gen AI for strategic decision-making at corporate executive levels. While AI-related risks have been identified, mitigation measures are still in early stages. Leaders in Gen AI adoption anticipate workforce changes and re-skilling needs. Gen AI is primarily used for text functions, big data analysis, and customer services, with the strongest impact in knowledge-based sectors. High-performing AI companies prioritize revenue generation over cost reduction, rapidly expand the use of Gen AI across various business functions, and link business value to organizational performance and structure. There is a notable lack of attention to addressing broader societal risks and the impact on the labor force. Gen AI creates new job opportunities and improves productivity in key areas. Future investment in AI is expected to rise. Concerns about the potential AI singularity, where machines surpass human intelligence, are subject to debate. Some view singularity as a risk, others are more optimistic based on human control and societal constraints. Leading experts in Gen AI predict that the coming decade can be the most prosperous in history if we manage to harness the benefits of Gen AI and control its downside.

AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining

AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining

Sounding Video Generator: A Unified Framework for Text-Guided Sounding Video Generation

As a combination of visual and audio signals, video is inherently multi-modal. However, existing video generation methods are primarily intended for the synthesis of visual frames, whereas audio signals in realistic videos are disregarded. In this work, we concentrate on a rarely investigated problem of text-guided sounding video generation and propose the Sounding Video Generator (SVG), a unified framework for generating realistic videos along with audio signals. Specifically, we present the SVG-VQGAN to transform visual frames and audio mel-spectrograms into discrete tokens. SVG-VQGAN applies a novel hybrid contrastive learning method to model inter-modal and intra-modal consistency and improve the quantized representations. A cross-modal attention module is employed to extract associated features of visual frames and audio signals for contrastive learning. Then, a Transformer-based decoder is used to model associations between texts, visual frames, and audio signals at token level for auto-regressive sounding video generation. AudioSet-Cap, a human annotated text-video-audio paired dataset, is produced for training SVG. Experimental results demonstrate the superiority of our method when compared with existing text-to-video generation methods as well as audio generation methods on Kinetics and VAS datasets.