Audio Modalities Research Articles

A mindfulness intervention to reduce maternal distress in neonatal intensive care: a mixed methods pilot study.

Mothers with an infant in the neonatal intensive care unit (NICU) are at risk for depression, anxiety, and trauma symptoms, with negative implications for maternal-infant bonding, maternal well-being, and infant development. Few interventions to promote NICU mothers' mental health, however, have been developed or tested. This pre-post pilot study assessed feasibility, acceptability, and preliminary outcomes of a mindfulness intervention for NICU mothers. Twenty-seven mothers were recruited from a university NICU and offered a mindfulness intervention via introductory video and audio-recorded practices. Participants completed a baseline self-report survey. After 2weeks of engaging with intervention materials, participants completed a second survey and in-depth interview. Quantitative data were analyzed using paired t tests; qualitative data were analyzed using thematic coding. Twenty-four women (89%) completed the study. Quantitative data indicated significant improvements in depressive, anxiety, and trauma symptoms, negative coping, NICU-related stress, and sleep (p < 0.05). Qualitative data identified themes of perceived improvements in psychological distress and stress symptoms, self-care, and relationships. Findings support the mindfulness intervention's feasibility, acceptability, and potential promise for reducing maternal distress and promoting well-being. Use of video and audio modalities may facilitate program sustainability and scale up. Further research on the program is merited.

Integrating Both Visual and Audio Cues for Enhanced Video Caption

Video caption refers to generating a descriptive sentence for a specific short video clip automatically, which has achieved remarkable success recently. However, most of the existing methods focus more on visual information while ignoring the synchronized audio cues. We propose three multimodal deep fusion strategies to maximize the benefits of visual-audio resonance information. The first one explores the impact on cross-modalities feature fusion from low to high order. The second establishes the visual-audio short-term dependency by sharing weights of corresponding front-end networks. The third extends the temporal dependency to long-term through sharing multimodal memory across visual and audio modalities. Extensive experiments have validated the effectiveness of our three cross-modalities fusion strategies on two benchmark datasets, including Microsoft Research Video to Text (MSRVTT) and Microsoft Video Description (MSVD). It is worth mentioning that sharing weight can coordinate visual- audio feature fusion effectively and achieve the state-of-art performance on both BELU and METEOR metrics. Furthermore, we first propose a dynamic multimodal feature fusion framework to deal with the part modalities missing case. Experimental results demonstrate that even in the audio absence mode, we can still obtain comparable results with the aid of the additional audio modality inference module.

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Visual and audio modalities are two symbiotic modalities underlying videos, which contain both common and complementary information. If they can be mined and fused sufficiently, performances of related video tasks can be significantly enhanced. However, due to the environmental interference or sensor fault, sometimes, only one modality exists while the other is abandoned or missing. By recovering the missing modality from the existing one based on the common information shared between them and the prior information of the specific modality, great bonus will be gained for various vision tasks. In this paper, we propose a Cross-Modal Cycle Generative Adversarial Network (CMCGAN) to handle cross-modal visual-audio mutual generation. Specifically, CMCGAN is composed of four kinds of subnetworks: audio-to-visual, visual-to-audio, audio-to-audio and visual-to-visual subnetworks respectively, which are organized in a cycle architecture. CMCGAN has several remarkable advantages. Firstly, CMCGAN unifies visual-audio mutual generation into a common framework by a joint corresponding adversarial loss. Secondly, through introducing a latent vector with Gaussian distribution, CMCGAN can handle dimension and structure asymmetry over visual and audio modalities effectively. Thirdly, CMCGAN can be trained end-to-end to achieve better convenience. Benefiting from CMCGAN, we develop a dynamic multimodal classification network to handle the modality missing problem. Abundant experiments have been conducted and validate that CMCGAN obtains the state-of-the-art cross-modal visual-audio generation results. Furthermore, it is shown that the generated modality achieves comparable effects with those of original modality, which demonstrates the effectiveness and advantages of our proposed method.

Audio Visual Attribute Discovery for Fine-Grained Object Recognition

Current progresses on fine-grained recognition are mainly focus on learning the discriminative feature representation via introducing the visual supervisions e.g. part labels. However, it is time-consuming and needs the professional knowledge to obtain the accuracy annotations. Different from these existing methods based on the visual supervisions, in this paper, we introduce a novel feature named audio visual attributes via discovering the correlations between the visual and audio representations. Specifically, our unified framework is training with video-level category label, which consists of two important modules, the encoder module and the attribute discovery module, to encode the image and audio into vectors and learn the correlations between audio and images, respectively. On the encoder module, we present two types of feed forward convolutional neural network for the image and audio modalities. While an attention driven framework based on recurrent neural network is developed to generate the audio visual attribute representation. Thus, our proposed architecture can be implemented end-to-end in the step of inference. We exploit our models for the problem of fine-grained bird recognition on the CUB200-211 benchmark. The experimental results demonstrate that with the help of audio visual attribute, we achieve the superior or comparable performance to that of strongly supervised approaches on the bird recognition.

Effects of stimulus response compatibility on covert imitation of vowels

When we observe someone else speaking, we tend to automatically activate the corresponding speech motor patterns. When listening, we therefore covertly imitate the observed speech. Simulation theories of speech perception propose that covert imitation of speech motor patterns supports speech perception. Covert imitation of speech has been studied with interference paradigms, including the stimulus–response compatibility paradigm (SRC). The SRC paradigm measures covert imitation by comparing articulation of a prompt following exposure to a distracter. Responses tend to be faster for congruent than for incongruent distracters; thus, showing evidence of covert imitation. Simulation accounts propose a key role for covert imitation in speech perception. However, covert imitation has thus far only been demonstrated for a select class of speech sounds, namely consonants, and it is unclear whether covert imitation extends to vowels. We aimed to demonstrate that covert imitation effects as measured with the SRC paradigm extend to vowels, in two experiments. We examined whether covert imitation occurs for vowels in a consonant–vowel–consonant context in visual, audio, and audiovisual modalities. We presented the prompt at four time points to examine how covert imitation varied over the distracter’s duration. The results of both experiments clearly demonstrated covert imitation effects for vowels, thus supporting simulation theories of speech perception. Covert imitation was not affected by stimulus modality and was maximal for later time points.

Психологические особенности курсантов в связи с различиями по критериям «минимальные мозговые дисфункции» и «минимальные повреждения мозга»

The traditional view of the extremely negative role of minimal dysfunction and brain damage in the psychological status of a person requires experimental verification on a large and relatively homogeneous sample of examinees. The study presents the test results for 500 male student pilots aged 18–19 who are physically and mentally healthy according to the Pathocharacterologic Diagnostic Questionnaire, the 16PF Questionnaire by R. Cattell, and the Neuropsychic Instability and Accentuation Questionnaire. The cognitive indicators, military and professional motivation and minimal brain dysfunction rates are recorded. The results are differentiated by the minimal brain criteria. The most complex form of minimal brain dysfunction (MBD) was evident in 14.7% of cases, and minimal brain damage (Index B) was typical for 6.8%. There is no relationship among these criteria. High values of Index B indicating the organic nature of psychopathy is positively correlated with the indicators of emotional stability, credulity, expressiveness, non-anxiety, self-control, attention, intelligence, insight, military and professional motivation. When Index B is high, the functions of the right side of the body, audio modality of perception, left-brain thinking are less evident. MBD indicator is positively correlated with the practical thinking, self-sufficiency, audio modality of perception and the left type of action. MBD plays the negative role for intelligence, image memory, visual modality of perception, and such indicators of military and professional motivation as the developed worldview and self-regulation of impulses. These qualities are subject to correction by means of psychological training of personal development. The results obtained suggest that both criteria are informative when assessing professional suitability.

Multi-modal learning for affective content analysis in movies

Affective content analysis is an important research topic in video content analysis, and has extensive applications in many fields. However, it is a challenging task to design a computational model for predicting emotions induced by videos, since the elicited emotions can be considered relatively subjective. Intuitively, several features of different modalities can depict the elicited emotions, but the correlation and influence of these features are still not well studied. To address this issue, we propose a multi-modal learning framework, which classifies affective contents in the valence-arousal space. In particular, we utilize the features extracted by the methods of motion keypoint trajectory and convolutional neural networks to depict the visual modality of elicited emotions, and extract a global audio feature by the openSMILE toolkit to describe the audio modality. Then, the linear support vector machine and support vector regression are employed to learn the affective models. By comparing these three features with five baseline features, we discover that the three features are significant for describing affective content. Experimental results also demonstrate that the three features complement each other. Moreover, the proposed framework obtains the state-of-the-art results on two challenging datasets of video affective content analysis.

The Relative Importance of Aural and Visual Information in the Evaluation of Western Canon Music Performance by Musicians and Nonmusicians

Aural and visual information have been shown to affect audience evaluations of music performance (Griffiths, 2010; Juslin, 2000); however, it is not fully understood which modality has the greatest relative impact upon judgements of performance or if the evaluator’s musical expertise mediates this effect. An opportunity sample of thirty-four musicians (8 male, 26 female Mage = 26.4 years) and 26 nonmusicians (6 male, 20 female, Mage = 44.0 years) rated four video clips for technical proficiency, musicality, and overall performance quality using 7-point Likert scales. Two video performances of Debussy’s Clare de lune (one professional, one amateur) were used to create the four video clips, comprising two clips with congruent modality information, and two clips with incongruent modality information. The incongruent clips contained the visual modality of one quality condition with the audio modality of the other. It was possible to determine which modality was most important in participants’ evaluative judgements based on the modality of the professional quality condition in the clip that was rated most highly. The current study confirms that both aural and visual information can affect audience members’ experience of musical performance. We provide evidence that visual information has a greater impact than aural information on evaluations of performance quality, as the incongruent clip with amateur audio + professional video was rated significantly higher than that with professional audio + amateur video. Participants’ level of musical expertise was found to have no effect on their judgements of performance quality.

Facial Expression Recognition in Video with Multiple Feature Fusion

Video based facial expression recognition has been a long standing problem and attracted growing attention recently. The key to a successful facial expression recognition system is to exploit the potentials of audiovisual modalities and design robust features to effectively characterize the facial appearance and configuration changes caused by facial motions. We propose an effective framework to address this issue in this paper. In our study, both visual modalities (face images) and audio modalities (speech) are utilized. A new feature descriptor called Histogram of Oriented Gradients from Three Orthogonal Planes (HOG-TOP) is proposed to extract dynamic textures from video sequences to characterize facial appearance changes. And a new effective geometric feature derived from the warp transformation of facial landmarks is proposed to capture facial configuration changes. Moreover, the role of audio modalities on recognition is also explored in our study. We applied the multiple feature fusion to tackle the video-based facial expression recognition problems under lab-controlled environment and in the wild, respectively. Experiments conducted on the extended Cohn-Kanade (CK+) database and the Acted Facial Expression in Wild (AFEW) 4.0 database show that our approach is robust in dealing with video-based facial expression recognition problems under lab-controlled environment and in the wild compared with the other state-of-the-art methods.

Auxiliary Loss Multimodal GRU Model in Audio-Visual Speech Recognition

Audio-visual speech recognition (AVSR) utilizes both audio and video modalities for the robust automatic speech recognition. Most deep neural network (DNN) has achieved promising performances in AVSR owing to its generalized and nonlinear mapping ability. However, these DNN models have two main disadvantages: 1) the first disadvantage is that most models alleviate the AVSR problems neglecting the fact that the frames are correlated; and 2) the second disadvantage is the feature learned by the mentioned models is not credible. This is because the joint representation learned by the fusion fails to consider the specific information of categories, and the discriminative information is sparse, while the noise, reverberation, irrelevant image objection, and background are redundancy. Aiming at relieving these disadvantages, we propose the auxiliary loss multimodal GRU (alm-GRU) model including three parts: feature extraction, data augmentation, and fusion & recognition. The feature extraction and data augmentation are a complete effective solution for the processing raw complete video and training, and precondition for later core part: fusion & recognition using alm-GRU equipped with a novel loss which is an end-to-end network combining both fusion and recognition, furthermore considering the modal and temporal information. The experiments show the superiority of our model and necessity of the data augmentation and generative component in the benchmark data sets.

Amused speech components analysis and classification: Towards an amusement arousal level assessment system

In this paper, we present our work on analysis and classification of smiled vowels, chuckling (or shaking) vowels and laughter syllables. This work is part of a larger framework that aims at assessing the level of amusement in speech using the audio modality only. Indeed all of these three categories occur in amused speech and are considered to contribute in the expression of different levels of amusement. We first analyze these three amused speech components on the acoustic level. Then, we improve a classification system we previously developed. With a limited amount of data and features, we are able to obtain good classification results with different systems. Among the compared systems, the best one achieved 82.8% of accuracy, therefore outperforming chance.

Does facilitation by phonological cuing in picture naming depend on the modality of the cue?

ABSTRACTBackground: Phonological cuing in spoken picture naming is widely used to improve word production accuracy in aphasic participants and to investigate word planning processes in psycholinguistic experiments with healthy participants. However, different paradigms and different modalities have been used across studies and populations, with some contradictory results.Aims: Here, we investigate whether and how the modality in which a phonological cue is delivered impacts on word production in picture naming tasks using the exact same paradigm with healthy and with aphasic participants.Methods & Procedure: In experiment 1, behavioural and electroencephalographic event-related potential (ERP) responses to auditory, visual and audiovisual cues in a picture naming task were investigated in healthy participants. In experiment 2, the impact of the same cue modalities was studied on errors and latencies in 15 aphasic participants.Outcomes & Results: The modality in which the cue is delivered influenced phonological facilitation effects in both populations, the audio modality being the most facilitative on production latencies in healthy participants and in participants with mild aphasia. No further gains were obtained by a double modality audiovisual cue on production latencies, whereas the double cues interfered with production accuracy in aphasic participants. ERP results not only were compatible with facilitation of phonological encoding in all experimental conditions but also showed modulation of a late encoding time period for both the visual and audiovisual cues, likely reflecting interference of the visual modality during monitoring or motor speech encoding. The latter observation is indeed coherent with the interference of the audiovisual modality that was observed on error rates in the more severely impaired aphasic participants.Conclusions: These results suggest that unimodal auditory cuing is the facilitatory condition, while visual cues in unimodal or bimodal conditions give rise to both facilitation and interference which manifest behaviourally depending on the balance of these opposing effects.

Multi-modal Multimedia Big Data Analyzing Architecture and Resource Allocation on Cloud Platform

Multimedia big data analyzing is the new topic that focus on all features of distributed computing systems that contains of a combination of text, visual and audio modalities. The traditional method to transcoding multi-modal multimedia big data needs expensive hardware and the amount of data increases transcoding executes a significant burden on the computing infrastructure. Therefore we illustrate a novel implementation for multimedia big data analyzing and data distribution. Our proposed architecture contains three layers such as service layer, platform layer and infrastructure layer. We design and implement the platform layer of the system by using a MapReduce framework running on a hadoop distributed file system (HDFS) and the media processing libraries Xuggler. In this way, our proposed system reduces the time for transcoding large amounts of data into specific formats depending on the user requirements. It provides flexible multimedia record/write interface and we can build large scale multimedia big data analytic applications based on Hadoop cloud platform. Moreover, we proposed the ant colony optimization (ACO) algorithm for efficient resource allocation in infrastructure layer. The simulation results demonstrate that the proposed algorithm can optimally allocate VM to achieve a minimal response time.

Multimodal Stereoscopic Movie Summarization Conforming to Narrative Characteristics.

Video summarization is a timely and rapidly developing research field with broad commercial interest, due to the increasing availability of massive video data. Relevant algorithms face the challenge of needing to achieve a careful balance between summary compactness, enjoyability, and content coverage. The specific case of stereoscopic 3D theatrical films has become more important over the past years, but not received corresponding research attention. In this paper, a multi-stage, multimodal summarization process for such stereoscopic movies is proposed, that is able to extract a short, representative video skim conforming to narrative characteristics from a 3D film. At the initial stage, a novel, low-level video frame description method is introduced (frame moments descriptor) that compactly captures informative image statistics from luminance, color, optical flow, and stereoscopic disparity video data, both in a global and in a local scale. Thus, scene texture, illumination, motion, and geometry properties may succinctly be contained within a single frame feature descriptor, which can subsequently be employed as a building block in any key-frame extraction scheme, e.g., for intra-shot frame clustering. The computed key-frames are then used to construct a movie summary in the form of a video skim, which is post-processed in a manner that also considers the audio modality. The next stage of the proposed summarization pipeline essentially performs shot pruning, controlled by a user-provided shot retention parameter, that removes segments from the skim based on the narrative prominence of movie characters in both the visual and the audio modalities. This novel process (multimodal shot pruning) is algebraically modeled as a multimodal matrix column subset selection problem, which is solved using an evolutionary computing approach. Subsequently, disorienting editing effects induced by summarization are dealt with, through manipulation of the video skim. At the last step, the skim is suitably post-processed in order to reduce stereoscopic video defects that may cause visual fatigue.

It’s Not Just What You Say, But How You Say It: Muiltimodal Sentiment Analysis Via Crowdsourcing

This paper examines the effect of various modalities of expression on the reliability of crowdsourced sentiment polarity judgments. A novel corpus of YouTube video reviews was created, and sentiment judgments were obtained via Amazon Mechanical Turk. We created a system for isolating text, video, and audio modalities from YouTube videos to ensure that annotators could only see the particular modality or modalities being evaluated. Reliability of judgments was assessed using Fleiss Kappa inter-annotator agreement values. We found that the audio only modality produced the most reliable judgments for video fragments and that across modalities video fragments are less ambiguous than full videos.

Classification Study of Sound and Image Events Using Event Detection Systems

For intelligent systems to make best use of the audio modality, it is important that they can recognize not just speech and music, which have been researched as specific tasks, but also general sounds in everyday environments. To stimulate research in this field we conducted a public research challenge: the IEEE Audio and Acoustic Signal Processing Technical Committee challenge on Detection and Classification of Acoustic Scenes and Events (DCASE). In this paper, we report on the state of the art in automatically classifying audio scenes, and automatically detecting and classifying audio events. We survey prior work as well as the state of the art represented by the submissions to the challenge from various research groups. We also provide detail on the organization of the challenge, so that our experience as challenge hosts may be useful to those organizing challenges in similar domains. We created new audio datasets and baseline systems for the challenge; these, as well as some submitted systems, are publicly available under open licenses, to serve as benchmarks for further research in general-purpose machine listening.

Discrimination Between Native and Non-Native Speech Using Visual Features Only.

Accent is a soft biometric trait that can be inferred from pronunciation and articulation patterns characterizing the speaking style of an individual. Past research has addressed the task of classifying accent, as belonging to a native language speaker or a foreign language speaker, by means of the audio modality only. However, features extracted from the visual stream of speech have been successfully used to extend or substitute audio-only approaches that target speech or language recognition. Motivated by these findings, we investigate to what extent temporal visual speech dynamics attributed to accent can be modeled and identified when the audio stream is missing or noisy, and the speech content is unknown. We present here a fully automated approach to discriminating native from non-native English speech, based exclusively on visual cues. A systematic evaluation of various appearance and shape features for the target problem is conducted, with the former consistently yielding superior performance. Subject-independent cross-validation experiments are conducted on mobile phone recordings of continuous speech and isolated word utterances spoken by 56 subjects from the challenging MOBIO database. High performance is achieved on a text-dependent (TD) protocol, with the best score of 76.5% yielded by fusion of five hidden Markov models trained on appearance features. Our framework is also efficient even when tested on examples of speech unseen in the training phase, although performing less accurately compared to the TD case.

Special issue on spatial sound in virtual, augmented, and mixed-reality environments

The rapid advances in the technology and science of presenting spatial sound in virtual, augmented, and mixedreality environments seem to be underrepresented in recent literature. The goal of this special issue of the Virtual Reality Journal is twofold: to provide a state-of-the-art review of progress in spatial sound as applied to virtual reality (VR) and augmented reality (AR), and to stimulate further research in this emerging and important field. In this special issue, we are pleased to present papers representing a range of topics, from basic research on the perception of spatial sound to more applied papers on the use of spatial sound in real-world settings. It is the hope of the guest editors that the special issue will also encourage scientists to incorporate spatial sound into their own research and to take the next step beyond the innovative projects described here. Virtual reality, mixed reality, and augmented reality are often technology-driven. On this point, the recent emergence of affordable head-mounted displays—as exemplified by HTC and Valve Vive, Samsung Gear VR, Sony (PlayStation 4) VR, Samsung-sponsored eye-tracking FOVE, Google Cardboard, and the Facebook-acquired Oculus Rift—signals a mass diffusion of VR-style applications such as games, programs, and visualization. Presumably, tracking, alignment, and omnidirectional capabilities often found in these visual displays will foster similar developments on personalized audio displays. Considering the range of information that human senses can detect, multimodal systems that present users with a coordinated display are needed to increase realism and performance. In R&D of virtual and mixed-reality systems, the audio modality has been somewhat underemphasized, partly because the ‘‘original field’’ of virtual reality focused on the dominant (at least for some tasks) visual modality. For instance, despite its ostensible ambitions for multimodal interfaces, the Virtual Reality journal is still classified by its publisher as ‘‘Image Processing: Computer Imaging, Graphics, and Vision,’’ whereas Springer’s category ‘‘HCI: User Interfaces, HCI and Ergonomics’’ would probably be a better fit. The peer-reviewed articles presented in this special issue represent a broad consideration of themes on spatial sound, including basic research papers which we include to help provide a baseline of established scientific data in the field and ‘‘application’’ papers which are not only interesting but can be used to evaluate how well systems with spatial audio perform in realistic scenarios. These papers will be especially useful for those who design systems for realworld and pragmatic applications. Representing expanding interest in the field, the papers presented in this special issue come from innovative researchers in Asia, Europe, and North America, with a focus on recent advances in spatial ‘‘virtual’’ sound, including spatialized audio interfaces, perception, presence and cognition, navigation and way-finding, and applications of spatialized sound for VR, AR, mixed reality (MR), and presence. A deeper review of many of these topics can be found in a just-published anthology of augmented reality and wearable computers, edited by one of the special issue guest editors (Barfield 2016) and including contributions about spatial sound and augmented audio reality by the other two (Michael Cohen and Julian Villegas). & Michael Cohen mcohen@u-aizu.ac.jp

Detection and Classification of Acoustic Scenes and Events

For intelligent systems to make best use of the audio modality, it is important that they can recognize not just speech and music, which have been researched as specific tasks, but also general sounds in everyday environments. To stimulate research in this field we conducted a public research challenge: the IEEE Audio and Acoustic Signal Processing Technical Committee challenge on Detection and Classification of Acoustic Scenes and Events (DCASE). In this paper, we report on the state of the art in automatically classifying audio scenes, and automatically detecting and classifying audio events. We survey prior work as well as the state of the art represented by the submissions to the challenge from various research groups. We also provide detail on the organization of the challenge, so that our experience as challenge hosts may be useful to those organizing challenges in similar domains. We created new audio datasets and baseline systems for the challenge; these, as well as some submitted systems, are publicly available under open licenses, to serve as benchmarks for further research in general-purpose machine listening.

Use of Video and Audio Texts in EFL Listening Test

The study aims to discover whether audio or video modality in a listening test is more beneficial to test takers. In this study, the posttest-only control group design was utilized and quantitative data were collected in order to measure participant performances concerning two types of modality (audio or video) in a listening test. The participants, first grade students from an ELT program, were recruited and randomly assigned to two groups: audio-only text (AOT) (n=30) and video-only text (VOT) (n=27). Audio-only text (AOT) and video-only text (VOT) posttests were administered to the two randomly selected groups. Based on the results, the spread of the scores was wide in the post tests. In a nutshell, apart from texts 1 and 2, the AOT group performed significantly higher than the VOT group, despite the visual elements of the video. When considered all twenty items of the four texts, the significant difference found indicates that the audio modality was more favorable. This study examined differences in the effects of video listening text or audio-only listening text in terms of their effect on L2 test-taker performance. The quantitative results showed significantly higher success for AOT test takers. In other words, a consistent pattern presented in the listening comprehension test towards audio modality. However, the findings of the current research are not conclusive since various elements may have affected the outcome, such as motivation, physical factors, and topic familiarity, note-taking habits, and initial preference for audio or video. Therefore, further empirical research comparing AOT and VOT listening comprehension assessments is suggested to take into account these variables.