Audio-visual Data Research Articles

Electroencephalography can provide advance warning of technical errors during laparoscopic surgery.

Intraoperative adverse events lead to patient injury and death, and are increasing. Early warning systems (EWSs) have been used to detect patient deterioration and save lives. However, few studies have used EWSs to monitor surgical performance and caution about imminent technical errors. Previous (non-surgical) research has investigated neural activity to predict future motor errors using electroencephalography (EEG). The present proof-of-concept cohort study investigates whether EEG could predict technical errors in surgery. In a large academic hospital, three surgical fellows performed 12 elective laparoscopic general surgeries. Audiovisual data of the operating room and the surgeon's neural activity were recorded. Technical errors and epochs of good surgical performance were coded into events. Neural activity was observed 40s prior and 10s after errors and good events to determine how far in advance errors were detected. A hierarchical regression model was used to account for possible clustering within surgeons. This prospective, proof-of-concept, cohort study was conducted from July to November 2021, with a pilot period from February to March 2020 used to optimize the technique of data capture and included participants who were blinded from study hypotheses. Forty-five technical errors, mainly due to too little force or distance (n = 39), and 27 good surgical events were coded during grasping and dissection. Neural activity representing error monitoring (p = .008) and motor uncertainty (p = .034) was detected 17s prior to errors, but not prior to good surgical performance. These results show that distinct neural signatures are predictive of technical error in laparoscopic surgery. If replicated with low false-alarm rates, an EEG-based EWS of technical errors could be used to improve individualized surgical training by flagging imminent unsafe actions-before errors occur and cause patient harm.

Investigating Teacher’s Linguistic Competence in Teaching English for Academic Purposes

Being a language teacher means they are not only fluent in using the target language but also possess competence in the study of language. Hence, linguistic knowledge is one essential skill a language teacher needs to possess. Using a descriptive qualitative research design of the content analysis, this article aimed to investigate the linguistic competence of an English teacher and its implementation in the classroom. The recordings of online classroom sessions of an English for Academic Purposes course were observed. The teacher, as the main object of the study, was also interviewed to enrich understanding through open-ended questions. The collected audio-visual data was then analyzed and categorized based on the linguistic features employed by the teacher. While existing studies focus on the linguistic competence of pre-service teachers, the findings are inclined to negative results. Hence, by investigating the teaching practice of a teacher with 30 years of experience, the findings revealed the descriptions of linguistic features evident in the data and explained how those features helped the teacher in carrying out the teaching and learning session. Subsequently, it is recommended that English teachers increase their language awareness as well as linguistic competence so that they can perform better in meeting the students’ needs in acquiring and improving English language skills.

Video-Based Cross-Modal Auxiliary Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis has a wide range of applications due to its information complementarity in multimodal interactions. Previous works focus more on investigating efficient joint representations, but they rarely consider the insufficient unimodal features extraction and data redundancy of multimodal fusion. In this paper, a Video-based Cross-modal Auxiliary Network (VCAN) is proposed, which is comprised of an audio features map module and a cross-modal selection module. The first module is designed to substantially increase feature diversity in audio feature extraction, aiming to improve classification accuracy by providing more comprehensive acoustic representations. To empower the model to handle redundant visual features, the second module is addressed to efficiently filter the redundant visual frames during integrating audiovisual data. Moreover, a classifier group consisting of several image classification networks is introduced to predict sentiment polarities and emotion categories. Extensive experimental results on RAVDESS, CMU-MOSI, and CMU-MOSEI benchmarks indicate that VCAN is significantly superior to the state-of-the-art methods for improving the classification accuracy of multimodal sentiment analysis.

Self-Supervised Learning of Audio Representations From Audio-Visual Data Using Spatial Alignment

Learning from audio-visual data offers many possibilities to express correspondence between the audio and visual content, similar to the human perception that relates aural and visual information. In this work, we present a method for self-supervised representation learning based on audio-visual spatial alignment (AVSA), a more sophisticated alignment task than the audio-visual correspondence (AVC). In addition to the correspondence, AVSA also learns from the spatial location of acoustic and visual content. Based on 360$^\text{o}$ video and Ambisonics audio, we propose selection of visual objects using object detection, and beamforming of the audio signal towards the detected objects, attempting to learn the spatial alignment between objects and the sound they produce. We investigate the use of spatial audio features to represent the audio input, and different audio formats: Ambisonics, mono, and stereo. Experimental results show a 10 $\%$ improvement on AVSA for the first order ambisonics intensity vector (FOA-IV) in comparison with log-mel spectrogram features; the addition of object-oriented crops also brings significant performance increases for the human action recognition downstream task. A number of audio-only downstream tasks are devised for testing the effectiveness of the learnt audio feature representation, obtaining performance comparable to state-of-the-art methods on acoustic scene classification from ambisonic and binaural audio.

Audio-video database from subacute stroke patients for dysarthric speech intelligence assessment and preliminary analysis

Early, objective, and accurate assessment and identification of dysarthria caused by neurological diseases are essential in neurorehabilitation. This could be achieved by a robust smart system. However, developing such a system requires a standard training database that is properly labelled, which unfortunately is currently lacking. The present study aimed to establish a standardized, audio-visual integrated speech database of subacute stroke patients with dysarthria, named “The Mandarin Subacute Stroke Dysarthria Multimodal (MSDM) Database”, which included audio-visual data from 25 subacute stroke patients and 25 healthy participants. In addition, comprehensive subjective clinical assessment information of speech-motor function and ecological psychology of each patient was also provided. Based on this database, a pilot study was conducted to detect the significant acoustic and visual characteristics that revealed the severity of dysarthria related to subacute stroke. The present study offered a novel perspective to objectively quantify and identify the pathological differences in speech production. It can serve as a baseline for the development of an automatic intelligent system for assessing severity of dysarthria. In conclusion, the establishment and analysis of high-quality database on articulation errors associated with dysarthria will benefit clinical treatments and contribute to the realization of automatic diagnostic tools that can be implemented for clinical telehealth services.

Audio-Visual Stress Classification Using Cascaded RNN-LSTM Networks.

The purpose of this research is to emphasize the importance of mental health and contribute to the overall well-being of humankind by detecting stress. Stress is a state of strain, whether it be mental or physical. It can result from anything that frustrates, incenses, or unnerves you in an event or thinking. Your body’s response to a demand or challenge is stress. Stress affects people on a daily basis. Stress can be regarded as a hidden pandemic. Long-term (chronic) stress results in ongoing activation of the stress response, which wears down the body over time. Symptoms manifest as behavioral, emotional, and physical effects. The most common method involves administering brief self-report questionnaires such as the Perceived Stress Scale. However, self-report questionnaires frequently lack item specificity and validity, and interview-based measures can be time- and money-consuming. In this research, a novel method used to detect human mental stress by processing audio-visual data is proposed. In this paper, the focus is on understanding the use of audio-visual stress identification. Using the cascaded RNN-LSTM strategy, we achieved 91% accuracy on the RAVDESS dataset, classifying eight emotions and eventually stressed and unstressed states.

A Model of Normality Inspired Deep Learning Framework for Depression Relapse Prediction Using Audiovisual Data

Background: Depression (Major Depressive Disorder) is one of the most common mental illnesses. According to the World Health Organization, more than 300 million people in the world are affected. A first depressive episode can be solved by a spontaneous remission within 6 to 12 months. It has been shown that depression affects speech production and facial expressions. Although numerous studies are proposed in the literature for depression recognition using audiovisual cues, depression relapse using audiovisual cues has not been studied in the literature. Method: In this paper, we propose a deep learning-based approach for depression recognition and depression relapse prediction using audiovisual data. For more versatility and reusability, the proposed approach is based on a Model of Normality inspired framework where we define depression relapse by the closeness of the audiovisual patterns of a subject after a symptom-free period to the audiovisual patterns of depressed subjects. A model of Normality is an anomaly detection distance-based approach that computes a distance of normality between the deep audiovisual encoding of a test sample and a learned representation from audiovisual encodings of anomaly-free data. Results: The proposed approach shows a very promising results with an accuracy of 87.4% and a F1-score of 82.3% for relapse/depression prediction using a Leave-One-Subject-Out training strategy on the DAIC-Woz dataset. Conclusion: The proposed model of normality-based framework is accurate in detecting depression and in predicting depression relapse. A prospective monitoring system is proposed for assisting depressed patients. The proposed framework is easily extensible and others modalities will be integrated in future works.

Didaktik attention. Preschool teaching from a didaktik and variation theory perspective using examples of mathematical aspects of programming

This article is based on a partial study of an R&D programme with various theory-informed teaching arrangements aimed at describing and further developing knowledge of what characterizes preschool teaching. The aim of this study is to contribute knowledge about how teachers and children can create attention in relation to an example when teaching focuses on mathematical aspects and programming from a didaktik and variation theory perspective. The guiding question is: What can characterize attention-forming acts of teaching in preschool? The analysed material in this article consists of an example with just over ten minutes of audio-visual data – videos of teaching. An abductive analysis has been carried out. Preschool teachers’ interactions with children while teaching is studied and analysed based on didaktik questions and with a didaktik triangle as the didaktik model and based on three dimensions representations of attention: sharing, creating and alternating attention and attention-forming acts. Concepts of variation theory is connec­ted to the didaktik questions and didaktik triangle. Overall, the representations of sharing, creating and alternating attention and attention-forming acts of teaching can be summarized in the concept of didaktik attention.

Assessing endovascular team performances in a hybrid room using the Black Box system: a prospective cohort study.

The hybrid room (HR) is a complex, high-risk environment, requiring teams (surgeons, anesthesiologists, nurses, technologists) to master various skills, including the 'As Low As Reasonably Achievable' principle of radiation safety. This prospective single center cohort reports the first use of the Operating Room Black Box (ORBB) in a HR. This medical data recording system captures procedural and audio-visual data to facilitate structured team performance analysis. Patients planned for endovascular repair of an infrarenal abdominal aortic aneurysm (EVAR) or treatment of symptomatic iliac-femoral-popliteal atherosclerotic disease (Peripheral Vascular Interventions or PVI) were included. Validated measures and established assessment tools were used to assess (non-)technical skills, radiation safety performance and environmental distractions. Six EVAR and sixteen PVI procedures were captured. Technical performance for one EVAR was rated 19/35 on the procedure-specific scale, below the 'acceptable' score of 21. Technical skills were rated above acceptable in all PVI procedures. Shared decision making and leadership were rated highly in 12/22 cases, whereas surgical communication and nurses' task management were rated low in 14/22 cases. Team members rarely stepped back from the C-arm during digital subtraction angiography. Radiation safety behavior was scored below 'acceptable' in 14/22 cases. A median (interquartile range) number of 12 (6-23) auditory distractions was observed per procedure. The ORBB facilitates holistic workplace-based assessment of endovascular performance in a HR by combining objective assessment parameters and rating scale-based evaluations. Strengths and weaknesses were identified in team members' (non-)technical and radiation safety practices. This technology has the potential to improve vascular surgical practice, though human input remains crucial. (NCT04854278).

Single-layer vision transformers for more accurate early exits with less overhead

Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well as audiovisual crowd counting, we show that our method works for both classification and regression problems, and in both single- and multi-modal settings. Additionally, we introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis, that can lead to a more fine-grained dynamic inference.

Emotion Dependent Domain Adaptation for Speech Driven Affective Facial Feature Synthesis

Although speech driven facial animation has been studied extensively in the literature, works focusing on the affective content of the speech are limited. This is mostly due to the scarcity of affective audio-visual data. In this article, we improve the affective facial animation using domain adaptation by partially reducing the data scarcity. We first define a domain adaptation to map affective and neutral speech representations to a common latent space in which cross-domain bias is smaller. Then the domain adaptation is used to augment affective representations for each emotion category, including angry, disgust, fear, happy, sad, surprise, and neutral, so that we can better train emotion-dependent deep audio-to-visual (A2V) mapping models. Based on the emotion-dependent deep A2V models, the proposed affective facial synthesis system is realized in two stages: first, speech emotion recognition extracts soft emotion category likelihoods for the utterances; then a soft fusion of the emotion-dependent A2V mapping outputs form the affective facial synthesis. Experimental evaluations are performed on the SAVEE audio-visual dataset. The proposed models are assessed with objective and subjective evaluations. The proposed affective A2V system achieves significant MSE loss improvements in comparison to the recent literature. Furthermore, the resulting facial animations of the proposed system are preferred over the baseline animations in the subjective evaluations.

Subjective Evaluation of Basic Emotions from Audio–Visual Data

Understanding of the perception of emotions or affective states in humans is important to develop emotion-aware systems that work in realistic scenarios. In this paper, the perception of emotions in naturalistic human interaction (audio–visual data) is studied using perceptual evaluation. For this purpose, a naturalistic audio–visual emotion database collected from TV broadcasts such as soap-operas and movies, called the IIIT-H Audio–Visual Emotion (IIIT-H AVE) database, is used. The database consists of audio-alone, video-alone, and audio–visual data in English. Using data of all three modes, perceptual tests are conducted for four basic emotions (angry, happy, neutral, and sad) based on category labeling and for two dimensions, namely arousal (active or passive) and valence (positive or negative), based on dimensional labeling. The results indicated that the participants’ perception of emotions was remarkably different between the audio-alone, video-alone, and audio–video data. This finding emphasizes the importance of emotion-specific features compared to commonly used features in the development of emotion-aware systems.

Assessment of crowdsourced social media data and numerical modelling as complementary tools for urban flood mitigation

ABSTRACT This paper explores how crowdsourced social media data complements urban flood modelling to improve model performance and achieve a better classification of impacts. In addition to georeferencing flood impacts, Twitter allows monitoring the events in terms of hazards and impacts, and YouTube facilitates a retrospective analysis from audiovisual data. The analysis of 2800 tweets collected during four storm events and of almost 900 videos of the recent history of the basin, together with the implementation of a high-resolution model, contributed to the expansion of the capacity to represent the temporal and spatial scales of the problem. The complementation of crowdsourced social media data and urban modelling enhances the understanding of the flood dynamics, thus offering a framework of greater certainty for the generation of flood risk management products.

Raga Classification Based on Novel Method of Pitch Co-Occurrence

Automatic identification of raga is a growing research area and has captured significant attention from movie making industry. It is the need of time to develop efficient tools for data mining the vast audio visual data on internet. In particular, to search for a specific raga. Applications of raga search are in musicological studies, similarity based search. Ascending and descending pattern of swaras is an important feature in the raga classification. Pitch tracks of swaras are obtained from raw audio recordings. This research has utilised the pattern developed due to co-occurrence of pitches of swaras for classification. This pattern gives a concise representation of the signal which contains time and frequency information of the raga. K Nearest Neighbour (KNN) has been used as the classifier.

Technological affordances versus narrative delivery?: the practice of recent virtual reality storytelling

ABSTRACT Subsequent to virtual reality (VR) technologies becoming cheaper and more accessible, a new generation of practitioners have focused on the affordances and opportunities that VR can offer to storytelling. A recurring concern is how to understand the value of VR’s interactivity and how its additional, spatialised, audiovisual data can aid storytelling rather than provide surplus distraction. By focusing on three specific areas – interactive affordance; authorial and user agency; and first and third-person perspectives – this article outlines how practice in the field of VR production can lead to inductive responses to these issues, as well as their relationship with the historical questions that are raised around the implementation of new audiovisual technologies. Through a discussion of a recent VR project, Embodied, this article considers how VR technology and content can combine to explore new interactive narrative possibilities.

CREATIVE PROCESSES AND EMOTIONS IN COVID-19 PANDEMIC

The objective was to analyse, from the perspective of the participants, creative processes developed in isolation and social distancing by COVID-19 pandemic. The main interest was to study transformations in daily, family and work life, focusing on the creation of new practices and projects. Likewise, it was relevant to study emotions, obstacles and difficulties perceived by the participants. The study was qualitative, the participants (N = 25) were selected by non-probabilistic convenience sampling. All the participants lived in intermediate or small towns in Argentina. Data collection was carried out in September, 2020. Semi-structured interviews and online qualitative research were the data collection instruments. The Atlas.ti 8 program was used for the qualitative analysis of textual and audio-visual data. The results indicated significant modifications in the lives of the participants from isolation and distancing. These changes demand flexibility and adaptability as well as creative processes for generating alternatives and solving problems. In the context of the home, transformations of times, spaces and relationships were observed. Work and leisure activities also were reconfigured. Sadness appeared as the predominant emotion. However, some participants experienced enthusiasm and joy for the new projects. The purpose of this study was to build knowledge that contributes to the design of health promotion project.

Investigating New Sensory Methods Related to Taste Sensitivity, Preferences, and Diet of Mother-Infant Pairs and Their Relationship With Body Composition and Biomarkers: Protocol for an Explorative Study.

BackgroundEarly experiences with different flavors play an important role in infant development, including food and taste acceptance. Flavors are already perceived in utero with the development of the taste and olfactory system and are passed on to the child through breast and bottle feeding. Therefore, the first 1000 days of life are considered a critical window for infant developmental programming.ObjectiveThe objective of our study is to investigate, both in the prenatal and postnatal period, taste sensitivity, preferences, and dietary diversity of mother-infant pairs. The explorative study design will also report on the impact of these variables on body composition (BC) and biomarkers. In contrast to conventional methods, this study involves long-term follow-up data collection from mother-infant pairs; moreover, the integration of audiovisual tools for recording infants' expressions pertaining to taste stimuli is a novelty of this study. Considering these new methodological approaches, the study aims to assess taste-related data in conjunction with BC parameters like fat-free mass or fat mass, biomarkers, and nutritional intake in infants and children.MethodsHealthy pregnant women aged between 18 and 50 years (BMI≥18.5 kg/m2 to ≤30 kg/m2; <28 weeks of gestation) were recruited from January 2014 to October 2014. The explorative design implies 2 center visits during pregnancy (24-28 weeks of gestation and 32-34 weeks of gestation) and 2 center visits after delivery (6-8 weeks postpartum and 14-16 weeks postpartum) as well as follow-up visits at 1, 3-3.5, and 6 years after delivery. Data collection encompasses anthropometric and biochemical measurements as well as BC analyses with air displacement plethysmography, taste perception assessments, and multicomponent questionnaires on demographics, feeding practices, and nutritional and lifestyle behaviors. Audiovisual data from infants’ reactions to sensory stimuli are collected and coded by trained staff using Baby Facial Action Coding and the Body Action Posture System. Birth outcomes and weight development are obtained from medical records, and additional qualitative data are gathered from 24 semistructured interviews.ResultsOur cohort represents a homogenous group of healthy women with stringent exclusion criteria. A total of 54 women met the eligibility criteria, whereas 47 mother-child pairs completed data collection at 4 center visits during and after pregnancy. Follow-up phases, data analyses, and dissemination of the findings are scheduled for the end of 2023. The study was approved by the ethics committee of the Medical University of Graz (EC No 26–066 ex 13/14), and all participants provided informed consent.ConclusionsThe results of this study could be useful for elucidating the connections between maternal and infant statuses regarding diet, taste, biomarkers, and prenatal and postnatal weight development. This study may also be relevant to the establishment of further diagnostic and interventional strategies targeting childhood obesity and early body fat development.International Registered Report Identifier (IRRID)DERR1-10.2196/37279

Deducing health cues from biometric data

Medical diagnosis involves the expert opinion of trained health care professionals based on causal inference from medical data. While medical data are typically collected using specialized medical-grade sensors, similar data characteristics useful for medical diagnosis are sometimes present in biometric data (e.g., face images, ocular images, and speech signals). In this paper, we explore the biometrics and medical literature to study the following questions. 1) What kind of health cues are embedded in the commonly utilized forms of audio-visual biometric data? 2) How can these health cues be gleaned from the biometric data, and what kind of diseases can it help diagnose? 3) What are some of the implications of using biometric data for medical diagnosis?

It\u2019s time for a minimum synoptic operation template in patients undergoing laparoscopic cholecystectomy: a systematic review

BackgroundDespite the call to enhance accuracy and value of operation records few international recommended minimal standards for operative notes documentation have been described. This study undertook a systematic review of existing operative reporting systems for laparoscopic cholecystectomy (LC) to fashion a comprehensive, synoptic operative reporting template for the future.MethodsA search for all relevant articles was conducted using PubMed version of Medline, Scopus and Web of Science databases in June 2021, for publications from January 1st 2011 to October 25th 2021, using the keywords: laparoscopic cholecystectomy AND operation notes OR operative notes OR proforma OR documentation OR report OR narrative OR audio-visual OR synoptic OR digital. Two reviewers (NOC, GMC) independently assessed each published study using a MINORS score of ≥ 16 for comparative and ≥ 10 for non-comparative for inclusion. This systematic review followed PRISMA guidelines and was registered with PROSPERO. Synoptic operative templates from published data were assimilated into one “ideal” laparoscopic operative report template following international input from the World Society of Emergency Surgery board.ResultsA total of 3567 articles were reviewed. Following MINORS grading 25 studies were selected spanning 14 countries and 4 continents. Twenty-two studies were prospective. A holistic overview of the operative procedure documentation was reported in 6/25 studies and a further 19 papers dealt with selective surgical aspects of LC. A unique synoptic LC operative reporting template was developed and translated into Chinese/Mandarin, French and Arabic.ConclusionThis systematic review identified a paucity of publications dealing with operative reporting of LC. The proposed new template may be integrated digitally with hospitals’ medical systems and include additional narrative text and audio-visual data. The template may help define new OR (operating room) recording standards and impact on care for patients undergoing LC.

A study of the interactions and conversations of families visiting the museum of microbiology of the Butantan institute, São Paulo, Brazil

ABSTRACT Science museums are key locations for informal education. They also enable research that explores how families structure their activities and conversations and how these contribute to learning experiences. The aim of this exploratory study, using qualitative and quantitative methodologies, was to analyse the process of family interactions during visits to a Brazilian science museum focused on microbiology and how their conversations mediated their learning experiences. Four groups, each consisting of a single family with children, participated in the study. Their visits were recorded using a subjective camera, and the audiovisual data were analysed regarding types of interactions and conversational content, based on an analysis protocol. The results reveal that the interactions and social dynamics that family groups engaged in during their visit provided cognitive and social learning experiences. The study also offers evidence on the role of children, the importance of conversations with adults during the visits, and the relevance of objects on exhibition in promoting learning conversations.