Audiovisual Cues Research Articles

Deep learning for depression recognition with audiovisual cues: A review

With the acceleration of the pace of work and life, people are facing more and more pressure, which increases the probability of suffering from depression. However, many patients may fail to get a timely diagnosis due to the serious imbalance in the doctor–patient ratio in the world. A promising development is that physiological and psychological studies have found some differences in speech and facial expression between patients with depression and healthy individuals. Consequently, to improve current medical care, Deep Learning (DL) has been used to extract a representation of depression cues from audio and video for automatic depression detection. To classify and summarize such research, we introduce the databases and describe objective markers for automatic depression estimation. We also review the DL methods for automatic detection of depression to extract a representation of depression from audio and video. Lastly, we discuss challenges and promising directions related to the automatic diagnoses of depression using DL.

Exploring developmental changes in infant anticipation and perceptual processing: EEG responses to tactile stimulation.

There is an increasing interest in alpha-range rhythms in the electroencephalogram (EEG) in relation to perceptual and attentional processes. The infant mu rhythm has been extensively studied in the context of linkages between action observation and action production in infancy, but less is known about the mu rhythm in relation to cross-modal processes involving somatosensation. We investigated differences in mu responses to cued vibrotactile stimulation of the hand in two age groups of infants: From 6 to 7months and 13 to 14months. We were also interested in anticipatory neural responses in the alpha frequency range prior to tactile stimulation. Tactile stimulation of infants' left or right hand was preceded by an audiovisual cue signaling which hand would be stimulated. In response to the tactile stimulus, infants demonstrated significant mu desynchronization over the central areas contralateral to the hand stimulated, with higher mu peak frequency and greater contralateral mu desynchronization for older infants. Prior to the tactile stimulus, both age groups showed significant bilateral alpha desynchronization over frontocentral sites, which may be indicative of generalized anticipation of an upcoming stimulus. The findings highlight the potential of examining the sensorimotor mu rhythm in the context of infant attentional development.

Using Optogenetics to Reverse Neuroplasticity and Inhibit Cocaine Seeking in Rats

This protocol demonstrates the steps needed to use optogenetic tools to reverse cocaine-induced plasticity at thalamo-amygdala circuits to reduce subsequent cocaine seeking behaviors in the rat. In our research, we had found that when rats self-administer intravenous cocaine paired with an audiovisual cue, synapses formed at inputs from the medial geniculate nucleus of the thalamus (MGN) onto principal neurons of the lateral amygdala (LA) become stronger as the cue-cocaine association is learned. We hypothesized that reversal of the cocaine-induced plasticity at these synapses would reduce cue-motivated cocaine seeking behavior. In order to accomplish this type of neuromodulation in vivo, we wanted to induce synaptic long-term depression (LTD), which decreases the strength of MGN-LA synapses. To this end, we used optogenetics, which allows neuromodulation of brain circuits using light. The excitatory opsin oChiEF was expressed on presynaptic MGN terminals in the LA by infusing an AAV containing oChiEF into the MGN. Optical fibers were then implanted in the LA and 473 nm laser light was pulsed at a frequency of 1 Hz for 15 minutes to induce LTD and reverse cocaine induced plasticity. This manipulation produces a long-lasting reduction in the ability of cues associated with cocaine to induce drug seeking actions.

Using Optogenetics to Reverse Neuroplasticity and Inhibit Cocaine Seeking in Rats

This protocol demonstrates the steps needed to use optogenetic tools to reverse cocaine-induced plasticity at thalamo-amygdala circuits to reduce subsequent cocaine seeking behaviors in the rat. In our research, we had found that when rats self-administer intravenous cocaine paired with an audiovisual cue, synapses formed at inputs from the medial geniculate nucleus of the thalamus (MGN) onto principal neurons of the lateral amygdala (LA) become stronger as the cue-cocaine association is learned. We hypothesized that reversal of the cocaine-induced plasticity at these synapses would reduce cue-motivated cocaine seeking behavior. In order to accomplish this type of neuromodulation in vivo, we wanted to induce synaptic long-term depression (LTD), which decreases the strength of MGN-LA synapses. To this end, we used optogenetics, which allows neuromodulation of brain circuits using light. The excitatory opsin oChiEF was expressed on presynaptic MGN terminals in the LA by infusing an AAV containing oChiEF into the MGN. Optical fibers were then implanted in the LA and 473 nm laser light was pulsed at a frequency of 1 Hz for 15 minutes to induce LTD and reverse cocaine induced plasticity. This manipulation produces a long-lasting reduction in the ability of cues associated with cocaine to induce drug seeking actions.

Nicotine self-administration with menthol and audiovisual cue facilitates differential packaging of CYP2A6 and cytokines/chemokines in rat plasma extracellular vesicles

In this study, we investigated whether intravenously self-administered nicotine with menthol and audiovisual cue modulates nicotine-metabolizing CYP2A6, oxidative stress modulators, and cytokines/chemokines in plasma extracellular vesicles (EVs) in rats. We assigned rats to self-administered nicotine with: (a) audiovisual cue (AV), (b) menthol, and (c) menthol and AV cue. We found increased levels of CD9 in plasma EVs after self-administered nicotine with menthol and AV cue. Moreover, expression of CYP2A6 in plasma EVs was significantly increased after self-administered nicotine in response to menthol and AV cue. However, despite an upward trend on SOD1 and catalase, increase was not found to be statistically significant, while total antioxidant capacity was found to be significantly increased in plasma and plasma EVs obtained after self-administered nicotine with menthol and AV cue. Among cytokine and chemokine profiling, we found a significant increase in the levels of MCP-1 after self-administered nicotine with menthol and AV cue and complete packaging of IL-1β in EVs. Taken together, the study provides evidence that nicotine in response to menthol and AV cues can package altered levels of CYP2A6, and cytokines/chemokines in plasma EVs that may contribute to cell–cell communication, nicotine metabolism, and inflammation upon cigarette smoking.

Multisensory-Guided Associative Learning Enhances Multisensory Representation in Primary Auditory Cortex.

Sensory cortices, classically considered to represent modality-specific sensory information, are also found to engage in multisensory processing. However, how sensory processing in sensory cortices is cross-modally modulated remains an open question. Specifically, we understand little of cross-modal representation in sensory cortices in perceptual tasks and how perceptual learning modifies this process. Here, we recorded neural responses in primary auditory cortex (A1) both while freely moving rats discriminated stimuli in Go/No-Go tasks and when anesthetized. Our data show that cross-modal representation in auditory cortices varies with task contexts. In the task of an audiovisual cue being the target associating with water reward, a significantly higher proportion of auditory neurons showed a visually evoked response. The vast majority of auditory neurons, if processing auditory-visual interactions, exhibit significant multisensory enhancement. However, when the rats performed tasks with unisensory cues being the target, cross-modal inhibition, rather than enhancement, predominated. In addition, multisensory associational learning appeared to leave a trace of plastic change in A1, as a larger proportion of A1 neurons showed multisensory enhancement in anesthesia. These findings indicate that multisensory processing in principle sensory cortices is not static, and having cross-modal interaction in the task requirement can substantially enhance multisensory processing in sensory cortices.

Posttraumatic Stress Disorder Symptoms and Coping Motives are Independently Associated with Cannabis Craving Elicited by Trauma Cues.

Cannabis use is common among individuals with posttraumatic stress disorder (PTSD), although its use can ultimately worsen PTSD outcomes. Cannabis-use coping motives may help explain the PTSD-cannabis relationship. Frequent pairing of trauma cues with substance use to cope with negative affect can lead to conditioned substance craving. For the present cue-reactivity study, we examined if PTSD symptoms were associated with cannabis craving elicited by a personalized trauma cue and explored whether coping motives mediated this hypothesized relationship; enhancement motives were included as a comparison mediator. Participants (N = 51) were trauma-exposed cannabis users who completed validated assessments on PTSD symptom severity and cannabis use motives. They were then exposed to a personalized audiovisual cue based on their own traumatic experience after which they responded to questions on a standardized measure regarding their cannabis craving. The results demonstrated that PTSD symptoms were associated with increased cannabis craving following trauma cue exposure, B = 0.43, p = .004, 95% CI [0.14, 0.72]. However, the results did not support our hypothesis of an indirect effect through general coping motives, indirect effect = .03, SE = .08, 95% CI [-.10, .21]. We found an independent main effect of general coping motives on trauma cue-elicited cannabis craving, B = 1.86, p = .002, 95% CI [0.72, 3.01]. These findings have important clinical implications suggesting that clinicians should target both PTSD symptoms and general coping motives to prevent the development of conditioned cannabis craving to trauma reminders in trauma-exposed cannabis users.

A Vision-Based Social Distancing and Critical Density Detection System for COVID-19.

Social distancing (SD) is an effective measure to prevent the spread of the infectious Coronavirus Disease 2019 (COVID-19). However, a lack of spatial awareness may cause unintentional violations of this new measure. Against this backdrop, we propose an active surveillance system to slow the spread of COVID-19 by warning individuals in a region-of-interest. Our contribution is twofold. First, we introduce a vision-based real-time system that can detect SD violations and send non-intrusive audio-visual cues using state-of-the-art deep-learning models. Second, we define a novel critical social density value and show that the chance of SD violation occurrence can be held near zero if the pedestrian density is kept under this value. The proposed system is also ethically fair: it does not record data nor target individuals, and no human supervisor is present during the operation. The proposed system was evaluated across real-world datasets.

Sign tracking predicts suboptimal behavior in a rodent gambling task.

Reward-associated cues can promote maladaptive behavior, including risky decision-making in a gambling setting. A propensity for sign tracking over goal tracking-i.e., interaction with a reward-predictive cue rather than the site of reward-demonstrates an individual's tendency to transfer motivational value to a cue. However, the relationship of sign tracking to risky decision-making remains unclear. To determine whether sign tracking predicts risky choice, we used a Pavlovian conditioned approach task to evaluate the tendency of male rats to sign track to a lever cue and then trained rats on a rodent gambling task (rGT) with win-associated cues. We also tested the effects of D-amphetamine, quinpirole (a D2/D3 receptor agonist), and PD128907 (a D3 receptor agonist) on gambling behavior in sign tracker and goal tracker individuals. Increased sign tracking relative to goal tracking was associated with suboptimal performance on the rGT, including decreased selection of the optimal choice, increased selection of a high-risk/high-reward option, and increased impulsive premature choices. Amphetamine increased choices of a low-risk/low-reward option at the expense of optimal and high-risk choices, whereas quinpirole and PD128907 had little effect on choice allocation, but reduced impulsivity. Drug effects were similar across sign tracker and goal tracker individuals. Cue reactivity, as measured by sign tracking, is predictive and may be an important driver of risky and impulsive choices in a gambling setting laden with salient audiovisual cues. Evaluating an individual's sign tracking behavior may be an avenue to predict vulnerability to pathological gambling and the efficacy of treatments.

Directing attention to event changes improves memory updating for older adults.

People use memory for observed actions to guide current perceptions. When actions change from one situation to the next, one must register the change to update memory. Research suggests that older adults may sometimes update memory for naturalistic action changes less effectively than younger adults. We examined whether this deficit reflects age differences in attention allocation by cuing attention to changed action features and testing memory for those features. Older (N = 47) and younger (N = 73) adults watched movies of an actor performing everyday activities on two fictive "days" in her life. Some activities began identically on both days (e.g., reaching for dessert) and ended with features that changed across days (e.g., cookie vs. brownie). Half of the changed activities included audio-visual cues on both days that signaled changed features, whereas the other half did not include cues. Memory updating was assessed through cued recall and two-alternative forced choice recognition (2AFC recognition) of recent action features. Cuing attention improved cued recall but not 2AFC recognition of recent action features for both older and younger adults. These recall benefits were associated with improved recollection that changes had earlier occurred. The present findings suggest that although older adults sometimes experience deficits in aspects of attention, using cues to guide their attention to features of everyday activities can enhance their event memory updating when the later memory test emphasizes recollection-based retrieval. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Lip-Reading Driven Deep Learning Approach for Speech Enhancement

This paper proposes a novel lip-reading driven deep learning framework for speech enhancement. The approach leverages the complementary strengths of both deep learning and analytical acoustic modeling (filtering-based approach) as compared to benchmark approaches that rely only on deep learning. The proposed audio-visual (AV) speech enhancement framework operates at two levels. In the first level, a novel deep learning based lip-reading regression model is employed. In the second level, lip-reading approximated clean-audio features are exploited, using an enhanced, visually-derived Wiener filter (EVWF), for estimating the clean audio power spectrum. Specifically, a stacked long-short-term memory (LSTM) based lip-reading regression model is designed for estimating the clean audio features using only temporal visual features (i.e., lip reading), by considering a range of prior visual frames. For clean speech spectrum estimation, a new filterbank-domain EVWF is formulated, which exploits the estimated speech features. The EVWF is compared with conventional spectral subtraction and log-minimum mean-square error methods using both ideal AV mapping and LSTM driven AV mapping approaches. The potential of the proposed AV speech enhancement framework is evaluated under four different dynamic real-world scenarios [cafe, street junction, public transport, and pedestrian area] at different SNR levels (ranging from low to high SNRs) using benchmark grid and ChiME3 corpora. For objective testing, perceptual evaluation of speech quality is used to evaluate the quality of restored speech. For subjective testing, the standard mean-opinion-score method is used with inferential statistics. Comparative simulation results demonstrate significant lip-reading and speech enhancement improvements in terms of both speech quality and speech intelligibility. Ongoing work is aimed at enhancing the accuracy and generalization capability of the deep learning driven lip-reading model, using contextual integration of AV cues, leading to context-aware, autonomous AV speech enhancement.

Application of Immersive Virtual Reality to Pragmatics Data Collection Methods

This exploratory study investigated the usability of immersive virtual reality (VR) as a way of creating a role-play task to examine pragmatic competence, specifically the ability to produce the speech act of request. The study created a closed role-play task in two versions. One was a standard computer-based version in which participants read a written scenario displayed on the screen and produced the target speech act for the computer. The other one was a VR version in which participants put on a VR headset and produced the target speech act for the interlocutor in the virtual space. Five native and five non-native speakers of English completed both versions and participated in a follow-up interview. The purpose of the interview was to examine similarities and differences in participants' perceptions of the two role-play tasks in four areas: (1) thought processes (what they were thinking during the task), (2) recall (what they remembered about the task), (3) difficulty (what made the task difficult), and (4) enjoyment (whether the task was fun). Results revealed that the participants attended to various audio-visual cues in the VR scene and used them to guide their actions. The VR version also evoked greater emotional reactions from the participants.

Exogenous Bimodal Cues Attenuate Age-Related Audiovisual Integration.

Previous studies have demonstrated that exogenous attention decreases audiovisual integration (AVI); however, whether the AVI is different when exogenous attention is elicited by bimodal and unimodal cues and its aging effect remain unclear. To clarify this matter, 20 older adults and 20 younger adults were recruited to conduct an auditory/visual discrimination task following bimodal audiovisual cues or unimodal auditory/visual cues. The results showed that the response to all stimulus types was faster in younger adults compared with older adults, and the response was faster when responding to audiovisual stimuli compared with auditory or visual stimuli. Analysis using the race model revealed that the AVI was lower in the exogenous-cue conditions compared with the no-cue condition for both older and younger adults. The AVI was observed in all exogenous-cue conditions for the younger adults (visual cue > auditory cue > audiovisual cue); however, for older adults, the AVI was only found in the visual-cue condition. In addition, the AVI was lower in older adults compared to younger adults under no- and visual-cue conditions. These results suggested that exogenous attention decreased the AVI, and the AVI was lower in exogenous attention elicited by bimodal-cue than by unimodal-cue conditions. In addition, the AVI was reduced for older adults compared with younger adults under exogenous attention.

Keeping in time with social and non-social stimuli: Synchronisation with auditory, visual, and audio-visual cues

Everyday social interactions require us to closely monitor, predict, and synchronise our movements with those of an interacting partner. Experimental studies of social synchrony typically examine the social-cognitive outcomes associated with synchrony, such as affiliation. On the other hand, research on the sensorimotor aspects of synchronisation generally uses non-social stimuli (e.g. a moving dot). To date, the differences in sensorimotor aspects of synchronisation to social compared to non-social stimuli remain largely unknown. The present study aims to address this gap using a verbal response paradigm where participants were asked to synchronise a ‘ba’ response in time with social and non-social stimuli, which were presented auditorily, visually, or audio-visually combined. For social stimuli a video/audio recording of an actor performing the same verbal ‘ba’ response was presented, whereas for non-social stimuli a moving dot, an auditory metronome or both combined were presented. The impact of autistic traits on participants’ synchronisation performance was examined using the Autism Spectrum Quotient (AQ). Our results revealed more accurate synchronisation for social compared to non-social stimuli, suggesting that greater familiarity with and motivation in attending to social stimuli may enhance our ability to better predict and synchronise with them. Individuals with fewer autistic traits demonstrated greater social learning, as indexed through an improvement in synchronisation performance to social vs non-social stimuli across the experiment.

Individuals With Mild Cognitive Impairment and Alzheimer's Disease Benefit From Audiovisual Speech Cues and Supportive Sentence Context

Purpose Speech perception in noise becomes difficult with age but can be facilitated by audiovisual (AV) speech cues and sentence context in healthy older adults. However, individuals with Alzheimer's disease (AD) may present with deficits in AV integration, potentially limiting the extent to which they can benefit from AV cues. This study investigated the benefit of these cues in individuals with mild cognitive impairment (MCI), individuals with AD, and healthy older adult controls. Method This study compared auditory-only and AV speech perception of sentences presented in noise. These sentences had one of two levels of context: high (e.g., "Stir your coffee with a spoon") and low (e.g., "Bob didn't think about the spoon"). Fourteen older controls (M age = 72.71 years, SD = 9.39), 13 individuals with MCI (M age = 79.92 years, SD = 5.52), and nine individuals with probable Alzheimer's-type dementia (M age = 79.38 years, SD = 3.40) completed the speech perception task and were asked to repeat the terminal word of each sentence. Results All three groups benefited (i.e., identified more terminal words) from AV and sentence context. Individuals with MCI showed a smaller AV benefit compared to controls in low-context conditions, suggesting difficulties with AV integration. Individuals with AD showed a smaller benefit in high-context conditions compared to controls, indicating difficulties with AV integration and context use in AD. Conclusions Individuals with MCI and individuals with AD do benefit from AV speech and semantic context during speech perception in noise (albeit to a lower extent than healthy older adults). This suggests that engaging in face-to-face communication and providing ample context will likely foster more effective communication between patients and caregivers, professionals, and loved ones.

Social facilitation for conservation planning: understanding fairy tern behavior and site selection in response to conspecific audio-visual cues

Simulated social facilitation techniques (e.g. decoys and call playbacks) are commonly used to attract seabirds to restored and artificially created nesting habitats. However, a lack of social stimuli and conspecific cueing at these habitats may limit the use of these sites, at least in the short term. Therefore, testing the effectiveness of simulated audio-visual cues for attracting gregarious birds is important for conservation planning. In this study, we (1) assessed whether call playback and decoys were associated with an increased likelihood of Australian fairy terns Sternula nereis nereis visiting potentially suitable nesting habitats; (2) tested their behavioral response to different cues; and (3) documented whether social facilitation had the potential to encourage colony establishment. A full cross-over study design consisting of all possible pairings of decoy and call playback treatments (control [no attractants], decoys, call playback, both decoys and playback), allocated as part of a random block design, was undertaken at 2 sites. Linear modeling suggested that call playback was important in explaining the time spent aerial prospecting as well as the maximum number of fairy terns aerial prospecting, although this only appeared to be the case for 1 of the 2 sites. Decoys, on the other hand, did not appear to have any effect on time spent aerial prospecting. The results from this study suggest that audio cues have the potential to encourage site selection by increasing social stimuli, but attractants may be required over several breeding seasons before colonies are established.

The Effectiveness of the Interaction of Interactive Book Cues and Levels of Information Processing on Learning Retention and External Cognitive Load

This study explores interactive e-book cues and Information Processing Levels (IPL)’s effectiveness on Learning Retention (LR) and External Cognitive Load (ECL). 117 middle school pupils (MSP) were divided into six experimental groups based on their IPL and cues during the second term of the academic year 2019–2020. Visual Cues (VC)/Audiovisual Cues (VAC) and Auditory Cues (AC)/Audiovisual Cues (VAC) statistically varied in the Ie-book in LR test and ECL scale, same for the average scores when testing the LR in Science for MSP due to the difference between IPL for the DL. There is a statistically significant effect of cue types' interaction in Ie-book with IPL in ECL scale for MSP, at its highest peak in the case of the AVC with DL, followed by the interaction resulting from the VC with DL then AC with SL. Also, cues interaction in Ie-book with IPL immensely affect the LR test for MEP, which is at its highest peak in the case of the AVC with DL. The interactions between (DL–SL) and (AC–VC) seem to equally influence the ELC

Audiovisual integration in macaque face patch neurons

SummaryPrimate social communication depends on the perceptual integration of visual and auditory cues, reflected in the multimodal mixing of sensory signals in certain cortical areas. The macaque cortical face patch network, identified through visual, face-selective responses measured with fMRI, is assumed to contribute to visual social interactions. However, whether face patch neurons are also influenced by acoustic information, such as the auditory component of a natural vocalization, remains unknown. Here, we recorded single-unit activity in the anterior fundus (AF) face patch, in the superior temporal sulcus, and anterior medial (AM) face patch, on the undersurface of the temporal lobe, in macaques presented with audiovisual, visual-only, and auditory-only renditions of natural movies of macaques vocalizing. The results revealed that 76% of neurons in face patch AF were significantly influenced by the auditory component of the movie, most often through enhancement of visual responses but sometimes in response to the auditory stimulus alone. By contrast, few neurons in face patch AM exhibited significant auditory responses or modulation. Control experiments in AF used an animated macaque avatar to demonstrate, first, that the structural elements of the face were often essential for audiovisual modulation and, second, that the temporal modulation of the acoustic stimulus was more important than its frequency spectrum. Together, these results identify a striking contrast between two face patches and specifically identify AF as playing a potential role in the integration of audiovisual cues during natural modes of social communication.

Dopamine neurons gate the intersection of cocaine use, decision making, and impulsivity.

Gambling and substance use disorders are highly comorbid. Both clinical populations are impulsive and exhibit risky decision-making. Drug-associated cues have long been known to facilitate habitual drug-seeking, and the salient audiovisual cues embedded within modern gambling products may likewise encourage problem gambling. The dopamine neurons of the ventral tegmental area (VTA) are exquisitely sensitive to drugs of abuse, uncertain rewards, and reward-paired cues and may therefore be the common neural substrate mediating synergistic features of both disorders. To test this hypothesis, we first gained specific inhibitory control over VTA dopamine neurons by transducing a floxed inhibitory DREADD (AAV5-hSyn-DIO-hM4D(Gi)-mCherry) in rats expressing Cre recombinase in tyrosine hydroxylase neurons. We then trained rats in our cued rat gambling task (crGT), inhibiting dopamine neurons throughout task acquisition and performance, before allowing them to self-administer cocaine in the same diurnal period as crGT sessions. The trajectories of addiction differ in women and men, and the dopamine system may differ functionally across the sexes; therefore, we used male and female rats here. We found that inhibition of VTA dopamine neurons decreased cue-induced risky choice and reduced motor impulsivity in males, but surprisingly, enhanced risky decision making in females. Inhibiting VTA dopamine neurons also prevented cocaine-induced changes in decision making in both sexes, but nevertheless drove all animals to consume more cocaine. These findings show that chronic dampening of dopamine signalling can have both protective and deleterious effects on addiction-relevant behaviours, depending on biological sex and dependent variable of interest.

Recognition of valence using QRS complex in children with Autism Spectrum Disorder (ASD)

In children with a diagnosis of autism spectrum disorder (ASD), the absence of emotional inexpressiveness continues to be a consistent issue. This causes them to an unexpected emotional outbursts and meltdowns. This study utilizes the QRS complex derived from the Electrocardiogram (ECG) signals to investigate the positive (“Like”) and negative (“Dislike”) valence using a personalized emotion elicitation protocol using audio- and audio-visual cues in children who have ASD under the ages between 5 and 11 years. The sample size consisted of 15 controls and 15 children who have ASD.The acquired raw ECG signals are cleaned using various digital filters and the useful valence specific features are extracted from the QRS complex and are classified using the K nearest neighbour (KNN) and Ensemble learner. The control children were able to exhibit largely to the valence states, in contrast the children who have ASD did not show much difference in their valence states which was also verified using the Ensemble learner which achieved a maximum mean accuracy of 75.5 % in controls and 70.5% in children who have ASD.