Audiovisual Memory Research Articles

DialogMCF: Multimodal Context Flow for Audio Visual Scene-Aware Dialog

In recent years, Audio Visual Scene-Aware Dialog (AVSD) has been an active research task in the multimodal dialogue community and has also been a core part of the Dialog System Technology Challenge (DSTC). This task is an extension of conventional visual question answering, where video-relevant answers must be generated taking into account multimodal contextual information from previous dialogue rounds. Despite recent advances in the AVSD task, there are still two major challenges in developing such a system: how to model the multimodal contextual information of multiple rounds of dialogues and how to integrate audio-visual information into the generation of textual responses. To tackle these two challenges, in this paper we propose a novel model, named DialogMCF, which constructs a multimodal context flow model to generate responses that are relevant to video scenes. This proposed context flow modeling can track the dynamics of the topic information across multiple rounds of dialogue history. To achieve an effective fusion of multimodal information, we propose an audio-visual memory network with cross-modality aligned features to model long multimodal dialogue context, and thus enhance the flow modeling. Furthermore, we make attempts to improve the performance of the proposed DialogMCF model with manual descriptions and explore the incorporation of temporal reasoning information. Extensive experiments on the DSTC AVSD datasets show that, compared to a range of baseline methods, the proposed method can yield state-of-art dialogue generation performance on most metrics when integrating the video descriptions.

Selective short-term memory impairment for verbalizable visual objects in children with Developmental Language Disorder

BackgroundDevelopmental language disorder (DLD) affects the ability to acquire and make use of native language. Possible underlying cognitive mechanisms are related to memory functions. AimsThe aim was examination of the relationship between visual short-term memory of objects as well as audiovisual short-term memory, and particular nonverbal and language abilities. Methods and proceduresThe study included 7–9-year-old children with DLD and matched control group. Participants completed the Language Development Test, the Stanford-Binet IQ scale (SB5), and two short-term memory tasks: immediate recall of the visually presented pictograms and immediate recall of audiovisually presented sequences of syllables. Outcomes and resultsThe results revealed diminished levels of short-term visual memory for objects as well as audiovisual memory in children with DLD. However, there were no group differences in the control task of WM. Conclusions and implicationsResults supported the idea of diminished abilities in children with DLD to perform mental operations on verbalizable visual objects. Importantly non-verbal working memory ability, which cannot easily be supported by verbal representations, is at typical levels. This suggests that verbalization ability should be taken into account in the assessment of seemingly non-verbal cognitive functions among children with DLD.

Digitisation of films and selection of films for digitisation from the perspective of users

Purpose. The selection of films that archivists carry out today has an important impact on the future configuration of digitalised and virtually accessible film collections. Given the responsibility that this task entails, we consider it interesting to explore what the different profiles of the users of film archives think about selection and digitisation of films. Approach/Methodology/Design. To this end, structured interviews were conducted with 31 subjects belonging to four groups of users: film researchers, film professionals, journalists/ film critics, and last year audio-visual communication undergraduates from the Universityof Seville. Findings. The most notable results, which do not differ in essence from what is known in the archival field, include, on the one hand, the unanimous view that digitisation is an effective way of conserving audio-visual memory and fostering public interest in film collections and archives and, on the other, that when films have to be selected for their digitisation, this should be done following rigorous and systematic selection criteria, thus avoiding the detrimental effects of random methods. Lastly, some future research lines are offered. Originality/Value. All the bibliography consulted view topics relating to film selection and digitisation exclusively through the prism of archival and library and information sciences. Namely, these studies are restricted to these specific areas of knowledge and aimed at professional and experts in the field. For that reason, it may be interesting to explore other viewpoints and perspectives that go beyond purely technical, theoretical or methodological approach to film curation.

Psychodynamic-based virtual reality cognitive training system with personalized emotional arousal elements for mild cognitive impairment patients

Background and objectiveMild cognitive impairment (MCI) is a serious threat to the physical health and quality of life of the elderly, as well as a heavy burden on families and society. The current computer-based rehabilitation training ignores the role of emotions in cognitive impairment rehabilitation, making it difficult to improve patient engagement and efficiency. To address this, a psychodynamics-based cognitive rehabilitation training method with personalized emotional arousal elements was proposed using virtual reality technology. MethodsOur proposed method contains four training tasks, which cover (audiovisual memory, attention & processing, working memory, abstract & Logic, spatial pathfinding) and six positive emotional arousal elements (sensory feedback, achievement system, multiplayer interaction, score comparison, relaxation scenarios, and peaceful videos) to motivate participants to persist during cognitive training continuously and maintain a positive mental attitude toward training. The six emotional arousal elements were divided into two personalized combinations—full combination and half combination—based on the results of the pre-assessment and were dynamically distributed throughout both the training tasks and post-training. ResultsFifteen participants with MCI were recruited to complete the proposed experiment and validate the effectiveness of the system. They were first asked to complete two assessments (e.g., the big five scale and the positive and negative affect scale) to investigate their personalities. Based on the results of the assessments, they were provided with a full or half combination of arousal elements in the training tasks and post-training. Finally, the acceptability of the system and task experience were assessed using questionnaires. Notably, there was a significant increase in training scores for participants who completed a six-week training period (66.7%, 33.4%, and 25.0% for attention and processing, working memory, and abstraction and logic, respectively). The results show that positive emotional arousal had a positive effect on the MCI participants. The training tasks and arousal elements can improve cognitive function and enhance the confidence and engagement of participants. There were no significant differences in cognitive domain training scores between the two groups. ConclusionsThis personalized cognitive training system has the potential to serve as a convenient solution for complementary treatment of MCI.

Influence of gaze cues on audiovisual memory: Study of event-related potentials

Influence of gaze cues on audiovisual memory: Study of event-related potentials

Recreated Memories

“Recreating Memories” is one of the shortest and most challenging videos I’ve ever made. Drawing on Agnes Varda’s Les plages D’agnès (2008) and the given written text, I engage in an associative audiovisual memory exploration.

Selective Compassion: Public Memory, Persona, and “The Early Years” of the Southern Baptist Convention

ABSTRACT Founded in 1845 to defend the appointment of slaveholders to missionary posts, the Southern Baptist Convention’s origins complicate its efforts to communicate its history to its members. Developing a public memory-based approach to persona criticism, this essay analyzes a 2006 documentary entitled “Forged by Faith: The Early Years.” By selectively narrating SBC history to construct a “legacy of compassion,” the documentary offers a harmful post-racial ideology to its auditors. The video’s substantive claims and stylistic tokens construct Southern Baptist commitments to personal freedom, church autonomy, and cooperation while minimizing historical commitments to slavery and negating African American history. This essay proposes persona criticism as an innovative method for critiquing the ideology-shaping work performed by audiovisual memory texts such as documentaries.

Intersezioni di immaginari in un iper-immaginario postmoderno e postmediale: il caso della serie Once Upon a Time = Intersections of imaginaries in a postmodern and postmedial hyper-imaginary: the case of the series Once Upon a Time

Within the contemporary mediascape, Once Upon a Time (ABC, seven seasons, 2011-2018) represents a rather original serial experiment: starting from the shared westerly (and not only) fairy-tale imagery, fueled and sometimes created by Disney, the series, thanks to its ecosystem structure, is configured in its essence by intersections with other imaginaries. The result is the creation of a contaminated hyper-imaginary novel that rethinks, re-semantizes and reformulates characters and stories already imprinted in the collective imaginary. The architecture of the series resorts to the starting imaginaries as architraves and foundations; the semantic construction, instead, weaves a constant dialogue with the spectators, questioning stories, characters, musical moments, costumes, iconographies, even entire sequences of other films - all materials well impressed in the collective imaginaries of viewers, whose memory is solicited. It is only thanks to the links generated in the audiovisual memory of the spectators – in turn, well evoked by the narrative and metanarrative strategies applied by the authors – it the series manages to carve out its own originality. Therefore, the series under discussion creates, from scratch, a reflective imaginary of imaginaries in a constant evolution. So, the traditional fairy-tale imagery is not forgotten, but reshaped and re-semantized in favor of the new perspectives that dominate in the contemporary mediascape (gender perspectives, villains back history, transmedia storytelling, replicability), combined with an original metanarrative: therefore Once Upon a Time, referring to the paradigm of complexity, proposes itself as a new point of reference that can be useful on several levels, and could be seen as a new postmodern and postmedial myth valid from childhood to adulthood.

Impaired Audiovisual Representation of Phonemes in Children with Developmental Language Disorder.

We examined whether children with developmental language disorder (DLD) differed from their peers with typical development (TD) in the degree to which they encode information about a talker’s mouth shape into long-term phonemic representations. Children watched a talker’s face and listened to rare changes from [i] to [u] or the reverse. In the neutral condition, the talker’s face had a closed mouth throughout. In the audiovisual violation condition, the mouth shape always matched the frequent vowel, even when the rare vowel was played. We hypothesized that in the neutral condition no long-term audiovisual memory traces for speech sounds would be activated. Therefore, the neural response elicited by deviants would reflect only a violation of the observed audiovisual sequence. In contrast, we expected that in the audiovisual violation condition, a long-term memory trace for the speech sound/lip configuration typical for the frequent vowel would be activated. In this condition then, the neural response elicited by rare sound changes would reflect a violation of not only observed audiovisual patterns but also of a long-term memory representation for how a given vowel looks when articulated. Children pressed a response button whenever they saw a talker’s face assume a silly expression. We found that in children with TD, rare auditory changes produced a significant mismatch negativity (MMN) event-related potential (ERP) component over the posterior scalp in the audiovisual violation condition but not in the neutral condition. In children with DLD, no MMN was present in either condition. Rare vowel changes elicited a significant P3 in both groups and conditions, indicating that all children noticed auditory changes. Our results suggest that children with TD, but not children with DLD, incorporate visual information into long-term phonemic representations and detect violations in audiovisual phonemic congruency even when they perform a task that is unrelated to phonemic processing.

Effects of Audiovisual Memory Cues on Working Memory Recall.

Previous studies have focused on topics such as multimodal integration and object discrimination, but there is limited research on the effect of multimodal learning in memory. Perceptual studies have shown facilitative effects of multimodal stimuli for learning; the current study aims to determine whether this effect persists with memory cues. The purpose of this study was to investigate the effect that audiovisual memory cues have on memory recall, as well as whether the use of multiple memory cues leads to higher recall. The goal was to orthogonally evaluate the effect of the number of self-generated memory cues (one or three), and the modality of the self-generated memory-cue (visual: written words, auditory: spoken words, or audiovisual). A recall task was administered where participants were presented with their self-generated memory cues and asked to determine the target word. There was a significant main effect for number of cues, but no main effect for modality. A secondary goal of this study was to determine which types of memory cues result in the highest recall. Self-reference cues resulted in the highest accuracy score. This study has applications to improving academic performance by using the most efficient learning techniques.

A memória enquanto ato teórico dos 'making of' documentários

Este artigo discute a dimensão da memória nos processos de criação cinematográficos através dos making ofs documentários, compreendendo-os enquanto índices de pensamento teórico e criativo, cujos vestígios temporais reverberam e disseminam sentidos de uma memória do (e para o) cinema, ultrapassando o texto fílmico ao qual se refere. Admitimos o making of como processo e obra, donde a memória desponta enquanto um ato teórico no ambiente de criação cinematográfica possível de desvelamento. O percurso teórico-metodológico revela uma breve explanação acerca dos conceitos de memória e dialética do olhar para, num segundo momento, conjugar determinados aspectos com as noções de imagem crítica e dupla distância, a partir de Didi-Huberman (2010). A reflexão sobre os agenciamentos da memória audiovisual nos making ofs documentários se dá através do seguinte corpus: Burden of Dreams (de 1982, direção de Les Blank) e The Hamster Factor and Other Tales of Twelve Monkeys (de 1996, direção de Keith Fulton e Louis Peppe).

Debates electorales en los VOD de las RTV Públicas de Europa

Introducción: En los medios públicos, el debate electoral es de especial interés por las reglas que deben adoptar estas televisiones para cumplir con sus compromisos de servicio público y mantener los valores de diversidad, universalidad y participación. En el nuevo escenario que plantean las plataformas propias de VOD la inclusión de estos debates electorales televisados y las variaciones en su formato se plantea como un ejercicio de transparencia y un compromiso con la memoria audiovisual del país que refuerza los valores de diversidad, proximidad y universalidad representativa. Metodología: El artículo revisa de un modo comparativo el uso de este formato en siete de las principales emisoras públicas de televisión de Europa siguiendo los modelos de Hallin y Mancini y con una subdivisión regional dentro del modelo democrático-corporatista para obtener resultados con un corte étnico-político. Resultados y conclusiones: Existe una notable disparidad entre las televisiones públicas que tienen accesibles los debates electorales en sus plataformas de VOD. El análisis desvela que, a pesar de su trascendencia, los debates no se encuentran disponibles en los entornos online del medio, aunque sí son accesibles desde plataformas ajenas, principalmente YouTube.

Movimento Feminino pela Anistia em reportagens da Rede Globo Minas

Este artigo visa analisar como o Movimento Feminino pela Anistia é abordado em cinco reportagens feitas pela Rede Globo Minas em Belo Horizonte (BH) entre os anos de 1977 e 1979. A análise será feita a partir do acervo da emissora presente no Museu da Imagem e do Som de BH. Essa abordagem é um recorte da pesquisa BH em movimento – Registros audiovisuais do período da ditadura na capital de Minas, que busca localizar e analisar reportagens televisivas realizadas entre 1968 e 1983 em BH que façam referência a pessoas e/ou eventos ligados ao período militar. Localizar esses registros revela-se tão importante quanto entender de que forma eles se inserem na memória audiovisual do período.

Comparison of audio-visual short-term and active memory in multiple sclerosis patients and non-patients regarding their depression, stress and anxiety level

Background: Multiple sclerosis is an autoimmune disease in the central nervous system that is accompanied by a number of cognitive deficiencies. The present study is aimed at comparing audio-visual short-term and active memory in multiple sclerosis patients and non-patients regarding their depression, stress and anxiety level. Methods: It is causal-comparative study comprising 108 participants (54 subjects with relapsing-recovering MS and 54 non-patient subjects). Depression, anxiety and stress scale and Wechsler Memory Scale were measured on all subjects. Results: Study results revealed a significant difference between the two groups in audio-visual short-term and active memories. Similarly, a significant difference was found between the two groups in terms of their levels of depression, anxiety and stress. Conclusions: MS disease leads to deficiency in audio-visual short-term and active memories. With the addition of depression, anxiety and stress to the disease, deficiency in memory will get intensified.

Examination of tablet usage by 4 years old pre-school student

Accurate usage of tablet etc. devices in growth period is essential in terms of development performance. Tablet usage for education and teaching supports audio-visual memory; however, an examination of the consequences of continued usage reveals serious problems. Technology is essential in terms of communication and reaching information in the life of every individual regardless of their age and period. However, especially for small children, tablet and other technological devices should be used under the surveillance of parents. In terms of development, tablets are used in education in areas such as linguistic, social, cognitive and academic development. However, it is known that using them for long hours has negative impacts on the development of children. In this paper, interviews were held with 10 mothers whose children are going to pre-school institutions. The research used face-to-face interview technique. The purpose of the study is to examine the attitude of mothers as regards usage of tablets by their children. During the interviews with mothers, the “interview form prepared for tablet usage” was used which was prepared by the researcher this form was prepared by the researcher. It consisted of question and answer technique mothers who constituted the study group were interviewed in person and analysis were made according to the obtained answers. Sub-theme was created and analysis were concluded based on these data. Obtained findings show that mothers have both negative and positive attitude as regards tablet usage. They think that tablets are effective in teaching their children important concepts, but are also worried that it can lead to addiction and problems in their relations with peers.

Examination of tablet usage by 4 years old pre-school student

Accurate usage of tablet etc. devices in growth period is essential in terms of development performance. Tablet usage for education and teaching supports audio-visual memory; however, an examination of the consequences of continued usage reveals serious problems. Technology is essential in terms of communication and reaching information in the life of every individual regardless of their age and period. However, especially for small children, tablet and other technological devices should be used under the surveillance of parents. In terms of development, tablets are used in education in areas such as linguistic, social, cognitive and academic development. However, it is known that using them for long hours has negative impacts on the development of children. In this paper, interviews were held with 10 mothers whose children are going to pre-school institutions. The research used face-to-face interview technique. The purpose of the study is to examine the attitude of mothers as regards usage of tablets by their children. During the interviews with mothers, the “interview form prepared for tablet usage” was used which was prepared by the researcher this form was prepared by the researcher. It consisted of question and answer technique mothers who constituted the study group were interviewed in person and analysis were made according to the obtained answers. Sub-theme was created and analysis were concluded based on these data. Obtained findings show that mothers have both negative and positive attitude as regards tablet usage. They think that tablets are effective in teaching their children important concepts, but are also worried that it can lead to addiction and problems in their relations with peers.

The effect of antidepressant treatment on the working memory of Parkinson's disease patients with depression

Objective To investigate the impaired characteristics of working memory in Parkinson's disease patients with depression and the recovery status of working memory after antidepressant treatment. Methods Totally 60 cases were enrolled in this study and divided into three groups: 30 Parkinson's disease patients with depression (PD-D), 15 Parkinson's disease patients without depression (PD) and 15 persons without Parkinson's disease as normal controls. All subjects were evaluated with Wechsler Memory Scale (WMS-R) audio-visual span, Tower of Hanoi (TOH) and Wisconsin Card Sorting Test (WCST). 30 PD-D patients were randomly divided into two subgroups: routine treatment subgroup (8 weeks levodopa therapy) and antidepressant combined treatment subgroup (8 weeks levodopa + citalopram therapy). The evaluation of Hamilton Depression Scale (HAMD), WMS-R audio-visual span, TOH and WCST were performed on these two groups before and after treatment. Results ① There were significant differences comparing PD-D and PD groups with the normal control group in scores of WMS-R, TOH and WCST indicators (P<0.05). There were significant differences in WMS-R visual span, accuracy rate and speed of TOH, as well as the percent errors and percent perseverative responses of WCST comparing PD-D group(11.88±5.91, 0.420±0.345, 0.408±0.334, 0.882±0.253, 0.565±0.229) with PD group (15.87±5.21, 0.768±0.167, 0.634±0.232, 0.493±0.161, 0.327±0.122) (P<0.05). ② Before and after treatment in PD-D routine treatment subgroup, there were significant differences in digital memory and audio-visual memory of WMS-R((6.73±3.72, 5.95±3.13) vs (3.77±2.16, 1.91±1.58)), accuracy rate of TOH(0.45±0.26 vs 0.23±0.13), as well as percent errors((-0.58±0.17) vs (-0.37±0.14)), percent perseverative responses((-0.32±0.15) vs (-0.14±0.09)), percent conceptual level responses(0.38±0.09 vs 0.13±0.07) and number of categories completed(3.27±1.56 vs 1.06±0.91) of WCST(P<0.05). ③ After 4-week treatment and 8-week treatment, there were significant differences in HAMD score comparing PD-D antidepressant combined treatment group(16.33±2.72, 10.27±2.66) with PD-D routine treatment group(21.73±2.28, 18.4±2.47) (P<0.05). Conclusion Comparing Parkinson's disease patients without depression, the impaired working memory is more serious and extensive in Parkinson's disease patients with depression. Antidepressant treatment can improve the working memory of Parkinson's disease patients with depression. Key words: Parkinson's disease; Depression; Working memory; Levodopa; Citalopram

“É trabalho de Formiguinha”: Hernani Heffner, conservação e preservação na Cinemateca do Museu de Arte Moderna do Rio de Janeiro

Entrevista com Hernani Heffner, conservador-chefe da Cinemateca do Museu de Arte Moderna do Rio de Janeiro. A entrevista aborda a trajetória profissional do conservador, os desafios e conquistas vividos na Cinemateca do MAM, as especificidades deste acervo e os dilemas enfrentados pelas instituições que trabalham pela conservação e difusão da memória audiovisual brasileira.

Emotion Regulation of Neuroticism: Emotional Information Processing Related to Psychosomatic State Evaluated by Electroencephalography and Exact Low-Resolution Brain Electromagnetic Tomography

Emotion regulation is the process that adjusts the type or amount of emotion when we experience an emotional situation. The aim of this study was to reveal quantitative changes in brain activity during emotional information processing related to psychosomatic states and to determine electrophysiological features of neuroticism. Twenty-two healthy subjects (mean age 25 years, 14 males and 8 females) were registered. Electroencephalography (EEG) was measured during an emotional audiovisual memory task under three conditions (neutral, pleasant and unpleasant sessions). We divided the subjects into two groups using the Cornell Medical Index (CMI): (CMI-I: control group, n = 10: CMI-II, III or IV: neuroticism group, n = 12). We analyzed the digital EEG data using exact low-resolution brain electromagnetic tomography (eLORETA) current source density (CSD) and functional connectivity analysis in several frequency bands (δ, θ, α, β, γ and whole band). In all subjects, bilateral frontal α CSD in the unpleasant session increased compared to the pleasant session, especially in the control group (p < 0.05). CSD of the neuroticism group was significantly higher than that of the control group in the full band at the amygdala and inferior temporal gyrus, and in the α band at the right temporal lobe (p < 0.05). Additionally, we found an increase in functional connectivity between the left insular cortex and right superior temporal gyrus in all subjects during the unpleasant session compared to the pleasant session (p < 0.05). In this study, using EEG analysis, we could find a novel cortical network related to brain mechanisms underlying emotion regulation. Overall findings indicate that it is possible to characterize neuroticism electrophysiologically, which may serve as a neurophysiological marker of this personality trait.

Separating lexical-semantic access from other mnemonic processes in picture-name verification

We present a novel paradigm to identify shared and unique brain regions underlying non-semantic, non-phonological, abstract, audio-visual (AV) memory vs. naming using a longitudinal functional magnetic resonance imaging experiment. Participants were trained to associate novel AV stimulus pairs containing hidden linguistic content. Half of the stimulus pairs were distorted images of animals and sine-wave speech versions of the animal's name. Images and sounds were distorted in such a way as to make their linguistic content easily recognizable only after being made aware of its existence. Memory for the pairings was tested by presenting an AV pair and asking participants to verify if the two stimuli formed a learned pairing. After memory testing, the hidden linguistic content was revealed and participants were tested again on their recollection of the pairings in this linguistically informed state. Once informed, the AV verification task could be performed by naming the picture. There was substantial overlap between the regions involved in recognition of non-linguistic sensory memory and naming, suggesting a strong relation between them. Contrasts between sessions identified left angular gyrus and middle temporal gyrus as key additional players in the naming network. Left inferior frontal regions participated in both naming and non-linguistic AV memory suggesting the region is responsible for AV memory independent of phonological content contrary to previous proposals. Functional connectivity between angular gyrus and left inferior frontal gyrus and left middle temporal gyrus increased when performing the AV task as naming. The results are consistent with the hypothesis that, at the spatial resolution of fMRI, the regions that facilitate non-linguistic AV associations are a subset of those that facilitate naming though reorganized into distinct networks.