Incongruent Objects Research Articles

The Sensory-Dependent Nature of Audio-Visual Interactions for Semantic Knowledge

The Sensory-Dependent Nature of Audio-Visual Interactions for Semantic Knowledge Guillaume Vallet (guillaume.vallet@univ-lyon2.fr) Universit´e Lumi`ere Lyon 2. Laboratoire EMC, 5 avenue Pierre Mend`es France, 69676, Bron cedex, France & Laval University, School of Psychololgy, 2325 rue des Biblioth`eques Quebec City (Quebec), G1V 0A6 Canada Benoit Riou (benoit.riou@univ-lyon2.fr), R´emy Versace (remy.versace@univ-lyon2.fr) Universit´e Lumi`ere Lyon 2. Laboratoire EMC, 5 avenue Pierre Mend`es France, 69676, Bron cedex, France Martine Simard (martine.simard@psy.ulaval.ca) Laval University, School of Psychololgy, 2325 rue des Biblioth`eques Quebec City (Quebec), G1V 0A6 Canada Abstract The nature of audio-visual interactions is poorly understood for meaningful objects. These interactions would be indirect through semantic memory according to the amodal nature of knowledge, whereas these interactions would be direct accord- ing to the modal nature of knowledge. This question, cen- tral for both memory and multisensory frameworks, was as- sessed using a cross-modal priming paradigm from auditory to visual modalities tested on familiar objects. For half of the sound primes, a visual abstract mask was simultaneously pre- sented to the participants. The results showed a cross-modal priming effect for semantically congruent objects compared to semantically incongruent objects presented without the mask. The mask interfered in the semantically congruent condition, but had no effect in the semantically incongruent condition. The semantic specificity of the mask effect demonstrates a memory-related effect. The results suggest that audio-visual interactions are direct. The data support the modal approach of knowledge and the grounded cognition theory. Keywords: Memory; Perception; Audio-visual; Masking; Priming; Grounded Cognition. Introduction Our environment is filled with meaningful objects repre- senting semantic knowledge. These objects are perceptually processed using several sensory channels in which the auditory and visual modalities dominate the other senses in Human (for a review see Spence, 2007). The sensory infor- mation is mainly integrated on the basis of the temporal and spatial relationships between the stimuli (Calvert & Thesen, 2004), and also on the basis of the semantic relationships existing between them (Laurienti, Kraft, Maldjian, Burdette, & Wallace, 2004). Yet it remains uncertain how semantic memory aspects are involved in multisensory perception (for a review see Doehrmann & Naumer, 2008). This issue depends on the perceptual or semantic nature of cross-modal interactions, and thus questions the modal or amodal nature of knowledge (Vallet, Brunel, & Versace, 2010). The present study therefore aims at assessing the nature of audio-visual interactions using an innovative masking procedure. Communication between different modalities is called in- teraction (or interplay). If this interaction involves a repre- sentation of higher level, this interaction is called integration (Driver & Noesselt, 2008). An integrated object is a represen- tation that is more than the sum of its part. Previous research in the multisensory perception theoretical framework princi- pally studied the neural basis of the integration mechanism using meaningless stimuli (for review see Calvert & Thesen, 2004; Koelewijn, Bronkhorst, & Theeuwes, 2010). Fewer studies were conducted with meaningful stimuli, and the goal of these studies was also to determine the brain substrates of multisensory integration (Doehrmann & Naumer, 2008). The semantic constraint is generally assessed by manipulating the semantic congruency. A congruent trial is when the prime and the target refer to the same semantic object (meowing sound - cat’s picture). Semantic congruent stimuli usually fa- cilitate information processing (Chen & Spence, 2010), and may enhance memory performances in semantic (Laurienti et al., 2004) and episodic tasks (Lehmann & Murray, 2005). In the memory theoretical framework, cross-modal inter- actions tested on meaningful stimuli are generally studied by inserting a delay between the stimuli. The most famous paradigm in this field is the cross-modal priming paradigm. The cross-modal priming effect is the facilitation of the pro- cessing of one stimulus in one modality (the target) by the previous presentation of another stimulus in another modal- ity (the prime). The cross-modal priming effect may be ob- served between different modalities, such as the haptic and vi- sual modalities (Easton, Srinivas, & Greene, 1997), but most of the studies were realized between the auditory and visual modalities (for a review see Schneider, Engel, & Debener, 2008). The increasing number of studies on the audio-visual interactions involving meaningful stimuli are aimed at a bet- ter understanding of these effects. Nevertheless, the nature of audio-visual interactions, which is the central issue underly- ing these effects, remains poorly understood. The nature of these interactions depends on the nature

Scene incongruity and attention.

Does scene incongruity, (a mismatch between scene gist and a semantically incongruent object), capture attention and lead to conscious perception? We explored this question using 4 different procedures: Inattention (Experiment 1), Scene description (Experiment 2), Change detection (Experiment 3), and Iconic Memory (Experiment 4). We found no differences between scene incongruity and scene congruity in Experiments 1, 2, and 4, although in Experiment 3 change detection was faster for scenes containing an incongruent object. We offer an explanation for why the change detection results differ from the results of the other three experiments. In all four experiments, participants invariably failed to report the incongruity and routinely mis-described it by normalizing the incongruent object. None of the results supports the claim that semantic incongruity within a scene invariably captures attention and provide strong evidence of the dominant role of scene gist in determining what is perceived.

Stuck on semantics: Irrelevant object-scene inconsistencies modulate ongoing eye movement behavior during letter search

People have an amazing ability to identify objects and scenes within a glimpse of an eye. How automatic is this scene and object identification? Are scene and object semantics —let alone their semantic congruity— processed to a degree that modulates ongoing gaze behavior even if they are irrelevant to the current task? Objects that do not fit the semantics of the scene (e.g. a toothbrush in an office) are typically fixated longer and more often than consistent controls. In Experiment 1, we overlaid a letter T on photographs of indoor scenes and instructed fourteen participants to search for it. Some of the background images contained scene incongruent objects, but these provided no information about the target position. Despite their lack of relevance to the search, we found that subjects looked at incongruent objects longer and more often compared to congruent objects in the same position of the scene. In Experiment 2, we replicated these findings and — to better understand how aware participants had been of the objects — subsequently tested participants' memory for the critical objects in both a free recall and a 2AFC task. Participants did not remember more incongruent objects than congruent objects and performed no better than chance for incongruent objects during 2AFC suggesting that while being "stuck" the observers had no explicit memory of these objects. Attempting to diminish this object congruency effect, in Experiment 3 we overlaid a grid of search elements on the background images. Even though no longer statistically significant, the same pattern of results emerged for each measure of gaze behavior. Based on these results we argue that when we view natural scenes, scene and object identities are processed automatically. Moreover, even when irrelevant to our current search, a semantic mismatch between scene and object identity can modulate ongoing eye movement behavior. Meeting abstract presented at VSS 2016

Semantically incongruent objects attract eye gaze when viewing scenes for change

ABSTRACTPast research has shown that change detection performance is often more efficient for target objects that are semantically incongruent with a surrounding scene context than for target objects that are semantically congruent with the scene context. One account of these findings is that attention is attracted to objects for which the identity of the object conflicts with the meaning of the scene, perhaps as a violation of expectancies created by earlier recruitment of scene gist information. An alternative account of the performance benefit for incongruent objects is that attention is more apt to linger on incongruent objects, as perhaps identifying these objects is more difficult due to conflicting information from the scene context. In the current experiment, we present natural scenes in a change detection task while monitoring eye movements. We find that eye gaze is attracted to these objects relatively early during scene processing.

Gaze Following Is Modulated by Expectations Regarding Others’ Action Goals

Humans attend to social cues in order to understand and predict others’ behavior. Facial expressions and gaze direction provide valuable information to infer others’ mental states and intentions. The present study examined the mechanism of gaze following in the context of participants’ expectations about successive action steps of an observed actor. We embedded a gaze-cueing manipulation within an action scenario consisting of a sequence of naturalistic photographs. Gaze-induced orienting of attention (gaze following) was analyzed with respect to whether the gaze behavior of the observed actor was in line or not with the action-related expectations of participants (i.e., whether the actor gazed at an object that was congruent or incongruent with an overarching action goal). In Experiment 1, participants followed the gaze of the observed agent, though the gaze-cueing effect was larger when the actor looked at an action-congruent object relative to an incongruent object. Experiment 2 examined whether the pattern of effects observed in Experiment 1 was due to covert, rather than overt, attentional orienting, by requiring participants to maintain eye fixation throughout the sequence of critical photographs (corroborated by monitoring eye movements). The essential pattern of results of Experiment 1 was replicated, with the gaze-cueing effect being completely eliminated when the observed agent gazed at an action-incongruent object. Thus, our findings show that covert gaze following can be modulated by expectations that humans hold regarding successive steps of the action performed by an observed agent.

Time-Driven Effects on Processing Relative Clauses.

The present response time study investigated how a hypothesized time-based working memory constraint of 2–3 s affects the resolution of grammatical and semantic dependencies. Congruent and incongruent object relative (OR) and subject relative sentences were read at different presentation rates so that the distance between dependent words was either shorter or longer than 2–3 s. Incongruent OR sentences showed an effect of presentation rate. Experiment 1 focused on grammatical dependencies. Processing of adjectives with agreement features mismatching those of the preceding dependent word showed rapid agreement resolution at a time-interval below 2 s. Dependency intervals over 3 s reflected a different, more time-consuming process possibly due to extended search in sentence semantic representations as the grammatical form of the first word in the dependency fades away. In experiment 2, focusing on semantic dependencies, incongruent OR sentences displayed a different pattern: a gradual increase in processing time as a function of distance between dependent words. Thus, the 2–3 s long time-window seems to constrain the maintenance of grammatical forms in working memory.

What you see depends on what you saw, and what else you saw: the interactions between motion priming and object priming.

Both visual object priming and motion priming have been reported independently, but the interactions between the two are still largely unexplored. Here we investigated this question using a novel type of SFM stimuli, 3-D helixes, and found that the motion direction perception of an ambiguous helix can be biased by the motion direction of a preceding SFM stimulus - a classic motion priming effect. However, the effectiveness of motion priming depends on object priming: a neutral object priming produced a weak motion priming, a congruent object priming led to a strong motion priming, and critically, an incongruent object priming abolished and overpowered the motion priming. In contrast, object priming alone (in the absence of motion overlap) had little effects biasing motion perception. Taken together, these results suggest that there exists an integrated neural representation of motion and structure of 3-D SFM stimuli, and motion priming of 3-D SFM stimuli might happen at an intermediate stage between MT/V5 (which is not shape selective) and LO (lateral occipital, which is not motion selective). This novel type of stimuli, 3-D helixes, along with the prime-target paradigm, thus might offer a unique tool to examine neural bases underlying the perception of 3-D SFM stimuli and perceptual priming.

Do simultaneously viewed objects influence scene recognition individually or as groups? Two perceptual studies.

The ability to quickly categorize visual scenes is critical to daily life, allowing us to identify our whereabouts and to navigate from one place to another. Rapid scene categorization relies heavily on the kinds of objects scenes contain; for instance, studies have shown that recognition is less accurate for scenes to which incongruent objects have been added, an effect usually interpreted as evidence of objects' general capacity to activate semantic networks for scene categories they are statistically associated with. Essentially all real-world scenes contain multiple objects, however, and it is unclear whether scene recognition draws on the scene associations of individual objects or of object groups. To test the hypothesis that scene recognition is steered, at least in part, by associations between object groups and scene categories, we asked observers to categorize briefly-viewed scenes appearing with object pairs that were semantically consistent or inconsistent with the scenes. In line with previous results, scenes were less accurately recognized when viewed with inconsistent versus consistent pairs. To understand whether this reflected individual or group-level object associations, we compared the impact of pairs composed of mutually related versus unrelated objects; i.e., pairs, which, as groups, had clear associations to particular scene categories versus those that did not. Although related and unrelated object pairs equally reduced scene recognition accuracy, unrelated pairs were consistently less capable of drawing erroneous scene judgments towards scene categories associated with their individual objects. This suggests that scene judgments were influenced by the scene associations of object groups, beyond the influence of individual objects. More generally, the fact that unrelated objects were as capable of degrading categorization accuracy as related objects, while less capable of generating specific alternative judgments, indicates that the process by which objects interfere with scene recognition is separate from the one through which they inform it.

The Interplay of Bottom-Up and Top-Down Mechanisms in Visual Guidance during Object Naming

An ongoing issue in visual cognition concerns the roles played by low- and high-level information in guiding visual attention, with current research remaining inconclusive about the interaction between the two. In this study, we bring fresh evidence into this long-standing debate by investigating visual saliency and contextual congruency during object naming (Experiment 1), a task in which visual processing interacts with language processing. We then compare the results of this experiment to data of a memorization task using the same stimuli (Experiment 2). In Experiment 1, we find that both saliency and congruency influence visual and naming responses and interact with linguistic factors. In particular, incongruent objects are fixated later and less often than congruent ones. However, saliency is a significant predictor of object naming, with salient objects being named earlier in a trial. Furthermore, the saliency and congruency of a named object interact with the lexical frequency of the associated word and mediate the time-course of fixations at naming. In Experiment 2, we find a similar overall pattern in the eye-movement responses, but only the congruency of the target is a significant predictor, with incongruent targets fixated less often than congruent targets. Crucially, this finding contrasts with claims in the literature that incongruent objects are more informative than congruent objects by deviating from scene context and hence need a longer processing. Overall, this study suggests that different sources of information are interactively used to guide visual attention on the targets to be named and raises new questions for existing theories of visual attention.

Synchronous contextual irregularities affect early scene processing: Replication and extension

Whether contextual regularities facilitate perceptual stages of scene processing is widely debated, and empirical evidence is still inconclusive. Specifically, it was recently suggested that contextual violations affect early processing of a scene only when the incongruent object and the scene are presented a-synchronously, creating expectations. We compared event-related potentials (ERPs) evoked by scenes that depicted a person performing an action using either a congruent or an incongruent object (e.g., a man shaving with a razor or with a fork) when scene and object were presented simultaneously. We also explored the role of attention in contextual processing by using a pre-cue to direct subjects׳ attention towards or away from the congruent/incongruent object. Subjects׳ task was to determine how many hands the person in the picture used in order to perform the action. We replicated our previous findings of frontocentral negativity for incongruent scenes that started ~210ms post stimulus presentation, even earlier than previously found. Surprisingly, this incongruency ERP effect was negatively correlated with the reaction times cost on incongruent scenes. The results did not allow us to draw conclusions about the role of attention in detecting the regularity, due to a weak attention manipulation. By replicating the 200–300ms incongruity effect with a new group of subjects at even earlier latencies than previously reported, the results strengthen the evidence for contextual processing during this time window even when simultaneous presentation of the scene and object prevent the formation of prior expectations. We discuss possible methodological limitations that may account for previous failures to find this an effect, and conclude that contextual information affects object model selection processes prior to full object identification, with semantic knowledge activation stages unfolding only later on.

Differential processing of invisible congruent and incongruent scenes: A case for unconscious integration

Integration is held to be a key feature of conscious awareness. Some even argue that the latter cannot occur without the former. We tested this claim by presenting masked scenes depicting a person performing an action with a congruent or an incongruent object (e.g., a man pouring coffee into a mug or into a roll of toilet paper). The masked scenes were then followed by briefly flashed targets that again included a congruent or an incongruent object, and subjects were asked to judge targets' congruency as fast as possible. Reaction times (RTs) for targets preceded by perceptually invisible scenes that included an incongruent object were longer than RTs for targets that were preceded by congruent images. This implicit measure suggests that subjects processed certain relations between the object and its background--or at least between the object and another object in the scene--despite being unaware of either. Subjective and objective measures confirmed the invisibility of the masked scenes, ruling out partial awareness. These results suggest that incongruency between scene elements can be unconsciously processed even at impoverished presentation conditions, with reduced contrast and exposure durations as short as 33 ms. They provide evidence for ongoing contextual influences of unseen stimuli on the processing of a subsequent target.

Multisensory Interactions between Auditory and Haptic Object Recognition

Object manipulation produces characteristic sounds and causes specific haptic sensations that facilitate the recognition of the manipulated object. To identify the neural correlates of audio-haptic binding of object features, healthy volunteers underwent functional magnetic resonance imaging while they matched a target object to a sample object within and across audition and touch. By introducing a delay between the presentation of sample and target stimuli, it was possible to dissociate haptic-to-auditory and auditory-to-haptic matching. We hypothesized that only semantically coherent auditory and haptic object features activate cortical regions that host unified conceptual object representations. The left fusiform gyrus (FG) and posterior superior temporal sulcus (pSTS) showed increased activation during crossmodal matching of semantically congruent but not incongruent object stimuli. In the FG, this effect was found for haptic-to-auditory and auditory-to-haptic matching, whereas the pSTS only displayed a crossmodal matching effect for congruent auditory targets. Auditory and somatosensory association cortices showed increased activity during crossmodal object matching which was, however, independent of semantic congruency. Together, the results show multisensory interactions at different hierarchical stages of auditory and haptic object processing. Object-specific crossmodal interactions culminate in the left FG, which may provide a higher order convergence zone for conceptual object knowledge.

ERP correlates of spatially incongruent object identification during scene viewing: Contextual expectancy versus simultaneous processing

Object processing is affected by the gist of the scene within which it is embedded. Previous ERP research has suggested that manipulating the semantic congruency between an object and the surrounding scene affects the high level (semantic) representation of that object emerging after the presentation of the scene (Ganis & Kutas, 2003). In two ERP experiments, we investigated whether there would be a similar electrophysiological response when spatial congruency of an object in a scene was manipulated while the semantic congruency remained the same. Apart from the location of the object, all other object features were congruent with the scene (e.g., in a bedroom scene, either a painting or a cat appeared on the wall). In the first experiment, participants were shown a location cue and then a scene image for 300ms, after which an object image appeared on the cued location for 300ms. Spatially incongruent objects elicited a stronger centro-frontal N300–N400 effect in the 275–500ms window relative to the spatially congruent objects. We also found early ERP effects, dominant on the left hemisphere electrodes. Strikingly, LORETA analysis revealed that these activations were mainly located in the superior and middle temporal gyrus of the right hemisphere. In the second experiment, we used a paradigm similar to Mudrik, Lamy, and Deouell (2010). The scene and the object were presented together for 300ms after the location cue. This time, we did not observe either an early or the pronounced N300–N400 effect. In contrast to Experiment 1, LORETA analysis on the N400 time-window revealed that the generators of these weak ERP effects were mainly located in the temporal lobe of the left hemisphere. Our results suggest that, when the scene is presented before the object, top-down spatial encoding processes are initiated and the right superior temporal gyrus is activated, as previously suggested (Ellison, Schindler, Pattison, & Milner, 2004). Mismatch between the actual object features and the spatially driven top-down structural and functional features could lead to the early effect, and then to the N300–N400 effect. In contrast, when the scene is not presented before the object, the spatial encoding could not happen early and strong enough to initiate spatial object-integration effects. Our results indicate that spatial information is an early and essential part in scene–object integration, and it primes structural as well as semantic features of an object.

The evocative power of words: activation of concepts by verbal and nonverbal means.

A major part of learning a language is learning to map spoken words onto objects in the environment. An open question is what are the consequences of this learning for cognition and perception? Here, we present a series of experiments that examine effects of verbal labels on the activation of conceptual information as measured through picture verification tasks. We find that verbal cues, such as the word "cat," lead to faster and more accurate verification of congruent objects and rejection of incongruent objects than do either nonverbal cues, such as the sound of a cat meowing, or words that do not directly refer to the object, such as the word "meowing." This label advantage does not arise from verbal labels being more familiar or easier to process than other cues, and it does extends to newly learned labels and sounds. Despite having equivalent facility in learning associations between novel objects and labels or sounds, conceptual information is activated more effectively through verbal means than through nonverbal means. Thus, rather than simply accessing nonverbal concepts, language activates aspects of a conceptual representation in a particularly effective way. We offer preliminary support that representations activated via verbal means are more categorical and show greater consistency between subjects. These results inform the understanding of how human cognition is shaped by language and hint at effects that different patterns of naming can have on conceptual structure.

Listening to “flying ducks”: Individual differences in sentence–picture verification investigated with ERPs

The present ERP study investigated individual differences in the integration of verbal descriptions and visual object representations. Participants saw pictures of objects (e.g., a swimming duck) after listening to noun phrases describing the same object in the identical state, a shape-mismatching state ("flying duck"), or an incongruent object (e.g., "sliced bread"). Individual differences in the vividness of mental imagery and preference for mental imagery were assessed after the experiment. ERP effects of context arose 170 ms after picture onset, differentiating the incongruent-object context from the other two. The N400 mirrored these context effects. Self-rated vividness of imagery affected responses to pictures already after 100 ms, and modulated the N400 effect. Participants with highly vivid imagery showed larger context effects than participants low in imagery. The context effects at 170 ms were not modulated by individual differences.

Integration without awareness: expanding the limits of unconscious processing

Human conscious awareness is commonly seen as the climax of evolution. However, what function—if any—it serves in human behavior is still debated. One of the leading suggestions is that the cardinal function of conscious awareness is to integrate numerous inputs—including the multitude of features and objects in a complex scene—across different levels of analysis into a unified, coherent, and meaningful perceptual experience. Here we demonstrate, however, that integration of objects with their background scenes can be achieved without awareness of either. We used a binocular rivalry technique known as continuous flash suppression to induce perceptual suppression in a group of human observers. Complex scenes that included incongruent objects escaped perceptual suppression faster than normal scenes did. We conclude that visual awareness is not needed for object-background integration or for processing the likelihood of an object to appear within a given semantic context, but may be needed for dealing with n...

Integration Without Awareness

Human conscious awareness is commonly seen as the climax of evolution. However, what function—if any—it serves in human behavior is still debated. One of the leading suggestions is that the cardinal function of conscious awareness is to integrate numerous inputs—including the multitude of features and objects in a complex scene—across different levels of analysis into a unified, coherent, and meaningful perceptual experience. Here we demonstrate, however, that integration of objects with their background scenes can be achieved without awareness of either. We used a binocular rivalry technique known as continuous flash suppression to induce perceptual suppression in a group of human observers. Complex scenes that included incongruent objects escaped perceptual suppression faster than normal scenes did. We conclude that visual awareness is not needed for object-background integration or for processing the likelihood of an object to appear within a given semantic context, but may be needed for dealing with novel situations.

Scene congruency biases Binocular Rivalry

Contextual regularities, that is, objects' tendency to appear with certain other objects, facilitate the processing of visual scenes and confer contextually incongruent objects with a special attentional status. This study was aimed at investigating the mechanisms underlying this attentional advantage using Binocular Rivalry (BR). In two experiments, congruent and incongruent images (e.g., a man drinking from a glass vs. a man "drinking" from a hairbrush) were pitted against each other, yielding a version of BR in which two objects rival within a given scene. Incongruent objects predominated in awareness longer than congruent ones. This effect stemmed from the fact that their dominance epochs lasted longer on the average than those of congruent objects, suggesting a difficulty to disengage attention from such objects. On the other hand, no support was found for the notion that incongruent objects also attract attention.

Dynamics of inter‐organizational knowledge creation and information technology use across object worlds: the case of an innovative construction project

Organizational research argues that under relational forms of governance a high degree of both information pooling and physical interaction are necessary for inter‐organizational knowledge creation. Yet, recent studies of information and communication technologies (ICTs) suggest that both practices at the same time are sometimes unnecessary. We address this discrepancy by developing a framework whereby the intensity and proportion of these inter‐organizational practices are affected by the object world congruence between designers within and across partnering firms, and the level to which a common information technology platform is embedded in their activity. Through a multi‐level case study of a Frank Gehry construction project we illustrate how designers with highly congruent object worlds, due to a strongly embedded common information technology platform, could jointly create knowledge despite decreased physical interaction. Conversely, designers from firms with incongruent object worlds or with congruent object worlds lacking a strongly embedded common ICT platform demanded a higher degree of physical interaction for effective knowledge creation. Our research suggests a dynamic, evolutionary model of inter‐organizational knowledge creation influenced by variation in object world congruence and the levels of embedding a common ICT platform.

Odors enhance visual attention to congruent objects

Although it is well known that visual stimuli affect olfactory performance, little is known about the reverse case: the influence of odor on visual performance. This study aimed to determine whether odors can enhance attention towards visually presented objects congruent with the odors. Sixty healthy participants were presented with four odors (orange, lavender, coffee, and liquorice) before and during the presentation of photographic slides containing one congruent and three incongruent objects with the presented odors. The participants’ visual attention was assessed as the total number and time of eye fixations by using an eye tracking system. When the participants smelled an odor, they looked more frequently and longer at a corresponding object as compared to the odorless condition. In conclusion, our findings demonstrate for the first time an olfactory priming effect on visual selective attention: odor can increase attention towards a congruent visual object as compared to a non-odor condition.