Semantic and Syntactic Inconsistencies in Scenes Elicit Differential ERP Signatures.

Abstract

We have evolved to be efficient searchers by making use of the regularities of our visual world. We rapidly register two types of regularity violation: Incongruous objects in scenes are “semantic inconsistencies” (a fire-hydrant in the kitchen). Misplaced objects are “syntactic inconsistencies” (a fork on the kitchen chair). ERPs differentially reflect semantic and syntactic errors in language. Does the brain honor such differences in vision? Previous studies (Ganis & Kutas, 2004; Mudrik, Lamy, & Deouell, 2010) have shown an “N390/N400 scene congruity effect” for semantic inconsistencies, similar to the semantic N400 effect found in sentence processing. We compared ERPs for semantic and syntactic anomalies in scenes. We collected EEG data from 64 electrodes while participants attempted to memorize scenes containing an object that was either syntactically inconsistent, semantically inconsistent, both, or neither. First, the scene without the critical object was presented (500ms), followed by a location cue that indicated where to move the eyes (500ms). Then the critical object appeared at the cued location (2000ms). We found an N400 for semantic inconsistencies between 300 and 500 ms after object onset. Interestingly, we observed a prolonged deflection of the opposite (positive) sign for syntactic inconsistencies, which might resemble the P600 found for syntax manipulations in sentence processing. When we repeated the same experiment, but presented words in the scene, naming the objects, instead of the objects themselves, we found evidence for an N400 in response to semantically incongruous words in scenes, but no modulation attributable to the syntactically incongruous positioning of words. Together, these findings demonstrate that, as in sentence processing, semantic and syntactic inconsistencies of objects in scenes elicit differential brain responses. This implies that the semantic/syntax distinction is meaningful in the visual system as it is in sentence processing.