Attention and Salience

Abstract

Attention guides the focus of our perceptual experience: during visual perception, for example, attention helps us filter out relevant information and allocate cognitive resources where they are needed. Attention is strongly related to another concept, that is salience. Salience is a very widely used term in linguistics, but it was also used in visual cognition, where it is typically described as the property of (part of) a perceptual input that leaps out to us, attracting attention. The term salience itself suggest some sort of bottom-up process, implying that it is the input properties which determine what part of the input attracts our attention (think of a spot of color in a black and white picture, for example). In linguistics, bottom-up salience is related to phonological prominence or to thematization—something that is noticeable (cognitively salient) for its perceptual features and/or for its being an infrequent variety. The field of visual cognition has widely addressed attentional processes, identifying a (fast) bottom-up mechanism based on the salience (or saliency) of the input, as well as a top-down mechanism driven by our goals and intents. If a perceptual input leaps out to us because of its intrinsic properties, we may not even notice it if we are allocating our cognitive resources for a specific task, for example during visual search. In linguistic tasks, a top-down goal may also determine if we notice an incongruent input or not. More recently, in computational linguistics, attention has been used as an architectural property of encoder-decoder neural networks, for example in machine translation. Interestingly, even if this property is not aimed at realistically modeling human cognitive processes, attention is the mechanism that helps some deep learning models to identify the properties of the input that are most informative to perform a certain task. For example, while generating a French translation for an English sentence, the attention mechanism may help a neural machine translation model focus on the right part of the English input required to sequentially generate the French output.