Category-specific Representations Research Articles

Effects of motor preparation on category-specific representations in human cortex

Effects of motor preparation on category-specific representations in human cortex

Voxelwise encoding models of body stimuli reveal a representational gradient from low-level visual features to postural features in occipitotemporal cortex

Previous research on body representation in the brain has focused on category-specific representation, using fMRI to investigate the response pattern to body stimuli in occipitotemporal cortex. But the central question of the specific computations involved in body selective regions has not been addressed so far. This study used ultra-high field fMRI and banded ridge regression to investigate the computational mechanisms of coding body images, by comparing the performance of three encoding models in predicting brain activity in occipitotemporal cortex and specifically in the extrastriate body area (EBA). Our results indicate that bodies are encoded in occipitotemporal cortex and in the EBA according to a combination of low-level visual features and postural features.

Semantic guided level-category hybrid prediction network for hierarchical image classification

Hierarchical classification (HC) assigns each object with multiple labels organized into a hierarchical structure. The existing deep learning-based HC methods usually predict an instance starting from the root node until a leaf node is reached. However, in the real world, images impaired by noise, occlusion, blur, or low resolution may not provide sufficient information for the classification at subordinate levels. To address this issue, we propose a novel Semantic Guided level-category Hybrid Prediction Network (SGHPN) that can jointly perform the level and category prediction in an end-to-end manner. SGHPN comprises two modules: a visual transformer that extracts feature vectors from the input images, and a semantic guided cross-attention module that uses categories word embeddings as queries to guide learning category-specific representations. In order to evaluate the proposed method, we construct two new datasets in which images are at a broad range of quality and thus are labeled to different levels (depths) in the hierarchy according to their individual quality. Experimental results demonstrate the effectiveness of our proposed HC method.

Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels

Training the multi-label image recognition models with partial labels, in which merely some labels are known while others are unknown for each image, is a considerably challenging and practical task. To address this task, current algorithms mainly depend on pre-training classification or similarity models to generate pseudo labels for the unknown labels. However, these algorithms depend on sufficient multi-label annotations to train the models, leading to poor performance especially with low known label proportion. In this work, we propose to blend category-specific representation across different images to transfer information of known labels to complement unknown labels, which can get rid of pre-training models and thus does not depend on sufficient annotations. To this end, we design a unified semantic-aware representation blending (SARB) framework that exploits instance-level and prototype-level semantic representation to complement unknown labels by two complementary modules: 1) an instance-level representation blending (ILRB) module blends the representations of the known labels in an image to the representations of the unknown labels in another image to complement these unknown labels. 2) a prototype-level representation blending (PLRB) module learns more stable representation prototypes for each category and blends the representation of unknown labels with the prototypes of corresponding labels to complement these labels. Extensive experiments on the MS-COCO, Visual Genome, Pascal VOC 2007 datasets show that the proposed SARB framework obtains superior performance over current leading competitors on all known label proportion settings, i.e., with the mAP improvement of 4.6%, 4.6%, 2.2% on these three datasets when the known label proportion is 10%. Codes are available at https://github.com/HCPLab-SYSU/HCP-MLR-PL.

An Attention-based Deep Relevance Model for Few-shot Document Filtering

With the large quantity of textual information produced on the Internet, a critical necessity is to filter out the irrelevant information and organize the rest into categories of interest (e.g., an emerging event). However, supervised-learning document filtering methods heavily rely on a large number of labeled documents for model training. Manually identifying plenty of positive examples for each category is expensive and time-consuming. Also, it is unrealistic to cover all the categories from an evolving text source that covers diverse kinds of events, user opinions, and daily life activities. In this article, we propose a novel attention-based deep relevance model for few-shot document filtering (named ADRM), inspired by the relevance feedback methodology proposed for ad hoc retrieval. ADRM calculates the relevance score between a document and a category by taking a set of seed words and a few seed documents relevant to the category. It constructs the category-specific conceptual representation of the document based on the corresponding seed words and seed documents. Specifically, to filter irrelevant yet noisy information in the seed documents, ADRM employs two types of attention mechanisms (namely whole-match attention and max-match attention ) and generates category-specific representations for them. Then ADRM is devised to extract the relevance signals by modeling the hidden feature interactions in the word embedding space. The relevance signals are extracted through a gated convolutional process, a self-attention layer, and a relevance aggregation layer. Extensive experiments on three real-world datasets show that ADRM consistently outperforms the existing technical alternatives, including the conventional classification and retrieval baselines, and the state-of-the-art deep relevance ranking models for few-shot document filtering. We also perform an ablation study to demonstrate that each component in ADRM is effective for enhancing filtering performance. Further analysis shows that ADRM is robust under varying parameter settings.

Spoken words can make the invisible visible-Testing the involvement of low-level visual representations in spoken word processing.

The notion that processing spoken (object) words involves activation of category-specific representations in visual cortex is a key prediction of modality-specific theories of representation that contrasts with theories assuming dedicated conceptual representational systems abstracted away from sensorimotor systems. In the present study, we investigated whether participants can detect otherwise invisible pictures of objects when they are presented with the corresponding spoken word shortly before the picture appears. Our results showed facilitated detection for congruent ("bottle" → picture of a bottle) versus incongruent ("bottle" → picture of a banana) trials. A second experiment investigated the time-course of the effect by manipulating the timing of picture presentation relative to word onset and revealed that it arises as soon as 200-400 ms after word onset and decays at 600 ms after word onset. Together, these data strongly suggest that spoken words can rapidly activate low-level category-specific visual representations that affect the mere detection of a stimulus, that is, what we see. More generally, our findings fit best with the notion that spoken words activate modality-specific visual representations that are low level enough to provide information related to a given token and at the same time abstract enough to be relevant not only for previously seen tokens but also for generalizing to novel exemplars one has never seen before. (PsycINFO Database Record

Early-latency categorical speech sound representations in the left inferior frontal gyrus

Efficient speech perception requires the mapping of highly variable acoustic signals to distinct phonetic categories. How the brain overcomes this many-to-one mapping problem has remained unresolved. To infer the cortical location, latency, and dependency on attention of categorical speech sound representations in the human brain, we measured stimulus-specific adaptation of neuromagnetic responses to sounds from a phonetic continuum. The participants attended to the sounds while performing a non-phonetic listening task and, in a separate recording condition, ignored the sounds while watching a silent film. Neural adaptation indicative of phoneme category selectivity was found only during the attentive condition in the pars opercularis (POp) of the left inferior frontal gyrus, where the degree of selectivity correlated with the ability of the participants to categorize the phonetic stimuli. Importantly, these category-specific representations were activated at an early latency of 115–140ms, which is compatible with the speed of perceptual phonetic categorization. Further, concurrent functional connectivity was observed between POp and posterior auditory cortical areas. These novel findings suggest that when humans attend to speech, the left POp mediates phonetic categorization through integration of auditory and motor information via the dorsal auditory stream.

Specific and common brain regions involved in the perception of faces and bodies and the representation of their emotional expressions

Many studies provide support for the role of the fusiform gyrus in face recognition and its sensitivity to emotional expressions. Recently, category-specific representation was also observed for neutral human bodies in the middle temporal/middle occipital gyrus (extrastriate body area) but it is not clear whether this area is also sensitive to emotional bodily expressions. Besides these areas, other regions that process the affective information carried by the face and the body may be common and/or specific to the face or the body. To clarify these issues we performed a systematic comparison of how the whole brain processes faces and bodies and how their affective information is represented. Participants categorized emotional facial and bodily expressions while brain activity was measured using functional magnetic resonance imaging. Our results show that, first, the amygdala and the fusiform gyrus are sensitive to recognition of facial and bodily fear signals. Secondly, the extrastriate body area–area V5/MT is specifically involved in processing bodies without being sensitive to the emotion displayed. Thirdly, other important areas such as the superior temporal sulcus, the parietal lobe and subcortical structures represent selectively facial and bodily expressions. Finally, some face/body differences in activation are a function of the emotion expressed.

Category specificity in the processing of color-related and form-related words: An ERP study

In this study, we investigated the spatio-temporal patterns of category-specific cortical activation elicited by the visual presentation of words whose meaning relates to a color or to a visual form or shape. We recorded the event-related potentials (ERPs) of ten healthy, right-handed volunteers while they passively read words presented tachistoscopically. As early as 150 ms after stimulus onset, the ERPs revealed significant neurophysiological differences between words and strings of hash marks. Around 200 ms after stimulus onset, we found significant differences in the ERPs elicited by color- and form-related words. We used minimum norm current estimates to investigate the spatial location of these differences. This revealed that, at the 150 ms peak, the activation advantage of words over sequences of hash marks was located in a left posterior area, proximal to what has previously been called the visual word form area. At the 200 ms peak, the advantage of words related to colors over words related to forms seemed to reside in temporal cortical areas, whereas the form-related words elicited greater activation in frontal areas than color-related words. These results provide evidence for early access to detailed category-specific representations of word meaning, with subtle differences in meaning being reflected in the activation of different cortical areas, as early as 200 ms after stimulus presentation. In line with previous studies, these differences can be related to the areas involved in the conceptual processing of sensory (visual) and action-related information.

Dissociation of numbers and objects in corticobasal degeneration and semantic dementia.

Semantic memory is thought to consist of category-specific representations of knowledge that may be selectively compromised in patients with neurodegenerative diseases, but this has been difficult to demonstrate reliably across object categories. The authors evaluated performance on several simple measures requiring number representations (including addition and magnitude judgments of single digits), and on a task that requires object representations (an object naming task) in patients with corticobasal degeneration (CBD; n = 13) and semantic dementia (SD; n = 15). They also examined regional cortical atrophy using voxel-based morphometric analyses of high resolution structural MRI in subgroups of five CBD patients and three SD patients. CBD patients were consistently more impaired on simple addition and magnitude judgment tasks requiring number representations compared to object representations. Impaired performance with numbers in CBD was associated with cortical atrophy in right parietal cortex. By comparison, SD patients demonstrated a greater impairment on a naming task requiring object representations relative to their performance on measures involving number representations. This was associated with left anterior temporal cortical atrophy. The cognitive and neuroanatomic dissociations between CBD and SD are consistent with the hypothesis that number and object representations constitute distinct domains in semantic memory, and these domains appear to be associated with distinct neural substrates.

Category-specific representation and processing of numbers and animal names across semantic tasks: A PET study

Category-specific representation and processing of numbers and animal names across semantic tasks: A PET study