Mid-level Representation Research Articles

Learning Invariant Inter-pixel Correlations for Superpixel Generation

Deep superpixel algorithms have made remarkable strides by substituting hand-crafted features with learnable ones. Nevertheless, we observe that existing deep superpixel methods, serving as mid-level representation operations, remain sensitive to the statistical properties (e.g., color distribution, high-level semantics) embedded within the training dataset. Consequently, learnable features exhibit constrained discriminative capability, resulting in unsatisfactory pixel grouping performance, particularly in untrainable application scenarios. To address this issue, we propose the Content Disentangle Superpixel (CDS) algorithm to selectively separate the invariant inter-pixel correlations and statistical properties, i.e., style noise. Specifically, We first construct auxiliary modalities that are homologous to the original RGB image but have substantial stylistic variations. Then, driven by mutual information, we propose the local-grid correlation alignment across modalities to reduce the distribution discrepancy of adaptively selected features and learn invariant inter-pixel correlations. Afterwards, we perform global-style mutual information minimization to enforce the separation of invariant content and train data styles. The experimental results on four benchmark datasets demonstrate the superiority of our approach to existing state-of-the-art methods, regarding boundary adherence, generalization, and efficiency. Code and pre-trained model are available at https://github.com/rookiie/CDSpixel.

Decoding face recognition abilities in the human brain.

Why are some individuals better at recognizing faces? Uncovering the neural mechanisms supporting face recognition ability has proven elusive. To tackle this challenge, we used a multimodal data-driven approach combining neuroimaging, computational modeling, and behavioral tests. We recorded the high-density electroencephalographic brain activity of individuals with extraordinary face recognition abilities-super-recognizers-and typical recognizers in response to diverse visual stimuli. Using multivariate pattern analyses, we decoded face recognition abilities from 1 s of brain activity with up to 80% accuracy. To better understand the mechanisms subtending this decoding, we compared representations in the brains of our participants with those in artificial neural network models of vision and semantics, as well as with those involved in human judgments of shape and meaning similarity. Compared to typical recognizers, we found stronger associations between early brain representations of super-recognizers and midlevel representations of vision models as well as shape similarity judgments. Moreover, we found stronger associations between late brain representations of super-recognizers and representations of the artificial semantic model as well as meaning similarity judgments. Overall, these results indicate that important individual variations in brain processing, including neural computations extending beyond purely visual processes, support differences in face recognition abilities. They provide the first empirical evidence for an association between semantic computations and face recognition abilities. We believe that such multimodal data-driven approaches will likely play a critical role in further revealing the complex nature of idiosyncratic face recognition in the human brain.

FERMixNet: An Occlusion Robust Facial Expression Recognition Model with Facial Mixing Augmentation and Mid-Level Representation Learning

FERMixNet: An Occlusion Robust Facial Expression Recognition Model with Facial Mixing Augmentation and Mid-Level Representation Learning

Depth- and semantics-aware multi-modal domain translation: Generating 3D panoramic color images from LiDAR point clouds

This work presents a new depth- and semantics-aware conditional generative model, named TITAN-Next, for cross-domain image-to-image translation in a multi-modal setup between LiDAR and camera sensors. The proposed model leverages scene semantics as a mid-level representation and is able to translate raw LiDAR point clouds to RGB-D camera images by solely relying on semantic scene segments. We claim that this is the first framework of its kind and it has practical applications in autonomous vehicles such as providing a fail-safe mechanism and augmenting available data in the target image domain. The proposed model is evaluated on the large-scale and challenging Semantic-KITTI dataset, and experimental findings show that it considerably outperforms the original TITAN-Net and other strong baselines by 23.7% margin in terms of IoU.

Image-based Navigation in Real-World Environments via Multiple Mid-level Representations: Fusion Models, Benchmark and Efficient Evaluation

AbstractRobot visual navigation is a relevant research topic. Current deep navigation models conveniently learn the navigation policies in simulation, given the large amount of experience they need to collect. Unfortunately, the resulting models show a limited generalization ability when deployed in the real world. In this work we explore solutions to facilitate the development of visual navigation policies trained in simulation that can be successfully transferred in the real world. We first propose an efficient evaluation tool to reproduce realistic navigation episodes in simulation. We then investigate a variety of deep fusion architectures to combine a set of mid-level representations, with the aim of finding the best merge strategy that maximize the real world performances. Our experiments, performed both in simulation and on a robotic platform, show the effectiveness of the considered mid-level representations-based models and confirm the reliability of the evaluation tool. The 3D models of the environment and the code of the validation tool are publicly available at the following link: https://iplab.dmi.unict.it/EmbodiedVN/.

A Texture Statistics Encoding Model Reveals Hierarchical Feature Selectivity across Human Visual Cortex.

Midlevel features, such as contour and texture, provide a computational link between low- and high-level visual representations. Although the nature of midlevel representations in the brain is not fully understood, past work has suggested a texture statistics model, called the P-S model (Portilla and Simoncelli, 2000), is a candidate for predicting neural responses in areas V1-V4 as well as human behavioral data. However, it is not currently known how well this model accounts for the responses of higher visual cortex to natural scene images. To examine this, we constructed single-voxel encoding models based on P-S statistics and fit the models to fMRI data from human subjects (both sexes) from the Natural Scenes Dataset (Allen et al., 2022). We demonstrate that the texture statistics encoding model can predict the held-out responses of individual voxels in early retinotopic areas and higher-level category-selective areas. The ability of the model to reliably predict signal in higher visual cortex suggests that the representation of texture statistics features is widespread throughout the brain. Furthermore, using variance partitioning analyses, we identify which features are most uniquely predictive of brain responses and show that the contributions of higher-order texture features increase from early areas to higher areas on the ventral and lateral surfaces. We also demonstrate that patterns of sensitivity to texture statistics can be used to recover broad organizational axes within visual cortex, including dimensions that capture semantic image content. These results provide a key step forward in characterizing how midlevel feature representations emerge hierarchically across the visual system.SIGNIFICANCE STATEMENT Intermediate visual features, like texture, play an important role in cortical computations and may contribute to tasks like object and scene recognition. Here, we used a texture model proposed in past work to construct encoding models that predict the responses of neural populations in human visual cortex (measured with fMRI) to natural scene stimuli. We show that responses of neural populations at multiple levels of the visual system can be predicted by this model, and that the model is able to reveal an increase in the complexity of feature representations from early retinotopic cortex to higher areas of ventral and lateral visual cortex. These results support the idea that texture-like representations may play a broad underlying role in visual processing.

Self-information of radicals: A new clue for zero-shot Chinese character recognition

Zero-shot Chinese character recognition (ZSCCR) is an important research topic in Chinese character recognition as it attempts to recognize unseen Chinese characters. As basic components and mid-level representations, radicals are significant for ZSCCR. However, previous methods treat the importance of radicals equally, ignoring the different contributions of radicals in distinguishing characters. In this paper, we propose the self-information of radicals (SIR) to measure the importance of radicals in recognizing Chinese characters. The proposed SIR can be easily adopted by two commonly used radical-based ZSCCR frameworks, i.e., sequence matching based and attribute embedding based. For sequence matching based ZSCCR, we propose a novel Chinese character uncertainty elimination (CUE) framework to alleviate the radical sequence mismatch problem. For attribute embedding based ZSCCR, we propose a novel radical information embedding (RIE) method that can highlight the importance of indispensable radicals and weaken the influence of some unnecessary radicals. We conducted comprehensive experiments on the CASIA-HWDB, ICDAR2013, CTW datasets, and AHCDB datasets to evaluate the proposed method. Experiments show that our proposed methods can achieve superior performance to the state-of-the-art methods, which demonstrate the effectiveness and the high extensibility of the proposed SIR.

Joint multimodal sentiment analysis based on information relevance

Social media users are increasingly turning to express opinions with both images and text, while the visual content and text description may cover some conflicting information diverse from each other. Information relevance refers to the matching degree between cross-modal features at the emotional semantic level, which is not systematically studied. In order to exploit the discriminative features and the internal correlation among different modalities, the mid-level representation extracted by a visual sentiment concept classifier is used to determine information relevance, with the integration of other features, including attended textual and visual features. Then grid search is applied to tune weighting coefficients of the decision fusion scheme, followed by a multimodal adaptive method for joint sentiment analysis based on image-text relevance. The superiority of our architecture approach is demonstrated experimentally by comparing it with several state-of-the-art baselines, such as the vision-aware language modeling approach and contrastive learning-based model. The results indicate that fused multiple features lead to more precise classification than unimodal ones, while the contributions of each single modality differ obviously in emotional expression. Besides, the performance of every involved model varies markedly per dataset of different content correlation, which proves that it is of extensive theoretical significance and application prospect to introduce the image-text relevance classifier into a multimodal task.

Increasing the Efficiency of Policy Learning for Autonomous Vehicles by Multi-Task Representation Learning

Driving in a dynamic, multi-agent, and complex urban environment is a difficult task requiring a complex decision-making policy. The learning of such a policy requires a state representation that can encode the entire environment. Mid-level representations that encode a vehicle’s environment as images have become a popular choice. Still, they are quite high-dimensional, limiting their use in data-hungry approaches such as reinforcement learning. In this article, we propose to learn a low-dimensional and rich latent representation of the environment by leveraging the knowledge of relevant semantic factors. To do this, we train an encoder-decoder deep neural network to predict multiple application-relevant factors such as the trajectories of other agents and the ego car. Furthermore, we propose a hazard signal based on other vehicles’ future trajectories and the planned route which is used in conjunction with the learned latent representation as input to a down-stream policy. We demonstrate that using the multi-head encoder-decoder neural network results in a more informative representation than a standard single-head model. In particular, the proposed representation learning and the hazard signal help reinforcement learning to learn faster, with increased performance and less data than baseline methods.

Image-Based Navigation in Real-World Environments Via Multiple Mid-Level Representations: Fusion Models, Benchmark and Efficient Evaluation

Image-Based Navigation in Real-World Environments Via Multiple Mid-Level Representations: Fusion Models, Benchmark and Efficient Evaluation

Segment-based bag of visual words model for urban land cover mapping using polarimetric SAR data

Producing high-resolution urban land cover maps is essential for decision-making and urban management. In this regard, Synthetic Aperture Radar (SAR), especially Polarimetric SAR (PolSAR), has served as a valuable data source to fulfill this task. Direct conversion of low-level features into high-level Land Cover (LC) concept may reduce the final classification accuracy. Therefore, mid-level representation models, such as Bag of Visual Words (BOVW), were employed to resolve the existing semantic gap challenge. In this paper, a Segment-based BOVW (Seg-BOVW) model was developed for urban land cover classification using PolSAR data. To this end, two PolSAR data over San Francisco Bay (SFB) and Flevoland (FL) acquired by RADARSAT-2 were employed to comprehensively evaluate the Seg-BOVW model's performance. First, to exploit the full potential of PolSAR data, 169 low-level features in four categories: (1) original, (2) polarimetric, (3) texture, and (4) decomposition features were extracted. Afterward, a Multi-Objective Genetic Algorithm (MOGA) was implemented to investigate the importance of low-level features for urban land cover mapping. This step resulted in selecting 14 and 6 low-level features, as the high contributing features, for SFB and FL datasets, respectively. The Seg-BOVW model achieved significant overall accuracies of 96.02% and 98.82% for SFB and FL, respectively, indicating the high potential of the proposed method for urban land cover classification. Furthermore, a comparison with other well-known algorithms of Support Vector Machine (SVM), Artificial Neural Networks (ANN), and Convolutional Neural Networks (CNN) was made, suggesting the capability of the Seg-BOVW model to improve the urban land cover classification results. Finally, the Seg-BOVW model was tested with two other PolSAR datasets acquired with different sensors over SFB to examine its applicability with different datasets.

A Modified HSIFT Descriptor for Medical Image Classification of Anatomy Objects

Modeling low level features to high level semantics in medical imaging is an important aspect in filtering anatomy objects. Bag of Visual Words (BOVW) representations have been proven effective to model these low level features to mid level representations. Convolutional neural nets are learning systems that can automatically extract high-quality representations from raw images. However, their deployment in the medical field is still a bit challenging due to the lack of training data. In this paper, learned features that are obtained by training convolutional neural networks are compared with our proposed hand-crafted HSIFT features. The HSIFT feature is a symmetric fusion of a Harris corner detector and the Scale Invariance Transform process (SIFT) with BOVW representation. The SIFT process is enhanced as well as the classification technique by adopting bagging with a surrogate split method. Quantitative evaluation shows that our proposed hand-crafted HSIFT feature outperforms the learned features from convolutional neural networks in discriminating anatomy image classes.

Deep neural network models of visual cortex reveal curvature and real-world size as organizing principles of mid-level representation

Mid-level visual features directly support an array of behaviors; thus, they may be critical for understanding the functional organization of visual cortex. However, attempts at characterizing mid-level features have been hampered by the difficulty of describing these features in words—they exist in an “ineffable valley” between the describable patterns of low-level vision (e.g., edges) and the commonsense concepts of visual cognition (e.g., objects). Here we developed a novel approach to identify interpretable emergent properties of mid-level representations in deep neural network (DNN) models of visual cortex. Using this approach, we examined DNN models that were fit to scene-evoked fMRI responses in category-selective regions of visual cortex—specifically, scene-selective cortex (sceneDNN) and object-selective cortex (objectDNN). Our method uses a semantically-guided image-occlusion procedure to systematically characterize how DNN activations are driven by the classes of objects within a scene. We examined the relationship between mid-level features and several object properties that have previously been associated with response preferences in visual cortex: curvature, real-world size, animacy, naturalness, and spatial stability. We found that while mid-level features appear complex and difficult to describe at a surface level, large-scale computational analyses can reveal a latent underlying relationship to interpretable object properties. Specifically, we found that the mid-level representations of the sceneDNN support a latent preference for objects that are boxy and large in real-world size. In contrast, mid-level representations of the objectDNN support a complementary preference for objects that are curvy and small in real-world size. These effects were robust to variations of model hyperparameters and were reproducible across different DNN models. Our findings show that curvature and real-world size are emergent organizing principles of mid-level visual representation, and they suggest that differences in mid-level feature tuning may be critical for understanding the organization of visual cortex into category-selective patches.

DeepFlux for Skeleton Detection in the Wild

The medial axis, or skeleton, is a fundamental object representation that has been extensively used in shape recognition. Yet, its extension to natural images has been challenging due to the large appearance and scale variations of objects and complex background clutter that appear in this setting. In contrast to recent methods that address skeleton extraction as a binary pixel classification problem, in this article we present an alternative formulation for skeleton detection. We follow the spirit of flux-based algorithms for medial axis recovery by training a convolutional neural network to predict a two-dimensional vector field encoding the flux representation. The skeleton is then recovered from the flux representation, which captures the position of skeletal pixels relative to semantically meaningful entities (e.g., image points in spatial context, and hence the implied object boundaries), resulting in precise skeleton detection. Moreover, since the flux representation is a region-based vector field, it is better able to cope with object parts of large width. We evaluate the proposed method, termed DeepFlux, on six benchmark datasets, consistently achieving superior performance over state-of-the-art methods. Finally, we demonstrate an application of DeepFlux, augmented with a skeleton scale estimation module, to detect objects in aerial images. This combination yields results that are competitive with models trained specifically for object detection, showcasing the versatility and effectiveness of mid-level representations in high-level tasks. An implementation of our method is available at https://github.com/YukangWang/DeepFlux .

CTC-Based Learning of Chroma Features for Score–Audio Music Retrieval

This paper deals with a scoreaudio music retrieval task where the aim is to find relevant audio recordings of Western classical music, given a short monophonic musical theme in symbolic notation as a query. Strategies for comparing score and audio data are often based on a common mid-level representation, such as chroma features, which capture melodic and harmonic properties. Recent studies demonstrated the effectiveness of deep neural networks that learn task-specific mid-level representations. Usually, such supervised learning approaches require scoreaudio pairs where individual note events of the score are aligned to the corresponding time positions of the audio excerpt. However, in practice, it is tedious to generate such strongly aligned training pairs. As one contribution of this paper, we show how to apply the Connectionist Temporal Classification (CTC) loss in the training procedure, which only uses weakly aligned training pairs. In such a pair, only the time positions of the beginning and end of a theme occurrence are annotated in an audio recording, rather than requiring local alignment annotations. We evaluate the resulting features in our theme retrieval scenario and show that they improve the state of the art for this task. As a main result, we demonstrate that with the CTC-based training procedure using weakly annotated data, we can achieve results almost as good as with strongly annotated data. Furthermore, we assess our chroma features in depth by inspecting their temporal smoothness or granularity as an important property and by analyzing the impact of different degrees of musical complexity on the features.

Cogmic space for narrative-based world representation

Representing a world or a physical/social environment in an agent’s cognitive system is essential for creating human-like artificial intelligence. This study takes a story-centered approach to this issue. In this context, a story refers to an internal representation involving a narrative structure, which is assumed to be a common form of organizing past, present, future, and fictional events and situations. In the artificial intelligence field, a story or narrative is traditionally treated as a symbolic representation. However, a symbolic story representation is limited in its representational power to construct a rich world. For example, a symbolic story representation is unfit to handle the sensory/bodily dimension of a world. In search of a computational theory for narrative-based world representation, this study proposes the conceptual framework of a Cogmic Space for a comic strip-like representation of a world. In the proposed framework, a story is positioned as a mid-level representation, in which the conceptual and sensory/bodily dimensions of a world are unified. The events and their background situations that constitute a story are unified into a sequence of panels. Based on this structure, a representation (i.e., a story) and the represented environment are connected via an isomorphism of their temporal, spatial, and relational structures. Furthermore, the framework of a Cogmic Space is associated with the generative aspect of representations, which is conceptualized in terms of unconscious- and conscious-level processes/representations. Finally, a proof-of-concept implementation is presented to provide a concrete account of the proposed framework.

Visual sentiment analysis via deep multiple clustered instance learning

The increasing tendency of people expressing opinions via images online has motivated the development of automatic assessment of sentiment from visual contents. Based on the observation that visual sentiment is conveyed through many visual elements in images, we put forward to tackle visual sentiment analysis under multiple instance learning (MIL) formulation. We propose a deep multiple clustered instance learning formulation, under which a deep multiple clustered instance learning network (DMCILN) is constructed for visual sentiment analysis. Specifically, the input image is converted into a bag of instances through visual instance generation module, which is composed of a pre-trained convolutional neural network (CNN) and two adaptation layers. Then, a fuzzy c-means routing algorithm is introduced for generating clustered instances as semantic mid-level representation to bridge the instance-to-bag gap. To explore the relationships between clustered instances and bags, we construct an attention based MIL pooling layer for representing bag features. A multi-head mechanism is integrated to form MIL ensembles, which enables to weigh the contribution of each clustered instance in different subspaces for generating more robust bag representation. Finally, we conduct extensive experiments on several datasets, and the experimental results verify the feasibility of our proposed approach for visual sentiment analysis.

Learning Contour-Based Mid-Level Representation for Shape Classification

This article proposes a novel contour-based mid-level shape description method for shape classification. This method resolves the shortcomings of low-level shape descriptors in dealing with the shapes of objects with large intra-class changes and non-linear deformation (articulation, occlusion and noise), thus improving the accuracy of shape classification. First, we extract the outer contour of an object and sample it. We next describe each sampling point on the shape contour with a triangular feature and regard it as a local feature. Then, a shape codebook is learned, and the Fisher vector encoding method is used to produce a compact mid-level shape feature. Finally, the learned mid-level shape features are sent to the linear support vector machine (SVM) classifier for shape recognition. The proposed method has been extensively tested on several standard shape datasets, and the experimental results show that our approach attains high accuracy of shape classification. Comparisons to other state-of-the-art shape classification approaches further prove the superiority and effectiveness of our method.

SSNet: Learning Mid-Level Image Representation Using Salient Superpixel Network

In the standard bag-of-visual-words (BoVW) model, the burstiness problem of features and the ignorance of high-order information often weakens the discriminative power of image representation. To tackle them, we present a novel framework, named the Salient Superpixel Network, to learn the mid-level image representation. For reducing the impact of burstiness occurred in the background region, we use the salient regions instead of the whole image to extract local features, and a fast saliency detection algorithm based on the Gestalt grouping principle is proposed to generate image saliency maps. In order to introduce the high-order information, we propose a weighted second-order pooling (WSOP) method, which is capable of exploiting the high-order information and further alleviating the impact of burstiness in the foreground region. Then, we conduct experiments on six image classification benchmark datasets, and the results demonstrate the effectiveness of the proposed framework with either the handcrafted or the off-the-shelf CNN features.

Heterogeneous Transfer Learning for Hyperspectral Image Classification Based on Convolutional Neural Network

Deep convolutional neural networks (CNNs) have shown their outstanding performance in the hyperspectral image (HSI) classification. The success of CNN-based HSI classification relies on the availability sufficient training samples. However, the collection of training samples is expensive and time consuming. Besides, there are many pretrained models on large-scale data sets, which extract the general and discriminative features. The proper reusage of low-level and midlevel representations will significantly improve the HSI classification accuracy. The large-scale ImageNet data set has three channels, but HSI contains hundreds of channels. Therefore, there are several difficulties to simply adapt the pretrained models for the classification of HSIs. In this article, heterogeneous transfer learning for HSI classification is proposed. First, a mapping layer is used to handle the issue of having different numbers of channels. Then, the model architectures and weights of the CNN trained on the ImageNet data sets are used to initialize the model and weights of the HSI classification network. Finally, a well-designed neural network is used to perform the HSI classification task. Furthermore, attention mechanism is used to adjust the feature maps due to the difference between the heterogeneous data sets. Moreover, controlled random sampling is used as another training sample selection method to test the effectiveness of the proposed methods. Experimental results on four popular hyperspectral data sets with two training sample selection strategies show that the transferred CNN obtains better classification accuracy than that of state-of-the-art methods. In addition, the idea of heterogeneous transfer learning may open a new window for further research.