High-level Visual Representations Research Articles

Learning high-level visual representations from a child’s perspective without strong inductive biases

Learning high-level visual representations from a child’s perspective without strong inductive biases

Klasifikacija vrsta drva prema slikama uz pomoć dvodimenzionalne konvolucijske neuronske mreže

The woodworking industry’s recognition and classification of timber is essential for trade, production and timber science. Traditional methods of identifying wood types are complex, time-consuming, costly and require expertise in wood science. Traditional techniques have been replaced by convolutional neural networks (CNNs), a deep learning tool to better identify wood species. In contrast to earlier studies that used pretrained models, a novel architecture designed explicitly for the WOOD-AUTH dataset was proposed in this study to develop a new 2D CNN model. The data collection encompasses high-level visual representations of 12 distinct types of timber. It is aimed to create a simpler and faster model as an alternative to time-consuming and heavy wood classification models. Compared to previous studies, this research worked with a newly structured 2D CNN network based on 12 wood species. High accuracy and fast computation time were achieved using fewer numbers (three layers) of the convolutional neural network. The proposed model achieved 94 % accuracy, 87 % precision, 81 % recall, 80 % F1 score and 112 minutes 27 seconds computation time. The 2D CNN model performed better than the transfer learning models regarding training epochs. The primary benefit of the model is its ability to achieve high accuracy with lower computation time, even at high epochs compared to other models. The introduced 2D CNN model produced satisfactory outcomes for wood species classification.

A Texture Statistics Encoding Model Reveals Hierarchical Feature Selectivity across Human Visual Cortex.

Midlevel features, such as contour and texture, provide a computational link between low- and high-level visual representations. Although the nature of midlevel representations in the brain is not fully understood, past work has suggested a texture statistics model, called the P-S model (Portilla and Simoncelli, 2000), is a candidate for predicting neural responses in areas V1-V4 as well as human behavioral data. However, it is not currently known how well this model accounts for the responses of higher visual cortex to natural scene images. To examine this, we constructed single-voxel encoding models based on P-S statistics and fit the models to fMRI data from human subjects (both sexes) from the Natural Scenes Dataset (Allen et al., 2022). We demonstrate that the texture statistics encoding model can predict the held-out responses of individual voxels in early retinotopic areas and higher-level category-selective areas. The ability of the model to reliably predict signal in higher visual cortex suggests that the representation of texture statistics features is widespread throughout the brain. Furthermore, using variance partitioning analyses, we identify which features are most uniquely predictive of brain responses and show that the contributions of higher-order texture features increase from early areas to higher areas on the ventral and lateral surfaces. We also demonstrate that patterns of sensitivity to texture statistics can be used to recover broad organizational axes within visual cortex, including dimensions that capture semantic image content. These results provide a key step forward in characterizing how midlevel feature representations emerge hierarchically across the visual system.SIGNIFICANCE STATEMENT Intermediate visual features, like texture, play an important role in cortical computations and may contribute to tasks like object and scene recognition. Here, we used a texture model proposed in past work to construct encoding models that predict the responses of neural populations in human visual cortex (measured with fMRI) to natural scene stimuli. We show that responses of neural populations at multiple levels of the visual system can be predicted by this model, and that the model is able to reveal an increase in the complexity of feature representations from early retinotopic cortex to higher areas of ventral and lateral visual cortex. These results support the idea that texture-like representations may play a broad underlying role in visual processing.

Human Posture Detection Using Image Augmentation and Hyperparameter-Optimized Transfer Learning Algorithms

With the advancement in pose estimation techniques, human posture detection recently received considerable attention in many applications, including ergonomics and healthcare. When using neural network models, overfitting and poor performance are prevalent issues. Recently, convolutional neural networks (CNNs) were successfully used for human posture recognition from human images due to their superior multiscale high-level visual representations over hand-engineering low-level characteristics. However, calculating millions of parameters in a deep CNN requires a significant number of annotated examples, which prohibits many deep CNNs such as AlexNet and VGG16 from being used on issues with minimal training data. We propose a new three-phase model for decision support that integrates CNN transfer learning, image data augmentation, and hyperparameter optimization (HPO) to address this problem. The model is used as part of a new decision support framework for the optimization of hyperparameters for AlexNet, VGG16, CNN, and multilayer perceptron (MLP) models for accomplishing optimal classification results. The AlexNet and VGG16 transfer learning algorithms with HPO are used for human posture detection, while CNN and Multilayer Perceptron (MLP) were used as standard classifiers for contrast. The HPO methods are essential for machine learning and deep learning algorithms because they directly influence the behaviors of training algorithms and have a major impact on the performance of machine learning and deep learning models. We used an image data augmentation technique to increase the number of images to be used for model training to reduce model overfitting and improve classification performance using the AlexNet, VGG16, CNN, and MLP models. The optimal combination of hyperparameters was found for the four models using a random-based search strategy. The MPII human posture datasets were used to test the proposed approach. The proposed models achieved an accuracy of 91.2% using AlexNet, 90.2% using VGG16, 87.5% using CNN, and 89.9% using MLP. The study is the first HPO study executed on the MPII human pose dataset.

CheXGAT: A disease correlation-aware network for thorax disease diagnosis from chest X-ray images

Chest X-ray (CXR) imaging is one of the most common diagnostic imaging techniques in clinical diagnosis and is usually used for radiological examinations to screen for thorax diseases. In this paper, we propose a novel computer-aided diagnosis (CAD) system based on a hybrid deep learning model composed of a convolutional neural network (CNN) and a graph neural network (GNN). The system is intended to explore implicit correlations between thorax diseases to aid in the multilabel chest X-ray image classification task, which we term ‶CheXGAT‶. Specifically, the proposed CheXGAT framework comprises two main modules: an image representation learning (IRL) module and a graph representation learning (GRL) module. We employ the IRL module to learn high-level visual representation features from the CXR image. From the GRL module, the self-attention mechanism aggregates neighborhood features from the graphic structure to enhance the implicit correlation between thorax diseases. We adopted a data-driven method to create a disease correlation matrix that works on the message passing and aggregation process for the nodes in the GRL module. After end-to-end training, the GRL module enhances the correlation between thorax diseases to improve diagnosis performance. We performed experiments on the NIH Chest X-ray14 dataset, which contains 112,120 frontal-view radiographs; each image has multiple thorax disease labels. In the experimental results, the average AUC score of our proposed CheXGAT model reached 0.8266, and the AUC scores of Emphysema and Hernia reached 0.9447 and 0.9313, respectively. In addition, we visualized explanations via a gradient-based localization method in our proposed CheXGAT framework. Compared with previous studies, the experimental results show the competitive performance of our framework. Specifically, we propose a CAD system that uses a hybrid model to help radiologists identify CXR images from 14 chest diseases for clinical diagnosis.

Value of Temporal Dynamics Information in Driving Scene Segmentation

Semantic scene segmentation has primarily been addressed by forming high-level visual representations of single images. The problem of semantic segmentation in dynamic scenes has begun to receive attention with the video object segmentation and tracking problem. While there has been some recent work attempt to use deep learning models on the video level, what is not known is how the temporal dynamics information is contributing to the full scene segmentation. Moreover, most existing datasets only provide full scene annotation on non-consecutive images to ensure the variability of scenes, making it even harder to explore novel methods on video-level modeling. To address the above issues, our work takes steps to explore the behavior of modern spatiotemporal modeling approaches by: 1) constructing the MIT DriveSeg dataset, a large-scale video driving scene segmentation dataset, densely annotated for pixel-level semantic classes with 5000 consecutive video frames, and 2) proposing a joint-learning framework that reveals the contribution of temporal dynamics information in regard to different semantic classes in the driving scene. This work is intended to help assess current methods and support further exploration of the value of temporal dynamics information in video-level scene segmentation.

A connectivity-constrained computational account of topographic organization in primate high-level visual cortex

Inferotemporal (IT) cortex in humans and other primates is topographically organized, containing multiple hierarchically organized areas selective for particular domains, such as faces and scenes. This organization is commonly viewed in terms of evolved domain-specific visual mechanisms. Here, we develop an alternative, domain-general and developmental account of IT cortical organization. The account is instantiated in interactive topographic networks (ITNs), a class of computational models in which a hierarchy of model IT areas, subject to biologically plausible connectivity-based constraints, learns high-level visual representations optimized for multiple domains. We find that minimizing a wiring cost on spatially organized feedforward and lateral connections, alongside realistic constraints on the sign of neuronal connectivity within model IT, results in a hierarchical, topographic organization. This organization replicates a number of key properties of primate IT cortex, including the presence of domain-selective spatial clusters preferentially involved in the representation of faces, objects, and scenes; columnar responses across separate excitatory and inhibitory units; and generic spatial organization whereby the response correlation of pairs of units falls off with their distance. We thus argue that topographic domain selectivity is an emergent property of a visual system optimized to maximize behavioral performance under generic connectivity-based constraints.

Closing the gap between single-unit and neural population codes: Insights from deep learning in face recognition.

Single-unit responses and population codes differ in the “read-out” information they provide about high-level visual representations. Diverging local and global read-outs can be difficult to reconcile with in vivo methods. To bridge this gap, we studied the relationship between single-unit and ensemble codes for identity, gender, and viewpoint, using a deep convolutional neural network (DCNN) trained for face recognition. Analogous to the primate visual system, DCNNs develop representations that generalize over image variation, while retaining subject (e.g., gender) and image (e.g., viewpoint) information. At the unit level, we measured the number of single units needed to predict attributes (identity, gender, viewpoint) and the predictive value of individual units for each attribute. Identification was remarkably accurate using random samples of only 3% of the network's output units, and all units had substantial identity-predicting power. Cross-unit responses were minimally correlated, indicating that single units code non-redundant identity cues. Gender and viewpoint classification required large-scale pooling of units—individual units had weak predictive power. At the ensemble level, principal component analysis of face representations showed that identity, gender, and viewpoint separated into high-dimensional subspaces, ordered by explained variance. Unit-based directions in the representational space were compared with the directions associated with the attributes. Identity, gender, and viewpoint contributed to all individual unit responses, undercutting a neural tuning analogy. Instead, single-unit responses carry superimposed, distributed codes for face identity, gender, and viewpoint. This undermines confidence in the interpretation of neural representations from unit response profiles for both DCNNs and, by analogy, high-level vision.

Face Recognition by Humans and Machines: Three Fundamental Advances from Deep Learning.

Deep learning models currently achieve human levels of performance on real-world face recognition tasks. We review scientific progress in understanding human face processing using computational approaches based on deep learning. This review is organized around three fundamental advances. First, deep networks trained for face identification generate a representation that retains structured information about the face (e.g., identity, demographics, appearance, social traits, expression) and the input image (e.g., viewpoint, illumination). This forces us to rethink the universe of possible solutions to the problem of inverse optics in vision. Second, deep learning models indicate that high-level visual representations of faces cannot be understood in terms of interpretable features. This has implications for understanding neural tuning and population coding in the high-level visual cortex. Third, learning in deep networks is a multistep process that forces theoretical consideration of diverse categories of learning that can overlap, accumulate over time, and interact. Diverse learning types are needed to model the development of human face processing skills, cross-race effects, and familiarity with individual faces.

Representational Content of Oscillatory Brain Activity during Object Recognition: Contrasting Cortical and Deep Neural Network Hierarchies.

Numerous theories propose a key role for brain oscillations in visual perception. Most of these theories postulate that sensory information is encoded in specific oscillatory components (e.g., power or phase) of specific frequency bands. These theories are often tested with whole-brain recording methods of low spatial resolution (EEG or MEG), or depth recordings that provide a local, incomplete view of the brain. Opportunities to bridge the gap between local neural populations and whole-brain signals are rare. Here, using representational similarity analysis (RSA) in human participants we explore which MEG oscillatory components (power and phase, across various frequency bands) correspond to low or high-level visual object representations, using brain representations from fMRI, or layer-wise representations in seven recent deep neural networks (DNNs), as a template for low/high-level object representations. The results showed that around stimulus onset and offset, most transient oscillatory signals correlated with low-level brain patterns (V1). During stimulus presentation, sustained β (∼20 Hz) and γ (>60 Hz) power best correlated with V1, while oscillatory phase components correlated with IT representations. Surprisingly, this pattern of results did not always correspond to low-level or high-level DNN layer activity. In particular, sustained β band oscillatory power reflected high-level DNN layers, suggestive of a feed-back component. These results begin to bridge the gap between whole-brain oscillatory signals and object representations supported by local neuronal activations.

Seeing through disguise: Getting to know you with a deep convolutional neural network

People use disguise to look unlike themselves (evasion) or to look like someone else (impersonation). Evasion disguise challenges human ability to see an identity across variable images; Impersonation challenges human ability to tell people apart. Personal familiarity with an individual face helps humans to see through disguise. Here we propose a model of familiarity based on high-level visual learning mechanisms that we tested using a deep convolutional neural network (DCNN) trained for face identification. DCNNs generate a face space in which identities and images co-exist in a unified computational framework, that is categorically structured around identity, rather than retinotopy. This allows for simultaneous manipulation of mechanisms that contrast identities and cluster images. In Experiment 1, we measured the DCNN's baseline accuracy (unfamiliar condition) for identification of faces in no disguise and disguise conditions. Disguise affected DCNN performance in much the same way it affects human performance for unfamiliar faces in disguise (cf. Noyes & Jenkins, 2019). In Experiment 2, we simulated familiarity for individual identities by averaging the DCNN-generated representations from multiple images of each identity. Averaging improved DCNN recognition of faces in evasion disguise, but reduced the ability of the DCNN to differentiate identities of similar appearance. In Experiment 3, we implemented a contrast learning technique to simultaneously teach the DCNN appearance variation and identity contrasts between different individuals. This facilitated identification with both evasion and impersonation disguise. Familiar face recognition requires an ability to group images of the same identity together and separate different identities. The deep network provides a high-level visual representation for face recognition that supports both of these mechanisms of face learning simultaneously.

Serial dependence of facial identity reflects high-level face coding

Serial dependence of facial identity is a type of bias where the perceived identity of a face is biased towards a previously presented face. There are individual differences in serial dependence strength and tuning (how the strength varies depending on stimuli similarity), and previous research has shown that both stronger and more narrowly tuned serial dependence of facial identity is associated with better face recognition abilities. These results are consistent with the idea that this bias plays a functional role in face perception. It is important, therefore, to determine whether serial dependence of facial identity reflects a high-level face-coding mechanism acting on the identity of a face or instead predominantly reflects a bias in low-level features, which are also subject to serial dependence. We first sought evidence that serial dependence of facial identity survived changes in low-level visual features, by varying face viewpoint between successive stimuli. We found that serial dependence persisted across changes in viewpoint, arguing against an entirely low-level locus for this bias. We next tested whether the bias was affected by inversion, as sensitivity to inversion is argued to be a characteristic of high-level face-selective processing. Serial dependence was stronger and more narrowly tuned for upright than inverted faces. Taken together, our results are consistent with the view that serial dependence of facial identity affects high-level visual representations and may reflect a face-coding mechanism that is operating at the level of facial identity.

Blind image quality assessment with hierarchy: Degradation from local structure to deep semantics

Though blind image quality assessment (BIQA) is highly desired in perceptual-oriented image processing systems, it is extremely difficult to design a reliable BIQA method. With the help of the prior knowledge, the human visual system (HVS) hierarchically perceives the quality degradation during the visual recognition. Inspired by this, we suggest different levels of distortion generate individual degradations on hierarchical features, and propose to consider the degradations on both low and high level features for quality prediction. By mimicking the orientation selectivity (OS) mechanism in the primary visual cortex, an OS based local structure is designed for low-level visual information representation. At the meantime, the deep residual network, which possesses multiple levels for feature integration, is employed to extract the deep semantics for high-level visual content representation. By fusing the local structure and the deep semantics, a hierarchical feature set is acquired. Next, the correlations between the degradations of image qualities and their corresponding hierarchical feature sets are analyzed, and a novel hierarchical feature degradation (HFD) based BIQA (HFD-BIQA) method is built. Experimental results on the legacy and wild image quality assessment databases demonstrate the prediction accuracy of the proposed HFD-BIQA method, and verify that the HFD-BIQA performs highly consistent with the subjective perception.

Picture it in your mind: generating high level visual representations from textual descriptions

In this paper we tackle the problem of image search when the query is a short textual description of the image the user is looking for. We choose to implement the actual search process as a similarity search in a visual feature space, by learning to translate a textual query into a visual representation. Searching in the visual feature space has the advantage that any update to the translation model does not require to reprocess the, typically huge, image collection on which the search is performed. We propose Text2Vis, a neural network that generates a visual representation, in the visual feature space of the fc6-fc7 layers of ImageNet, from a short descriptive text. Text2Vis optimizes two loss functions, using a stochastic loss-selection method. A visual-focused loss is aimed at learning the actual text-to-visual feature mapping, while a text-focused loss is aimed at modeling the higher-level semantic concepts expressed in language and countering the overfit on non-relevant visual components of the visual loss. We report preliminary results on the MS-COCO dataset.

On Hardware Programmable Network Dynamics With a Chemistry-Inspired Abstraction

Chemical algorithms are statistical control algorithms described and represented as chemical reaction networks. They are analytically tractable, they reinforce a deterministic state-to-dynamics relation, they have configurable stability properties, and they are directly implemented in state space using a high-level visual representation. These properties make them attractive solutions for traffic shaping and generally the control of dynamics in computer networks. In this paper, we present a framework for deploying chemical algorithms on field programmable gate arrays. Besides substantial computational acceleration, we introduce a low-overhead approach for hardware-level programmability and re-configurability of these algorithms at runtime , and without service interruption. We believe that this is a promising approach for expanding the control-plane programmability of software defined networks (SDN), to enable programmable network dynamics . To this end, the simple high-level abstractions of chemical algorithms offer an ideal northbound interface to the hardware, aligned with other programming primitives of SDN ( e.g. , flow rules).

Neuropsychological evidence for the temporal dynamics of category-specific naming

ABSTRACTMultiple accounts have been proposed to explain category-specific recognition impairments. Some suggest that category-specific deficits may be caused by a deficit in recurrent processing between the levels of a hierarchically organized visual object recognition system. Here, we tested predictions of interactive processing theories on the emergence of category-selective naming deficits in neurologically intact observers and in patient GA, a single case showing a category-specific impairment for natural objects after a herpes simplex encephalitis infection. Fragmented object outlines were repeatedly presented until correct naming occurred (maximum 10 times), and the fragments increased in length with every repetition. We studied how shape complexity, object category, and fragment curvature influence the timing of correct object identification. The results of a survival analysis are consistent with the idea that deficits in recurrent processing between low- and high-level visual object representations can cause category-selective impairments.

Sparse representation-based 3D model retrieval

In this study, we leveraged the sparse representation for multi-modal information fusion to handle 3D model retrieval problem. First, SIFT feature is extracted to represent the visual appearance of 2D view images for each 3D models. With this low-level feature representation, the Latent Dirichlet Allocation model is learned to generate the high-level & discriminative visual representation for individual 3D model. Then, we utilize the sparse representation framework to handle the key problem, the similarity measure between two different 3D models, for model retrieval. The performance of the proposed method is evaluated on the novel MV-RED 3D object dataset, which contains both RGB and depth 3D model data. The comparison experiments demonstrate the proposed sparse representation-based framework can benefit from multi-modal information fusion and consequently augment the performance.

Metagene Profiles Analyses Reveal Regulatory Element's Factor-Specific Recruitment Patterns.

ChIP-Sequencing (ChIP-Seq) provides a vast amount of information regarding the localization of proteins across the genome. The aggregation of ChIP-Seq enrichment signal in a metagene plot is an approach commonly used to summarize data complexity and to obtain a high level visual representation of the general occupancy pattern of a protein. Here we present the R package metagene, the graphical interface Imetagene and the companion package similaRpeak. Together, they provide a framework to integrate, summarize and compare the ChIP-Seq enrichment signal from complex experimental designs. Those packages identify and quantify similarities or dissimilarities in patterns between large numbers of ChIP-Seq profiles. We used metagene to investigate the differential occupancy of regulatory factors at noncoding regulatory regions (promoters and enhancers) in relation to transcriptional activity in GM12878 B-lymphocytes. The relationships between occupancy patterns and transcriptional activity suggest two different mechanisms of action for transcriptional control: i) a “gradient effect” where the regulatory factor occupancy levels follow transcription and ii) a “threshold effect” where the regulatory factor occupancy levels max out prior to reaching maximal transcription. metagene, Imetagene and similaRpeak are implemented in R under the Artistic license 2.0 and are available on Bioconductor.

Semantics of the Visual Environment Encoded in Parahippocampal Cortex.

Semantic representations capture the statistics of experience and store this information in memory. A fundamental component of this memory system is knowledge of the visual environment, including knowledge of objects and their associations. Visual semantic information underlies a range of behaviors, from perceptual categorization to cognitive processes such as language and reasoning. Here we examine the neuroanatomic system that encodes visual semantics. Across three experiments, we found converging evidence indicating that knowledge of verbally mediated visual concepts relies on information encoded in a region of the ventral-medial temporal lobe centered on parahippocampal cortex. In an fMRI study, this region was strongly engaged by the processing of concepts relying on visual knowledge but not by concepts relying on other sensory modalities. In a study of patients with the semantic variant of primary progressive aphasia (semantic dementia), atrophy that encompassed this region was associated with a specific impairment in verbally mediated visual semantic knowledge. Finally, in a structural study of healthy adults from the fMRI experiment, gray matter density in this region related to individual variability in the processing of visual concepts. The anatomic location of these findings aligns with recent work linking the ventral-medial temporal lobe with high-level visual representation, contextual associations, and reasoning through imagination. Together, this work suggests a critical role for parahippocampal cortex in linking the visual environment with knowledge systems in the human brain.

Recurrent Processing during Object Recognition.

How does the brain learn to recognize objects visually, and perform this difficult feat robustly in the face of many sources of ambiguity and variability? We present a computational model based on the biology of the relevant visual pathways that learns to reliably recognize 100 different object categories in the face of naturally occurring variability in location, rotation, size, and lighting. The model exhibits robustness to highly ambiguous, partially occluded inputs. Both the unified, biologically plausible learning mechanism and the robustness to occlusion derive from the role that recurrent connectivity and recurrent processing mechanisms play in the model. Furthermore, this interaction of recurrent connectivity and learning predicts that high-level visual representations should be shaped by error signals from nearby, associated brain areas over the course of visual learning. Consistent with this prediction, we show how semantic knowledge about object categories changes the nature of their learned visual representations, as well as how this representational shift supports the mapping between perceptual and conceptual knowledge. Altogether, these findings support the potential importance of ongoing recurrent processing throughout the brain’s visual system and suggest ways in which object recognition can be understood in terms of interactions within and between processes over time.