Low-level Visual Features Research Articles

Predictive neural representations of naturalistic dynamic input

Adaptive behavior such as social interaction requires our brain to predict unfolding external dynamics. While theories assume such dynamic prediction, empirical evidence is limited to static snapshots and indirect consequences of predictions. We present a dynamic extension to representational similarity analysis that uses temporally variable models to capture neural representations of unfolding events. We applied this approach to source-reconstructed magnetoencephalography (MEG) data of healthy human subjects and demonstrate both lagged and predictive neural representations of observed actions. Predictive representations exhibit a hierarchical pattern, such that high-level abstract stimulus features are predicted earlier in time, while low-level visual features are predicted closer in time to the actual sensory input. By quantifying the temporal forecast window of the brain, this approach allows investigating predictive processing of our dynamic world. It can be applied to other naturalistic stimuli (e.g., film, soundscapes, music, motor planning/execution, social interaction) and any biosignal with high temporal resolution.

Rethinking Alignment and Uniformity in Unsupervised Image Semantic Segmentation

Unsupervised image segmentation aims to match low-level visual features with semantic-level representations without outer supervision. In this paper, we address the critical properties from the view of feature alignments and feature uniformity for UISS models. We also make a comparison between UISS and image-wise representation learning. Based on the analysis, we argue that the existing MI-based methods in UISS suffer from representation collapse. By this, we proposed a robust network called Semantic Attention Network(SAN), in which a new module Semantic Attention(SEAT) is proposed to generate pixel-wise and semantic features dynamically. Experimental results on multiple semantic segmentation benchmarks show that our unsupervised segmentation framework specializes in catching semantic representations, which outperforms all the unpretrained and even several pretrained methods.

Understanding and predicting cross-cultural food preferences with online recipe images

Predicting food preferences is challenging due to the numerous factors that can influence individual taste. Cultural influences are one such factor that can significantly impact food preferences. Irrespective of culture, however, food visual aesthetics drive food choice. With this in mind, we study 15,000 images of food from prominent recipe portals in China, the US and Germany with the aim of identifying common visual trends related to food choice. We report that distinguishing between appreciated and less appreciated recipes based on visual features through multiple machine learning experiments is possible, with a maximum accuracy of 67%. The classifiers trained on one culture, when applied to other cultures, reveal that appreciated recipes from US and German portals share visual similarities, while the visual aspects making recipes attractive for Chinese users differ. Complementing our machine learning experiments, we conducted a user study with 450 participants from the three cultures. Participants are asked to rank recipes based on their appeal and provide justification through answering 19 questions. The results reveal significant predictors of preference, including perceived appearance and taste across all countries, and negative perception of perceived healthiness for US and German participants. Analysis of 77 consistently labelled appealing/non-appealing images reveal significant variations in low-level visual features such as colourfulness, sharpness, and contrast. These findings suggest that these low-level visual features may play a critical role in determining whether an image is perceived as appealing or not across different cultures. Our results offer promising insights into the development of cross-culture food recommender systems. By demonstrating common ground across food cultures, we believe that these systems can provide personalised meal suggestions that reflect user’s preferences while incorporating ideals such as healthfulness and sustainability.

Voxelwise encoding models of body stimuli reveal a representational gradient from low-level visual features to postural features in occipitotemporal cortex

Previous research on body representation in the brain has focused on category-specific representation, using fMRI to investigate the response pattern to body stimuli in occipitotemporal cortex. But the central question of the specific computations involved in body selective regions has not been addressed so far. This study used ultra-high field fMRI and banded ridge regression to investigate the computational mechanisms of coding body images, by comparing the performance of three encoding models in predicting brain activity in occipitotemporal cortex and specifically in the extrastriate body area (EBA). Our results indicate that bodies are encoded in occipitotemporal cortex and in the EBA according to a combination of low-level visual features and postural features.

Memorable beginnings, but forgettable endings: Intrinsic memorability alters our subjective experience of time

ABSTRACT Time is the fabric of experience – yet it is incredibly malleable in the mind of the observer: seeming to drag on, or fly right by at different moments. One of the most influential drivers of temporal distortions is attention, where heightened attention dilates subjective time. But an equally important feature of subjective experience involves not just the objects of attention, but also what information will naturally be remembered or forgotten, independent of attention (i.e., intrinsic image memorability). Here we test how memorability influences time perception. Observers viewed scenes in an oddball paradigm, where the last scene could be a forgettable “oddball” amidst memorable ones, or vice versa. Subjective time dilation occurred only for forgettable oddballs, but not memorable ones – demonstrating an oddball effect where the oddball did not differ in low-level visual features, image category, or even subjective memorability. But more importantly, these results emphasize how memory can interact with temporal experience: memorable beginnings may put people in an efficient encoding state, which may in turn influence which moments are dilated in time.

A facial depression recognition method based on hybrid multi-head cross attention network.

Deep-learn methods based on convolutional neural networks (CNNs) have demonstrated impressive performance in depression analysis. Nevertheless, some critical challenges need to be resolved in these methods: (1) It is still difficult for CNNs to learn long-range inductive biases in the low-level feature extraction of different facial regions because of the spatial locality. (2) It is difficult for a model with only a single attention head to concentrate on various parts of the face simultaneously, leading to less sensitivity to other important facial regions associated with depression. In the case of facial depression recognition, many of the clues come from a few areas of the face simultaneously, e.g., the mouth and eyes. To address these issues, we present an end-to-end integrated framework called Hybrid Multi-head Cross Attention Network (HMHN), which includes two stages. The first stage consists of the Grid-Wise Attention block (GWA) and Deep Feature Fusion block (DFF) for the low-level visual depression feature learning. In the second stage, we obtain the global representation by encoding high-order interactions among local features with Multi-head Cross Attention block (MAB) and Attention Fusion block (AFB). We experimented on AVEC2013 and AVEC2014 depression datasets. The results of AVEC 2013 (RMSE = 7.38, MAE = 6.05) and AVEC 2014 (RMSE = 7.60, MAE = 6.01) demonstrated the efficacy of our method and outperformed most of the state-of-the-art video-based depression recognition approaches. We proposed a deep learning hybrid model for depression recognition by capturing the higher-order interactions between the depression features of multiple facial regions, which can effectively reduce the error in depression recognition and gives great potential for clinical experiments.

On the dynamics of gain and loss: Electrophysiological evidence from associative learning

Associated relevance affects the sensory encoding of low-level visual features of symbolic stimuli. It is unclear, however, which dimension of low-level visual features benefits from prioritized processing, and how these effects develop throughout the course of relevance acquisition. Moreover, previous evidence is inconclusive regarding the preservation of processing advantage once the association is no longer relevant, as well as its generalization to perceptually similar but novel stimuli. The present study addresses these questions by employing an associative learning paradigm. In two experiments (N = 24 each, between-subject design), different dimensions of low-level visual features of symbolic stimuli were associated with monetary gain, loss, or zero outcome. In a consecutive old/new decision task, associated stimuli were presented together with perceptually similar but novel stimuli. Event-related brain potentials (P1, EPN, LPC) were measured throughout both sessions. Early sensory encoding (P1) was boosted by loss association and appeared to be sensitive to the dimension of the associated low-level visual features. Gain association influenced post-perceptual processing stages (LPC), arising over the course of the learning phase, and are preserved even when the associated outcome was no longer relevant. Gain association also resulted in EPN modulations similar to the effects observed in the case of emotional words. None of the observed effects generalized to perceptually similar stimuli. These results show that acquired relevance can influence the sensory processing of specific dimensions of low-level visual features. Moreover, this study extends previous evidence of a dissociation of early and late neural effects of associated motivational relevance.

Fast saliency prediction based on multi-channels activation optimization

The saliency prediction precision has improved rapidly with the development of deep learning technology, but the inference speed is slow due to the continuous deepening of networks. Hence, this paper proposes a fast saliency prediction model. Concretely, the siamese network backbone based on tailored EfficientNetV2 accelerates the inference speed while maintaining high performance. The shared parameters strategy further curbs parameter growth. Furthermore, we add multi-channel activation maps to optimize the fine features considering different channels and low-level visual features, which improves the interpretability of the model. Extensive experiments show that the proposed model achieves competitive performance on the standard benchmark datasets, and prove the effectiveness of our method in striking a balance between prediction accuracy and inference speed. Moreover, the small model size allows our method to be applied in edge devices. The code is available at: https://github.com/lscumt/fast-fixation-prediction.

Contrast sensitivity function in deep networks

The contrast sensitivity function (CSF) is a fundamental signature of the visual system that has been measured extensively in several species. It is defined by the visibility threshold for sinusoidal gratings at all spatial frequencies. Here, we investigated the CSF in deep neural networks using the same 2AFC contrast detection paradigm as in human psychophysics. We examined 240 networks pretrained on several tasks. To obtain their corresponding CSFs, we trained a linear classifier on top of the extracted features from frozen pretrained networks. The linear classifier is exclusively trained on a contrast discrimination task with natural images. It has to find which of the two input images has higher contrast. The network’s CSF is measured by detecting which one of two images contains a sinusoidal grating of varying orientation and spatial frequency. Our results demonstrate characteristics of the human CSF are manifested in deep networks both in the luminance channel (a band-limited inverted U-shaped function) and in the chromatic channels (two low-pass functions of similar properties). The exact shape of the networks’ CSF appears to be task-dependent. The human CSF is better captured by networks trained on low-level visual tasks such as image-denoising or autoencoding. However, human-like CSF also emerges in mid- and high-level tasks such as edge detection and object recognition. Our analysis shows that human-like CSF appears in all architectures but at different depths of processing, some at early layers, while others in intermediate and final layers. Overall, these results suggest that (i) deep networks model the human CSF faithfully, making them suitable candidates for applications of image quality and compression, (ii) efficient/purposeful processing of the natural world drives the CSF shape, and (iii) visual representation from all levels of visual hierarchy contribute to the tuning curve of the CSF, in turn implying a function which we intuitively think of as modulated by low-level visual features may arise as a consequence of pooling from a larger set of neurons at all levels of the visual system.

Are social interactions preferentially attended in real-world scenes? Evidence from change blindness.

In change detection paradigms, changes to social or animate aspects of a scene are detected better and faster compared with non-social or inanimate aspects. While previous studies have focused on how changes to individual faces/bodies are detected, it is possible that individuals presented within a social interaction may be further prioritised, as the accurate interpretation of social interactions may convey a competitive advantage. Over three experiments, we explored change detection to complex real-world scenes, in which changes either occurred by the removal of (a) an individual on their own, (b) an individual who was interacting with others, or (c) an object. In Experiment 1 (N = 50), we measured change detection for non-interacting individuals versus objects. In Experiment 2 (N = 49), we measured change detection for interacting individuals versus objects. Finally, in Experiment 3 (N = 85), we measured change detection for non-interacting versus interacting individuals. We also ran an inverted version of each task to determine whether differences were driven by low-level visual features. In Experiments 1 and 2, we found that changes to non-interacting and interacting individuals were detected better and more quickly than changes to objects. We also found inversion effects for both non-interaction and interaction changes, whereby they were detected more quickly when upright compared with inverted. No such inversion effect was seen for objects. This suggests that the high-level, social content of the images was driving the faster change detection for social versus object targets. Finally, we found that changes to individuals in non-interactions were detected faster than those presented within an interaction. Our results replicate the social advantage often found in change detection paradigms. However, we find that changes to individuals presented within social interaction configurations do not appear to be more quickly and easily detected than those in non-interacting configurations.

Cross-modal transformer with language query for referring image segmentation

Referring image segmentation (RIS) aims to predict a segmentation mask for a target specified by a natural language expression. However, the existing methods failed to implement deep interaction between vision and language is needed in RIS, resulting inaccurate segmentation. To address the problem, a cross-modal transformer (CMT) with language queries for referring image segmentation is proposed. First, a cross-modal encoder of CMT is designed for intra-modal and inter-modal interaction, capturing context-aware visual features. Secondly, to generate compact visual-aware language queries, a language-query encoder (LQ) embeds key visual cues into linguistic features. In particular, the combination of the cross-modal encoder and language query encoder realizes the mutual guidance of vision and language. Finally, the cross-modal decoder of CMT is constructed to learn multimodal features of the referent from the context-aware visual features and visual-aware language queries. In addition, a semantics-guided detail enhancement (SDE) module is constructed to fuse the semantic-rich multimodal features with detail-rich low-level visual features, which supplements the spatial details of the predicted segmentation masks. Extensive experiments on four referring image segmentation datasets demonstrate the effectiveness of the proposed method.

Let’s Unleash the Network Judgment: A Self-Supervised Approach for Cloud Image Analysis

Abstract Accurate cloud type identification and coverage analysis are crucial in understanding the Earth’s radiative budget. Traditional computer vision methods rely on low-level visual features of clouds for estimating cloud coverage or sky conditions. Several handcrafted approaches have been proposed; however, scope for improvement still exists. Newer deep neural networks (DNNs) have demonstrated superior performance for cloud segmentation and categorization. These methods, however, need expert engineering intervention in the preprocessing steps—in the traditional methods—or human assistance in assigning cloud or clear sky labels to a pixel for training DNNs. Such human mediation imposes considerable time and labor costs. We present the application of a new self-supervised learning approach to autonomously extract relevant features from sky images captured by ground-based cameras, for the classification and segmentation of clouds. We evaluate a joint embedding architecture that uses self-knowledge distillation plus regularization. We use two datasets to demonstrate the network’s ability to classify and segment sky images—one with ∼ 85,000 images collected from our ground-based camera and another with 400 labeled images from the WSISEG database. We find that this approach can discriminate full-sky images based on cloud coverage, diurnal variation, and cloud base height. Furthermore, it semantically segments the cloud areas without labels. The approach shows competitive performance in all tested tasks, suggesting a new alternative for cloud characterization.

The visual size is enough to automatically induce the potentiation of grasping behaviours.

Seeing objects usually grasped with a power or a precision grip (e.g., an apple vs a cherry) potentiates power- and precision-grip responses, respectively. An embodied account suggests that this effect occurs because object conceptual representations would lie on a motor simulation process. A new account, named the size-coding account, argues that this effect could be rather due to an overlapping of size codes used to represent both manipulable objects and response options. In this article, we investigate whether this potentiation effect could be merely due to a low-level visual feature that favours a size-coding of stimuli: the visual size in which objects are presented. Accordingly, we conducted two experiments in which we presented highly elementary and non-graspable stimuli (i.e., ink spots) either large or small rather than graspable objects. Our results showed that the mere visual size automatically potentiates power- and precision-grip responses that are in line with the size-coding account of the potentiation effect of grasping behaviours. Moreover, these results appeal to improve the methodological control of the size of stimuli especially when researchers try to support the embodied account.

Social context modulates autonomic responses to direct eye contact

Eye contact with another person (social gaze) can evoke emotions, produce autonomic arousal, and influence behavior. Gaze cues can be evocative even when presented in static pictures of faces suggesting that responses depend on low-level visual features of gaze stimuli. The current study examined whether emotional gaze responses depend on the physical stimulus properties of an eye contact experience versus the cognitive evaluation of the social context of gaze. This was done by comparing skin conductance responses (SCR), an index of emotional arousal, during episodes of social gaze and 'self-gaze' (gazing at one's own eyes in a mirror), keeping other aspects of the viewing conditions constant. We compared SCRs during social gaze and self-gaze in forty participant pairs. Each participant engaged in ten, 20 second eye contact trials, alternating between social and self-gaze. Self-gaze episodes produced significant SCRs but social gaze SCR's were larger and occurred more reliably. SCRs decreased across trials (habituation effect) in both conditions. We speculated that social gaze between opposite sex partners might yield larger SCRs but this was not found. Overall, these results conceptually replicate previous findings of (likely top-town) cognitive regulation of autonomic gaze responses based on evaluation of the social context.

A Remote Sensing Image Fusion Method Combining Low-Level Visual Features and Parameter-Adaptive Dual-Channel Pulse-Coupled Neural Network

Remote sensing image fusion can effectively solve the inherent contradiction between spatial resolution and spectral resolution of imaging systems. At present, the fusion methods of remote sensing images based on multi-scale transform usually set fusion rules according to local feature information and pulse-coupled neural network (PCNN), but there are problems such as single local feature, as fusion rule cannot effectively extract feature information, PCNN parameter setting is complex, and spatial correlation is poor. To this end, a fusion method of remote sensing images that combines low-level visual features and a parameter-adaptive dual-channel pulse-coupled neural network (PADCPCNN) in a non-subsampled shearlet transform (NSST) domain is proposed in this paper. In the low-frequency sub-band fusion process, a low-level visual feature fusion rule is constructed by combining three local features, local phase congruency, local abrupt measure, and local energy information to enhance the extraction ability of feature information. In the process of high-frequency sub-band fusion, the structure and parameters of the dual-channel pulse-coupled neural network (DCPCNN) are optimized, including: (1) the multi-scale morphological gradient is used as an external stimulus to enhance the spatial correlation of DCPCNN; and (2) implement parameter-adaptive representation according to the difference box-counting, the Otsu threshold, and the image intensity to solve the complexity of parameter setting. Five sets of remote sensing image data of different satellite platforms and ground objects are selected for experiments. The proposed method is compared with 16 other methods and evaluated from qualitative and quantitative aspects. The experimental results show that, compared with the average value of the sub-optimal method in the five sets of data, the proposed method is optimized by 0.006, 0.009, 0.009, 0.035, 0.037, 0.042, and 0.020, respectively, in the seven evaluation indexes of information entropy, mutual information, average gradient, spatial frequency, spectral distortion, ERGAS, and visual information fidelity, indicating that the proposed method has the best fusion effect.

GDRL: An interpretable framework for thoracic pathologic prediction

Deep learning methods have shown significant performance in medical image analysis tasks. However, they generally act like “black box” without explanations in both feature extraction and decision processes, leading to lack of clinical insights and high risk assessments. To aid deep learning in envisioning diseases with visual clues, we propose a novel Group-Disentangled Representation Learning framework (GDRL). The key contribution is that GDRL completely disentangles latent space into disease concepts with abundant and non-overlapping feature related explanations, thus enhancing interpretability in feature extraction and decision processes. Furthermore, we introduce an implicit group-swap structure by emphasizing the linking relationship between semantical concepts of disease and low-level visual features, other than explicit explanations on general objects and their attributes. We demonstrate our framework on predicting four categories of diseases from chest X-ray images. The AUROC of GDRL on ChestX-ray14 for thoracic pathologic prediction are 0.8630, 0.8980, 0.9269 and 0.8653 respectively, and we showcase the potential of our framework in enhancing interpretability of the factors contributing to different diseases.

A Large Video Set of Natural Human Actions for Visual and Cognitive Neuroscience Studies and Its Validation with fMRI.

The investigation of the perception of others' actions and underlying neural mechanisms has been hampered by the lack of a comprehensive stimulus set covering the human behavioral repertoire. To fill this void, we present a video set showing 100 human actions recorded in natural settings, covering the human repertoire except for emotion-driven (e.g., sexual) actions and those involving implements (e.g., tools). We validated the set using fMRI and showed that observation of the 100 actions activated the well-established action observation network. We also quantified the videos' low-level visual features (luminance, optic flow, and edges). Thus, this comprehensive video set is a valuable resource for perceptual and neuronal studies.

Computer vision recognition and tracking algorithm based on convolutional neural network

Abstract In the past few decades, target tracking algorithm has been paid great attention by peers at home and abroad in the field of computer vision because of its potential for in-depth research and practical value. Typical applications of target tracking algorithms include intelligent video surveillance, autonomous vehicles, human-computer interaction and so on. Given the initial state of a target object, the task of the target tracking algorithm is to estimate the state of the target in the subsequent video. Despite years of efforts, designing a target tracking algorithm is still a very challenging problem, because it poses changes, particularly illumination changes, and in addition, occlusion, complex environments, and the moving background will also cause changes in the appearance of the target. The traditional target tracking algorithm based on manually designed features or shallow classifiers uses manually designed low-level visual features or shallow classifiers to build the target apparent model, so the semantic information prediction ability of the target apparent model is limited. Given the defect that the above traditional target tracking algorithm is difficult to capture the semantic information of visual data in the target apparent model, inspired by the great success of deep convolution networks in image classification and speech recognition, a target tracking algorithm based on convolution neural network is proposed in this paper.

Up close and emotional: Electrophysiological dynamics of approaching angry faces

Recent evidence suggests that looming emotional faces are processed rapidly by the neural system, and that this apparent approach further interacts with emotion, causing an enhanced neural response for angry expressions. However, previous research has not demonstrated unequivocally if these effects are due to low-level visual features, or if they are indeed due to the emotional content of the stimuli. To address this question, the current study presented upright and inverted angry and neutral faces, which either expanded or contracted in size on a constant depth-cued background, such that they appeared to approach or retreat from the viewer. EEG/ERP measures were used to identify the time course of brain activity for these stimuli. The results showed that when faces were upright, both the P1 and N170 were enhanced for angry expressions, with the P1 being further increased with looming angry faces. The inversion of the faces caused an increase in both the P1 and N170 amplitudes, but no modulation was found for emotions. These findings show an early modulation of brain activity for upright looming angry faces and rule out the influence of low-level visual features as a contributing factor.

Face identity and facial expression representations with adaptation paradigms: New directions for potential applications.

Adaptation and aftereffect are well-known procedures for exploring our neural representation of visual stimuli. It has been reported that they occur in face identity, facial expressions, and low-level visual features. This method has two primary advantages. One is to reveal the common or shared process of faces, that is, the overlapped or discrete representation of face identities or facial expressions. The other is to investigate the coding system or theory of face processing that underlies the ability to recognize faces. This study aims to organize recent research to guide the reader into the field of face adaptation and its aftereffect and to suggest possible future expansions in the use of this paradigm. To achieve this, we reviewed the behavioral short-term aftereffect studies on face identity (i.e., who it is) and facial expressions (i.e., what expressions such as happiness and anger are expressed), and summarized their findings about the neural representation of faces. First, we summarize the basic characteristics of face aftereffects compared to simple visual features to clarify that facial aftereffects occur at a different stage and are not inherited or combinations of low-level visual features. Next, we introduce the norm-based coding hypothesis, which is one of the theories used to represent face identity and facial expressions, and adaptation is a commonly used procedure to examine this. Subsequently, we reviewed studies that applied this paradigm to immature or impaired face recognition (i.e., children and individuals with autism spectrum disorder or prosopagnosia) and examined the relationships between their poor recognition performance and representations. Moreover, we reviewed studies dealing with the representation of non-presented faces and social signals conveyed via faces and discussed that the face adaptation paradigm is also appropriate for these types of examinations. Finally, we summarize the research conducted to date and propose a new direction for the face adaptation paradigm.