High-level Visual Features Research Articles

Multimodal Sentiment Analysis in Natural Disaster Data on Social Media

INTRODUCTION: With the development of the Internet, users tend to express their opinions and emotions through text, visual and/or audio content. This has increased the interest in multimodal analysis methods. OBJECTIVES: This study addresses multimodal sentiment analysis on tweets related to natural disasters by combining textual and visual embeddings.METHODS: The use of textual representations together with the emotional expressions of the visual content provides a more comprehensive analysis. To investigate the impact of high-level visual and texual features, a three-layer neural network is used in the study, where the first two layers collect features from different modalities and the third layer is used to analyze sentiments. RESULTS: According to experimental tests on our dataset, the highest performance values (77% Accuracy, 71% F1-score) are achieved by using the CLIP model in the image and the RoBERTa model in the text. CONCLUSION: Such analyzes can be used in different application areas such as agencies, advertising, social/digital media content producers, humanitarian aid organizations and can provide important information in terms of social awareness.

Retinotopy drives the variation in scene responses across visual field map divisions of the occipital place area.

The occipital place area (OPA) is a scene-selective region on the lateral surface of human occipitotemporal cortex that spatially overlaps multiple visual field maps, as well as portions of cortex that are not currently defined as retinotopic. Here we combined population receptive field modeling and responses to scenes in a representational similarity analysis (RSA) framework to test the prediction that the OPA's visual field map divisions contribute uniquely to the overall pattern of scene selectivity within the OPA. Consistent with this prediction, the patterns of response to a set of complex scenes were heterogeneous between maps. To explain this heterogeneity, we tested the explanatory power of seven candidate models using RSA. These models spanned different scene dimensions (Content, Expanse, Distance), low- and high-level visual features, and navigational affordances. None of the tested models could account for the variation in scene response observed between the OPA's visual field maps. However, the heterogeneity in scene response was correlated with the differences in retinotopic profiles across maps. These data highlight the need to carefully examine the relationship between regions defined as category-selective and the underlying retinotopy, and they suggest that, in the case of the OPA, it may not be appropriate to conceptualize it as a single scene-selective region.

GAT-Based Bi-CARU with Adaptive Feature-Based Transformation for Video Summarisation

Nowadays, video is a common social media in our lives. Video summarisation has become an interesting task for information extraction, where the challenge of high redundancy of key scenes leads to difficulties in retrieving important messages. To address this challenge, this work presents a novel approach called the Graph Attention (GAT)-based bi-directional content-adaptive recurrent unit model for video summarisation. The model makes use of the graph attention approach to transform the visual features of interesting scene(s) from a video. This transformation is achieved by a mechanism called Adaptive Feature-based Transformation (AFT), which extracts the visual features and elevates them to a higher-level representation. We also introduce a new GAT-based attention model that extracts major features from weight features for information extraction, taking into account the tendency of humans to pay attention to transformations and moving objects. Additionally, we integrate the higher-level visual features obtained from the attention layer with the semantic features processed by Bi-CARU. By combining both visual and semantic information, the proposed work enhances the accuracy of key-scene determination. By addressing the issue of high redundancy among major information and using advanced techniques, our method provides a competitive and efficient way to summarise videos. Experimental results show that our approach outperforms existing state-of-the-art methods in video summarisation.

Anchor objects drive realism while diagnostic objects drive categorization in GAN generated scenes

Our visual surroundings are highly complex. Despite this, we understand and navigate them effortlessly. This requires transforming incoming sensory information into representations that not only span low- to high-level visual features (e.g., edges, object parts, objects), but likely also reflect co-occurrence statistics of objects in real-world scenes. Here, so-called anchor objects are defined as being highly predictive of the location and identity of frequently co-occuring (usually smaller) objects, derived from object clustering statistics in real-world scenes, while so-called diagnostic objects are predictive of the larger semantic context (i.e., scene category). Across two studies (N1 = 50, N2 = 44), we investigate which of these properties underlie scene understanding across two dimensions – realism and categorisation – using scenes generated from Generative Adversarial Networks (GANs) which naturally vary along these dimensions. We show that anchor objects and mainly high-level features extracted from a range of pre-trained deep neural networks (DNNs) drove realism both at first glance and after initial processing. Categorisation performance was mainly determined by diagnostic objects, regardless of realism, at first glance and after initial processing. Our results are testament to the visual system’s ability to pick up on reliable, category specific sources of information that are flexible towards disturbances across the visual feature-hierarchy.

Quantifying urban environments: Aesthetic preference through the lens of prospect-refuge theory

Prospect-refuge theory suggests that people prefer environments that offer both prospect, the ability to scan for resources, and refuge, a safe place to hide. Urban planners, architects and researchers alike have had a tendency to use prospect-refuge theory research on natural scenes to inform on the design of urban environments. Despite the large body of prospect-refuge theory research, the degree to which prospect and refuge impact preference in urban environments remain unclear. Here, we aim to first evaluate the relationship between prospect, refuge and preference for urban scene images. Secondly, we aim to evaluate the contributions of visual features and streetscape quality ratings to subjective ratings of prospect and refuge in order to create proxy values of prospect and refuge. Finally, we aim to understand how the proxy values impact preference for urban scenes, and if the proxy values created replicate the relationship between subjective measures of prospect, refuge and preference. First, we used participant ratings of prospect and refuge to predict participants' preference for 552 images of urban street scenes. Higher ratings of both prospect and refuge predicted greater image preference. We next used principal components analysis to summarize these images' low- and high-level visual features as well as participant ratings of streetscape qualities, such as walkability and disorder. Visual feature and streetscape quality principal components predicted prospect and refuge ratings in this first image set, providing “proxy measures' for prospect and refuge. In an independent set of 1119 images from Talen et al. (2022) for which prospect and refuge ratings were not available, we asked whether these proxies for prospect and refuge predicted preference. Findings replicated the effect that more refuge in an image predicts more preference. However, the proxy measure of prospect did not predict preference. In summary, our results show that refuge ratings do relate to preferences in urban environments, which extends prospect-refuge theory to more urban environments. Future work is needed to understand if prospect has different implications in more urban environments.

Testing the flexibility of ensemble coding: Limitations in cross-modal ensemble perception.

Ensemble coding (the brain's ability to rapidly extract summary statistics from groups of items) has been demonstrated across a range of low-level (e.g., average color) to high-level (e.g., average facial expression) visual features, and even on information that cannot be gleaned solely from retinal input (e.g., object lifelikeness). There is also evidence that ensemble coding can interact with other cognitive systems such as long-term memory (LTM), as observers are able to derive the average cost of items. We extended this line of research to examine if different sensory modalities can interact during ensemble coding. Participants made judgments about the average sweetness of groups of different visually presented foods. We found that, when viewed simultaneously, observers were limited in the number of items they could incorporate into their cross-modal ensemble percepts. We speculate that this capacity limit is caused by the cross-modal translation of visual percepts into taste representations stored in LTM. This was supported by findings that (a) participants could use similar stimuli to form capacity-unlimited ensemble representations of average screen size and (b) participants could extract the average sweetness of displays when items were viewed in sequence, with no capacity limitation (suggesting that spatial attention constrains the number of necessary visual cues an observer can integrate in a given moment to trigger cross-modal retrieval of taste). Together, the results of our study demonstrate that there are limits to the flexibility of ensemble coding, especially when multiple cognitive systems need to interact to compress sensory information into an ensemble representation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Predicting movies’ eudaimonic and hedonic scores: A machine learning approach using metadata, audio and visual features

In the task of modeling user preferences for movie recommender systems, recent research has demonstrated the benefits of describing movies with their eudaimonic and hedonic scores (E and H scores), which reflect the depth of their message and the level of fun experience they provide, respectively. So far, the labeling of movies with their E and H scores has been done manually using a dedicated instrument (a questionnaire), which is time-consuming. To address this issue, we propose an automatic approach for predicting E and H scores. Specifically, we collected E and H scores of 709 movies from 370 users (with a total of 3699 records), augmented this dataset with metadata, audio, and low-level and high-level visual features, and trained machine learning models for predicting the E and H scores of movies. This study investigates the use of machine learning models in predicting the E and H scores of movies using various feature sets, including audio, low-level and high-level visual features, and metadata. We compared the performance of predictive models using different combinations of features with the majority classifier as the baseline approach. The results demonstrate that our proposed machine learning-based models significantly outperform the baseline in predicting E and H scores, particularly when leveraging metadata features. Specifically, the random forest classifier achieved a 20% increase in ROC AUC compared to the baseline when predicting both the E score and the H score. These improvements were found to be statistically significant. Overall, our findings suggest that automated tools for predicting E and H scores in movies are promising alternatives to traditional questionnaire-based approaches.

Semantic manipulation through the lens of Geometric Algebra

Semantic image manipulation is a complex problem defined as the ability to change high-level features while keeping the final result visually similar to the original. Recent deep generative solutions show that manipulating semantic features in latent space produces compelling results. Even if we consider those advances, the question remains: Can we algebraically operate high-level visual semantic features in images with meaningful operations? In this paper, we demonstrate the feasibility of interpreting and manipulating image pseudovectors (n−1-dimensional subspaces) as the union of visual features (k-dimensional subspaces, for 0<k<n) operated using Geometric Algebra (GA). Depending on how the latent space is organized, any GA operation would be applicable, enabling the solution to handle an open set of problems without retraining generative models for specific tasks. As a proof of concept, in this paper, we demonstrate how GA operations can be applied to manipulate subspaces in the latent space of faces to perform operations like putting on or taking off clothing accessories, transferring age characteristics, changing hairstyles, and performing semantic queries in sets of images.

Visual Features for Improving Endoscopic Bleeding Detection Using Convolutional Neural Networks.

The presented paper investigates the problem of endoscopic bleeding detection in endoscopic videos in the form of a binary image classification task. A set of definitions of high-level visual features of endoscopic bleeding is introduced, which incorporates domain knowledge from the field. The high-level features are coupled with respective feature descriptors, enabling automatic capture of the features using image processing methods. Each of the proposed feature descriptors outputs a feature activation map in the form of a grayscale image. Acquired feature maps can be appended in a straightforward way to the original color channels of the input image and passed to the input of a convolutional neural network during the training and inference steps. An experimental evaluation is conducted to compare the classification ROC AUC of feature-extended convolutional neural network models with baseline models using regular color image inputs. The advantage of feature-extended models is demonstrated for the Resnet and VGG convolutional neural network architectures.

Visual perceptual learning is effective in the illusory far but not in the near space

Visual shape discrimination is faster for objects close to the body, in the peripersonal space (PPS), compared with objects far from the body. Visual processing enhancement in PPS occurs also when perceived depth is based on 2D pictorial cues. This advantage has been observed from relatively low-level (detection, size, orientation) to high-level visual features (face processing). While multisensory association also displays proximal advantages, whether PPS influences visual perceptual learning remains unclear. Here, we investigated whether perceptual learning effects vary according to the distance of visual stimuli (near or far) from the observer, illusorily induced by leveraging the Ponzo illusion. Participants performed a visual search task in which they reported whether a specific target object orientation (e.g., triangle pointing downward) was present among distractors. Performance was assessed before and after practicing the visual search task (30 minutes/day for 5 days) at either the close (near group) or far (far group) distance. Results showed that participants that performed the training in the near space did not improve. By contrast, participants that performed the training in the far space showed an improvement in the visual search task in both the far and near spaces. We suggest that such improvement following the far training is due to a greater deployment of attention in the far space, which could make the learning more effective and generalize across spaces.

Estimating urban noise along road network from street view imagery

Estimating road traffic noise is essential for examining the quality of sounding environment and mitigating such a non-negligible pollutant in urban areas. However, existing estimated models often have limited applicability to specific traffic conditions, while the required parameters may not be readily available for city-wide collection. This paper proposes a data-driven approach for measuring road-level acoustic information of traffic with street view imagery. Specifically, we utilize portable vehicle-equipped hardware for in-situ noise acquisition and employ a deep learning model ResNet to learn high-level visual features from street view images that are closely associated with road traffic noise. The ResNet captures meaningful patterns from the input data, and the output probability vectors are then fed into a Random-Forest regression algorithm to quantitatively estimate the noise in decibels for different road segments. The MAE and RMSE of the DCNN-RF model are 2.01 and 2.71, respectively. Additionally, we employ a gradient-weighted Class Active Mapping approach to visually interpret our deep learning model and explore the significant elements in streetscapes that contribute to the model's estimations. Our proposed framework facilitates low-cost and fine-scale road traffic noise estimations and sheds light on how auditory information could be inferred from street imagery, which may benefit practices in geography and urban planning.

High or low? Exploring the restorative effects of visual levels on campus spaces using machine learning and street view imagery

According to the Attentional Restoration Theory (ART), cognitive restoration (e.g., Fascination) may occur when the physical environment exhibits high restorative quality. However, these studies usually ignore the effect of different levels of visual features on restoration quality, and small-scale questionnaires are difficult to use to comprehensively evaluate the restoration quality of a space. In this study, we propose a machine learning based method for high-resolution, large-scale assessment of the restoration quality of campus environments using Street View Images (SVIs). First, visual features are extracted from campus SVIs using computer vision method. Second, an online survey using the PRS-11 questionnaire (containing four indicators: Being-away, Coherence, Scope, and Fascination) was conducted to label the images. Finally, we developed a regression model to predict campus restorative quality and to model the non-linear relationship between the visual features of SVIs and this quality. We studied 1088 SVIs in the Lihu campus of Jiangnan University (JNU) to verify the feasibility of our method, and the results showed that SVIs can accurately help us predict the restoration quality of the campus environment on a large scale (R2 = 0.726). Next, we examined the variance in visual features between campus spaces with different levels of restorative quality, and investigated the effect of different levels of visual features on restoration quality. We found that contributions of high-level visual features to restoration, such as trees, are robust (Adj R2 = 0.504) compared to low-level visual features (Adj R2 = 0.032) that included such as color information. This provides a new perspective for assessing recovery environments and designing healthy campus environments. The code is shared at: https://github.com/MMHHRR/Restorative_Quality

Transductive semantic knowledge graph propagation for zero-shot learning

The domain shift is a very common phenomenon in Zero-shot Learning (ZSL), because the data distribution between the source and target domain might not match well in real scenarios. This paper focuses on dealing with the domain drift problem in ZSL based on the semantic knowledge graph propagation. Our method consists of visual feature extraction (VFE) module, semantic feature extraction module (SFE) and feature mapping module (FM). In VFE module, the high-level visual features are extracted under Convolutional Neural Network (CNN) framework, where unseen samples participate in model training and stay away from seen ones under the premise of distinguishability. SFE module relies on the Graph Convolutional Network (GCN) framework and focuses on the transmission of semantic embeddings. A modified message aggregation and transformation strategy is proposed to effectively relieve the information smoothing phenomenon caused by the increase of GCN layers. In the recognition stage, AutoEncoder framework achieves a two-way visual and semantic interaction to further minimize the domain drift phenomenon. Simulation results on ImageNet dataset have demonstrated the performance of the proposed method.

Perceiving Multiple Representations for scene text image super-resolution guided by text recognizer

Single image super-resolution (SISR) aims to recover clear high-resolution images from low-resolution images, which has made great progress with the development of deep learning these years. Scene text image super-resolution (STISR) is a subfield of SISR with the goal of increasing the resolution of a low-resolution text image and enhancing the readability of characters in the image. Despite significant improvements in recent approaches, STISR remains a challenging task due to the diversity of background, text appearances and layouts, etc. This paper presents a Perceiving Multiple Representations (PerMR) method for better super-resolution performances in scene text images. PerMR is a unified network that combines super-resolution with text recognition and exploits the recognizer’s feedback to facilitate super-resolution. Specifically, contextual information from the text decoder is extracted to provide sequence-specific guidance and enable the super-resolution model to pay more attention to the text region. Meanwhile, low-level and high-level visual features from the vision backbone of the recognition network are integrated to further improve visual quality. Additionally, we incorporate a frequency branch into the vanilla convolution unit, which efficiently enhances global and local feature representations. Experiments on the STISR benchmark dataset TextZoom validate that PerMR can not only generate more distinguishable images, but also outperforms the current state-of-the-art methods. PerMR boosts the average recognition accuracy by 5.9% using ASTER, 5.8% using MORAN and 10.6% using CRNN compared to the baseline model TSRN. PerMR outperforms the advanced method TPGSR-3 by 1.4% on ASTER, 0.1% on MORAN, 0.2% on CRNN and boosts TATT by 0.6% on ASTER and 1.1% on MORAN respectively. Furthermore, PerMR demonstrates good robustness and generalization when tackling low-quality text images in multiple scene text recognition datasets. The experiment results verify the capabilities of PerMR to boost text recognition performance.

(Retracted) Mimicking human vision systems: deep-learning-based feature fusion for semantic image retrieval

Cross-form feature combination is an important multifeature fusion technique where the purpose is to implicitly discover the relationship between samples from different modalities, i.e., to retrieve another image encoded by similar semantics through one example image. In the past decade, cross-modal image retrieval has becoming a hotspot investigated by many academicians. Moreover, it is now a significant tool for the future performance enhancement of image retrieval. A long-short term memory (LSTM)-based feature fusion model is proposed. First, aiming at the competitiveness of nonmixed deep architecture for image retrieval, the mechanism of LSTM is introduced in detail. Among them, ground-truth-based methods are used to improve cross-modality. We notice that LSTM can mimic human visual understanding of image semantics well. To improve the accuracy of oblique-form image retrieval, systems based on binary representation are proposed to improve cross-modal similarity and effectiveness of message recovery. Second, we use a quality model to measure the commonly used image low-/high-level visual features, where the disqualified features are abandoned accordingly. This in turn achieves an optimal set of highly descriptive features for image retrieval. Furthermore, we use LSTM and the refined visual features to build a biological model for image retrieval, wherein the multimodel features can be optimally incorporated at the temporal level. Extensive experimental validations on multiple well-known image sets have shown the superiority of our method.

Perceptual awareness of natural scenes is limited by higher-level visual features: Evidence from deep neural networks.

Perceptual awareness of natural scenes is limited by higher-level visual features: Evidence from deep neural networks.

Self-Supervision-Augmented Deep Autoencoder for Unsupervised Visual Anomaly Detection.

Deep autoencoder (AE) has demonstrated promising performances in visual anomaly detection (VAD). Learning normal patterns on normal data, deep AE is expected to yield larger reconstruction errors for anomalous samples, which is utilized as the criterion for detecting anomalies. However, this hypothesis cannot be always tenable since the deep AE usually captures the low-level shared features between normal and abnormal data, which leads to similar reconstruction errors for them. To tackle this problem, we propose a self-supervised representation-augmented deep AE for unsupervised VAD, which can enlarge the gap of anomaly scores between normal and abnormal samples by introducing autoencoding transformation (AT). Essentially, AT is introduced to facilitate AE to learn the high-level visual semantic features of normal images by introducing a self-supervision task (transformation reconstruction). In particular, our model inputs the original and transformed images into the encoder for obtaining latent representations; afterward, they are fed to the decoder for reconstructing both the original image and applied transformation. In this way, our model can utilize both image and transformation reconstruction errors to detect anomaly. Extensive experiments indicate that the proposed method outperforms other state-of-the-art methods, which demonstrates the validity and advancement of our model.

Looking more masculine among females: Spatial context modulates gender perception of face and biological motion.

Perception of visual information highly depends on spatial context. For instance, perception of a low-level visual feature, such as orientation, can be shifted away from its surrounding context, exhibiting a simultaneous contrast effect. Although previous studies have demonstrated the adaptation aftereffect of gender, a high-level visual feature, it remains largely unknown whether gender perception can also be shaped by a simultaneously presented context. In the present study, we found that the gender perception of a central face or a point-light walker was repelled away from the gender of its surrounding faces or walkers. A norm-based opponent model of lateral inhibition, which accounts for the adaptation aftereffect of high-level features, can also excellently fit the simultaneous contrast effect. But different from the reported contextual effect of low-level features, the simultaneous contrast effect of gender cannot be observed when the centre and the surrounding stimuli are from different categories, or when the surrounding stimuli are suppressed from awareness. These findings on one hand reveal a resemblance between the simultaneous contrast effect and the adaptation aftereffect of high-level features, on the other hand highlight different biological mechanisms underlying the contextual effects of low- and high-level visual features.

Is it the best for barista robots to serve like humans? A multidimensional anthropomorphism perspective

The COVID-19 pandemic has accelerated the use of contactless service robots in hospitality industries. However, the key drivers of consumer behaviors against service robots have been ill-understood. This study examines the interactive relationships between the physical (visual features) and psychological (service autonomy) dimensions of service-robot anthropomorphism and their impacts on consumer acceptance of service robots. Adopting an experimental vignette method (EVM) with 402 participants, the study reveals that the impacts of visual features on consumers’ intention are affected by the level of service robots’ autonomy; particularly, consumers showed the highest intention when the robots have medium visual features and high autonomy while their intention became lower for the same level of visual features with low autonomy. Interestingly, consumers showed the lowest intention with high level visual features, regardless of the levels of autonomy. Our results also show that human identity threats and consumer resistance play a significant counterproductive mechanism between service robot anthropomorphism and consumers’ intention.

CheXGAT: A disease correlation-aware network for thorax disease diagnosis from chest X-ray images

Chest X-ray (CXR) imaging is one of the most common diagnostic imaging techniques in clinical diagnosis and is usually used for radiological examinations to screen for thorax diseases. In this paper, we propose a novel computer-aided diagnosis (CAD) system based on a hybrid deep learning model composed of a convolutional neural network (CNN) and a graph neural network (GNN). The system is intended to explore implicit correlations between thorax diseases to aid in the multilabel chest X-ray image classification task, which we term ‶CheXGAT‶. Specifically, the proposed CheXGAT framework comprises two main modules: an image representation learning (IRL) module and a graph representation learning (GRL) module. We employ the IRL module to learn high-level visual representation features from the CXR image. From the GRL module, the self-attention mechanism aggregates neighborhood features from the graphic structure to enhance the implicit correlation between thorax diseases. We adopted a data-driven method to create a disease correlation matrix that works on the message passing and aggregation process for the nodes in the GRL module. After end-to-end training, the GRL module enhances the correlation between thorax diseases to improve diagnosis performance. We performed experiments on the NIH Chest X-ray14 dataset, which contains 112,120 frontal-view radiographs; each image has multiple thorax disease labels. In the experimental results, the average AUC score of our proposed CheXGAT model reached 0.8266, and the AUC scores of Emphysema and Hernia reached 0.9447 and 0.9313, respectively. In addition, we visualized explanations via a gradient-based localization method in our proposed CheXGAT framework. Compared with previous studies, the experimental results show the competitive performance of our framework. Specifically, we propose a CAD system that uses a hybrid model to help radiologists identify CXR images from 14 chest diseases for clinical diagnosis.