Image Aesthetic Quality Research Articles

Image Aesthetics Assessment With Attribute-Assisted Multimodal Memory Network

Image aesthetics assessment (IAA) has attracted growing interest in recent years but is still challenging due to its highly abstract nature. Nowadays, more and more people tend to comment images shared on the social networks, which can provide rich aesthetics-aware semantic information from different aspects. Therefore, user comments of an image can be exploited as supplementary information for enhancing aesthetic representation learning. Previous researches have demonstrated that aesthetic attributes make significant effect on image aesthetic quality and humans’ aesthetic perception. Typically, people are used to give comments on an image from the perspective of aesthetic attributes, based on which the aesthetic quality of images can be inferred. Motivated by this, this paper presents an Attribute-assisted Multimodal Memory Network (AMM-Net) for image aesthetics assessment, which utilizes aesthetic attributes to model the interactions between visual and textual modalities. Specifically, we design two memory networks to capture the attribute-aware information most related to the image and associated comments respectively. Further, with multiple memory hops, attribute semantics shared by the two modalities are refined and cross-modal interactions are enhanced progressively. Finally, more discriminative aesthetic representations can be obtained for IAA. The experimental results and comparisons on two public multimodal IAA datasets demonstrate the superiority of the proposed model over the state-of-the-art methods. The source code is available at https://github.com/zhutong0219/AMM-Net.

Correct for Whom? Subjectivity and the Evaluation of Personalized Image Aesthetics Assessment Models

The problem of image aesthetic quality assessment is surprisingly difficult to define precisely. Most early work attempted to estimate the average aesthetic rating of a group of observers, while some recent work has shifted to an approach based on few-shot personalization. In this paper, we connect few-shot personalization, via Immanuel Kant's concept of disinterested judgment, to an argument from feminist aesthetics about the biased tendencies of objective standards for subjective pleasures. To empirically investigate this philosophical debate, we introduce PR-AADB, a relabeling of the existing AADB dataset with labels for pairs of images, and measure how well the existing groundtruth predicts our new pairwise labels. We find, consistent with the feminist critique, that both the existing groundtruth and few-shot personalized predictions represent some users' preferences significantly better than others, but that it is difficult to predict when and for whom the existing groundtruth will be correct. We thus advise against using benchmark datasets to evaluate models for personalized IAQA, and recommend caution when attempting to account for subjective difference using machine learning more generally.

Aesthetically Relevant Image Captioning

Image aesthetic quality assessment (AQA) aims to assign numerical aesthetic ratings to images whilst image aesthetic captioning (IAC) aims to generate textual descriptions of the aesthetic aspects of images. In this paper, we study image AQA and IAC together and present a new IAC method termed Aesthetically Relevant Image Captioning (ARIC). Based on the observation that most textual comments of an image are about objects and their interactions rather than aspects of aesthetics, we first introduce the concept of Aesthetic Relevance Score (ARS) of a sentence and have developed a model to automatically label a sentence with its ARS. We then use the ARS to design the ARIC model which includes an ARS weighted IAC loss function and an ARS based diverse aesthetic caption selector (DACS). We present extensive experimental results to show the soundness of the ARS concept and the effectiveness of the ARIC model by demonstrating that texts with higher ARS’s can predict the aesthetic ratings more accurately and that the new ARIC model can generate more accurate, aesthetically more relevant and more diverse image captions. Furthermore, a large new research database containing 510K images with over 5 million comments and 350K aesthetic scores, and code for implementing ARIC, are available at https://github.com/PengZai/ARIC

Styling Classification of Group Photos Fusing Head and Pose Features

Group photo images are everywhere and vary greatly by the shooting scene. Compared with common images, the Image Aesthetic Quality Assessment (IAQI) of group photo pays more attention to the relevant characteristics of the main population. Still, the existing methods do not make further special research on group photos. Therefore, we propose a new concept of group photo styling based on analyzing group photos and photographic theory. Besides that, by comparing and analyzing many group photos, we classify the group photos into five categories. In this paper, the main factors of the head and pose are considered simultaneously, and the method of Group Photo Styling Classification (GPSC) can classify different group photos automatically. To verify the effectiveness of our method, we collected a Group Photo Styling Dataset (GPSD). The dataset contains 998 group photo images, and each image’s group photo styling category is marked. The experimental results on GPSD show that the fusion of head features and pose features can classify different group photos well. The accuracy of GPSC reaches 93.9%, much higher than the previous classification model.

Pseudo-Labeling and Meta Reweighting Learning for Image Aesthetic Quality Assessment

In the tasks of image aesthetic quality assessment, it is difficult to reach both the high score area and low score area due to the normal distribution of aesthetic datasets. To reduce the error in labelling and solve the problem of normal data distribution, we propose a new aesthetic mixed dataset with classification and regression called AMD-CR, and we train a meta reweighting network to reweight the loss of training data differently. In addition, we provide a training strategy according to different stages based on pseudo labels, and then we use it for aesthetic training according to different stages in classification and regression tasks. In the construction of the network structure, we construct an aesthetic adaptive block (AAB) structure that can adapt to any size of the input images. Besides, we also use the efficient channel attention (ECA) to strengthen the feature extracting ability of each task. The experimental result shows that our method improves 0.1112 compared with the conventional method in SROCC. The method can also help to find best aesthetic path planning for unmanned aerial vehicles (UAV) and vehicles.

An interpretable approach for automatic aesthetic assessment of remote sensing images.

The increase of remote sensing images in recent decades has resulted in their use in non-scientific fields such as environmental protection, education, and art. In this situation, we need to focus on the aesthetic assessment of remote sensing, which has received little attention in research. While according to studies on human brain's attention mechanism, certain areas of an image can trigger visual stimuli during aesthetic evaluation. Inspired by this, we used convolutional neural network (CNN), a deep learning model resembling the human neural system, for image aesthetic assessment. So we propose an interpretable approach for automatic aesthetic assessment of remote sensing images. Firstly, we created the Remote Sensing Aesthetics Dataset (RSAD). We collected remote sensing images from Google Earth, designed the four evaluation criteria of remote sensing image aesthetic quality-color harmony, light and shadow, prominent theme, and visual balance-and then labeled the samples based on expert photographers' judgment on the four evaluation criteria. Secondly, we feed RSAD into the ResNet-18 architecture for training. Experimental results show that the proposed method can accurately identify visually pleasing remote sensing images. Finally, we provided a visual explanation of aesthetic assessment by adopting Gradient-weighted Class Activation Mapping (Grad-CAM) to highlight the important image area that influenced model's decision. Overall, this paper is the first to propose and realize automatic aesthetic assessment of remote sensing images, contributing to the non-scientific applications of remote sensing and demonstrating the interpretability of deep-learning based image aesthetic evaluation.

Semantic-aware multi-task learning for image aesthetic quality assessment

In recent years, image aesthetic quality assessment has attracted considerable attention due to the massive growth of digital images in social platforms and the Internet. However, automatically assessing aesthetic quality of an image is a challenging task, because image aesthetic is affected by various factors, and the criteria for judging the aesthetic of images with diverse semantic information are different. To this end, a Semantic-Aware Multi-task convolution neural network (SAM-CNN) for evaluating image aesthetic quality is proposed in this paper. The network can fuse intermediate features of different layers at different scales in CNN to obtain a more comprehensive and accurate aesthetic expression, under the joint supervision of image aesthetic quality assessment task and semantic classification task in a multi-task learning manner. Besides, by applying the attention mechanism, semantic information with a large receptive field extracted from deep layers is utilised to guide the network to focus on the key parts of features to be fused, to improve the effectiveness of feature fusion. Experimental results on the AVA dataset and Photo.net dataset demonstrate the effectiveness and superiority of the proposed SAM-CNN.

Theme-Aware Semi-Supervised Image Aesthetic Quality Assessment

Image aesthetic quality assessment (IAQA) has aroused considerable interest in recent years and is widely used in various applications, such as image retrieval, album management, chat robot and social media. However, existing methods need an excessive amount of labeled data to train the model. Collecting the enormous quantity of human scored training data is not always feasible due to a number of factors, such as the expensiveness of the labeling process and the difficulty in correctly classifying data. Previous studies have evaluated the aesthetic of a photo based only on image features, but have ignored the criterion bias associated with the themes. In this work, we present a new theme-aware semi-supervised image quality assessment method to address these difficulties. Specifically, the proposed method consists of two steps: a representation learning step and a label propagation step. In the representation learning step, we propose a robust theme-aware attention network (TAAN) to cope with the theme criterion bias problem. In the label propagation step, we use preliminary trained TAAN by step one to extract features and utilize the label propagation with a cumulative confidence (LPCC) algorithm to assign pseudo-labels to the unlabeled data. This enables use of both labeled and unlabeled data to train the TAAN model. To the best of our knowledge, this is the first time that a semi-supervised learning method to address image aesthetic assessment problems has been studied. We evaluate our approach on three benchmark datasets and show that it can achieve almost the same performance as a fully supervised learning method for a small number of samples. Furthermore, we show that our semi-supervised approach is robust to using varying quantities of labeled data.

Multimodal Image Aesthetic Prediction with Missing Modality

With the increasing growth of multimedia data on the Internet, multimodal image aesthetic assessment has attracted a great deal of attention in the image processing community. However, traditional multimodal methods often have the following two problems: (1) Existing multimodal image aesthetic methods are based on the assumption that full modalities are available in all samples, which is unapplicable in most cases since textual information is more difficult to obtain. (2) They only fuse multimodal information at a single level and ignore their interaction at different levels. To address these two challenges, we proposed a novel framework termed Missing-Modility-Multimodal-Bert networks (MMMB). To achieve the completeness, we first generate the missing textual modality conditioned on the available visual modality. We then project the image features to the token space of the text, and use the transformer’s self-attention mechanism to make the two different modalities information interact at different levels for earlier and more fine-grained fusion, rather than only at the final layer. A large number of experiments on two large benchmark datasets in the field of image aesthetic quality evaluation: AVA and Photo.net demonstrate that the proposed model significantly improves image aesthetic assessment performance under both textual missing modality condition and full-modality condition.

Image Aesthetic Description Based on Semantic Addition Transformer Model

Image aesthetic quality assessment has been a hot research topic in the field of image analysis during the last decade. Most recently, people have proposed comment type assessment to describe the aesthetics of an image using text automatically. However, existing works have rarely considered the quality of the aesthetic description. In this work, we propose a novel neural image aesthetic description network framework, named Deep Image Aesthetic Reviewer (DIAReviewer), based on Semantic Addition Transformer Model, the learning of Residual Network, and the Attention Mechanism in a single framework. Beyond that, we design a Semantic Addition module to compromise the image feature and semantic information to focus on the comment quality, such as fluency and complexity. We introduce a new image dataset named Aesthetic Review Dataset (ARD), which contains one or more aesthetic comments for each image. Finally, the experimental results on ARD show that our model outperforms other methods in content complexity and sentence fluency of aesthetic descriptions.

Rendering Natural Bokeh Effects Based on Depth Estimation to Improve the Aesthetic Ability of Machine Vision

Machine vision is the key to realizing computer-vision tasks such as human–computer interaction and autonomous driving. However, human perception of an image’s beauty is innate. If a machine can increase aesthetic awareness, it will greatly improve the comfort of human perception in human–computer interaction. The bokeh effect is one of the most important ways to improve the artistic beauty of photographic images and the image aesthetic quality. Bokeh rendering of an image can highlight the main object of the image and blur unnecessary or unattractive background details. The existing methods usually have unrealistic rendering effects with obvious artifacts around the foreground boundary. Therefore, we propose a natural bokeh-rendering method based on depth estimation that satisfies the following characteristics: objects in the focal plane are clear and out-of-focus objects are blurred; and the further away from the focal plane, the more blurred the objects are. Our method consists of three modules: depth estimation, background subdivision, and bokeh rendering. The background-subdivision module can select different focal planes to obtain different blur radii, making the bokeh-rendering effect more diverse, so that it does not oversegment objects. The bokeh-rendering module adjusts the degree of bokeh by adjusting the blur-radius factor. In the experimental section, we analyze the model results and present the visualization results.

Image Aesthetic Description Based on Semantic Addition Transformer Model

Image aesthetic quality assessment has been a hot research topic in the field of image analysis during the last decade. Most recently, people have proposed comment type assessment to describe the aesthetics of an image using text automatically. However, existing works have rarely considered the quality of the aesthetic description. In this work, we propose a novel neural image aesthetic description network framework, named Deep Image Aesthetic Reviewer (DIAReviewer), based on Semantic Addition Transformer Model, the learning of Residual Network, and the Attention Mechanism in a single framework. Beyond that, we design a Semantic Addition module to compromise the image feature and semantic information to focus on the comment quality, such as fluency and complexity. We introduce a new image dataset named Aesthetic Review Dataset (ARD), which contains one or more aesthetic comments for each image. Finally, the experimental results on ARD show that our model outperforms other methods in content complexity and sentence fluency of aesthetic descriptions.

Image Aesthetic Assessment Based on Latent Semantic Features

Image aesthetic evaluation refers to the subjective aesthetic evaluation of images. Computational aesthetics has been widely concerned due to the limitations of subjective evaluation. Aiming at the problem that the existing evaluation methods of image aesthetic quality only extract the low-level features of images and they have a low correlation with human subjective perception, this paper proposes an aesthetic evaluation model based on latent semantic features. The aesthetic features of images are extracted by superpixel segmentation that is based on weighted density POI (Point of Interest), which includes semantic features, texture features, and color features. These features are mapped to feature words by LLC (Locality-constrained Linear Coding) and, furthermore, latent semantic features are extracted using the LDA (Latent Dirichlet Allocation). Finally, the SVM classifier is used to establish the classification prediction model of image aesthetics. The experimental results on the AVA dataset show that the feature coding based on latent semantics proposed in this paper improves the adaptability of the image aesthetic prediction model, and the correlation with human subjective perception reaches 83.75%.

A Novel Feature Fusion Method for Computing Image Aesthetic Quality

Computationally, the aesthetic quality of an image means that the model automatically scores the aesthetic level of the image. However, there are many factors that determine beauty or ugliness for photographic photos. Therefore, extracting a variety of representative aesthetic features and fusing these features are still difficult tasks. In this paper, we design a two-stream network to calculate the aesthetic quality of the image. The upper stream of the network is an improved network with the SEResNet-50 and six skip connections added, which can improve the performance of the model without training to obtain deep convolutional neural network features. The lower stream of the network consists of the proposed algorithms for handcrafted extracting aesthetic features and multiple convolution layers to extract the aesthetic features. Finally, to fuse the features of the two-stream network without adding feature dimensions, a novel feature fusion layer is proposed. The results show that this novel feature fusion method can calculate results close to the artificial aesthetic evaluation.

Incremental Learning of Multi-Tasking Networks for Aesthetic Radar Map Prediction

It is difficult and challenging to evaluate the aesthetics quality of images from multiple angles. Since humans’ perception of images comes from many factors, the integrated image aesthetic quality assessment cannot be easily summarized by few attributes. A comprehensive evaluation is supposed to predict many aesthetic attributes across not only one dataset. This requires the model to have not only high accuracy, but also strong generalization ability, resulting in a better prediction on multiple models and datasets. Recent work shows that deep convolution neural network can be used to extract image features and further evaluate the total score of images, and the method of evaluation are lacking of sufficient detailed features. In this paper, we propose a multi-task convolution neural network with more incremental features. We show the results in the way of a hexagon map, which is called aesthetic radar map. This allows the network model to fit different attributes in various datasets better.

Two open-source projects for image aesthetic quality assessment

Two open-source projects for image aesthetic quality assessment

ILGNet: inception modules with connected local and global features for efficient image aesthetic quality classification using domain adaptation

In this study, the authors address a challenging problem of aesthetic image classification, which is to label an input image as high‐ or low‐aesthetic quality. We take both the local and global features of images into consideration. A novel deep convolutional neural network named ILGNet is proposed, which combines both the inception modules and a connected layer of both local and global features. The ILGnet is based on GoogLeNet. Thus, it is easy to use a pre‐trained GoogLeNet for large‐scale image classification problem and fine tune their connected layers on a large‐scale database of aesthetic‐related images: AVA, i.e. domain adaptation. The experiments reveal that their model achieves the state of the arts in AVA database. Both the training and testing speeds of their model are higher than those of the original GoogLeNet.

Deep Active Learning with Contaminated Tags for Image Aesthetics Assessment.

Image aesthetic quality assessment has becoming an indispensable technique that facilitates a variety of image applications, e.g., photo retargeting and non-realistic rendering. Conventional approaches suffer from the following limitations: 1) the inefficiency of semantically describing images due to the inherent tag noise and incompletion, 2) the difficulty of accurately reflecting how humans actively perceive various regions inside each image, and 3) the challenge of incorporating the aesthetic experiences of multiple users. To solve these problems, we propose a novel semi-supervised deep active learning (SDAL) algorithm, which discovers how humans perceive semantically important regions from a large quantity of images partially assigned with contaminated tags. More specifically, as humans usually attend to the foreground objects before understanding them, we extract a succinct set of BING (binarized normed gradients) [60]-based object patches from each image. To simulate human visual perception, we propose SDAL which hierarchically learns human gaze shifting path (GSP) by sequentially linking semantically important object patches from each scenery. Noticeably, SDLA unifies the semantically important regions discovery and deep GSP feature learning into a principled framework, wherein only a small proportion of tagged images are adopted. Moreover, based on the sparsity penalty, SDLA can optimally abandon the noisy or redundant low-level image features. Finally, by leveraging the deeply-learned GSP features, a probabilistic model is developed for image aesthetics assessment, where the experience of multiple professional photographers can be encoded. Besides, auxiliary quality-related features can be conveniently integrated into our probabilistic model. Comprehensive experiments on a series of benchmark image sets have demonstrated the superiority of our method. As a byproduct, eye tracking experiments have shown that GSPs generated by our SDAL are about 93% consistent with real human gaze shifting paths.

Image aesthetic quality evaluation using convolution neural network embedded learning

A way of convolution neural network (CNN) embedded fine-tune based on the image contents is proposed to evaluate the image aesthetic quality in this paper. Our approach can not only solve the problem of small-scale data but also quantify the image aesthetic quality. First, we chose Alexnet and VGG_S to compare which is more suitable for image aesthetic quality evaluation task. Second, to further boost the image aesthetic quality classification performance, we employ the image content to train aesthetic quality classification models. But the training samples become smaller and only using once fine-tune can not make full use of the small-scale dataset. Third, to solve the problem in second step, a way of using twice fine-tune continually based on the aesthetic quality label and content label respective, is proposed. At last, the categorization probability of the trained CNN models is used to evaluate the image aesthetic quality. We experiment on the small-scale dataset Photo Quality. The experiment results show that the classification accuracy rates of our approach are higher than the existing image aesthetic quality evaluation approaches.

Image Aesthetic Assessment: An experimental survey

This survey aims at reviewing recent computer vision techniques used in the assessment of image aesthetic quality. Image aesthetic assessment aims at computationally distinguishing high-quality photos from low-quality ones based on photographic rules, typically in the form of binary classification or quality scoring. A variety of approaches has been proposed in the literature trying to solve this challenging problem. In this survey, we present a systematic listing of the reviewed approaches based on visual feature types (hand-crafted features and deep features) and evaluation criteria (dataset characteristics and evaluation metrics). Main contributions and novelties of the reviewed approaches are highlighted and discussed. In addition, following the emergence of deep learning techniques, we systematically evaluate recent deep learning settings that are useful for developing a robust deep model for aesthetic scoring. Experiments are conducted using simple yet solid baselines that are competitive with the current state-of-the-arts. Moreover, we discuss the possibility of manipulating the aesthetics of images through computational approaches. We hope that our survey could serve as a comprehensive reference source for future research on the study of image aesthetic assessment.