Image Aesthetic Quality Assessment Research Articles

Aesthetically Relevant Image Captioning

Image aesthetic quality assessment (AQA) aims to assign numerical aesthetic ratings to images whilst image aesthetic captioning (IAC) aims to generate textual descriptions of the aesthetic aspects of images. In this paper, we study image AQA and IAC together and present a new IAC method termed Aesthetically Relevant Image Captioning (ARIC). Based on the observation that most textual comments of an image are about objects and their interactions rather than aspects of aesthetics, we first introduce the concept of Aesthetic Relevance Score (ARS) of a sentence and have developed a model to automatically label a sentence with its ARS. We then use the ARS to design the ARIC model which includes an ARS weighted IAC loss function and an ARS based diverse aesthetic caption selector (DACS). We present extensive experimental results to show the soundness of the ARS concept and the effectiveness of the ARIC model by demonstrating that texts with higher ARS’s can predict the aesthetic ratings more accurately and that the new ARIC model can generate more accurate, aesthetically more relevant and more diverse image captions. Furthermore, a large new research database containing 510K images with over 5 million comments and 350K aesthetic scores, and code for implementing ARIC, are available at https://github.com/PengZai/ARIC

Correct for Whom? Subjectivity and the Evaluation of Personalized Image Aesthetics Assessment Models

The problem of image aesthetic quality assessment is surprisingly difficult to define precisely. Most early work attempted to estimate the average aesthetic rating of a group of observers, while some recent work has shifted to an approach based on few-shot personalization. In this paper, we connect few-shot personalization, via Immanuel Kant's concept of disinterested judgment, to an argument from feminist aesthetics about the biased tendencies of objective standards for subjective pleasures. To empirically investigate this philosophical debate, we introduce PR-AADB, a relabeling of the existing AADB dataset with labels for pairs of images, and measure how well the existing groundtruth predicts our new pairwise labels. We find, consistent with the feminist critique, that both the existing groundtruth and few-shot personalized predictions represent some users' preferences significantly better than others, but that it is difficult to predict when and for whom the existing groundtruth will be correct. We thus advise against using benchmark datasets to evaluate models for personalized IAQA, and recommend caution when attempting to account for subjective difference using machine learning more generally.

Styling Classification of Group Photos Fusing Head and Pose Features

Group photo images are everywhere and vary greatly by the shooting scene. Compared with common images, the Image Aesthetic Quality Assessment (IAQI) of group photo pays more attention to the relevant characteristics of the main population. Still, the existing methods do not make further special research on group photos. Therefore, we propose a new concept of group photo styling based on analyzing group photos and photographic theory. Besides that, by comparing and analyzing many group photos, we classify the group photos into five categories. In this paper, the main factors of the head and pose are considered simultaneously, and the method of Group Photo Styling Classification (GPSC) can classify different group photos automatically. To verify the effectiveness of our method, we collected a Group Photo Styling Dataset (GPSD). The dataset contains 998 group photo images, and each image’s group photo styling category is marked. The experimental results on GPSD show that the fusion of head features and pose features can classify different group photos well. The accuracy of GPSC reaches 93.9%, much higher than the previous classification model.

Pseudo-Labeling and Meta Reweighting Learning for Image Aesthetic Quality Assessment

In the tasks of image aesthetic quality assessment, it is difficult to reach both the high score area and low score area due to the normal distribution of aesthetic datasets. To reduce the error in labelling and solve the problem of normal data distribution, we propose a new aesthetic mixed dataset with classification and regression called AMD-CR, and we train a meta reweighting network to reweight the loss of training data differently. In addition, we provide a training strategy according to different stages based on pseudo labels, and then we use it for aesthetic training according to different stages in classification and regression tasks. In the construction of the network structure, we construct an aesthetic adaptive block (AAB) structure that can adapt to any size of the input images. Besides, we also use the efficient channel attention (ECA) to strengthen the feature extracting ability of each task. The experimental result shows that our method improves 0.1112 compared with the conventional method in SROCC. The method can also help to find best aesthetic path planning for unmanned aerial vehicles (UAV) and vehicles.

Semantic-aware multi-task learning for image aesthetic quality assessment

In recent years, image aesthetic quality assessment has attracted considerable attention due to the massive growth of digital images in social platforms and the Internet. However, automatically assessing aesthetic quality of an image is a challenging task, because image aesthetic is affected by various factors, and the criteria for judging the aesthetic of images with diverse semantic information are different. To this end, a Semantic-Aware Multi-task convolution neural network (SAM-CNN) for evaluating image aesthetic quality is proposed in this paper. The network can fuse intermediate features of different layers at different scales in CNN to obtain a more comprehensive and accurate aesthetic expression, under the joint supervision of image aesthetic quality assessment task and semantic classification task in a multi-task learning manner. Besides, by applying the attention mechanism, semantic information with a large receptive field extracted from deep layers is utilised to guide the network to focus on the key parts of features to be fused, to improve the effectiveness of feature fusion. Experimental results on the AVA dataset and Photo.net dataset demonstrate the effectiveness and superiority of the proposed SAM-CNN.

Theme-Aware Semi-Supervised Image Aesthetic Quality Assessment

Image aesthetic quality assessment (IAQA) has aroused considerable interest in recent years and is widely used in various applications, such as image retrieval, album management, chat robot and social media. However, existing methods need an excessive amount of labeled data to train the model. Collecting the enormous quantity of human scored training data is not always feasible due to a number of factors, such as the expensiveness of the labeling process and the difficulty in correctly classifying data. Previous studies have evaluated the aesthetic of a photo based only on image features, but have ignored the criterion bias associated with the themes. In this work, we present a new theme-aware semi-supervised image quality assessment method to address these difficulties. Specifically, the proposed method consists of two steps: a representation learning step and a label propagation step. In the representation learning step, we propose a robust theme-aware attention network (TAAN) to cope with the theme criterion bias problem. In the label propagation step, we use preliminary trained TAAN by step one to extract features and utilize the label propagation with a cumulative confidence (LPCC) algorithm to assign pseudo-labels to the unlabeled data. This enables use of both labeled and unlabeled data to train the TAAN model. To the best of our knowledge, this is the first time that a semi-supervised learning method to address image aesthetic assessment problems has been studied. We evaluate our approach on three benchmark datasets and show that it can achieve almost the same performance as a fully supervised learning method for a small number of samples. Furthermore, we show that our semi-supervised approach is robust to using varying quantities of labeled data.

Image Aesthetic Description Based on Semantic Addition Transformer Model

Image aesthetic quality assessment has been a hot research topic in the field of image analysis during the last decade. Most recently, people have proposed comment type assessment to describe the aesthetics of an image using text automatically. However, existing works have rarely considered the quality of the aesthetic description. In this work, we propose a novel neural image aesthetic description network framework, named Deep Image Aesthetic Reviewer (DIAReviewer), based on Semantic Addition Transformer Model, the learning of Residual Network, and the Attention Mechanism in a single framework. Beyond that, we design a Semantic Addition module to compromise the image feature and semantic information to focus on the comment quality, such as fluency and complexity. We introduce a new image dataset named Aesthetic Review Dataset (ARD), which contains one or more aesthetic comments for each image. Finally, the experimental results on ARD show that our model outperforms other methods in content complexity and sentence fluency of aesthetic descriptions.

Image Aesthetic Description Based on Semantic Addition Transformer Model

Image aesthetic quality assessment has been a hot research topic in the field of image analysis during the last decade. Most recently, people have proposed comment type assessment to describe the aesthetics of an image using text automatically. However, existing works have rarely considered the quality of the aesthetic description. In this work, we propose a novel neural image aesthetic description network framework, named Deep Image Aesthetic Reviewer (DIAReviewer), based on Semantic Addition Transformer Model, the learning of Residual Network, and the Attention Mechanism in a single framework. Beyond that, we design a Semantic Addition module to compromise the image feature and semantic information to focus on the comment quality, such as fluency and complexity. We introduce a new image dataset named Aesthetic Review Dataset (ARD), which contains one or more aesthetic comments for each image. Finally, the experimental results on ARD show that our model outperforms other methods in content complexity and sentence fluency of aesthetic descriptions.

Incremental Learning of Multi-Tasking Networks for Aesthetic Radar Map Prediction

It is difficult and challenging to evaluate the aesthetics quality of images from multiple angles. Since humans’ perception of images comes from many factors, the integrated image aesthetic quality assessment cannot be easily summarized by few attributes. A comprehensive evaluation is supposed to predict many aesthetic attributes across not only one dataset. This requires the model to have not only high accuracy, but also strong generalization ability, resulting in a better prediction on multiple models and datasets. Recent work shows that deep convolution neural network can be used to extract image features and further evaluate the total score of images, and the method of evaluation are lacking of sufficient detailed features. In this paper, we propose a multi-task convolution neural network with more incremental features. We show the results in the way of a hexagon map, which is called aesthetic radar map. This allows the network model to fit different attributes in various datasets better.

Two open-source projects for image aesthetic quality assessment

Two open-source projects for image aesthetic quality assessment

Deep Active Learning with Contaminated Tags for Image Aesthetics Assessment.

Image aesthetic quality assessment has becoming an indispensable technique that facilitates a variety of image applications, e.g., photo retargeting and non-realistic rendering. Conventional approaches suffer from the following limitations: 1) the inefficiency of semantically describing images due to the inherent tag noise and incompletion, 2) the difficulty of accurately reflecting how humans actively perceive various regions inside each image, and 3) the challenge of incorporating the aesthetic experiences of multiple users. To solve these problems, we propose a novel semi-supervised deep active learning (SDAL) algorithm, which discovers how humans perceive semantically important regions from a large quantity of images partially assigned with contaminated tags. More specifically, as humans usually attend to the foreground objects before understanding them, we extract a succinct set of BING (binarized normed gradients) [60]-based object patches from each image. To simulate human visual perception, we propose SDAL which hierarchically learns human gaze shifting path (GSP) by sequentially linking semantically important object patches from each scenery. Noticeably, SDLA unifies the semantically important regions discovery and deep GSP feature learning into a principled framework, wherein only a small proportion of tagged images are adopted. Moreover, based on the sparsity penalty, SDLA can optimally abandon the noisy or redundant low-level image features. Finally, by leveraging the deeply-learned GSP features, a probabilistic model is developed for image aesthetics assessment, where the experience of multiple professional photographers can be encoded. Besides, auxiliary quality-related features can be conveniently integrated into our probabilistic model. Comprehensive experiments on a series of benchmark image sets have demonstrated the superiority of our method. As a byproduct, eye tracking experiments have shown that GSPs generated by our SDAL are about 93% consistent with real human gaze shifting paths.