A Comprehensive Survey on Computational Aesthetic Evaluation of Visual Art Images: Metrics and Challenges

Jiajing Zhang,Jinhui Yu,Yongwei Miao

doi:10.1109/access.2021.3083075

Abstract

Computational image aesthetic evaluation is a computable human aesthetic perception and judgment realized by machines, which has a significant impact on a variety of applications such as image advanced search and promotional exhibition of painting arts. Various approaches have been proposed in copious literature trying to solve this challenging problem. However, there have been few attempts in reviewing works from different types of visual arts, due to their significant differences in visual features and aesthetic principles. In this survey, we present a comprehensive listing of the reviewed works on aesthetic assessment of photographs and paintings, mainly highlighting the contributions and innovations of the existing approaches. We firstly introduce aesthetic assessment benchmark datasets in different categories. Then, conventional aesthetic evaluation approaches based on handcrafted features are reviewed. Besides, we systematically evaluate recent deep learning techniques that are useful for developing robust models for aesthetic prediction tasks in scoring, distribution, attribute, and description. Moreover, the possibility of aesthetic-aware color enhancement, recomposition of photo images, and automatic generation of aesthetic-guided art paintings through computational approaches are summarized. Finally, challenges and potential future directions for this field are discussed. We hope that our survey could serve as a comprehensive reference source for future research on computational aesthetics in visual media.

Highlights

Aesthetics is an important discipline in visual arts that research on the aesthetic categories such as beauty and ugliness, human aesthetic consciousness, aesthetic experience, creation, development and law of beauty [1]
AESTHETIC JUDGEMENT WITH DEEP LEARNING APPROACHES Beginning with the strong performance of Krizhevsky et al [18] in the image classification, the powerful feature representation learned with a growing amount of datasets, and feasible transfer learning [19] with fine-tuned Convolutional Neural Networks (CNN) [20], deep learning methods [21] have been applied to aesthetic quality assessment of visual art images, which can automatically learn effective aesthetic features from deep hidden layers to abstract image information without expert knowledge, showing outperformed evaluation capability than conventional handcrafted features
AESTHETIC-DRIVEN MANIPULATION OF VISUAL ART IMAGES One of the most common applications in computational aesthetic evaluation is aesthetic-aware image manipulation, the aim of which is using various editing operations to improve the aesthetics of visual art images, as shown in Figure Here we focus on recent literature in three aesthetic enhancement applications including color enhancement, photo recomposition, and aesthetic-guided generation of art paintings

Summary

INTRODUCTION

Aesthetics is an important discipline in visual arts that research on the aesthetic categories such as beauty and ugliness, human aesthetic consciousness, aesthetic experience, creation, development and law of beauty [1]. The challenges include (i) quantitative modeling the complicated photographic rules, or abstract art principles and appreciation languages, (ii) explaining the aesthetical differences in various images contents, subjects, or art genres (e.g. animal, still life, scenery, architecture, landscape, portrait), (iii) knowing the expression techniques used in capturing photos or drawing paintings (e.g. lighting, sharpness, depth-of-field, motion blur, colorfulness, ink shading, whitespace), and (iv) building a large scale of aesthetic evaluation datasets for various types of visual art images. 33 image features suitable for aesthetic classification are obtained, and the results show that the art elements related to the aesthetic feeling of Chinese paintings are ranked in order of importance: color, brush strokes, brightness, and lines While these handcrafted aesthetics features achieved good evaluation performances, they have some limitations: First, the manually designed aesthetic features based on specific photographic criteria have a limited range, it is impossible to cover exhaustive effective photographic attributes. We would summarize some recent literature in aesthetic assessment using the deep learning technique

AESTHETIC JUDGEMENT WITH DEEP LEARNING APPROACHES

OPEN PROBLEMS AND FUTURE DIRECTIONS

Findings

CONCLUSION