Image aesthetics prediction using multiple patches preserving the original aspect ratio of contents

Abstract

The spread of social networking services has created an increasing demand for selecting, editing, and generating impressive images. This trend increases the importance of evaluating image aesthetics as a fundamental function of automatic image processing. However, most existing methods for aesthetics score prediction require image rescaling for input, which can affect the prediction, especially for images with unusual aspect ratios. We propose a multi-patch method, called a multi-patch aggregation network (MPA-Net), to predict image aesthetics scores by maintaining the original aspect ratios of the contents in the images. One of our key contributions is the adoption of an equal-interval multi-patch selection approach for the prediction of the aesthetics score. The effectiveness of our strategy is shown through experiments involving the large-scale AVA dataset. Our MPA-Net outperformed the reported scores of the baseline methods and achieved a better performance in terms of the mean square error (MSE) than the state-of-the-art end-to-end continuous aesthetics score prediction methods. Most notably, MPA-Net yields a significantly lower MSE particularly for images with aspect ratios far from 1.0, indicating that MPA-Net is useful for a wide range of image aspect ratios. Moreover, MPA-Net has several benefits for training and evaluation procedures. MPA-Net meets the conditions of end-to-end learning and mini-batch learning simultaneously, and MPA-Net uses only images that do not require external information during the training or prediction stages. Thus, our easy-to-handle method improves the prediction of image aesthetics scores, outstandingly for images with extraordinary aspect ratios.