Using Principal Paths to Walk Through Music and Visual Art Style Spaces Induced by Convolutional Neural Networks

E Gardini,M J Ferrarotti,A Cavalli,S Decherchi

doi:10.1007/s12559-021-09823-y

E Gardini, M J Ferrarotti + Show 2 more

Open Access

https://doi.org/10.1007/s12559-021-09823-y

Copy DOI

Abstract

Computational intelligence, particularly deep learning, offers powerful tools for discriminating and generating samples such as images. Deep learning methods have been used in different artistic contexts for neural style transfer, artistic style recognition, and musical genre recognition. Using a constrained manifold analysis protocol, we discuss to what extent spaces induced by deep-learning convolutional neural networks can capture historical/stylistic progressions in music and visual art. We use a path-finding algorithm, called principal path, to move from one point to another. We apply it to the vector space induced by convolutional neural networks. We perform experiments with visual artworks and songs, considering a subset of classes. Within this simplified scenario, we recover a reasonable historical/stylistic progression in several cases. We use the principal path algorithm to conduct an evolutionary analysis of vector spaces induced by convolutional neural networks. We perform several experiments in the visual art and music spaces. The principal path algorithm finds reasonable connections between visual artworks and songs from different styles/genres with respect to the historical evolution when a subset of classes is considered. This approach could be used in many areas to extract evolutionary information from an arbitrary high-dimensional space and deliver interesting cognitive insights.

Highlights

Computational intelligence (CI) and cognitive computations are relatively young fields in science and engineering, with the first ideas dating back to Turing [1]
The proposed protocol has three main steps elucidated in Fig. 1: 1. In the first step, a Convolutional neural networks (CNNs) is used as a featurization tool
To visualize the points together with the paths, we reduced the dimensions via principal component analysis (PCA) [30] followed by t-Distributed stochastic neighbor embedding (t-SNE) [25]

Summary

Introduction

Computational intelligence (CI) and cognitive computations are relatively young fields in science and engineering, with the first ideas dating back to Turing [1]. Some of the most successful CI systems are based on deep artificial neural networks and their variations, which model networks of neurons to solve tasks such as pattern recognition, learning, memorization, and generalization [2]. The community has become interested in other creative and innovative applications of deep artificial neural networks. These applications include cognitive tasks such as sentiment analysis, neural language processing, neural style transfer, artistic style recognition, and musical genre identification. In the field of sentiment analysis, works like [9,10,11] and many others have tried to transform abstract concepts like emotions into images and sounds. Neuralstyle transfer applications use CNNs to recombine the content of one image with the style of another, like in [12]. In terms of recognizing styles of visual art, some of the most interesting solutions have been proposed by Lecoutre et al [13], Karayev et al [14], and Tan et al [15], who achieved promising results using two publicly available CNN architectures (AlexNet [7] and ResNet [16]) with the same dataset (Wikipainting)

Methods

Results

Discussion

Conclusion