POSE-ID-on—A Novel Framework for Artwork Pose Clustering

Valerio Marsocci,Lorenzo Lastilla

doi:10.3390/ijgi10040257

Abstract

In this work, we focus our attention on the similarity among works of art based on human poses and the actions they represent, moving from the concept of Pathosformel in Aby Warburg. This form of similarity is investigated by performing a pose clustering of the human poses, which are modeled as 2D skeletons and are defined as sets of 14 points connected by limbs. To build a dataset of properly annotated artwork images (that is, including the 2D skeletons of the human figures represented), we relied on one of the most popular, recent, and accurate deep learning frameworks for pose tracking of human figures, namely OpenPose. To measure the similarity between human poses, two alternative distance functions are proposed. Moreover, we developed a modified version of the K-Medians algorithm to cluster similar poses and to find a limited number of poses that are representative of the whole dataset. The proposed approach was also compared to two popular clustering strategies, that is, K-Means and the Nearest Point Algorithm, showing higher robustness to outliers. Finally, we assessed the validity of the proposed framework, which we named POSE-ID-on, in both a qualitative and in a quantitative way by simulating a supervised setting, since we lacked a proper reference for comparison.

Highlights

We move from the concept of Pathosformel in Aby Warburg, and, in particular, we focus our attention on the similarity among works of art based on human poses [8]
We proposed a novel method with the aim of clustering the human poses represented in artworks, starting from Aby Warburg’s concept of the Pathosformel
The German art historian developed this concept to explain the survival of figurative expressions that are found in artworks, even those that are very distant in time

Summary

Introduction

The Pathosformel describes the portrayal or communication of emotion, movement, and passion through a repeatable visual paradigm or formula [1] This concept culminated in the Bilderatlas Mnemosyne [2], which consisted of a work-inprogress series of wooden panels covered with black Hessian, on which he pinned clusters of images (photographic reproductions, photos, diagrams and sketches, postcards, and various kinds of printed materials, including advertisements and newspaper clippings) and which he developed in several stages [3]. This figurative Atlas combined images for similarities and relationships, setting up a method that is increasingly recognized as a profitable instrument of analysis, even outside the history of art [5,6], as proven by the diffusion of tools such as imgs.ai [7], a dataset-agnostic deep visual search engine for digital art history based on neural network embeddings and approximate k-Nearest

Objectives

Methods

Results

Conclusion