This is not an apple! Benefits and challenges of applying computer vision to museum collections

Elena Villaespesa,Oonagh Murphy

doi:10.1080/09647775.2021.1873827

Elena Villaespesa, Oonagh Murphy

Open Access

https://doi.org/10.1080/09647775.2021.1873827

Copy DOI

Abstract

ABSTRACT The application of computer vision on museum collection data is at an experimental stage with predictions that it will grow in significance and use in the coming years. This research, based on the analysis of five case studies and semi-structured interviews with museum professionals, examined the opportunities and challenges of these technologies, the resources and funding required, and the ethical implications that arise during these initiatives. The case studies examined in this paper are drawn from: The Metropolitan Museum of Art (USA), Princeton University Art Museum (USA), Museum of Modern Art (USA), Harvard Art Museums (USA), Science Museum Group (UK). The research findings highlight the possibilities of computer vision to offer new ways to analyze, describe and present museum collections. However, their actual implementation on digital products is currently very limited due to the lack of resources and the inaccuracies created by algorithms. This research adds to the rapidly evolving field of computer vision within the museum sector and provides recommendations to operationalize the usage of these technologies, increase the transparency on their application, create ethics playbooks to manage potential bias and collaborate across the museum sector.

Highlights

When is an apple not an apple? Is the central question asked by Trevor Paglen in his 2019 exhibition From ‘Apple’ to ‘Anomaly’ (Pictures and Labels) Selections from the ImageNet dataset for object recognition at the Barbican Centre, London
Paglen has added an additional layer to the photographic reproduction of this painting, and included the categories, or tags that the machine vision training set, ImageNet applied to the painting when it was analyzed by its algorithm
The adoption of these technologies has been slower than other sectors, this study identifies a number of underlying tensions that have created sticking points, including a lack of resources, challenges with algorithmic biases and internal resistance to the accuracy of data outputs

Summary

Introduction

When is an apple not an apple? Is the central question asked by Trevor Paglen in his 2019 exhibition From ‘Apple’ to ‘Anomaly’ (Pictures and Labels) Selections from the ImageNet dataset for object recognition at the Barbican Centre, London. Paglen has added an additional layer to the photographic reproduction of this painting, and included the categories, or tags that the machine vision training set, ImageNet applied to the painting when it was analyzed by its algorithm. These tags include the nouns ‘red and green apple’ (Crawford and Paglen 2019). Paglens work provides a helpful contextual foundation from which to begin to examine the relationship between computer vision, taxonomy, art and objects as culturally variable constructs (McKim 2019) This example helps to situate the conversation, around basic objects and simple nouns; museums deal with vast and complex collections, that engage with challenging and disputed histories, diverse cultures and contemporary society. The problems that arise through the application of these technologies, if acknowledged, documented and critically engaged with, can be mitigated, and the opportunities of computer vision can be utilized to create a more robust, documented and discoverable collection (Murphy and Villaespesa 2020)

Objectives

Methods

Results

Discussion

Conclusion