Content-based image descriptors for enhanced person annotation in personal digital photo archives

Abstract

In this paper we investigate the use of content-based image descriptors for enhancing the performance of person annotation in personal photo management applications. The descriptors examined are related to the context of person recognition through face and body-patch feature matching in personal digital photos. In order to identify the best performing content-based descriptors, we first study a number of colour and texture descriptors for body-patch matching and face recognition descriptors for face matching using a suitably chosen data set taken from typical personal photo collections. We then analyse the performance of three different fusion schemes to identify the best combination of colour, texture and face recognition descriptors. Finally, we apply those descriptors to the problem of person annotation and measure their performance using a test data set, which comprises 7 different real-life personal photo collections. The experimental results illustrate that combining body-patch feature matching with face recognition significantly improves the performance of person annotation. We further show that combining colour with texture leads to improved performance of body-patch matching. The content-based image descriptors identified in this paper show great potential for person annotation in personal photo management applications.