SpyGAN sketch: Heterogeneous Face Matching in video for crime investigation

Abstract

Automatic retrieval of faces from videos based on query images effectively helps during the investigation. When the suspect’s image is unavailable, a face sketch, drawn based on eyewitness’s memory recollection, is used to search against photos. Present research works primarily focus on Heterogeneous Face Matching (HFM) sketches to mugshot images in databases. This paper proposes a sketch-face matching in a video that includes profile faces, different illumination, and poses, using a new Generative Adversarial Network called SpyGAN. Faces in the video detected using YOLOv3 are converted into realistic sketches by the proposed SpyGAN focusing on key facial regions. The generated sketches are represented using PCA-SIFT descriptors and are matched based on the cosine distance metric. Experimental results show that the proposed methodology has achieved an accuracy of 88.9% on the Chokepoint dataset and 78% on the OWN Short face-video linked dataset and has demonstrated effectiveness over the state-of-the-art methods.