Content selection using frontalness evaluation of multiple frames

Abstract

This paper addresses the problem of selecting instances of a planar object in a video or from a set of images based on an evaluation of its “frontalness”. We introduce the idea of “evaluating the frontalness” by computing how close the object's surface normal aligns with the optical axis of a camera. The unique and novel aspect of our method is that unlike previous planar object pose estimation methods, our method does not require the true frontal image as a reference. The intuition is that a true frontal image can be used to produce other non-frontal images by perspective projection, while the non-frontal images have limited ability to produce other non-frontal images. We show that this intuition of comparing ‘frontal’ and ‘non-frontal’ can be extended to comparing ‘more frontal’ and ‘less frontal’ images. Based on this observation, our method estimates the relative frontalness of an image by exploiting the objective space error. We also propose the usage of K-invariant space to evaluate the frontalness even when the camera intrinsic parameters are unknown (e.g., images/videos from the web). We show that our method outperforms the homography decomposition-based method which also does not require reference images. In addition, a qualitative evaluation is carried out to show that our method can be applied in selecting the most frontal characters from a set of images captured in various viewpoints.