Evaluating Locality Sensitive Hashing for Matching Partial Image Patches in a Social Media Setting

Abstract

Images posted to a social media site can employ image completion techniques to efficiently and seamlessly remove sensitive content and safeguard privacy. Image completion algorithms typically employ a time consuming patch matching stage derived nearest neighbour search algorithms. Typical patch matching processes perform poorly in the social media context which performs once-off edits on a range of high resolution images with plentiful exemplar material. We make use of hash tables to accelerate the matching stage. Our refinement is the development of a set of perceptually inspired hash functions that can exploit locality and provide a categorization consistent across any exemplar image. Descriptors derived from principal component analysis (PCA), after training on exemplar database, are used for comparison. Aggregation of descriptors improves accuracy and we adapt a probabilistic approach using randomly oriented hyperplanes to employ multiple descriptors in a single hash table. Hash table strategies demonstrate a substantial improvement in performance over a brute force strategy, and perceptually inspired features provide levels of accuracy comparable with those trained on the data using PCA descriptors. The aggregation strategies further improve accuracy although measurement of this is confounded by non-uniform distribution of the aggregated keys. Evaluation with increasing levels of missing data demonstrates that the use of hashing continues to perform well relative to the Euclidean metric benchmark. The patch matching process using aggregated perceptually inspired descriptors produces comparable results with substantial reduction in matching time when used for image completion in photographic images. While sensitivity to structural elements is identified as an issue, the complexity of the resulting process is well suited to bulk manipulation of high resolution images for use in social media.