Face Image Retrieval Research Articles

Similarity-based face image retrieval using sparsely embedded deep features and binary code learning

Similarity-based face image retrieval using sparsely embedded deep features and binary code learning

Vision-language joint representation learning for sketch less facial image retrieval

The traditional sketch-based facial image retrieval (SBFIR) framework assumes that a high-quality facial sketch has been prepared prior to the retrieval task. However, drawing such a sketch requires considerable skills and is time consuming, resulting in limited applicability. Sketch less facial image retrieval (SLFIR) framework aims to break these barriers through human–computer interaction during the sketching process. The primary challenges for the SLFIR problem can be noted that initial sketches (at early sketching) contain only local details and exhibit significant differences among users, resulting in poor performance at early stages and weak generalization abilities in practical testing. In this study, we developed a vision–language pretraining model to align the representation of facial images and their associated semantics. Based on this framework, we proposed a method for learning joint representations by fusing sketches with prior semantics, thereby enriching the information of initial sketches. Specifically, (1) we developed a series of well-designed operations to improve the quality of facial image–text pairs in the LAION-Face dataset; we trained a facial vision–language pretraining (FVLP) model to align the facial image and its semantics at the feature level. (2) subsequently, using FVLP as the backbone, we designed a convolutional attention module to fuse the multiscale features extracted from image encoder. This facilitated the learning of a multimodal representation crucial for the final retrieval process. In experiments, our proposed method achieved state-of-the-art performance at early stages on two public datasets; moreover, it exhibited good generalization capabilities during practical testing. Thus, our method significantly outperforms other baselines in terms of early retrieval performance. Codes are available at: https://github.com/ddw2AIGROUP2CQUPT/FVLP.

ExplainLFS: Explaining neural architectures for similarity learning from local perturbations in the latent feature space

Despite the increasing development in recent years of explainability techniques for deep neural networks, only some are dedicated to explaining the decisions made by neural networks for similarity learning. While existing approaches can explain classification models, their adaptation to generate visual similarity explanations is not trivial. Neural architectures devoted to this task learn an embedding that maps similar examples to nearby vectors and non-similar examples to distant vectors in the feature space. In this paper, we propose a post-hoc agnostic technique that explains the inference of such architectures on a pair of images. The proposed method establishes a relation between the most important features of the abstract feature space and the input feature space (pixels) of an image. For this purpose, we employ a relevance assignment and a perturbation process based on the most influential latent features in the inference. Then, a reconstruction process of the images of the pair is carried out from the perturbed embedding vectors. This process relates the latent features to the original input features. The results indicate that our method produces “continuous” and “selective” explanations. A sharp drop in the value of the function (summarized by a low value of the area under the curve) indicates its superiority over other explainability approaches when identifying features relevant to similarity learning. In addition, we demonstrate that our technique is agnostic to the specific type of similarity model, e.g., we show its applicability in two similarity learning tasks: face recognition and image retrieval.

Once and for All: Universal Transferable Adversarial Perturbation against Deep Hashing-Based Facial Image Retrieval

Deep Hashing (DH)-based image retrieval has been widely applied to face-matching systems due to its accuracy and efficiency. However, this convenience comes with an increased risk of privacy leakage. DH models inherit the vulnerability to adversarial attacks, which can be used to prevent the retrieval of private images. Existing adversarial attacks against DH typically target a single image or a specific class of images, lacking universal adversarial perturbation for the entire hash dataset. In this paper, we propose the first universal transferable adversarial perturbation against DH-based facial image retrieval, a single perturbation can protect all images. Specifically, we explore the relationship between clusters learned by different DH models and define the optimization objective of universal perturbation as leaving from the overall hash center. To mitigate the challenge of single-objective optimization, we randomly obtain sub-cluster centers and further propose sub-task-based meta-learning to aid in overall optimization. We test our method with popular facial datasets and DH models, indicating impressive cross-image, -identity, -model, and -scheme universal anti-retrieval performance. Compared to state-of-the-art methods, our performance is competitive in white-box settings and exhibits significant improvements of 10%-70% in transferability in all black-box settings.

An improved facial image retrieval using Hybrid Cheetah Optimization Algorithm

With the advent of social media, the volume of photographs uploaded on the internet has increased exponentialy. The task of efficiently recognizing and retrieving human facial images is inevitable and essential at this time. In this work, a feature selection approach for recognizing and retrieving human face images using Hybrid Cheetah Optimization algorithm is proposed. The deep feature extraction from the images is done using Deep Convolutional Neural Networks. Hybrid Cheetah Optimization algorithm, an improvised version of Cheetah Optimization algorithm fused with genetic algorithm is used, to choose optimum features from the extracted deep features. The chosen features are used for finding the best matching images from the image database. The image matching is performed by Approximate Nearest Neighbor search for the query image over the image database and similar images are retrieved. By constructing a k-NN graph for the images, the efficiency of image retrieval is enhanced. The proposed system’s performance is evaluated against benchmark datasets such as LFW, MultiePie, ColorFERET, DigiFace-1M and CelebA. The evaluation results has shown that the proposed methodology is superior to various existing methodologies.

Automated facial characterization and image retrieval by convolutional neural networks.

Developing efficient methods to infer relations among different faces consisting of numerous expressions or on the same face at different times (e.g., disease progression) is an open issue in imaging related research. In this study, we present a novel method for facial feature extraction, characterization, and identification based on classical computer vision coupled with deep learning and, more specifically, convolutional neural networks. We describe the hybrid face characterization system named FRetrAIval (FRAI), which is a hybrid of the GoogleNet and the AlexNet Neural Network (NN) models. Images analyzed by the FRAI network are preprocessed by computer vision techniques such as the oriented gradient-based algorithm that can extract only the face region from any kind of picture. The Aligned Face dataset (AFD) was used to train and test the FRAI solution for extracting image features. The Labeled Faces in the Wild (LFW) holdout dataset has been used for external validation. Overall, in comparison to previous techniques, our methodology has shown much better results on k-Nearest Neighbors (KNN) by yielding the maximum precision, recall, F1, and F2 score values (92.00, 92.66, 92.33, and 92.52%, respectively) for AFD and (95.00% for each variable) for LFW dataset, which were used as training and testing datasets. The FRAI model may be potentially used in healthcare and criminology as well as many other applications where it is important to quickly identify face features such as fingerprint for a specific identification target.

ETLBP and ERDLBP descriptors for efficient facial image retrieval in CBIR systems

ETLBP and ERDLBP descriptors for efficient facial image retrieval in CBIR systems

Orthonormal product quantization network for scalable face image retrieval

Existing deep quantization methods provided an efficient solution for large-scale image retrieval. However, the significant intra-class variations, like pose, illumination, and expressions in face images, still pose a challenge. In light of this, face image retrieval requires sufficiently powerful learning metrics, which are absent in current deep quantization works. Moreover, to tackle the growing unseen identities in the query stage, face image retrieval drives more demands regarding model generalization and scalability than general image retrieval tasks. This paper integrates product quantization with orthonormal constraints into an end-to-end deep learning framework to effectively retrieve face images. Specifically, we propose a novel scheme that uses predefined orthonormal vectors as codewords to enhance the quantization informativeness and reduce codewords’ redundancy. A tailored loss function maximizes discriminability among identities in each quantization subspace for both the quantized and original features. An entropy-based regularization term is imposed to reduce the quantization error. Experiments are conducted on four commonly-used face datasets under both seen and unseen identity retrieval settings. Our method outperforms all the compared state-of-the-art under both settings. The proposed orthonormal codewords consistently boost both models’ standard retrieval performance and generalization ability, demonstrating the superiority of our method for scalable face image retrieval.

Locality-Sensitive Hashing of Soft Biometrics for Efficient Face Image Database Search and Retrieval

As multimedia technology has advanced in recent years, the use of enormous image libraries has dramatically expanded. In applications for image processing, image retrieval has emerged as a crucial technique. Content-based face image retrieval is a well-established technology in many real-world applications, such as social media, where dependable retrieval capabilities are required to enable quick search among large numbers of images. Humans frequently use faces to recognize and identify individuals. Face recognition from official or personal photos is becoming increasingly popular as it can aid crime detectives in identifying victims and criminals. Furthermore, a large number of images requires a large amount of storage, and the process of image comparison and matching, consequently, takes longer. Hence, the query speed and low storage consumption of hash-based image retrieval techniques have garnered a considerable amount of interest. The main contribution of this work is to try to overcome the challenge of performance improvement in image retrieval by using locality-sensitive hashing (LSH) for retrieving top-matched face images from large-scale databases. We use face soft biometrics as a search input and propose an effective LSH-based method to replace standard face soft biometrics with their corresponding hash codes for searching a large-scale face database and retrieving the top-k of the matching face images with higher accuracy in less time. The experimental results, using the Labeled Faces in the Wild (LFW) database together with the corresponding database of attributes (LFW-attributes), show that our proposed method using LSH face soft biometrics (Soft BioHash) improves the performance of face image database search and retrieval and also outperforms the LSH hard face biometrics method (Hard BioHash).

Image Retrieval Using Convolutional Autoencoder, InfoGAN, and Vision Transformer Unsupervised Models

Query by Image Content (QBIC), subsequently known as Content-Based Image Retrieval (CBIR) systems, may offer a more advantageous solution in a variety of applications, including medical, meteorological, search by image, and others. Such systems primarily use similarity matching algorithms to compare image content to get their relevance from databases. They are essentially measuring the spatial distance between extracted visual features from a query image and their correspondence in the dataset. One of the most challenging query retrieval problems is Facial Sketched-Real Image Retrieval (FSRIR), which is content similarity matching based. These facial retrieving systems are employed in a variety of contexts, including criminal justice. The difficulties of retrieving such sorts come from the composition of the human face and its distinctive parts. In addition, the comparison between these images is made from two different domains. Besides, to our knowledge, there is a rare existence of large-scale facial datasets that can be used to evolve the performance of the retrieving system. The success of the retrieval process is governed by the method used to calculate similarity and the efficient representation of compared images. However, by effectively representing visual features, the main challenge-posing component of such approaches might be resolved. Hence, this paper has several contributions that fill the research gap in content-based similarity matching and retrieving as follows: 1) The first contribution is extending the Chinese University Face Sketch (CUFS) dataset by including augmented images, introducing to the community a novel dataset named Extended Sketched-Real Image Retrieval (ESRIR). The CUFS dataset has been extended from 100 images to include 53,000 facial sketches and 53,000 real facial images. 2) The paper’s second contribution is proposing three new algorithms for sketched-real image retrieving based on convolutional autoencoder, Infogan, and Vision Transformer unsupervised models for large datasets. 3) Furthermore, to meet the subjective demands of the users because of the prevalence of multiple query formats. The third contribution of this paper is to train and assess the proposed algorithms across two additional facial datasets of various image sorts. 4) Recently, the majority of people have preferred searching for brand logo images, but it may be tricky to separate certain brand logo features from their alternatives and even from other features in an image. Thus, the fourth contribution is to compare logo image retrieval performance based on visual features derived from each of the three suggested retrieving systems. 5) The paper also proposes cloud-based energy-saving and computational complexity approaches in large-scale datasets. Due to the ubiquity of touchscreen devices, users often make drawings based on their fantasies for certain object image searches. Thus, the proposed algorithms were tested and assessed on a tough dataset of doodle-scratched human artworks. They are also studied for a multi-category dataset to cover practically all possible image types and situations. The results are compared with the most recent algorithms found in the literature. The results show that the proposed approaches are outperforming the recent algorithms.

Rapid Face Image Retrieval for Large Scale Based on Spark and Machine Learning

This paper proposed a distributed computing environment large-scale face image retrieval method. Based on Spark platform, a face retrieval method based on machine learning is established, which takes advantage of parallel computing to improve the efficiency of face image retrieval. In the method, SIFT algorithm is used for feature coding, PCA dimension reduction processing, HBase database is used for data storage, and KD-Tree query algorithm is used to match images similar to the query images. Meanwhile, large-scale computing engine uses Spark to process data to improve the retrieval efficiency. The CelebA dataset is selected to test the method, and the experimental results show the effectiveness of the method.

ANN softmax

Thanks to the popularity of GPU and the growth of its computational power, more and more deep learning tasks, such as face recognition, image retrieval and word embedding, can take advantage of extreme classification to improve accuracy. However, it remains a big challenge to train a deep model with millions of classes efficiently due to the huge memory and computation consumption in the last layer. By sampling a small set of classes to avoid the total classes calculation, sampling-based approaches have been proved to be an effective solution. But most of them suffer from the following two issues: i) the important classes are ignored or only partly sampled, such as the methods using random sampling scheme or retrieval techniques of low recall (e.g., locality-sensitive hashing), resulting in the degradation of accuracy; ii) inefficient implementation owing to incompatibility with GPU, like selective softmax. It uses hashing forest to help select classes, but the search process is implemented in CPU. To address the above issues, we propose a new sampling-based softmax called ANN Softmax in this paper. Specifically, we employ binary quantization with inverted file system to improve the recall of important classes. With the help of dedicated kernel design, it can be totally parallelized in mainstream training framework. Then, we find the size of important classes that are recalled by each training sample has a great impact on the final accuracy, so we introduce sample grouping optimization to well approximate the full classes training. Experimental evaluations on two tasks (Embedding Learning and Classification) and ten datasets (e.g., MegaFace, ImageNet, SKU datasets) demonstrate our proposed method maintains the same precision as Full Softmax for different loss objectives, including cross entropy loss, ArcFace, CosFace and D-Softmax loss, with only 1/10 sampled classes, which outperforms the state-of-the-art techniques. Moreover, we implement ANN Soft-max in a complete GPU pipeline that can accelerate the training more than 4.3X. Equipped our method with a 256 GPUs cluster, the time of training a classifier of 300 million classes on our SKU-300M dataset can be reduced to ten days.

Facial expression analysis using local directional stigma mean patterns and convolutional neural networks

This paper represents automatic facial expression analysis method named Local Directional Stigma Mean Patterns (LDSMP) for automatic facial expression analysis and image retrieval using content based facial expression image retrieval and CNN. The traditional local patterns such as Local Binary Patterns (LBP) and Local Ternary Patterns (LTP) are applied for face recognition and expression analysis, calculated using relationship between the center pixel and neighboring pixels. The proposed method calculates the eight directional difference values then divided into the three ranges based on threshold values. Thus, the values are substituted with basic three positive values (+3, +2, +1) and three negative values (-3, -2, -1) to get more sensitive information from an image rather than aforementioned methods. The threshold can be select either static which is selected by user or dynamic is evaluated from image itself and supports to improve the efficiency. The performance of the proposed method is further improved by giving this patterns as input to the Convolutional Neural Networks (CNN) and compared with the existing methods LBP, LTP, and Directional Binary Code (DBC) in terms of Average Precision (AP), Average Recall (AR), and Average Retrieval Rate (ARR) using standard databases COREL 10K (DB1) and JAFFE (The Japanese Female Facial Expression) (DB2) and Extended Cohn-Kanade (CK +) (DB3) dataset.

Deep center-based dual-constrained hashing for discriminative face image retrieval

With the advantages of low storage cost and extremely fast retrieval speed, deep hashing methods have attracted much attention for image retrieval recently. However, large-scale face image retrieval with significant intra-class variations is still challenging. Neither existing pairwise/triplet labels-based nor softmax classification loss-based deep hashing works can generate compact and discriminative binary codes. Considering these issues, we propose a center-based framework integrating end-to-end hashing learning and class centers learning simultaneously. The framework minimizes the intra-class variance by clustering intra-class samples into a learnable class center. To strengthen inter-class separability, it additionally imposes a novel regularization term to enlarge the Hamming distance between pairwise class centers. Moreover, a simple yet effective regression matrix is introduced to encourage intra-class samples to generate the same binary codes, which further enhances the hashing codes compactness. Experiments on four large-scale datasets show the proposed method outperforms state-of-the-art baselines under various code lengths and commonly-used evaluation metrics.

Improved loss function strategy for multi source face image retrieval

Improved loss function strategy for multi source face image retrieval

Short search space and synthesized-reference re-ranking for face image retrieval

Face image retrieval underpins numerous applications in many computer vision domains, however facial appearance variations including age, gender and race make this task challenging. Prior art methods rely on geometric properties and relationship between local features. However, their performance is still short of what is needed, mainly because (1) they ignore the demographic information, and (2) lack age-invariant re-ranking while retrieving face images. In this paper, we aim to build a two-stage face retrieval approach. First, we search for candidate face images using demographic-assisted clustering resulting in a short search space. Second, we develop a generative model to compensate aging variations between query and candidate face images resulting into an independent aging synthesized face images reference set. We then use this reference set to re-rank candidate face images resulting into the final retrieval of face images. We show that the proposed face retrieval approach outperforms the state-of-the-art methods in terms of both the precision and scalability on publicly available longitudinal datasets including CACD and MORPH II.

Two-dimensional Subclass Discriminant Analysis for face recognition

Dimensionality reduction plays a major role in face recognition. Discriminant analysis (DA) and principal component analysis (PCA) are two of the most important approaches in this field. In particular, subclass discriminant analysis (SDA) is a well-known scheme for feature extraction and dimensionality reduction. It is widely used in many high-dimensional data-driven applications, namely face recognition and image retrieval. It is also found to be applicable under various scenarios. However, it has high cost in time and space given the need for an eigendecomposition involving the scatter matrices, known as the singularity problem. This limitation is caused by the high-dimensional space of data, particularly when dimensions exceed the number of observations. Recent advances widely reported that 2D methods with matrix-based representation perform better than the traditional 1D vector-based ones. In this paper, we propose a novel 2D-SDA algorithm to avoid the “curse of dimensionality” and address the singularity issue. The performance of the proposed algorithm is evaluated for face recognition in terms of recognition performance and computational cost. Experiments are conducted on four benchmark face databases and compared to several competitive 1D and 2D methods based on PCA and DA. Results show that 2DSVD achieves the best recognition performance at low dimensions. In particular, 2D-SDA works significantly better on large-sized data sets where intra-class variation is the most important.

Learning Representations for Facial Actions From Unlabeled Videos.

Facial actions are usually encoded as anatomy-based action units (AUs), the labelling of which demands expertise and thus is time-consuming and expensive. To alleviate the labelling demand, we propose to leverage the large number of unlabelled videos by proposing a twin-cycle autoencoder (TAE) to learn discriminative representations for facial actions. TAE is inspired by the fact that facial actions are embedded in the pixel-wise displacements between two sequential face images (hereinafter, source and target) in the video. Therefore, learning the representations of facial actions can be achieved by learning the representations of the displacements. However, the displacements induced by facial actions are entangled with those induced by head motions. TAE is thus trained to disentangle the two kinds of movements by evaluating the quality of the synthesized images when either the facial actions or head pose is changed, aiming to reconstruct the target image. Experiments on AU detection show that TAE can achieve accuracy comparable to other existing AU detection methods including some supervised methods, thus validating the discriminant capacity of the representations learned by TAE. TAE's ability in decoupling the action-induced and pose-induced movements is also validated by visualizing the generated images and analyzing the facial image retrieval results qualitatively and quantitatively.

Dynamic multi-attribute priority based face attribute detection for robust face image retrieval system

The increase in the popularity of social websites and smartphones has helped people to easily click photos and upload them on the internet. Almost 60% of the photos are human faces. There might be an increase in face search in future because human faces are closely associated with social media where people show special interest on specific personalities. The main issue raised in the face retrieval system is the various intra-class variances like expression, pose and illumination. Most of the conventional approaches lack the accuracy to meet the human intuition for retrieving images. The proposed approach develops a Dynamic Multi-Attribute Priority-based Face Attribute Detection (DMAP-FAD) method based on contextual information of the face (Race and Gender) by detecting the attributes dynamically. This approach can provide better discrimination of the face image retrieval system by detecting the attributes dynamically. The digital binning technique is applied for improving the lighting differences for illumination. The proposed method effectively minimized the semantic gap and achieved high accuracy by focusing on the variations involved in the pose, illumination, and expression with dynamically detected an optimal set of attributes from the original set of attributes by using contextual relationship. In the experimental results, the proposed method has been tested efficiently by the metrics such as F-Score, Precision and Recall with help of Pubfig and LFW datasets and it is observed that the proposed method obtained an overall accuracy of 95.63% for higher discrimination of face image retrieval system. The results are comparable with those of the state-of-the-art methods.

Face image retrieval via sum and difference histograms of elliptical local ternary pattern

This paper presents a new feature extraction method called sum and difference histograms of elliptical local ternary pattern (SDH-ELTP) for face image retrieval. This technique first calculates sparse local ternary pattern (LTP) in an elliptical shaped neighborhood and then higher order statistical texture information is extracted via sum and difference histograms (SDH) of elliptical LTP features. In sparse elliptical LTP, a 4 point LTP from horizontal and vertical elliptical neighborhoods and 4 point LTP from simple diagonal neighborhood is considered. The SDH is calculated only in relevant directions. Since the sum and difference histogram provides higher order statistical information, the calculation of SDH of elliptical LTP features further enhances the discriminativeness of proposed descriptor. The SDH-ELTP is finally tested on two popular face image databases and the results are compared with several recent state of the art techniques. The SDH-ELTP is low dimensional and show the best retrieval results as compared to all other face image retrieval techniques.