Quadruplet Loss Research Articles

Deep Adaptive Quadruplet Hashing With Probability Sampling for Large-Scale Image Retrieval

With the preferable efficiency in storage and computation, hashing has shown potential application in large-scale multimedia retrieval. Compared with traditional hashing algorithms using hand-crafted characteristics, deep hashing inherits the representational capacity of deep neural networks to jointly learn semantic features and hash functions, encoding raw data into compact binary codes with significant discrimination. Generally, most of the current multi-wise hashing methods view the similarity margins between image pairs as constant values in training process. When the distance between sample pairs exceeds the fixed margin, the hashing network would not learn anything. Besides, available hashing methods commonly introduce the random sampling strategy to build training batches and ignore the sample distribution, which is harmful to parameter optimization. In this paper, we propose a novel Deep Adaptive Quadruplet Hashing with probability sampling (DAQH) for discriminative binary code learning. Specifically, with exploring the distribution relationship of raw samples, a non-uniform probability sampling strategy is proposed to build more informative and representative training batches, while maintaining the diversity of training samples. By introducing the prior similarity of sample pairs to calculate corresponding margins, an adaptive margin quadruplet loss is designed to dynamically preserve the underlying semantic relationships with its neighbors. To tune the attributes of binary codes, by combining quadruple regularization and orthogonality optimization, binary code constraint is developed to make the learned embedding with significant discrimination. Extensive experimental results on various benchmark datasets demonstrate our proposed DAQH framework achieves state-of-the-art visual similarity search performance.

FakePoI: A Large-Scale Fake Person of Interest Video Detection Benchmark and a Strong Baseline

Deepfake technique can synthesize realistic images, audios, and videos, facilitating the thriving of entertainment, education, healthcare, and other industries. However, its abuse may pose potential threats to personal privacy, social stability, and even national security. Therefore, the development of deepfake detection methods is attracting more and more attention. Existing works mainly focus on the detection of common videos for entertainment purposes. In contrast, fake videos maliciously synthesized for Person of Interest (PoI, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., who is in an authoritative position and has broadly public influences) are much more harmful to society because of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">celebrity endorsement</i> . However, there is no particular benchmark for driving related research in the community. Motivated by this observation, we present the first large-scale benchmark dataset, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> , to enable the research on fake PoI detection. It contains numerous fake videos of important people from all walks of life, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., police chiefs, city mayors, famous artists, and well-known Internet bloggers. In summary, our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> includes 11,092 synthesized videos where only a few clips rather than the entire are fake. Previous fake detection algorithms deteriorate heavily or even fail on our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> due to two main challenges. On the one hand, the rich diversity of our fake videos makes it pretty difficult to find universally applicable patterns for detection. On the other hand, the high credibility contributed by the presence of real frames easily confuses a common detector. To tackle these challenges, we present an amplifier framework, highlighting the feature gap between real and generated video frames. Specifically, we present a quadruplet loss to narrow the distance of all real PoIs and meanwhile push away each real and fake PoI in embedding space. We implement our framework and conduct extensive experiments on the proposed benchmark. The quantitative results demonstrate that our approach outperforms existing methods significantly, setting a strong baseline on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> . The qualitative analysis also shows its superiority. We will release our dataset and code at https://github.com/cslltian/deepfake-detection to encourage future research on this valuable area.

Boosting facial recognition capability for faces wearing masks using attention augmented residual model with quadruplet loss

Boosting facial recognition capability for faces wearing masks using attention augmented residual model with quadruplet loss

LIDN: A novel light invariant image dehazing network

With the increasing demand for efficient image dehazing solutions in computer vision algorithms, particularly for autonomous systems, a research focus on low-inference time yet high-performance dehazing techniques has emerged. Existing dehazing methods predominantly rely on either daytime or nighttime haze models, limiting their effectiveness in handling haziness under varying lighting conditions. To address this limitation, this research paper introduces the Light Invariant Dehazing Network (LIDN), an end-to-end image dehazing network consisting of four sub-modules: Feature Extractor, Deep Global Atmospheric Light Estimator, Medium Transmission Extractor, and Encoder-Decoder. The proposed model, trained using Quadruplet loss, effectively reduces artifacts and produces sharper dehazed images. Extensive experiments conducted under diverse lighting conditions demonstrate the superior performance of the proposed LIDN model compared to state-of-the-art daytime and nighttime dehazing approaches. Remarkably, the proposed model achieves these exceptional results with a runtime of only 0.24s per image, making it highly efficient than existing dehazing algorithms.

PCCT: Progressive Class-Center Triplet Loss for Imbalanced Medical Image Classification.

Imbalanced training data in medical image diagnosis is a significant challenge for diagnosing rare diseases. For this purpose, we propose a novel two-stage Progressive Class-Center Triplet (PCCT) framework to overcome the class imbalance issue. In the first stage, PCCT designs a class-balanced triplet loss to coarsely separate distributions of different classes. Triplets are sampled equally for each class at each training iteration, which alleviates the imbalanced data issue and lays solid foundation for the successive stage. In the second stage, PCCT further designs a class-center involved triplet strategy to enable a more compact distribution for each class. The positive and negative samples in each triplet are replaced by their corresponding class centers, which prompts compact class representations and benefits training stability. The idea of class-center involved loss can be extended to the pair-wise ranking loss and the quadruplet loss, which demonstrates the generalization of the proposed framework. Extensive experiments support that the PCCT framework works effectively for medical image classification with imbalanced training images. On four challenging class-imbalanced datasets (two skin datasets Skin7 and Skin 198, one chest X-ray dataset ChestXray-COVID, and one eye dataset Kaggle EyePACs), the proposed approach respectively obtains the mean F1 score 86.20, 65.20, 91.32, and 87.18 over all classes and 81.40, 63.87, 82.62, and 79.09 for rare classes, achieving state-of-the-art performance and outperforming the widely used methods for the class imbalance issue.

Quadruplet loss and SqueezeNets for Covid-19 detection from Chest-X ray

The Coronavirus Pandemic triggered by SARS-CoV-2 has wreaked havoc on the planet and is expanding exponentially. While scanning methods, including CT scans and chest X-rays, are commonly used, artificial intelligence implementations are also deployed for COVID-based pneumonia detection. Due to image biases in X-ray data, bilateral filtration and Histogram Equalization are used followed by lung segmentation by a U-Net, which successfully segmented 83.2\% of the collected dataset. The segmented lungs are fed into a Quadruplet Network with SqueezeNet encoders for increased computational efficiency and high-level embeddings generation. The embeddings are computed using a Multi-Layer Perceptron and visualized by T-SNE (T-Distributed Stochastic Neighbor Embedding) scatterplots. The proposed research results in a 94.6\% classifying accuracy which is 2\% more than the baseline Convolutional Neural Network and a 90.2\% decrease in prediction time.

Cross-Task Knowledge Distillation in Multi-Task Recommendation

Multi-task learning (MTL) has been widely used in recommender systems, wherein predicting each type of user feedback on items (e.g, click, purchase) are treated as individual tasks and jointly trained with a unified model. Our key observation is that the prediction results of each task may contain task-specific knowledge about user’s fine-grained preference towards items. While such knowledge could be transferred to benefit other tasks, it is being overlooked under the current MTL paradigm. This paper, instead, proposes a Cross-Task Knowledge Distillation framework that attempts to leverage prediction results of one task as supervised signals to teach another task. However, integrating MTL and KD in a proper manner is non-trivial due to several challenges including task conflicts, inconsistent magnitude and requirement of synchronous optimization. As countermeasures, we 1) introduce auxiliary tasks with quadruplet loss functions to capture cross-task fine-grained ranking information and avoid task conflicts, 2) design a calibrated distillation approach to align and distill knowledge from auxiliary tasks, and 3) propose a novel error correction mechanism to enable and facilitate synchronous training of teacher and student models. Comprehensive experiments are conducted to verify the effectiveness of our framework in real-world datasets.

Modeling Soil Temperature for Different Days Using Novel Quadruplet Loss-Guided LSTM.

Soil temperature (Ts), a key variable in geosciences study, has generated growing interest among researchers. There are many factors affecting the spatiotemporal variation of Ts, which poses immense challenges for the Ts estimation. To enrich processing information on loss function and achieve better performance in estimation, the paper designed a new long short-term memory model using quadruplet loss function as an intelligence tool for data processing (QL-LSTM). The model in this paper combined the traditional squared-error loss function with distance metric learning between the sample features. It can zoom analyze the samples accurately to optimize the estimation accuracy. We applied the meteorological data from Laegern and Fluehli stations at 5, 10, and 15 cm depth on the 1st, 5th, and 15th day separately to verify the performance of the proposed soil temperature estimation model. Meanwhile, this paper inputs the variables into the proposed model including radiation, air temperature, vapor pressure deficit, wind speed, air pressure, and past Ts data. The performance of the model was tested by several error evaluation indices, including root mean square error (RMSE), mean absolute error (MAE), Nash-Sutcliffe model efficiency coefficient (NS), Willmott Index of Agreement (WI), and Legates and McCabe index (LMI). As the test results at different soil depths show, our model generally outperformed the four existing advanced estimation models, namely, backpropagation neural networks, extreme learning machines, support vector regression, and LSTM. Furthermore, as experiments show, the proposed model achieved the best performance at the 15 cm depth of soil on the 1st day at Laegern station, which achieved higher WI (0.998), NS (0.995), and LMI (0.938) values, and got lower RMSE (0.312) and MAE (0.239) values. Consequently, the QL-LSTM model is recommended to estimate daily Ts profiles estimation on the 1st, 5th, and 15th days.

QuadNet: Quadruplet loss for multi-view learning in baggage re-identification

Recently, baggage re-identification (ReID) has become an attractive topic in computer vision because it plays an important role in intelligent surveillance. However, the wide variations in different views of baggage items degrade baggage ReID performance. In this paper, a novel QuadNet is proposed to solve the multi-view problem in baggage ReID at three levels. At the sample level, we propose a multi-view sampling strategy which samples hard examples from multiple identities in multiple views. The sampled baggage items are used to construct quadruplets. At the feature level, view-aware attentional local features are extracted from discriminative regions in each view. These local features are fused with global features to obtain better representations of the quadruplets. At the loss level, a multi-view quadruplet loss operating on the representations of quadruplets is proposed to reduce the intra-class distances caused by view variations and increase the inter-class distances of baggage images captured in the same view. A random local blur data augmentation is proposed to handle the motion blur which is often found in baggage images. The multi-task learning of materials is introduced to obtain discriminative features based on the materials of baggage surfaces. Extensive experiments on three ReID datasets, MVB, Market-1501 and VeRi-776, indicate the remarkable effectiveness and good generalization of the QuadNet model. It has achieved the state-of-the-art performance on the three datasets.

An efficient framework for zero-shot sketch-based image retrieval

Zero-shot sketch-based image retrieval (ZS-SBIR) has recently attracted the attention of the computer vision community due to its real-world applications, and the more realistic and challenging setting that it presents over SBIR. ZS-SBIR inherits the main challenges of multiple computer vision problems including content-based Image Retrieval (CBIR), zero-shot learning and domain adaptation. The majority of previous studies using deep neural networks have achieved improved results by either projecting sketch and images into a common low-dimensional space, or transferring knowledge from seen to unseen classes. However, those approaches are trained with complex frameworks composed of multiple deep convolutional neural networks (CNNs) and are dependent on category-level word labels. This increases the requirements for training resources and datasets. In comparison, we propose a simple and efficient framework that does not require high computational training resources, and learns the semantic embedding space from a vision model rather than a language model, as is done by related studies. Furthermore, at training and inference stages our method only uses a single CNN. In this work, a pre-trained ImageNet CNN (i.e., ResNet50) is fine-tuned with three proposed learning objects: domain-balanced quadruplet loss, semantic classification loss, and semantic knowledge preservation loss. The domain-balanced quadruplet and semantic classification losses are introduced to learn discriminative, semantic and domain invariant features by considering ZS-SBIR as an object detection and verification problem. To preserve semantic knowledge learned with ImageNet and exploit it for unseen categories, the semantic knowledge preservation loss is proposed. To reduce computational cost and increase the accuracy of the semantic knowledge distillation process, ground-truth semantic knowledge is prepared in a class-oriented fashion prior to training. Extensive experiments are conducted on three challenging ZS-SBIR datasets: Sketchy Extended, TU-Berlin Extended and QuickDraw Extended. The proposed method achieves state-of-the-art results, and outperforms the majority of related works by a substantial margin.

Acoustic Word Embedding Based on Multi-Head Attention Quadruplet Network

Acoustic word embedding (AWE) has become a mainstream method in low-resource Query-by-Example keywords search. This paper proposes an AWE based on a multi-head attention quadruplet network, which can learn the attention weight sequence for all time frames of bidirectional Long Short-Term Memory by a multi-head self-attentive mechanism to pay attention to the time position information. At the same time, we construct a differences order quadruplet loss to train the AWE model to adequately consider the relative and absolute distances between the positive and negative sample pairs. In addition, attention mechanism, differences order quadruplet loss, and word label information are combined to design an objective function so that the AWE vectors have a better feature expression in the embedded space. The experimental results show that the proposed method can improve the learning ability of the network and make the AWEs more identifiable. The above two points result in better performance in the word discrimination task.

A Quadruplet Deep Metric Learning model for imbalanced time-series fault diagnosis

Deep learning based methods are attractive and meaningful in the field of fault diagnosis recently. However, the sample size of different faults may be imbalanced in practical application scenarios, which results in performance degeneration. The optimization of data representation could be an effective way to alleviate such phenomenon. In this paper, a deep learning framework considering time series and distance metric called LSTM-Quadruplet Deep Metric Learning(LSTM-QDM) model is proposed. It maps original space to a feature space where the distribution of imbalance faults is more distinguishable. A novel quadruplet data pair is designed which adds a minor sample from imbalanced classes into traditional data pair. Based on such data pair, a quadruplet loss function is proposed to increase the distance between the imbalanced classes and other classes, and its combination with softmax loss would improve the representation ability and classification performance simultaneously. The proposed data pair and loss function encourage the full mining of imbalanced data in the training process. The experiments are carried out on two open-source datasets including TE and CWRU, and the fault diagnosis performance is validated under different imbalanced conditions. The experimental results indicate that our proposed model is effective and robust in the imbalanced fault diagnosis task.

Multi‐label based view learning for vehicle re‐identification

Due to the high similarity of different vehicles with similar appearances and the great diversity of camera viewpoints, vehicle re-identification (ReID) is still a challenging task. It commonly maps query set image into a high dimensional embedding space and then retrieve gallery vehicle images according to the distance. In this letter, a multi-label based view learning (MLVL) model to enhance the distinguishability of intra-class and inter-class through multiple labels learning, and decrease the distance of intra-class features caused by multi-view appearances of vehicles. Specifically, the model includes two main parts. First, the vehicle orientation estimation module is responsible for detecting key-points of the vehicle and predicting its orientation, which provides the orientation label for the multi-label based learning network to enhance view-invariant representation. Second, a novel dual-triplet and quadruplet loss function is designed to optimise intra-class and inter-class distance with the help of multi-label information, i.e. vehicle colour, type and orientation. In particular, a multi-label based sampler is proposed to generate training mini-batches instead of random sampler. Extensive experiments results show that the proposed MLVL model achieves 5.6 % $\%$ mAP improvement on VeRi-776 and 1.9 % $\%$ on VERI-Wild datasets compared with baseline model.

Hidden multi-distance loss-based full-convolution hashing

To improve retrieval efficiency and quality, learning to hash has been widely used in approximate nearest neighbor queries. Deep learning is characterized by high precision in extracting data features; therefore, deep-learning-based hashing has attracted more attention. Existing methods have some weaknesses, such as complex training and losing spatial information. We design a new deep hashing algorithm named HLFH, which is very simple technique but achieves amazingly good performance. HLFH is optimized and improved in two aspects: network structure and hashing loss. Concerning network structure, a new full convolutional hashing network is proposed to preserve spatial information of features. A smooth activation function is used in the hashing layer to reduce the quantization error. Concerning hashing loss, the semantic information of data is then used to generate binary codes by hidden multi-distance loss, i.e., combination of triplet loss and quadruplet loss. With these two new techniques, our method is more accurate than many other state-of-the-art methods.

A Robust Quadruplet and Faster Region-Based CNN for UAV Video-Based Multiple Object Tracking in Crowded Environment

Multiple object tracking (MOT) from unmanned aerial vehicle (UAV) videos has faced several challenges such as motion capture and appearance, clustering, object variation, high altitudes, and abrupt motion. Consequently, the volume of objects captured by the UAV is usually quite small, and the target object appearance information is not always reliable. To solve these issues, a new technique is presented to track objects based on a deep learning technique that attains state-of-the-art performance on standard datasets, such as Stanford Drone and Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking (UAVDT) datasets. The proposed faster RCNN (region-based convolutional neural network) framework was enhanced by integrating a series of activities, including the proper calibration of key parameters, multi-scale training, hard negative mining, and feature collection to improve the region-based CNN baseline. Furthermore, a deep quadruplet network (DQN) was applied to track the movement of the captured objects from the crowded environment, and it was modelled to utilize new quadruplet loss function in order to study the feature space. A deep 6 Rectified linear units (ReLU) convolution was used in the faster RCNN to mine spatial–spectral features. The experimental results on the standard datasets demonstrated a high performance accuracy. Thus, the proposed method can be used to detect multiple objects and track their trajectories with a high accuracy.

Quadruplet-Based Deep Cross-Modal Hashing.

Recently, benefitting from the storage and retrieval efficiency of hashing and the powerful discriminative feature extraction capability of deep neural networks, deep cross-modal hashing retrieval has drawn more and more attention. To preserve the semantic similarities of cross-modal instances during the hash mapping procedure, most existing deep cross-modal hashing methods usually learn deep hashing networks with a pairwise loss or a triplet loss. However, these methods may not fully explore the similarity relation across modalities. To solve this problem, in this paper, we introduce a quadruplet loss into deep cross-modal hashing and propose a quadruplet-based deep cross-modal hashing (termed QDCMH) method. Extensive experiments on two benchmark cross-modal retrieval datasets show that our proposed method achieves state-of-the-art performance and demonstrate the efficiency of the quadruplet loss in cross-modal hashing.

Video fingerprinting based on quadruplet convolutional neural network

In order to achieve fast and accurate retrieval of video copies, this paper proposes a compact video fingerprinting based on quadruplet convolutional neural network. The algorithm consists of four branch networks with shared weights, each branch network contains feature extraction and quantization coding. The projection and excitation network is combined with 3D convolution for feature extraction, which mainly learns feature weights to improve useful features and suppresses valueless features. The deep features learned are mapped into approximate real vectors in a fully connected form and quantized to generate binary codes. The model employs an improved quadruplet loss to divide the feature distance between the copied videos and the non-copied, and a quantization error term is added to ensure that the fingerprint codes contains as much similar information as the original data. The experimental results performed on the public dataset show that the algorithm can effectively improve the robustness and distinctiveness, and the average detection accuracy under multiple compound attacks is better than the compared algorithms.

A Quadruplet Loss for Enforcing Semantically Coherent Embeddings in Multi-Output Classification Problems

This article describes one objective function for learning semantically coherent feature embeddings in multi-output classification problems, i.e., when the response variables have dimension higher than one. Such coherent embeddings can be used simultaneously for different tasks, such as identity retrieval and soft biometrics labelling. We propose a generalization of the triplet loss that: 1) defines a metric that considers the number of agreeing labels between pairs of elements; 2) introduces the concept of similar classes, according to the values provided by the metric; and 3) disregards the notion of anchor, sampling four arbitrary elements at each time, from where two pairs are defined. The distances between elements in each pair are imposed according to their semantic similarity (i.e., the number of agreeing labels). Likewise the triplet loss, our proposal also privileges small distances between positive pairs. However, the key novelty is to additionally enforce that the distance between elements of any other pair corresponds inversely to their semantic similarity. The proposed loss yields embeddings with a strong correspondence between the classes centroids and their semantic descriptions. In practice, it is a natural choice to jointly infer coarse (soft biometrics) + fine (ID) labels, using simple rules such as k-neighbours. Also, in opposition to its triplet counterpart, the proposed loss appears to be agnostic with regard to demanding criteria for mining learning instances (such as the semi-hard pairs). Our experiments were carried out in five different datasets (BIODI, LFW, IJB-A, Megaface and PETA) and validate our assumptions, showing results that are comparable to the state-of-the-art in both the identity retrieval and soft biometrics labelling tasks.

QuadrupletNet: A Novel Local Descriptor for Brain Tumor Detection and Segmentation

Brain tumor detection and segmentation from Magnetic Resonance Imaging (MRI) images is being one of the emerging fields in the biomedicine. A formidable undertaking in brain tumor surgery, medical care, treatment programme and quantitative assessment of MRI images is to precisely diagnose its location and extent. Recently, the convolutional neural network (CNN) based detection and segmentation method on brain tumor MRI images is being one of the emerging fields in the medical imaging as an automatic clinic treatment and evaluation solution. In this article, we put forward a brand new quadruplet loss in CNN framework, which achieves higher accuracy in brain tumor detection and segmentation than other pairwise loss and triplet loss methods. By applying the proposed quadruplet loss to the original L2Net CNN architecture leads to a more compact descriptor named QuadrupletNet. From our experiments, QuadrupletNet shows higher performance than other state-of-the-art loss functions e.g., the Triplet loss, as indicated in experiments on Multimodal Brain Tumor Image Segmentation (BRATS 2018) datasets, and on our own collected MRI brain tumor datasets (named MBTD).

Cross-modal dual subspace learning with adversarial network

Cross-modal retrieval has recently attracted much interest along with the rapid development of multimodal data, and effectively utilizing the complementary relationship of different modal data and eliminating the heterogeneous gap as much as possible are the two key challenges. In this paper, we present a novel network model termed cross-modal Dual Subspace learning with Adversarial Network (DSAN). The main contributions are as follows: (1) Dual subspaces (visual subspace and textual subspace) are proposed, which can better mine the underlying structure information of different modalities as well as modality-specific information. (2) An improved quadruplet loss is proposed, which takes into account the relative distance and absolute distance between positive and negative samples, together with the introduction of the idea of hard sample mining. (3) Intra-modal constrained loss is proposed to maximize the distance of the most similar cross-modal negative samples and their corresponding cross-modal positive samples. In particular, feature preserving and modality classification act as two antagonists. DSAN tries to narrow the heterogeneous gap between different modalities, and distinguish the original modality of random samples in dual subspaces. Comprehensive experimental results demonstrate that, DSAN significantly outperforms 9 state-of-the-art methods on four cross-modal datasets.