Hard Triplet Research Articles

RTHEN: Unsupervised deep homography estimation based on dynamic attention for repetitive texture image stitching

Homography estimation is regarded as one of the key challenges in image alignment, where the goal is to estimate the projective transformation between two images on the same plane. Unsupervised learning methods are gradually becoming popular due to their excellent performance and lack of need for labeled data. However, in regional scenes with repeated textures, there may be ambiguity in the correspondence between local features, affecting homography estimation accuracy. This paper proposes a new unsupervised deep homography method RTHEN to solve such problems. In order to effectively obtain repeated texture features, a multi-scale Feature pyramid Siamese network (FPSN) is designed. Specifically, we dynamically allocate the weights of recited texture features through a dynamic attention module and introduce a channel attention module to provide rich contextual information for repeated texture areas. We propose a hard triplet loss function based on overlap constraints to optimize the matching results. At the same time, we collected a repetitive texture image dataset (RTID) for homography estimation training and evaluation. Experimental results show that our method outperforms existing learning methods in repetitive texture scenes and offers competitive performance with state-of-the-art traditional methods.

Pseudo labelling workflow, margin losses, hard triplet mining, and PENViT backbone for explainable age and biological gender estimation using dental panoramic radiographs

This study aimed to estimate human age and gender from panoramic radiographs using various deep learning techniques while using explainability to have a novel hybrid unsupervised model explain the decision-making process. The classification task involved training neural networks and vision transformers on 706 panoramic radiographs using different loss functions and backbone architectures namely ArcFace, a triplet network named TriplePENViT, and the subsequently developed model called PENViT. Pseudo labeling techniques were applied to train the models using unlabeled data. FullGrad Explainable AI was used to gain insights into the decision-making process of the developed PENViT model. The ViT Large 32 model achieved a validation accuracy of 68.21% without ArcFace, demonstrating its effectiveness in the classification task. The PENViT model outperformed other backbones, achieving the same validation accuracy without ArcFace and an improved accuracy of 70.54% with ArcFace. The TriplePENViT model achieved a validation accuracy of 67.44% using hard triplet mining techniques. Pseudo labeling techniques yielded poor performance, with a validation accuracy of 64.34%. Validation accuracy without ArcFace was established at 67.44% for Age and 84.49% for gender. The unsupervised model considered developing tooth buds, tooth proximity and mandibular shape for estimating age within deciduous and mixed dentitions. For ages 20–29, it factored permanent dentition, alveolar bone density, root apices, and third molars. Above 30, it notes occlusal deformity resulting from missing dentition and the temporomandibular joint complex as predictors for age estimation from panoramic radiographs.Graphical abstract

TFUN: Trilinear Fusion Network for Ternary Image-Text Retrieval

Recently, a particular type of image-text retrieval task named Ternary Image-Text Retrieval (TITR) has drawn increasing attention. In this task, the total inputs of query and target consist of three components, rather than two inputs in the widely-studied retrieval case. The typical TITR scenarios include recipe retrieval (e.g., ingredients text, instructions text and food images) and fashion search (e.g., original images, text and modified images). A few recently proposed TITR methods mainly focus on learning the semantic correlations of two modality data by projecting them to the same embedding space to capture the alignment between image and text modalities. Nevertheless, two limitations still exist in these methods: 1) the underlying difference between data in the same modality (e.g., ingredients and instructions) is neglected; and 2) the trilinear interaction among the three inputs is implicitly captured. To this end, we propose a novel fusion framework named Trilinear FUsion Network (TFUN) to utilize high-level associations between these three inputs simultaneously and learn an accurate cross-modal similarity function via bi-directional triplet loss explicitly, which is generic for the TITR task. To reduce the model complexity, we introduce the advanced method of tensor decomposition to ensure computational efficiency and accessibility. We also develop a three-stage hard triplet sampling scheme to ensure fast convergence. Extensive experiments on three large-scale TITR datasets Recipe1M, Fashion200k and FashionIQ demonstrate the superiority of our proposed TFUN model compared to the state-of-the-art approaches. The implementation code and additional instructions are provided at https://github.com/CFM-MSG/Code_TFUN.

Weighted triplet loss based on deep neural networks for loop closure detection in VSLAM

Smart living is an emerging technology that has attracted a lot of attention all around the world. As a key technology of smart space, which is the principal part of smart living, the SLAM system has effectively expanded the ability of space intelligent robots to explore unknown environments. Loop closure detection is an important part of SLAM system and plays a very important role in eliminating cumulative errors. The SLAM system without loop closure detection is degraded to an odometer. The state estimation solely relying on an odometer will be seriously deviated in the long-term and large-scale navigation and positioning. This paper proposes a metric learning method that uses deep neural networks for loop closure detection based on triplet loss. The map points obtained by metric learning are fused with all map points in the current keyframe, and the map points that do not meet the filtering conditions are eliminated. Based on the Batch Hard Triplet loss, the weighted triplet loss function avoids suboptimal convergence in the learning process by applying weighted value constraints. At the same time, considering that fixed boundary parameters cannot be well adapted to the diversity of scales between different samples, we use the semantic similarity of anchor samples and negative samples to redefine boundary parameters. Finally, a SLAM system based on metric learning is constructed, and the SLAM dataset TUM and KITTI are used to evaluate the proposed model’s accuracy rate and recall rate. The scene features in this method are extracted automatically through neural networks instead of being artificially set. Finally, a high-precision closed-loop detection method based on weight adaptive triple loss is effectively realised through the closed-loop detection experiment. The minimum relative pose error is 0.00048 m, which is 15.8% less than that of the closed-loop detection algorithm based on the word bag model.

Class-Balanced Contrastive Learning for Fine-Grained Airplane Detection

Airplane detection and fine-grained recognition in remote sensing images are challenging due to class imbalance and high inter-class indistinction. To alleviate these issues, we propose class-balanced contrastive learning (CBCL) approach for airplane detection to exploit the correlation between samples in different images, which is rarely explored in previous research. Specifically, we first dynamically build class-balanced memory queues during training, which mitigates class imbalance by memorizing training samples. Upon class-balanced memory queues, hard triplet contrastive learning is introduced to increase the inter-class discriminability, which enforces the maximum distance of the positive sample pair to be smaller than the minimum distance of the negative sample pair. We integrate the proposed CBCL strategy into oriented object detection frameworks for fine-grained airplane detection. The experimental results on the FAIR1M dataset reveal that several state-of-the-art algorithms with CBCL achieve significantly improvements.

A Novel Approach Based on Multi-Level Bottleneck Attention Modules Using Self-Guided Dropblock for Person Re-Identification

Person re-identification has inspired a lot of interest due to its significance in intelligent video surveillance. It is a difficult task due to the presence of critical challenges such as changes in appearance, misalignment, occlusion and background noise. Batch drop block layer (BDB) has been used recently in person re-identification by exploiting the feature erasing procedure. However, BDB drops a block of features randomly, resulting in the loss of contextual information, which makes the model difficult to train. Also, due to the random dropping of features, large area of discriminative information may lose during training, resulting in low efficiency and performance. To address this problem and to improve the model representation power, we propose a novel, lightweight, self-adaptive bottleneck attention module with a self-attention branch to improve the model performance by reducing the parameter overhead with negligible computation cost. The proposed approach entails bottleneck attention module (BAM) which is incorporated between ResNet layers to remove the background noise and to nominate the high-level semantic part. Further, dilated convolutions with batch normalization are used to tackle the contextual information loss problem and to avoid overfitting. In addition, an informative global branch is used which captures the global representation of the network, and the attention branch entails the multiscale local salient information. Two types of loss functions including softmax and batch hard triplet are used in the training process for each branch, forcing the network to encapsulate the common attribute within the similar identity and to maintain distance between distinct individuals. Compared with BDB, our network improves the mAp to 88.1%, and Rank-1 gets 96.3% for the market-1501 dataset. The results on Cuhk-03-Detected dataset showed 79.2% mAp score, with 81.4 %, Rank-1, whereas on Cuhk-03-labelled dataset, a mAP score of 81.3% and a Rank-1 score of 83.3% is achieved. Experiments reveal that ResNet model with addition of multiple BAM layers performs consistently over the state-of-the-art datasets using softmax and batch hard triplet loss.

Hybrid network with difference degree and attention mechanism combined with radiomics (H-DARnet) for MVI prediction in HCC

MVI is a risk assessment factor related to hepatocellular carcinoma (HCC) recurrence after hepatectomy or liver transplantation. The goal of this paper is to study the preoperative diagnosis of microvascular invasion (MVI) by using a deep learning algorithm in non-contrast T2 weighted magnetic resonance imaging (MRI) images instead of pathological images. Herein, an ensemble learning algorithm named H-DARnet—based on the difference degree and attention mechanism, combined with radiomics, for MVI prediction—is proposed. Our hybrid network combines the fine-grained, high-level semantic, and radiomics features and exhibits a rich multilevel-feature architecture composed of global-local-prior knowledge with suitable complementarity. The total loss function comprises two regularization items––the triplet and the cross-entropy loss function––which are selected for the triplet network and SE-DenseNet, respectively. The hard triplet sample selection strategy for a triplet network and data augmentation for small-scale liver image datasets in convolutional neural network (CNN) training is indispensable. For 200 patch level test samples (135 positive samples and 65 negative samples), our method can obtain the best prediction results, the AUC, sensitivity, and specificity were 0.826, 79.5% and 73.8%, respectively. The experiment results show that MVI can be predicted by using MRI images, and the proposed method is better than other deep learning algorithms and hand-crafted feature algorithms. The proposed ensemble learning algorithm is proved to be an effective method for MVI prediction.

Baggage Image Retrieval with Attention-Based Network for Security Checks

Baggage image search is the task of finding the same baggage across several cameras, which can improve the efficiency of security check. Since existing state-of-the-art image retrieval requires a significant amount of training data, we aim to investigate the few-shot situation, utilizing only a few samples for each baggage. In this paper, the framework we introduced is called AttentionNet, exploring the weak region-wise annotations as attention clues to improve the retrieval performance. Specifically, we integrate the semantic and recognition tasks using shared convolutional neural networks by multi-task learning. Semantic attention mechanism is used to explicitly re-weight the feature map for emphasizing the foreground. To prevent overfitting in few-shot training, we adopt a variant of the triplet loss to perform deep metric learning with an online hard triplet mining strategy. Once trained, AttentionNet identifies a probe image by computing its embedding’s cosine distance with images in the gallery. Experiments show that our method achieves state-of-the-art accuracy in image search. For instance, we obtain 82.9% Rank-1 score on the MVB dataset, dramatically outperforming the currently existing methods.

Biclustering Collaborative Learning for Cross-Domain Person Re-Identification

In the cross-domain person re-identification (ReID) method based on clustering, the performance of the model depends heavily on the quality of the information it obtains from clustering. To improve the reliability of the clustering information obtained by the model, we propose a biclustering collaborative learning (BCL) framework derived from an identity disentanglement adaptation network (IDA-Net). IDA-Net encodes the identity and style of the input image and transfers the style on the premise of maintaining identity consistency. By comparing the clustering results obtained on the same dataset before and after the transfer process, BCL can select hard samples with higher confidence for model optimization. In each iteration, we design a conditional batch hard triplet loss to optimize the two networks. Extensive experiments on large-scale datasets (Maket1501, DukeMTMC-reID and MSMT17) demonstrate the superior performance of BCL over the state-of-the-art methods.

Exhaustive hard triplet mining loss for Person Re-Identification

Exhaustive hard triplet mining loss for Person Re-Identification

Instance Hard Triplet Loss for In-video Person Re-identification

Traditional Person Re-identification (ReID) methods mainly focus on cross-camera scenarios, while identifying a person in the same video/camera from adjacent subsequent frames is also an important question, for example, in human tracking and pose tracking. We try to address this unexplored in-video ReID problem with a new large-scale video-based ReID dataset called PoseTrack-ReID with full images available and a new network structure called ReID-Head, which can extract multi-person features efficiently in real time and can be integrated with both one-stage and two-stage human or pose detectors. A new loss function is also required to solve this new in-video problem. Hence, a triplet-based loss function with an online hard example mining designed to distinguish persons in the same video/group is proposed, called instance hard triplet loss, which can be applied in both cross-camera ReID and in-video ReID. Compared with the widely-used batch hard triplet loss, our proposed loss achieves competitive performance and saves more than 30% of the training time. We also propose an automatic reciprocal identity association method, so we can train our model in an unsupervised way, which further extends the potential applications of in-video ReID. The PoseTrack-ReID dataset and code will be publicly released.

Training approach using the shallow model and hard triplet mining for person re‐identification

Multi-target tracking in a non-overlapping camera network is an active research field, and one of the important problems in it is the person re-identification problem. In this study, the authors propose an approach to improve the performance of the backbone model in the person re-identification. Their approach focuses on training a fusion model with a shallow model and making hard triplets with relationship matrices quickly and efficiently. The proposed approach is simple, but it improves the performance of the backbone. In addition, the hard triplet mining in their process is much faster than the conventional approach. Experimental evaluation shows that the proposed approach can improve the performances of the backbone model. The proposed approach improves rank-1 and mean average precision (mAP) performance by more than 12.54 and 15.44%, respectively, over the backbone models in the Market1501 and DukeMTMC-reID dataset. The approach also achieves competitive performances compared with state-of-the-art approaches.

Person Re-Identification Based on Heterogeneous Part-Based Deep Network in Camera Networks

In this paper, we propose a new deep learning model named heterogeneous part-based deep network for person re-identification in camera networks, which simultaneously learns the alignment and discrimination for parts of pedestrian images. Concretely, several parts are obtained through the uniform partition on the convolutional layer for each pedestrian image. Then, we present part-aligned distances to perform alignment by searching the shortest local distances between image parts in a certain range. Meanwhile, we utilize the batch hard triplet loss and cross-entropy loss to learn more discriminative part-based features in different aspects. Experiments are conducted on three challenging datasets, Market-1501, CUHK03, and DukeMTMC-reID, and we achieve 94.0%, 64.3%, and 83.6% rank-1 accuracy and 81.2%, 58.2%, and 68.0% mAP, outperforming the state-of-the-art methods by a large margin.

Isosceles Constraints for Person Re-identification.

In the existing works of person re-identification (ReID), batch hard triplet loss has achieved great success. However, it only cares about the hardest samples within the batch. For any probe, there are massive mismatched samples (crucial samples) outside the batch which are closer than the matched samples. To reduce the disruptive influence of crucial samples, we propose a novel isosceles contraint for triplet. Theoretically, we show that if a matched pair has equal distance to any one of mismatched sample, the matched pair should be infinitely close. Motivated by this, the isosceles constraint is designed for the two mismatched pairs of each triplet, to restrict some matched pairs with equal distance to different mismatched samples. Meanwhile, to ensure that the distance of mismatched pairs are larger than the matched pairs, margin constraints are necessary. Minimizing the isosceles and margin constraints with respect to the feature extraction network makes the matched pairs closer and the mismatched pairs farther away than the matched ones. By this way, crucial samples are effectively reduced and the performance on ReID is improved greatly. Likewise, our isosceles contraint can be applied to quadruplet as well. Comprehensive experimental evaluations on Market-1501, DukeMTMC-reID and CUHK03 datasets demonstrate the advantages of our isosceles constraint over the related state-of-the-art approaches.

Rich Features Embedding for Cross-Modal Retrieval: A Simple Baseline

During the past few years, significant progress has been made on cross-modal retrieval, benefiting from the development of deep neural networks. Meanwhile, the overall frameworks are becoming more and more complex, making the training as well as the analysis more difficult. In this paper, we provide a Rich Features Embedding (RFE) approach to tackle the cross-modal retrieval tasks in a simple yet effective way. RFE proposes to construct rich representations for both images and texts, which is further leveraged to learn the rich features embedding in the common space according to a simple hard triplet loss. Without any bells and whistles in constructing complex components, the proposed RFE is concise and easy to implement. More importantly, our RFE obtains the state-of-the-art results on several popular benchmarks such as MS COCO and Flickr 30 K. In particular, the image-to-text and text-to-image retrieval achieve 76.1% and 61.1% (R@1) on MS COCO, which outperform others more than 3.4% and 2.3%, respectively. We hope our RFE will serve as a solid baseline and help ease future research in cross-modal retrieval.

Semi-supervised center-based discriminative adversarial learning for cross-domain scene-level land-cover classification of aerial images

Supervised scene classification of aerial images is of great importance in land-cover classification. However, annotating the labeled data required for the conventional classifiers and convolutional neural networks (CNNs), costs much manpower and time. Domain adaptation methods can overcome this problem, to some extent, by transferring previously labeled data to the new images, but the classification models trained from the previously labeled data are not discriminative enough for classifying aerial images from other domains because of the data distribution differences caused by the variations in sensors, natural environments, seasons, angles, locations, and so on. In order to solve this problem, we propose a semi-supervised center-based discriminative adversarial learning (SCDAL) framework integrating three parts, namely filtering out easy triplets, proposed hard triplet loss, and the adversarial learning with center loss. In the SCDAL framework, a difficulty measure is proposed to remove easy triplets under the constraint of between-class dissimilarity and intra-class similarity and better distinguish hard triplets. The filtered triplets are then used to train a more discriminative source feature extractor with the proposed hard triplet loss combining the hardest triplet loss and semi-hard triplet loss. Adversarial learning with center loss is also proposed to reduce the feature distribution bias between the source and target feature extractors and increase the discriminative ability of the target feature extractor. The SCDAL framework is tested on two large aerial images as a case study. The experimental results demonstrate that when adequate previously labeled data but limited labeled target data exist, the SCDAL framework is superior to most of the existing domain adaptation methods, with an improvement of at least 3% in overall accuracy. It is also proved that removing easy triplets, proposed hard triplet loss, and the adversarial learning with center loss all help to improve the overall accuracy.

Transfer deep feature learning for face sketch recognition

Sketch-to-photo recognition is an important challenge in face recognition because it requires matching face images in different domains. Even the deep learning, which has recently been deployed in face recognition, is not efficient for face sketch recognition due to the limited sketch datasets. In this paper, we propose a novel face sketch recognition approach based on transfer learning. We design a three-channel convolutional neural network architecture in which the triplet loss is adopted in order to learn discriminative features and reduce intra-class variations. Moreover, we propose a hard triplet sample selection strategy to augment the number of training samples and avoid slow convergence. With the proposed method, facial features from digital photos and from sketches taken from the same person are closer; the opposite occurs if the digital photo and sketch are from different identities. Experimental results on multiple public datasets indicate that the proposed face sketch recognition method outperforms the existing approaches.

A joint loss function for deep face recognition

Convolutional neural networks (CNNs) have been widely used in computer vision community, and significantly improving the state-of-the-art. How to train an intra-class variant and inter-class discriminative feature is a central topic in face recognition. This paper proposes to learn an effective feature from face images by a joint loss function which combines the hard sample triplet (HST) and the absolute constraint triplet (ACT) loss, under the criteria that a maximum intra-class distance should be smaller than any inter-class distance. With the joint supervision of HST and ACT loss, CNNs is enable to learn discriminative features to improve face recognition performance. Experiments on labeled faces in the wild, IARPA Janus Benchmark (IJB-A) and YouTube Faces datasets achieve a comparable or superior performance to the state-of-the-arts.

The Three-Body Direct Correlation Function of Hard Sphere and Hard Ellipsoid Fluids

In previous work, the direct correlation functions of hard spheres and hard ellipsoidal fluids were calculated. In the present work, using Marko variational method and Pynn–Wulf approximation, the three-body direct correlation function of hard ellipsoid fluid is calculated. In these calculations, the triplet direct correlation function of hard spheres is required. The new expressions for the direct correlation function of hard spheres and hard ellipsoids and triplet correlation of hard spheres, introduced by Muller and coworker, are used. The Fourier transform of new triplet direct correlation of hard sphere fluid is compared with the simulation data. As mentioned in previous work, the expansion coefficients of two-body correlations of hard ellipsoids (components of three-body correlations) are fairly in agreement with the Monte Carlo simulation and the other theories. The Fourier transform of new triplet direct correlation of hard spheres and hard ellipsoid fluids is calculated. These Fourier transforms can be used for calculation of triplet structure factor of hard spheres and hard ellipsoid fluids. The Fourier transforms of triplet correlation of hard ellipsoids with elongations $$x = 1.50,\,\,1.20,\,\,1.10$$ as limited cases are compared with exact result of hard spheres of unit diameter. There is qualitative agreement between hard ellipsoids and hard spheres results of triplet correlations.

Triplet supertree heuristics for the tree of life

BackgroundThere is much interest in developing fast and accurate supertree methods to infer the tree of life. Supertree methods combine smaller input trees with overlapping sets of taxa to make a comprehensive phylogenetic tree that contains all of the taxa in the input trees. The intrinsically hard triplet supertree problem takes a collection of input species trees and seeks a species tree (supertree) that maximizes the number of triplet subtrees that it shares with the input trees. However, the utility of this supertree problem has been limited by a lack of efficient and effective heuristics.ResultsWe introduce fast hill-climbing heuristics for the triplet supertree problem that perform a step-wise search of the tree space, where each step is guided by an exact solution to an instance of a local search problem. To realize time efficient heuristics we designed the first nontrivial algorithms for two standard search problems, which greatly improve on the time complexity to the best known (naïve) solutions by a factor of n and n2 (the number of taxa in the supertree). These algorithms enable large-scale supertree analyses based on the triplet supertree problem that were previously not possible. We implemented hill-climbing heuristics that are based on our new algorithms, and in analyses of two published supertree data sets, we demonstrate that our new heuristics outperform other standard supertree methods in maximizing the number of triplets shared with the input trees.ConclusionWith our new heuristics, the triplet supertree problem is now computationally more tractable for large-scale supertree analyses, and it provides a potentially more accurate alternative to existing supertree methods.