Triplet Constraints Research Articles

Semi-supervised metric learning incorporating weighted triplet constraint and Riemannian manifold optimization for classification

Semi-supervised metric learning incorporating weighted triplet constraint and Riemannian manifold optimization for classification

ScTPC: a novel semisupervised deep clustering model for scRNA-seq data.

Continuous advancements in single-cell RNA sequencing (scRNA-seq) technology have enabled researchers to further explore the study of cell heterogeneity, trajectory inference, identification of rare cell types, and neurology. Accurate scRNA-seq data clustering is crucial in single-cell sequencing data analysis. However, the high dimensionality, sparsity, and presence of "false" zero values in the data can pose challenges to clustering. Furthermore, current unsupervised clustering algorithms have not effectively leveraged prior biological knowledge, making cell clustering even more challenging. This study investigates a semisupervised clustering model called scTPC, which integrates the triplet constraint, pairwise constraint, and cross-entropy constraint based on deep learning. Specifically, the model begins by pretraining a denoising autoencoder based on a zero-inflated negative binomial distribution. Deep clustering is then performed in the learned latent feature space using triplet constraints and pairwise constraints generated from partial labeled cells. Finally, to address imbalanced cell-type datasets, a weighted cross-entropy loss is introduced to optimize the model. A series of experimental results on 10 real scRNA-seq datasets and five simulated datasets demonstrate that scTPC achieves accurate clustering with a well-designed framework. scTPC is a Python-based algorithm, and the code is available from https://github.com/LF-Yang/Code or https://zenodo.org/records/10951780.

Toward Certified Robustness of Distance Metric Learning.

Metric learning aims to learn a distance metric such that semantically similar instances are pulled together while dissimilar instances are pushed away. Many existing methods consider maximizing or at least constraining a distance margin in the feature space that separates similar and dissimilar pairs of instances to guarantee their generalization ability. In this article, we advocate imposing an adversarial margin in the input space so as to improve the generalization and robustness of metric learning algorithms. We first show that the adversarial margin, defined as the distance between training instances and their closest adversarial examples in the input space, takes account of both the distance margin in the feature space and the correlation between the metric and triplet constraints. Next, to enhance robustness to instance perturbation, we propose to enlarge the adversarial margin through minimizing a derived novel loss function termed the perturbation loss. The proposed loss can be viewed as a data-dependent regularizer and easily plugged into any existing metric learning methods. Finally, we show that the enlarged margin is beneficial to the generalization ability by using the theoretical technique of algorithmic robustness. Experimental results on 16 datasets demonstrate the superiority of the proposed method over existing state-of-the-art methods in both discrimination accuracy and robustness against possible noise.

Triplet-constrained deep hashing for chest X-ray image retrieval in COVID-19 assessment

Radiology images of the chest, such as computer tomography scans and X-rays, have been prominently used in computer-aided COVID-19 analysis. Learning-based radiology image retrieval has attracted increasing attention recently, which generally involves image feature extraction and finding matches in extensive image databases based on query images. Many deep hashing methods have been developed for chest radiology image search due to the high efficiency of retrieval using hash codes. However, they often overlook the complex triple associations between images; that is, images belonging to the same category tend to share similar characteristics and vice versa. To this end, we develop a triplet-constrained deep hashing (TCDH) framework for chest radiology image retrieval to facilitate automated analysis of COVID-19. The TCDH consists of two phases, including (a) feature extraction and (b) image retrieval. For feature extraction, we have introduced a triplet constraint and an image reconstruction task to enhance discriminative ability of learned features, and these features are then converted into binary hash codes to capture semantic information. Specifically, the triplet constraint is designed to pull closer samples within the same category and push apart samples from different categories. Additionally, an auxiliary image reconstruction task is employed during feature extraction to help effectively capture anatomical structures of images. For image retrieval, we utilize learned hash codes to conduct searches for medical images. Extensive experiments on 30,386 chest X-ray images demonstrate the superiority of the proposed method over several state-of-the-art approaches in automated image search. The code is now available online.

Semi-supervised cross-modal retrieval with graph-based semantic alignment network

Semi-supervised cross-modal retrieval is an eclectic paradigm which learns common representations via exploiting underlying semantic information from both labeled and unlabeled data. Most existing methods ignore the rich semantic information of text data and are unable to fully utilize the text data in common representation learning. Moreover, they only considered the correlation of the data with the same semantic label, but ignored the correlation between the data with different semantic label. In this paper, we propose a novel semi-supervised cross-modal retrieval method, called Graph-based Semantic Alignment Network (GSAN), which learns common representation by aligning the features of different modalities with semantic embeddings of text data. Firstly, we design a Deep Supervised Semantic Encoding (DSSE) module to train the semantic projector and label predictor which can exploit the semantic embeddings and the predicted labels from unlabeled data of text modality. Then, GAN-based Bidirectional Fusion (GBF) module is designed to learn the mapping networks of two modalities (image and text). In order to make the mapping networks generate semantically discriminative and modality-invariant representations, we utilize the underlying semantic information exploited by DSSE to construct Graph-based Triplet Constraint (GTC) which can enforce feature embeddings from the semantically-matched (image and text) pairs to be more similar and push those mismatched ones away. By the benefit of fully using of semantic information, our approach can only use fewer label data and achieves the performance of state-of-the-art methods. In addition, since we only utilize the mapping networks trained in GBF module to generate common representations in referring stage, our approach is efficient and time saving in real world application. Extensive experiments on four widely-used datasets show the effectiveness of GSAN.

Fuzzy Information Discrimination Measures and Their Application to Low Dimensional Embedding Construction in the UMAP Algorithm.

Dimensionality reduction techniques are often used by researchers in order to make high dimensional data easier to interpret visually, as data visualization is only possible in low dimensional spaces. Recent research in nonlinear dimensionality reduction introduced many effective algorithms, including t-distributed stochastic neighbor embedding (t-SNE), uniform manifold approximation and projection (UMAP), dimensionality reduction technique based on triplet constraints (TriMAP), and pairwise controlled manifold approximation (PaCMAP), aimed to preserve both the local and global structure of high dimensional data while reducing the dimensionality. The UMAP algorithm has found its application in bioinformatics, genetics, genomics, and has been widely used to improve the accuracy of other machine learning algorithms. In this research, we compare the performance of different fuzzy information discrimination measures used as loss functions in the UMAP algorithm while constructing low dimensional embeddings. In order to achieve this, we derive the gradients of the considered losses analytically and employ the Adam algorithm during the loss function optimization process. From the conducted experimental studies we conclude that the use of either the logarithmic fuzzy cross entropy loss without reduced repulsion or the symmetric logarithmic fuzzy cross entropy loss with sufficiently large neighbor count leads to better global structure preservation of the original multidimensional data when compared to the loss function used in the original UMAP algorithm implementation.

No-Reference Image Quality Assessment by Hallucinating Pristine Features.

In this paper, we propose a no-reference (NR) image quality assessment (IQA) method via feature level pseudo-reference (PR) hallucination. The proposed quality assessment framework is rooted in the view that the perceptually meaningful features could be well exploited to characterize the visual quality, and the natural image statistical behaviors are exploited in an effort to deliver the accurate predictions. Herein, the PR features from the distorted images are learned by a mutual learning scheme with the pristine reference as the supervision, and the discriminative characteristics of PR features are further ensured with the triplet constraints. Given a distorted image for quality inference, the feature level disentanglement is performed with an invertible neural layer for final quality prediction, leading to the PR and the corresponding distortion features for comparison. The effectiveness of our proposed method is demonstrated on four popular IQA databases, and superior performance on cross-database evaluation also reveals the high generalization capability of our method. The implementation of our method is publicly available on https://github.com/Baoliang93/FPR.

TDCMR: Triplet-Based Deep Cross-Modal Retrieval for Geo-Multimedia Data

Mass multimedia data with geographical information (geo-multimedia) are collected and stored on the Internet due to the wide application of location-based services (LBS). How to find the high-level semantic relationship between geo-multimedia data and construct efficient index is crucial for large-scale geo-multimedia retrieval. To combat this challenge, the paper proposes a deep cross-modal hashing framework for geo-multimedia retrieval, termed as Triplet-based Deep Cross-Modal Retrieval (TDCMR), which utilizes deep neural network and an enhanced triplet constraint to capture high-level semantics. Besides, a novel hybrid index, called TH-Quadtree, is developed by combining cross-modal binary hash codes and quadtree to support high-performance search. Extensive experiments are conducted on three common used benchmarks, and the results show the superior performance of the proposed method.

Combined Rule-Based and Hypothesis-Based Method for Building Model Reconstruction from Photogrammetric Point Clouds

Three-dimensional (3D) building models play an important role in digital cities and have numerous potential applications in environmental studies. In recent years, the photogrammetric point clouds obtained by aerial oblique images have become a major source of data for 3D building reconstruction. Aiming at reconstructing a 3D building model at Level of Detail (LoD) 2 and even LoD3 with preferred geometry accuracy and affordable computation expense, in this paper, we propose a novel method for the efficient reconstruction of building models from the photogrammetric point clouds which combines the rule-based and the hypothesis-based method using a two-stage topological recovery process. Given the point clouds of a single building, planar primitives and their corresponding boundaries are extracted and regularized to obtain abstracted building counters. In the first stage, we take advantage of the regularity and adjacency of the building counters to recover parts of the topological relationships between different primitives. Three constraints, namely pairwise constraint, triplet constraint, and nearby constraint, are utilized to form an initial reconstruction with candidate faces in ambiguous areas. In the second stage, the topologies in ambiguous areas are removed and reconstructed by solving an integer linear optimization problem based on the initial constraints while considering data fitting degree. Experiments using real datasets reveal that compared with state-of-the-art methods, the proposed method can efficiently reconstruct 3D building models in seconds with the geometry accuracy in decimeter level.

Weighted-Attribute Triplet Hashing for Large-Scale Similar Judicial Case Matching.

Similar judicial case matching aims to enable an accurate selection of a judicial document that is most similar to the target document from multiple candidates. The core of similar judicial case matching is to calculate the similarity between two fact case documents. Owing to similar judicial case matching techniques, legal professionals can promptly find and judge similar cases in a candidate set. These techniques can also benefit the development of judicial systems. However, the document of judicial cases not only is long in length but also has a certain degree of structural complexity. Meanwhile, a variety of judicial cases are also increasing rapidly; thus, it is difficult to find the document most similar to the target document in a large corpus. In this study, we present a novel similar judicial case matching model, which obtains the weight of judicial feature attributes based on hash learning and realizes fast similar matching by using a binary code. The proposed model extracts the judicial feature attributes vector using the bidirectional encoder representations from transformers (BERT) model and subsequently obtains the weighted judicial feature attributes through learning the hash function. We further impose triplet constraints to ensure that the similarity of judicial case data is well preserved when projected into the Hamming space. Comprehensive experimental results on public datasets show that the proposed method is superior in the task of similar judicial case matching and is suitable for large-scale similar judicial case matching.

Deep learning-based remote and social sensing data fusion for urban region function recognition

Urban region function recognition is key to rational urban planning and management. Due to the complex socioeconomic nature of functional land use, recognizing urban region function in high-density cities using remote sensing images alone is difficult. The inclusion of social sensing has the potential to improve the function classification performance. However, effectively integrating the multi-source and multi-modal remote and social sensing data remains technically challenging. In this paper, we have proposed a novel end-to-end deep learning-based remote and social sensing data fusion model to address this issue. Two neural network based methods, one based on a 1-dimensional convolutional neural network (CNN) and the other based on a long short-term memory (LSTM) network, have been developed to automatically extract discriminative time-dependent social sensing signature features, which are fused with remote sensing image features extracted via a residual neural network. One of the major difficulties in exploiting social and remote sensing data is that the two data sources are asynchronous. We have developed a deep learning-based strategy to address this missing modality problem by enforcing cross-modal feature consistency (CMFC) and cross-modal triplet (CMT) constraints. We train the model in an end-to-end manner by simultaneously optimizing three costs, including the classification cost, the CMFC cost and the CMT cost. Extensive experiments have been conducted on publicly available datasets to demonstrate the effectiveness of the proposed method in fusing remote and social sensing data for urban region function recognition. The results show that the seemingly unrelated physically sensed image data and social activities sensed signatures can indeed complement each other to help enhance the accuracy of urban region function recognition.

LGSim: local task-invariant and global task-specific similarity for few-shot classification

Few-shot learning is one of the most challenging problems in computer vision due to the difficulty of sample collection in many real-world applications. It aims at classifying a sample when the number of training samples for each identity is limited. Most of the existing few-shot learning models learn a distance metric with pairwise or triplet constraints. In this paper, we make initial attempts on learning local and global similarities simultaneously to improve the few-shot classification performance in terms of accuracy. In particular, our system differs in two aspects. Firstly, we develop a neural network to learn the pairwise local relationship between each pair of samples in the union set that is composed of support set and query set, which fully utilize the supervision. Secondly, we design a global similarity function from the manifold perspective. The latent assumption is that if the neighbors of one sample are similar to those of another sample, the global similarity between them will be high. Otherwise, the global similarity of the two samples will become very low even if the local similarity between them is high. Meanwhile, we propose a new loss according to the pairwise local loss and task-specific global loss, encouraging the model toward better generalization. Extensive experiments on three popular benchmarks (Omniglot, miniImageNet and tieredImageNet) demonstrate that our simple, yet effective approach can achieve competitive accuracy compared to the state-of-the-art methods.

Data Augmentation with Suboptimal Warping for Time-Series Classification.

In this paper, a novel data augmentation method for time-series classification is proposed. In the introduced method, a new time-series is obtained in warped space between suboptimally aligned input examples of different lengths. Specifically, the alignment is carried out constraining the warping path and reducing its flexibility. It is shown that the resultant synthetic time-series can form new class boundaries and enrich the training dataset. In this work, the comparative evaluation of the proposed augmentation method against related techniques on representative multivariate time-series datasets is presented. The performance of methods is examined using the nearest neighbor classifier with the dynamic time warping (NN-DTW), LogDet divergence-based metric learning with triplet constraints (LDMLT), and the recently introduced time-series cluster kernel (NN-TCK). The impact of the augmentation on the classification performance is investigated, taking into account entire datasets and cases with a small number of training examples. The extensive evaluation reveals that the introduced method outperforms related augmentation algorithms in terms of the obtained classification accuracy.

Video Retrieval with Similarity-Preserving Deep Temporal Hashing

Despite the fact that remarkable progress has been made in recent years, Content-based Video Retrieval (CBVR) is still an appealing research topic due to increasing search demands in the Internet era of big data. This article aims to explore an efficient CBVR system by discriminately hashing videos into short binary codes. Existing video hashing methods usually encounter two weaknesses originating from the following sources: (1) Most works adopt the separated stages method or the frame-pooling based end-to-end architecture. However, the spatial-temporal properties of videos cannot be fully explored or kept well in the follow-up hashing step. (2) Discriminative learning based on pairwise or triplet constraints often suffers from slow convergence and poor local optimization, mainly because of the limited samples for each update. To alleviate these problems, we propose an end-to-end video retrieval framework called the Similarity-Preserving Deep Temporal Hashing (SPDTH) network. Specifically, we equip the model with the ability to capture spatial-temporal properties of videos and to generate binary codes by stacked Gated Recurrent Units (GRUs). It unifies video temporal modeling and learning to hash into one step to allow for maximum retention of information. We also introduce a deep metric learning objective called ℓ 2 All _ loss for network training by preserving intra-class similarity and inter-class separability, and a quantization loss between the real-valued outputs and the binary codes is minimized. Extensive experiments on several challenging datasets demonstrate that SPDTH can consistently outperform state-of-the-art methods.

Person Reidentification via Structural Deep Metric Learning.

Despite the promising progress made in recent years, person reidentification (re-ID) remains a challenging task due to the complex variations in human appearances from different camera views. This paper proposes to tackle this task by jointly learning feature representation and distance metric in an end-to-end manner. Existing deep metric learning-based re-ID methods usually encounter the following two weaknesses: 1) most works based on pairwise or triplet constraints often suffer from slow convergence and poor local optima, partially because they use very limited samples for each update and 2) hard negative sample mining has been widely applied in existing works. However, hard positive samples, which also contribute to the training of network, have not received enough attention. To alleviate these problems, we develop a novel structural metric learning objective for person re-ID, in which each positive pair is allowed to be compared against all negative pairs in a minibatch and each positive pair is adaptively assigned a hardness-aware weight to modulate its contribution. The introduced positive pair weighting strategy enables the algorithm to focus more on the hard positive samples. Furthermore, we propose to enhance the proposed loss function by adding a global loss term to reduce the variances of positive/negative pair distances, which is able to improve the generalization capability of the network model. By this approach, person images can be nonlinearly mapped into a low-dimensional embedding space where similar samples are kept closer and dissimilar samples are pushed farther apart. We implement the proposed algorithm using the inception architecture and evaluate it on three large-scale re-ID data sets. Experiment results demonstrate that our approach is able to outperform most state of the arts while using much lower dimensional deep features.

Similarity learning for person reidentification using projected gradient method

The problem of learning a similarity measure for cross-view person identification in a disjointed camera network is addressed. Our learning framework is based on the projected gradient approach and is suitable for large-scale applications. The hinge loss with the triplet constraints is used as the objective function. Contrary to other approaches where a dual formulation is used, we propose to optimize the objective function in the primal form, with the help of a minibatch gradient descent algorithm. The choice of learning rate and its schedule are nontrivial tasks in such optimization schemes. We have studied empirically the effect of using different strategies for learning rate schedules. Our experimental study includes conventional strategies such as fixed or diminishing learning rates and recently developed adaptive gradient methods such as root-mean-square propagation (RMSProp) for learning rate schedule. The experimental results are presented on three benchmark datasets of person reidentification, namely VIPeR, CUHK Campus, and CUHK03. Our experimental results demonstrate that the RMSProp is well suited for the proposed learning framework. A comparison with recent methods in the domain presents the justification of the proposed approach.

Urgent image-to-video person reidentification by cross-media transfer cycle generative adversarial networks

Recently, image-to-video person reidentification (IVPR) has attracted enormous research interest, and various models are proposed. IVPR is often applied to urgent situations, such as suspect tracking and lost-human locating. Existing IVPR models are under supervised frameworks, which require a large number of labeled image-to-video pairs. This severely limits their real-time efficiency in urgent situations, because annotation is much more time-consuming. To solve the urgent image-to-video person reidentification (UIVPR) problem, we propose a cross-media transfer cycle generative adversarial networks (CTC-GAN) network. Our model aims to alleviate the “media-gap” between image-to-video pairs without newly labeled pairs. We make an existing completely labeled dataset as guidance for CTC-GAN to achieve domain adaptation and make urgent image-to-video matching easier for person reidentification. We introduce cycle GANs for image(video)-to-video(image) translation and extract cross-media features using a triplet constraint in the source domain for different media features. Furthermore, we train the model in the labeled source domain by reconstructing the image (video) as its related video (image). Then, train the model in the unlabeled target domain by reconstructing itself along with source data, so as to ensure that the discriminative model can be used in target domain. Through CTC-GAN, our network can retain pedestrian discriminative information as much as possible, to ensure the matching rate in the target domain. To validate the effectiveness of our approach, we implement substantial experiments on two large-scale person reidentification datasets compared with six existing state-of-the-art unsupervised revised person reidentification models, and experimental results demonstrate that our method can solve UIVPR effectively.

Semi-Supervised Metric Learning-Based Anchor Graph Hashing for Large-Scale Image Retrieval.

Hashing-based image retrieval methods have become a cutting-edge topic in the information retrieval domain due to their high efficiency and low cost. In order to perform efficient hash learning by simultaneously preserving the semantic similarity and data structures in the feature space, this paper presents the semi-supervised metric learning-based anchor graph hashing method. Our proposed approach can be divided into three parts. First, we exploit a transformation matrix to construct the anchor-based similarity graph of the training set. Second, we propose the objective function based on the triplet relationship, in which the optimal transformation matrix can be learned by using the smoothness of labels and the margin hinge loss incurred by the triplet constraint. Moreover, the stochastic gradient descent (SGD) method leverages the gradient on each triplet to update the transformation matrix. Finally, a penalty factor is designed to accelerate the execution speed of SGD. Through comparison with the retrieval results of several state-of-the-art methods on several image benchmarks, the experiments validate the feasibility and advantages of our proposed methods.

Retrieving and Classifying Affective Images via Deep Metric Learning

Affective image understanding has been extensively studied in the last decade since more and more users express emotion via visual contents. While current algorithms based on convolutional neural networks aim to distinguish emotional categories in a discrete label space, the task is inherently ambiguous. This is mainly because emotional labels with the same polarity (i.e., positive or negative) are highly related, which is different from concrete object concepts such as cat, dog and bird. To the best of our knowledge, few methods focus on leveraging such characteristic of emotions for affective image understanding. In this work, we address the problem of understanding affective images via deep metric learning and propose a multi-task deep framework to optimize both retrieval and classification goals. We propose the sentiment constraints adapted from the triplet constraints, which are able to explore the hierarchical relation of emotion labels. We further exploit the sentiment vector as an effective representation to distinguish affective images utilizing the texture representation derived from convolutional layers. Extensive evaluations on four widely-used affective datasets, i.e., Flickr and Instagram, IAPSa, Art Photo, and Abstract Paintings, demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks.

Distance metric learning for ordinal classification based on triplet constraints

Ordinal classification is a problem setting in-between nominal classification and metric regression, where the goal is to predict classes of an ordinal scale. Usually, there is a clear ordering of the classes, but the absolute distances between them are unknown. Disregarding the ordering information, this kind of problems is commonly treated as multi-class classification problems, however, it often results in a significant loss of performance. Exploring such ordering information can help to improve the effectiveness of classifiers. In this paper, we propose a distance metric learning approach for ordinal classification by incorporating local triplet constraints containing the ordering information into a conventional large-margin distance metric learning approach. Specifically, our approach tries to preserve, for each training example, the ordinal relationship as well as the local geometry structure of its neighbors, which is suitable for use in local distance-based algorithms such as k-nearest-neighbor (k-NN) classification. Different from previous works that usually learn distance metrics by weighing the distances between training examples according to their class label differences, the proposed approach can directly satisfy the ordinal relationships where no assumptions about the distances between classes are made.