Latent Feature Space Research Articles

Deep Clustering and Visualization for End-to-End High-Dimensional Data Analysis.

High-dimensional data analysis for exploration and discovery includes two fundamental tasks: deep clustering and data visualization. When these two associated tasks are done separately, as is often the case thus far, disagreements can occur among the tasks in terms of geometry preservation. Namely, the clustering process is often accompanied by the corruption of the geometric structure, whereas visualization aims to preserve the data geometry for better interpretation. Therefore, how to achieve deep clustering and data visualization in an end-to-end unified framework is an important but challenging problem. In this article, we propose a novel neural network-based method, called deep clustering and visualization (DCV), to accomplish the two associated tasks end-to-end to resolve their disagreements. The DCV framework consists of two nonlinear dimensionality reduction (NLDR) transformations: 1) one from the input data space to latent feature space for clustering and 2) the other from the latent feature space to the final 2-D space for visualization. Importantly, the first NLDR transformation is mainly optimized by one Clustering Loss, allowing arbitrary corruption of the geometric structure for better clustering, while the second NLDR transformation is optimized by one Geometry-Preserving Loss to recover the corrupted geometry for better visualization. Extensive comparative results show that the DCV framework outperforms other leading clustering-visualization algorithms in terms of both quantitative evaluation metrics and qualitative visualization.

Self-supervised latent feature learning for partial point clouds recognition

3D vision perception, especially point clouds classification is fundamental and popular in safety-critical systems such as autonomous driving and robotics automation control. However, their robustness against incomplete partial point clouds in practical scenes is less studied, limiting real-world deployment. In this paper, we propose to improve the robustness and generalization of 3D models on partial point clouds by self-supervised latent feature learning. Unlike those data augmentation methods that generate partial point clouds by geometric transforms in coordinate space (e.g drop local structure or remove global points), we regard the partial data as a transformation in latent feature space. We explicitly learn the perspective transformation of partial point clouds and implicitly learn the occlusion transformation in the latent feature space by self-supervised learning. Different from previous methods that are validated on generated data, we test our method on point clouds completion datasets (e.g. PCN and MVP) which contain both complete and partial point clouds and have been widely used. Extensive experiments show that our proposed method consistently improves the robustness of state-of-the-art methods on the partial point clouds dataset.

Conditional feature generation for transductive open-set recognition via dual-space consistent sampling

Open-set recognition (OSR) aims to simultaneously detect unknown-class samples and classify known-class samples. Most of the existing OSR methods are inductive methods, which generally suffer from the domain shift problem that the learned model from the known-class domain might be unsuitable for the unknown-class domain. Addressing this problem, inspired by the success of transductive learning for alleviating the domain shift problem in many other visual tasks, we propose an Iterative Transductive OSR framework, called IT-OSR, which implements three explored modules iteratively, including a reliability sampling module, a feature generation module, and a baseline update module. Specifically at the initialization stage, a baseline method, which could be an arbitrary inductive OSR method, is used for assigning pseudo labels to the test samples. At the iteration stage, based on the consistency of the assigned pseudo labels between the output/logit space and the latent feature space of the baseline method, a dual-space consistent sampling approach is presented in the reliability sampling module for sampling some reliable ones from the test samples. Then in the feature generation module, a conditional dual-adversarial generative network is designed to generate discriminative features of both known and unknown classes. This generative network employs two discriminators for implementing fake/real and known/unknown-class discriminations respectively. And it is trained by jointly utilizing the test samples with their pseudo labels selected in the reliability sampling module and the labeled training samples. Finally in the baseline update module, the above baseline method is updated/re-trained for sample re-prediction by jointly utilizing the generated features, the selected test samples with pseudo labels, and the training samples. Extensive experimental results on both the standard-dataset and the cross-dataset settings demonstrate that the derived transductive methods, by introducing two typical inductive OSR methods into the proposed IT-OSR framework, achieve better performances than 19 state-of-the-art methods in most cases.

Coarse-to-fine online latent representations matching for one-stage domain adaptive semantic segmentation

Domain adaptive semantic segmentation is meaningful since collecting numerous labeled samples in different domains is expensive and time-consuming. Recent domain adaptation methods yield not so efficient performance compared with supervised learning. With the hypothesis that semantic feature can be shared across domains, this paper proposes a coarse-to-fine online matching architecture (COM) for one-stage domain adaptation. We consider subsequent learning stages progressively refining the task in the latent feature space, i.e. the finer set at each component is hierarchically derived from the coarser set of the previous components, including cross-domain global prototypes, categories and instances matching and anchor-points contrastive learning, which further achieve self-supervised learning with region-level pseudo label generated only in a single training step. Beforehand, feature refinement are performed to realize edge perception and inter-feature augmentation. Then, coarse-to-fine network fuses global and local consistency matching via specific distribution alignment between the source and target domain. Finally, the adversarial structure controls the uncertainty of generator prediction through the maximization of classification results and minimization of two classifiers discrepancy. This proposed method is evaluated in two unsupervised domain adaptation tasks, i.e. GTA5 → Cityscapes and SYNTHIA → Cityscapes. Extensive experiments verify the effectiveness of our proposed COM model and demonstrate its superiority over several state-of-the-art approaches.

Improving the Generalization of Visual Classification Models Across IoT Cameras via Cross-Modal Inference and Fusion

The performance of visual classification models across IoT devices is usually limited by the changes in local environments, resulted from the diverse appearances of the target objects and differences in light conditions and background scenes. To alleviate these problems, existing studies usually introduce the multimodal information to guide the learning process of the visual classification models, making the models extract the visual features from the discriminative image regions. Especially, cross-modal alignment between visual and textual features has been considered as an effective way for this task by learning a domain-consistent latent feature space for the visual and semantic features. However, this approach may suffer from the heterogeneity between multiple modalities, such as the multi-modal features and the differences in the learned feature values. To alleviate this problem, this paper first presents a comparative analysis of the functionality of various alignment strategies and their impacts on improving visual classification. Subsequently, a cross-modal inference and fusion framework (termed as CRIF) is proposed to align the heterogeneous features in both the feature distributions and values. More importantly, CRIF includes a cross-modal information enrichment module to improve the final classification and learn the mappings from the visual to the semantic space. We conduct experiments on four benchmarking datasets, i.e. the Vireo-Food172, NUS-WIDE, MSR-VTT, and ActivityNet Captions datasets. We report state-of-the-art results for basic classification tasks on the four datasets and conduct subsequent experiments on feature alignment and fusion. The experimental results verify that CRIF can effectively improve the learning ability of the visual classification models, and it is a model-agnostic framework that consistently improves the performance of state-of-the-art visual classification models.

A unified framework for visual domain adaptation with covariance matching

In the field of Machine Learning, it is widely acknowledged that training and test data should ideally stem from the same source and distribution. However, in the real world, this is not always feasible. Domain Adaptation (DA) techniques address this issue by adapting a classifier trained on annotated source domain data on unannotated target domain data while minimizing the impact of domain shift. Many recent DA approaches concentrate on learning a latent feature space that remains invariant to domain shift by mitigating various statistical and geometrical divergences. Although these methods have demonstrated effectiveness, they frequently overlook a crucial aspect of learning a domain-invariant discriminative latent feature space across diverse domains. To address this issue, we propose a novel framework called Unified Framework for Visual Domain Adaptation with Covariance Matching (UDACM) that aims to learn a domain invariant and class discriminative latent feature space by incorporating multiple objectives such as maximizing target domain variance, minimizing distribution and subspace divergence, performing manifold learning, preserving discriminative information of both the source and target domains, and measuring and matching covariance simultaneously. In our proposed framework, the measuring and matching of covariance play a crucial role in ensuring the learning of discriminative latent feature space by aligning the within-class and between-class covariance matrices of the source and target domains simultaneously. Detailed experiments demonstrate the outperformance of UDACM on baseline datasets such as CMU-PIE, Office+Caltech10, USPS+MNIST, VLCS, and Office–Home, compared to various established primitive, shallow, and deep domain adaptation methods on several image classification tasks.

Boosting One-Stage License Plate Detector via Self-Constrained Contrastive Aggregation

Scene Text Detection (STD) has applied in many fields successfully. One of the important applications of STD is License Plate Detection (LPD). As a unique identity of vehicle, License Plate (LP) facilitates the intelligent transportation in many fields, such as traffic enforcement, intelligent transportation dispatching, etc. However, there are many scene texts similar to LPs causing misjudgment of LP detector. To alleviate these disturbances, more discriminative features are necessary. In latent feature space, discriminative features should aggregate into a tight cluster to widen decision boundary. We assume three perspectives about how to aggregate features and boost feature expression. From these assumptions, a special contrastive triad is designed. Then, we propose a Self-Constrained Contrastive Aggregation (SCCA) method to lead the feature aggregation in latent space and boost the feature expression of backbone. The proposed SCCA is jointly trained with supervised learning for detection to improve the detection performance. The experiments show that our proposed SCCA prompts the baseline significantly and exceeds recent LP detectors, reaching 99.7 on both F1-score and AP on UFPR-ALPR dataset. Meanwhile, we compare the self-constrained contrastive learning with vanilla contrastive learning in experiments and visualize their LP features. The results show that our proposed SCCA reaches better performance and verifies our assumptions are reasonable.

As-Deformable-As-Possible Single-Image-Based View Synthesis Without Depth Prior

Depth-image-based rendering (DIBR) technologies have been widely employed to synthesize novel realistic views from a single image in 3D video applications. However, DIBR-oriented approaches heavily rely on the accuracy of depth maps, usually requiring the depth GT as a prior. Despite that, there might exist extensive float precision losses and invalid holes in the synthesized view due to warping error and occlusion. In this paper, we propose an end-to-end as-deformable-as-possible (ADAP) single-image-based view synthesis solution without depth prior. It addresses the above issues through two stages: alignment and reconstruction, where we first transform the input image to the latent feature space and then reconstruct the novel view in the image domain. In the first stage, the input image is deformed to align with the synthesized view at feature level. To this end, we propose an ADAP alignment mechanism through pixel-level warping to error-level quantization to feature-level alignment, progressively improving the deformable capability in handling challenging motion conditions in real-world scenes. In the second stage, we exploit an occlusion-aware reconstruction module to recover the content details from the deformed feature at pixel level. Extensive experiments demonstrate that our alignment-reconstruction approach is robust to the depth map. Even with a coarsely estimated depth map, our solution outperforms other SoTA schemes in the popular benchmarks.

Deep discriminative clustering and structural constraint for cross-domain fault diagnosis of rotating machinery

With the rapid development of intelligent manufacturing, fault diagnostic methods based on deep learning have achieved impressive results. However, most methods require plentiful annotated samples and are based on the assumption that data from the source and target domains has the same distribution. These two conditions are difficult to satisfy in practical engineering. In light of these problems, an unsupervised domain adaptation approach named Deep Discriminative Clustering network with Structural Constraint (DDCSC) is proposed in this article. In our method, a Convolutional Neural Network (CNN) module is exploited for learning feature representations of raw data. Then a softmax module is employed to simultaneously predict class probabilities and cluster assignments of the source and target data, respectively. The learnable cluster centroids are introduced into the latent feature space to alleviate the data distribution discrepancy while better capturing the discriminative structure of the target data. In addition, geometric properties of the source data in a feature space are constrained to expand the scope of each category, which facilitates to improve prediction accuracy. An information-theoretic metric is considered as the objective function of discriminative clustering. Diagnostic experiments on a rolling bearing dataset demonstrate that our approach outperforms other popular intelligent approaches and confirms the effectiveness of discriminative clustering.

IGRLDTI: An Improved Graph Representation Learning Method for Predicting Drug-Target Interactions over Heterogeneous Biological Information Network.

The task of predicting drug-target interactions (DTIs) plays a significant role in faciliating the development of novel drug discovery. Compared with laboratory-based approaches, computational methods proposed for DTI prediction are preferred due to their high-efficiency and low-cost advantages. Recently, much attention has been attracted to apply different graph neural network (GNN) models to discover underlying DTIs from hetergeneous biological information network (HBIN). Although GNN-based prediction methods achieve better performance, they are prone to encounter the over-smoothing simulation when learning the latent representations of drugs and targets with their rich neighborhood information in HBIN, and thereby reduce the discriminative ability in DTI prediction. In this work, an improved graph representation learning method, namely iGRLDTI, is proposed to address the above issue by better capturing more discriminative representations of drugs and targets in a latent feature space. Specifically, iGRLDTI first constructs a HBIN by integrating the biological knowledge of drugs and targets with their interactions. After that, it adopts a node-dependent local smoothing strategy to adaptively decide the propagation depth of each biomolecule in HBIN, thus significantly alleviating over-smoothing by enhancing the discriminative ability of feature represeantions of drugs and targets. Finally, a Gradient Boosting Decision Tree classifier is used by iGRLDTI to predict novel DTIs. Experimental results demonstrate that iGRLDTI yields better performance that several state-of-the-art computational methods on the benchmark dataset. Besides, our case study indicates that iGRLDTI can successfully identify novel DTIs with more distinguishable features of drugs and targets. Python codes and dataset are available at https://github.com/stevejobws/iGRLDTI/. Supplementary data are available at Bioinformatics online.

Locality Preserving Property Constrained Contrastive Learning for Object Classification in SAR Imagery

Robust unsupervised feature learning is a critical yet tough task for synthetic aperture radar (SAR) automatic target recognition (ATR) with limited labeled data. The developing contrastive self-supervised learning (CSL) method, which learns informative representations by solving an instance discrimination task, provides a novel method for learning discriminative features from unlabeled SAR images. However, the instance-level contrastive loss can magnify the differences between samples belonging to the same class in the latent feature space. Therefore, CSL can dispel these targets from the same class and affect the downstream classification tasks. In order to address this problem, this paper proposes a novel framework called locality preserving property constrained contrastive learning (LPPCL), which not only learns informative representations of data but also preserves the local similarity property in the latent feature space. In LPPCL, the traditional InfoNCE loss of the CSL models is reformulated in a cross-entropy form where the local similarity of the original data is embedded as pseudo labels. Furthermore, the traditional two-branch CSL architecture is extended to a multi-branch structure, improving the robustness of models trained with limited batch sizes and samples. Finally, the self-attentive pooling module is used to replace the global average pooling layer that is commonly used in most of the standard encoders, which provides an adaptive method for retaining information that benefits downstream tasks during the pooling procedure and significantly improves the performance of the model. Validation and ablation experiments using MSTAR datasets found that the proposed framework outperformed the classic CSL method and achieved state-of-the-art (SOTA) results.

Multi-Constraint Latent Representation Learning for Prognosis Analysis Using Multi-Modal Data.

The Cox proportional hazard model has been widely applied to cancer prognosis prediction. Nowadays, multi-modal data, such as histopathological images and gene data, have advanced this field by providing histologic phenotype and genotype information. However, how to efficiently fuse and select the complementary information of high-dimensional multi-modal data remains challenging for Cox model, as it generally does not equip with feature fusion/selection mechanism. Many previous studies typically perform feature fusion/selection in the original feature space before Cox modeling. Alternatively, learning a latent shared feature space that is tailored for Cox model and simultaneously keeps sparsity is desirable. In addition, existing Cox-based models commonly pay little attention to the actual length of the observed time that may help to boost the model's performance. In this article, we propose a novel Cox-driven multi-constraint latent representation learning framework for prognosis analysis with multi-modal data. Specifically, for efficient feature fusion, a multi-modal latent space is learned via a bi-mapping approach under ranking and regression constraints. The ranking constraint utilizes the log-partial likelihood of Cox model to induce learning discriminative representations in a task-oriented manner. Meanwhile, the representations also benefit from regression constraint, which imposes the supervision of specific survival time on representation learning. To improve generalization and alleviate overfitting, we further introduce similarity and sparsity constraints to encourage extra consistency and sparseness. Extensive experiments on three datasets acquired from The Cancer Genome Atlas (TCGA) demonstrate that the proposed method is superior to state-of-the-art Cox-based models.

Unsupervised Multi-Exposure Image Fusion Breaking Exposure Limits via Contrastive Learning

This paper proposes an unsupervised multi-exposure image fusion (MEF) method via contrastive learning, termed as MEF-CL. It breaks exposure limits and performance bottleneck faced by existing methods. MEF-CL firstly designs similarity constraints to preserve contents in source images. It eliminates the need for ground truth (actually not exist and created artificially) and thus avoids negative impacts of inappropriate ground truth on performance and generalization. Moreover, we explore a latent feature space and apply contrastive learning in this space to guide fused image to approximate normal-light samples and stay away from inappropriately exposed ones. In this way, characteristics of fused images (e.g., illumination, colors) can be further improved without being subject to source images. Therefore, MEF-CL is applicable to image pairs of any multiple exposures rather than a pair of under-exposed and over-exposed images mandated by existing methods. By alleviating dependence on source images, MEF-CL shows better generalization for various scenes. Consequently, our results exhibit appropriate illumination, detailed textures, and saturated colors. Qualitative, quantitative, and ablation experiments validate the superiority and generalization of MEF-CL. Our code is publicly available at https://github.com/hanna-xu/MEF-CL.

Learning a robust unified domain adaptation framework for cross-subject EEG-based emotion recognition

Over the last few years, unsupervised domain adaptation (UDA) based on deep learning has emerged as a solution to build cross-subject emotion recognition models from Electroencephalogram (EEG) signals, aligning the subject distributions within a latent feature space. However, most reported works have a common intrinsic limitation: the subject distribution alignment is coarse-grained, but not all of the feature space is shared between subjects. In this paper, we propose a robust unified domain adaptation framework, named Multi-source Feature Alignment and Label Rectification (MFA-LR), which performs a fine-grained domain alignment at subject and class levels, while inter-class separation and robustness against input perturbations are encouraged in coarse grain. As a complementary step, a pseudo-labeling correction procedure is used to rectify mislabeled target samples. Our proposal was assessed over two public datasets, SEED and SEED-IV, on each of the three available sessions, using leave-one-subject-out cross-validation. Experimental results show an accuracy performance of up to 89.11 ± 07.72% and 74.99 ± 12.10% for the best session on SEED and SEED-IV, as well as an average accuracy of 85.27% and 69.58% on all three sessions, outperforming state-of-the-art results.

Dual frame-level and region-level alignment for unsupervised video domain adaptation

Most existing unsupervised video domain adaptation methods focus on extracting frame-level features and using temporal attention to create a domain-invariant feature space. However, this ignores the importance of different image regions and temporal dynamics. In this study, we propose a method that considers both frame-level and image-region level alignment to learn a dual spatial and temporal domain-invariant feature space. This is achieved by: 1) extracting frame-level image features and aligning them into a domain-invariant latent feature space; 2) extracting region-level temporal features and aligning them into a domain-invariant latent feature space. Optimal frame-level correlation is achieved through video-frame level alignment, while optimal image-region level correlation is demonstrated through image-region level alignment between the source and target videos. We use an LSTM network to parameterize this dual feature alignment, which is more efficient than attention mechanisms and results in significantly improved performance and reduced GPU memory cost by over 40% compared to current state-of-the-art methods.

Multi-view subspace clustering via adaptive graph learning and late fusion alignment

Multi-view subspace clustering has attracted great attention due to its ability to explore data structure by utilizing complementary information from different views. Most of existing methods learn a sample representation coefficient matrix or an affinity graph for each single view, then the final clustering result is obtained from the spectral embedding of a consensus graph using certain traditional clustering techniques, such as k-means. However, clustering performance will be degenerated if the early fusion of partitions cannot fully exploit relationships between all samples. Different from existing methods, we propose a multi-view subspace clustering method via adaptive graph learning and late fusion alignment (AGLLFA). For each view, AGLLFA learns an affinity graph adaptively to capture the similarity relationship among samples. Moreover, a spectral embedding learning term is designed to exploit the latent feature space of different views. Furthermore, we design a late fusion alignment mechanism to generate an optimal clustering partition by fusing view-specific partitions obtained from multiple views. An alternate updating algorithm with validated convergence is developed to solve the resultant optimization problem. Extensive experiments on several benchmark datasets are conducted to illustrate the effectiveness of the proposed method when compared with other state-of-the-art methods. The demo code of this work is publicly available at https://github.com/tangchuan2000/AGLLFA.

Partial Unbalanced Feature Transport for Cross-Modality Cardiac Image Segmentation.

Deep learning based approaches have achieved great success on the automatic cardiac image segmentation task. However, the achieved segmentation performance remains limited due to the significant difference across image domains, which is referred to as domain shift. Unsupervised domain adaptation (UDA), as a promising method to mitigate this effect, trains a model to reduce the domain discrepancy between the source (with labels) and the target (without labels) domains in a common latent feature space. In this work, we propose a novel framework, named Partial Unbalanced Feature Transport (PUFT), for cross-modality cardiac image segmentation. Our model facilities UDA leveraging two Continuous Normalizing Flow-based Variational Auto-Encoders (CNF-VAE) and a Partial Unbalanced Optimal Transport (PUOT) strategy. Instead of directly using VAE for UDA in previous works where the latent features from both domains are approximated by a parameterized variational form, we introduce continuous normalizing flows (CNF) into the extended VAE to estimate the probabilistic posterior and alleviate the inference bias. To remove the remaining domain shift, PUOT exploits the label information in the source domain to constrain the OT plan and extracts structural information of both domains, which are often neglected in classical OT for UDA. We evaluate our proposed model on two cardiac datasets and an abdominal dataset. The experimental results demonstrate that PUFT achieves superior performance compared with state-of-the-art segmentation methods for most structural segmentation.

A multi-task learning-based optimization approach for finding diverse sets of microstructures with desired properties

Optimization along the chain processing-structure-properties-performance is one of the core objectives in data-driven materials science. In this sense, processes are supposed to manufacture workpieces with targeted material microstructures. These microstructures are defined by the material properties of interest and identifying them is a question of materials design. In the present paper, we addresse this issue and introduce a generic multi-task learning-based optimization approach. The approach enables the identification of sets of highly diverse microstructures for given desired properties and corresponding tolerances. Basically, the approach consists of an optimization algorithm that interacts with a machine learning model that combines multi-task learning with siamese neural networks. The resulting model (1) relates microstructures and properties, (2) estimates the likelihood of a microstructure of being producible, and (3) performs a distance preserving microstructure feature extraction in order to generate a lower dimensional latent feature space to enable efficient optimization. The proposed approach is applied on a crystallographic texture optimization problem for rolled steel sheets given desired properties.

Robust deep semi-supervised learning with label propagation and differential privacy

Semi-supervised learning (SSL) methods provide a powerful tool for utilizing abundant unlabeled data to strengthen standard supervised learning. Traditional graph-based SSL methods prevail in classical SSL problems for their intuitional implementation and effective performance. However, they encounter troubles when applying to image classification followed by modern deep learning, since the diffusion algorithms face the curse of dimensionality. In this study, we propose a simple and efficient SSL method, combining a graph-based SSL paradigm with differential privacy. We aim at developing coherent latent feature space of deep neural networks so that the diffusion algorithm in the latent space can give more precise predictions for unlabeled data. Our approach achieves state-of-the-art performance on the Cifar10, Cifar100, and Mini-imagenet benchmark datasets and obtains an error rate of 18.56% on Cifar10 using only 1% of all labels. Furthermore, our approach inherits the benefits of graph-based SSL methods with a simple training process and can be easily combined with any network architecture.

Deep expectation-maximization network for unsupervised image segmentation and clustering

Unsupervised learning, such as unsupervised image segmentation and clustering, are fundamental tasks in image representation learning. In this paper, we design a deep expectation-maximization (DEM) network for unsupervised image segmentation and clustering. It is based on the statistical modeling of image in its latent feature space by Gaussian mixture model (GMM), implemented in a novel deep learning framework. Specifically, in the unsupervised setting, we design an auto-encoder network and an EM module over the image latent features, for jointly learning the image latent features and GMM model of the latent features in a single framework. To construct the EM-module, we unfold the iterative operations of EM algorithm and the online EM algorithm in fixed steps to be differentiable network blocks, plugged into the network to estimate the GMM parameters of the image latent features. The proposed network parameters can be end-to-end optimized using losses based on log-likelihood of GMM, entropy of Gaussian component assignment probabilities and image reconstruction error. Extensive experiments confirm that our proposed networks achieve favorable results compared with several state-of-the-art methods in unsupervised image segmentation and clustering.