Unsupervised Learning Framework Research Articles

Expressway Vehicle Trajectory Prediction Based on Fusion Data of Trajectories and Maps from Vehicle Perspective

Research on vehicle trajectory prediction based on road monitoring video data often utilizes a global map as an input, disregarding the fact that drivers rely on the road structures observable from their own positions for path planning. This oversight reduces the accuracy of prediction. To address this, we propose the CVAE-VGAE model, a novel trajectory prediction approach. Initially, our method transforms global perspective map data into vehicle-centric map data, representing it through a graph structure. Subsequently, Variational Graph Auto-Encoders (VGAEs), an unsupervised learning framework tailored for graph-structured data, are employed to extract road environment features specific to each vehicle’s location from the map data. Finally, a prediction network based on the Conditional Variational Autoencoder (CVAE) structure is designed, which first predicts the driving endpoint and then fits the complete future trajectory. The proposed CVAE-VGAE model integrates a self-attention mechanism into its encoding and decoding modules to infer endpoint intent and incorporate road environment features for precise trajectory prediction. Through a series of ablation experiments, we demonstrate the efficacy of our method in enhancing vehicle trajectory prediction metrics. Furthermore, we compare our model with traditional and frontier approaches, highlighting significant improvements in prediction accuracy.

Image Intrinsic-Based Unsupervised Monocular Depth Estimation in Endoscopy.

Unsupervised monocular depth estimation plays a vital role for endoscopy-based minimally invasive surgery (MIS). However, it remains challenging due to the distinctive imaging characteristics of endoscopy which disrupt the assumption of photometric consistency, a foundation relied upon by conventional methods. Distinct from recent approaches taking image pre-processing strategy, this paper introduces a pioneering solution through intrinsic image decomposition (IID) theory. Specifically, we propose a novel end-to-end intrinsic-based unsupervised monocular depth learning framework that is comprised of an image intrinsic decomposition module and a synthesis reconstruction module. This framework seamlessly integrates IID with unsupervised monocular depth estimation, and dedicated losses are meticulously designed to offer robust supervision for network training based on this novel integration. Noteworthy, we rely on the favorable property of the resulting albedo map of IID to circumvent the challenging images characteristics instead of pre-processing the input frames. The proposed method is extensively validated on SCARED and Hamlyn datasets, and better results are obtained than state-of-the-art techniques. Beside, its generalization ability and the effectiveness of the proposed components are also validated. This innovative method has the potential to elevate the quality of 3D reconstruction in monocular endoscopy, thereby enhancing the accuracy and robustness of augmented reality navigation technology in MIS. Our code will be available at: https://github.com/bobo909/IID-SfmLearner.

An improved method for rapid un-collapsed building extraction from post-disaster high-resolution remote sensing imagery based on multi-scale feature alignment

ABSTRACT Quick and accurate extraction of un-collapsed buildings from post-disaster High-resolution Remote Sensing Images (HRSIs) is imperative for emergency response. Pre-disaster HRSIs could serve as auxiliary data for training models to expedite this extraction process. However, the effectiveness of models trained directly on pre-disaster HRSIs tends to diminish when applied to post-disaster scenarios, mainly due to the notable discrepancies between these datasets. The current popular approach to mitigate this issue involves aligning features from pre- and post-disaster images using an unsupervised domain adversarial learning framework. However, conventional methods often fall short in reducing the substantial disparity between pre- and post-disaster images, due to a lack of comprehensive alignment of category-level and multi-scale features. To overcome these limitations, we propose the Multi-scale Global and Category-attention Features Alignment Network (MGCAN). This novel approach further refines feature alignment strategies by concurrently aligning both multi-scale global and category-attention features, thus effectively narrowing the gap between pre- and post-disaster HRSIs. Extensive experiments have demonstrated that MGCAN significantly improves the accuracy of un-collapsed building extraction from post-disaster HRSIs. Moreover, compared to other state-of-the-art domain adversarial networks, MGCAN exhibits superior performance in different disaster scenarios.

Unveiling the potential of diffusion model-based framework with transformer for hyperspectral image classification

Hyperspectral imaging has gained popularity for analysing remotely sensed images in various fields such as agriculture and medical. However, existing models face challenges in dealing with the complex relationships and characteristics of spectral–spatial data due to the multi-band nature and data redundancy of hyperspectral data. To address this limitation, we propose a novel approach called DiffSpectralNet, which combines diffusion and transformer techniques. The diffusion method is able extract diverse and meaningful spectral–spatial features, leading to improvement in HSI classification. Our approach involves training an unsupervised learning framework based on the diffusion model to extract high-level and low-level spectral–spatial features, followed by the extraction of intermediate hierarchical features from different timestamps for classification using a pre-trained denoising U-Net. Finally, we employ a supervised transformer-based classifier to perform the HSI classification. We conduct comprehensive experiments on three publicly available datasets to validate our approach. The results demonstrate that our framework significantly outperforms existing approaches, achieving state-of-the-art performance. The stability and reliability of our approach are demonstrated across various classes in all datasets.

Optimizing IT FinOps and Sustainability through Unsupervised Workload Characterization

The widespread adoption of public and hybrid clouds, along with elastic resources and various automation tools for dynamic deployment, has accelerated the rapid provisioning of compute resources as needed. Despite these advancements, numerous resources persist unnecessarily due to factors such as poor digital hygiene, risk aversion, or the absence of effective tools, resulting in substantial costs and energy consumption. Existing threshold-based techniques prove inadequate in effectively addressing this challenge. To address this issue, we propose an unsupervised machine learning framework to automatically identify resources that can be de-provisioned completely or summoned on a schedule. Application of this approach to enterprise data has yielded promising initial results, facilitating the segregation of productive workloads with recurring demands from non-productive ones.

A clustering and graph deep learning-based framework for COVID-19 drug repurposing

Drug repurposing (or repositioning) is the process of finding new therapeutic uses for drugs already approved by drug regulatory authorities (e.g., the Food and Drug Administration (FDA) and Therapeutic Goods Administration (TGA)) for other diseases. This involves analysing the interactions between different biological entities, such as drug targets (genes/proteins and biological pathways) and drug properties, to discover novel drug–target or drug–disease relations. Machine learning and deep learning models have successfully analysed complex heterogeneous data with applications in the biomedical domain, and have also been used for drug repurposing. This study presents a novel unsupervised machine learning framework that utilizes a graph-based autoencoder for multi-feature type clustering on heterogeneous drug data. The dataset consists of 438 drugs, of which 224 are under clinical trials for COVID-19 (category A). The rest are systematically filtered to ensure the safety and efficacy of the treatment (category B). The framework solely relies on reported drug data, including its pharmacological properties, chemical/physical properties, interaction with the host, and efficacy in different publicly available COVID-19 assays. Our machine-learning framework revealed three clusters of interest and provided recommendations featuring the top 15 drugs for COVID-19 drug repurposing, which were shortlisted based on the predicted clusters that were dominated by category A drugs. Our framework can be extended to support other datasets and drug repurposing studies with the availability of our open-source code.

Uncertainty Quantification in CO2 Trapping Mechanisms: A Case Study of PUNQ-S3 Reservoir Model Using Representative Geological Realizations and Unsupervised Machine Learning

Evaluating uncertainty in CO2 injection projections often requires numerous high-resolution geological realizations (GRs) which, although effective, are computationally demanding. This study proposes the use of representative geological realizations (RGRs) as an efficient approach to capture the uncertainty range of the full set while reducing computational costs. A predetermined number of RGRs is selected using an integrated unsupervised machine learning (UML) framework, which includes Euclidean distance measurement, multidimensional scaling (MDS), and a deterministic K-means (DK-means) clustering algorithm. In the context of the intricate 3D aquifer CO2 storage model, PUNQ-S3, these algorithms are utilized. The UML methodology selects five RGRs from a pool of 25 possibilities (20% of the total), taking into account the reservoir quality index (RQI) as a static parameter of the reservoir. To determine the credibility of these RGRs, their simulation results are scrutinized through the application of the Kolmogorov–Smirnov (KS) test, which analyzes the distribution of the output. In this assessment, 40 CO2 injection wells cover the entire reservoir alongside the full set. The end-point simulation results indicate that the CO2 structural, residual, and solubility trapping within the RGRs and full set follow the same distribution. Simulating five RGRs alongside the full set of 25 GRs over 200 years, involving 10 years of CO2 injection, reveals consistently similar trapping distribution patterns, with an average value of Dmax of 0.21 remaining lower than Dcritical (0.66). Using this methodology, computational expenses related to scenario testing and development planning for CO2 storage reservoirs in the presence of geological uncertainties can be substantially reduced.

Unsupervised deep learning framework for temperature-compensated damage assessment using ultrasonic guided waves on edge device

Fueled by the rapid development of machine learning (ML) and greater access to cloud computing and graphics processing units, various deep learning based models have been proposed for improving performance of ultrasonic guided wave structural health monitoring (GW-SHM) systems, especially to counter complexity and heterogeneity in data due to varying environmental factors (e.g., temperature) and types of damages. Such models typically comprise of millions of trainable parameters, and therefore add to cost of deployment due to requirements of cloud connectivity and processing, thus limiting the scale of deployment of GW-SHM. In this work, we propose an alternative solution that leverages TinyML framework for development of light-weight ML models that could be directly deployed on embedded edge devices. The utility of our solution is illustrated by presenting an unsupervised learning framework for damage detection in honeycomb composite sandwich structure with disbond and delamination type of damages, validated using data generated by finite element simulations and experiments performed at various temperatures in the range 0–90 °C. We demonstrate a fully-integrated solution using a Xilinx Artix-7 FPGA for data acquisition and control, and edge-inference of damage. Despite the limited number of features, the lightweight model shows reasonably high accuracy, thereby enabling detection of small size defects with improved sensitivity on an edge device for online GW-SHM.

A deep clustering framework for load pattern segmentation

In recent years, the widespread use of smart meters in power networks has generated a wealth of data from electricity customers. However, much of this real-world smart meter dataset lacks labeled data, posing a significant challenge that can be addressed by clustering. Traditional clustering methods often struggle in high-dimensional spaces, leading to less accurate results and increased computational demands. In response to these challenges, this research introduces a framework that utilizes a deep-learning-based clustering approach to address the issue of accurate customer clustering based on usage patterns in unlabeled data. Using an autoencoder, our approach integrates dimensionality reduction and clustering into an end-to-end unsupervised learning framework. Our algorithm significantly improves load profiling by tackling challenges related to nonlinear decision boundaries at the autoencoder bottleneck. Unlike traditional approaches, we propose separating the optimization of reconstruction and cluster loss, bridging the gap between clustering quality and reconstruction efficiency. We rigorously analyze the performance of our approach by comparing classical and state-of-the-art algorithms using two real-world smart meter data. We provide a comprehensive comparative analysis of our method against five common dimension reduction techniques used in high-dimensional clustering. The experimental analysis concludes that the proposed algorithm significantly enhances load profiling more than others, as confirmed through detailed load curve analysis and clustering validity indexes. This comprehensive assessment highlights the effectiveness and versatility of our proposed methodology when compared to others. Moreover, this research advances load profiling in smart grid analytics, providing practical insights for utilities and stakeholders looking to optimize power network operations.

Composite encoder–decoder network for rapid bridge damage assessment using long-term monitoring acceleration data

In recent years, the deployment of structural health monitoring (SHM) systems has become paramount for safeguarding critical infrastructures. Notwithstanding, the development of an unsupervised deep learning framework capable of learning from long-term sensor data remains a critical challenge, particularly in accurately assessing the exact damage location. This study addresses this gap by proposing a novel approach for rapid bridge damage assessment. The proposed method employed a deep overcomplete encoder–decoder network (DOEDN) to reconstruct the acceleration data acquired from each sensor node on the bridge. The reconstruction losses generated by the DOEDN framework are then used as damage-sensitive features. Additionally, a damage indicator based on Gaussian processes is introduced to assess the damage location and evaluate its severity. The performance and sensitivity of the proposed DOEDN framework are evaluated through long-term monitoring acceleration data from a numerical highway bridge model and the well-known full-scale Z24 bridge. Furthermore, comparative assessments against the regular deep undercomplete encoder–decoder network are conducted using metrics including mean absolute error, coefficient of determination ( R2), and mean intersection over union. The results show that the proposed DOEDN framework can reasonably assess the damage location and evaluate its severity across various structural scenarios in the bridge, even in the presence of temperature variations, thus providing a practical and effective solution for bridge health monitoring.

An Unsupervised Deep Learning Framework for Retrospective Gating of Catheter-Based Cardiac Imaging

Motion artifacts are a major challenge in the in vivo application of catheter-based cardiac imaging modalities. Gating is a critical tool for suppressing motion artifacts. Electrocardiogram (ECG) gating requires a trigger device or synchronous ECG recordings for retrospective analysis. Existing retrospective software gating methods extract gating signals through separate steps based on changes in vessel morphology or image features, which require a high computational cost and are prone to error accumulation. In this paper, we report on an end-to-end unsupervised learning framework for retrospective image-based gating (IBG) of catheter-based intracoronary images, named IBG Network. It establishes a direct mapping from a continuously acquired image sequence to a gated subsequence. The network was trained on clinical data sets in an unsupervised manner, addressing the difficulty of obtaining the gold standard in deep learning-based motion suppression techniques. Experimental results of in vivo intravascular ultrasound and optical coherence tomography sequences show that the proposed method has better performance in terms of motion artifact suppression and processing efficiency compared with the state-of-the-art nonlearning signal-based and IBG methods.

Generative model for learning quantum ensemble with optimal transport loss

Generative modeling is an unsupervised machine learning framework, that exhibits strong performance in various machine learning tasks. Recently, we find several quantum versions of generative model, some of which are even proven to have quantum advantage. However, those methods are not directly applicable to construct a generative model for learning a set of quantum states, i.e., ensemble. In this paper, we propose a quantum generative model that can learn quantum ensemble, in an unsupervised machine learning framework. The key idea is to introduce a new loss function calculated based on optimal transport loss, which have been widely used in classical machine learning due to its good properties; e.g., no need to ensure the common support of two ensembles. We then give in-depth analysis on this measure, such as the scaling property of the approximation error. We also demonstrate the generative modeling with the application to quantum anomaly detection problem, that cannot be handled via existing methods. The proposed model paves the way for a wide application such as the health check of quantum devices and efficient initialization of quantum computation.

NN‐mCRE: A modified constitutive relation error framework for unsupervised learning of nonlinear state laws with physics‐augmented neural networks

AbstractThis article proposes a new approach to train physics‐augmented neural networks with observable data to represent mechanical constitutive laws. To train the neural network and learn thermodynamics potentials, the proposed method does not rely on strain‐stress or strain‐free energy pairs but needs only partial strain or displacement measurements inside the structure. The neural network is trained thanks to an unsupervised procedure in which the modified constitutive relation error (mCRE) is minimized. The mCRE functional provides a bias‐aware data assimilation framework with a rich physical sense as the constitutive relation error (CRE) part can be interpreted as a modeling error continuously defined over the structure, and can be used as a prediction quality in the inference phase. This article also extends previous works on the mCRE by introducing a new minimization procedure in the case of nonlinear state laws. As typical structural health monitoring applications may require that the neural networks should be trained online, an important focus is thus made on automatic and adaptive tuning of sensitive hyperparameters (learning rate, weighting between losses, number of epochs and initialization). It is shown that when the training database is rich enough with respect to the loading cases, the proposed method achieves remarkable performance regarding the quality of the learned model, noise robustness, and low sensitivity to user‐defined hyperparameters. The method is evaluated on two test cases: a non‐quadratic potential in the small strain regime with synthetic optic fiber measurements, and a Mooney–Rivlin model in the hyperelastic case with synthetic digital image correlation observations.

Graph-Neural-Network-Based Unsupervised Learning of the Temporal Similarity of Structural Features Observed in Molecular Dynamics Simulations.

Classification of molecular structures is a crucial step in molecular dynamics (MD) simulations to detect various structures and phases within systems. Molecular structures, which are commonly identified using order parameters, were recently identified using machine learning (ML), that is, the ML models acquire structural features using labeled crystals or phases via supervised learning. However, these approaches may not identify unlabeled or unknown structures, such as the imperfect crystal structures observed in nonequilibrium systems and interfaces. In this study, we proposed the use of a novel unsupervised learning framework, denoted temporal self-supervised learning (TSSL), to learn structural features and design their parameters. In TSSL, the ML models learn that the structural similarity is learned via contrastive learning based on minor short-term variations caused by perturbations in MD simulations. This learning framework is applied to a sophisticated architecture of graph neural network models that use bond angle and length data of the neighboring atoms. TSSL successfully classifies water and ice crystals based on high local ordering, and furthermore, it detects imperfect structures typical of interfaces such as the water-ice and ice-vapor interfaces.

3D Shape Contrastive Representation Learning with Adversarial Examples

Current supervised methods for 3D shape representation learning have achieved satisfying performance, yet require extensive human-labeled datasets. Unsupervised learning-based methods provide a viable solution by learning shape representations without using ground truth labels. In this study, we develop a contrastive learning framework for unsupervised representation learning of 3D shapes. Specifically, in order to encourage models to pay more attention to useful information during representation learning, we first introduce a new paradigm for critical points search based on the adversarial mechanism. We extract critical points with a larger impact on the global feature by attacking a pre-trained auto-encoder model, and apply data augmentations on these points to generate adversarial examples. Taking a pair of adversarial examples as inputs, we obtain their intermediate embeddings and global representations of corresponding inputs, which are then transformed into latent spaces by two predictor heads. Finally, we train the proposed model by maximizing the agreements on these latent spaces via Normalized Temperature-scaled Cross Entropy (NT-Xent) loss and a newly designed Cross-layer Normalized Temperature-scaled Cross Entropy (Cross-NT-Xent) loss, where the latter is proposed in this paper to enforce cross-layer feature similarities. The effectiveness, robustness, and transferability of learned representations are validated on three downstream tasks, including object classification, few-shot classification, and shape retrieval. Experiments on three benchmark datasets show that our learned representations achieve better or competitive performance than current state-of-the-art methods in these downstream tasks. Moreover, our model can easily be extended to 3D part segmentation and scene segmentation tasks.

GenURL: A General Framework for Unsupervised Representation Learning.

Unsupervised representation learning (URL) that learns compact embeddings of high-dimensional data without supervision has achieved remarkable progress recently. However, the development of URLs for different requirements is independent, which limits the generalization of the algorithms, especially prohibitive as the number of tasks grows. For example, dimension reduction (DR) methods, t-SNE and UMAP, optimize pairwise data relationships by preserving the global geometric structure, while self-supervised learning, SimCLR and BYOL, focuses on mining the local statistics of instances under specific augmentations. To address this dilemma, we summarize and propose a unified similarity-based URL framework, GenURL, which can adapt to various URL tasks smoothly. In this article, we regard URL tasks as different implicit constraints on the data geometric structure that help to seek optimal low-dimensional representations that boil down to data structural modeling (DSM) and low-dimensional transformation (LDT). Specifically, DSM provides a structure-based submodule to describe the global structures, and LDT learns compact low-dimensional embeddings with given pretext tasks. Moreover, an objective function, general Kullback-Leibler (GKL) divergence, is proposed to connect DSM and LDT naturally. Comprehensive experiments demonstrate that GenURL achieves consistent state-of-the-art performance in self-supervised visual learning, unsupervised knowledge distillation (KD), graph embeddings (GEs), and DR.

Unsupervised Test-Time Adaptation Learning for Effective Hyperspectral Image Super-Resolution With Unknown Degeneration.

Fusing a low-resolution hyperspectral image (HSI) with a high-resolution (HR) multi-spectral image has provided an effective way for HSI super-resolution (SR). The key lies on inferring the posteriori of the latent (i.e., HR) HSI using an appropriate image prior and the likelihood determined by the degeneration between the latent HSI and the observed images. However, in scenarios with complex imaging environments and various imaging scenes, the prior of HSIs can be prohibitively complicated and the degeneration is often unknown, which causes it difficult to accurately infer the posteriori of each latent HSI. To tackle this problem, we present an unsupervised test-time adaptation learning (UTAL) framework for HSI SR under unknown degeneration. Instead of directly modeling the complicated image prior, it first implicitly learns a content-agnostic prior shared across different images through supervisedly pre-training a mutual-guiding fusion module on extensive synthetic data. Then, it adapts the shared prior to those private characteristics in the latent HSI for posteriori inference through unsupervisedly learning a self-guiding adaptation module and a degeneration estimation network on two observed images in the test phase. Such a two-stage learning scheme models the complicated image prior in a divide-and-conquer manner, which eases the modeling difficulty and improves the prior accuracy. Moreover, the unknown degeneration can be estimated properly. Both of these two advantages empower us to accurately infer the posteriori of the latent HSI, thereby increasing the generalization performance in real applications. Additionally, in order to further mitigate the over-fitting in coping with more challenging cases (e.g., degenerations in both spectral and spatial domains are unknown) and speed up, we propose to meta-train UTAL on extensive synthetic SR tasks and solve it using an alternative optimization strategy such that UTAL learns to produce good generalization performance in real challenging cases with a small number of gradient descent steps. To verify the efficacy of UTAL, we evaluate it on HSI SR tasks with different unknown degenerations as well as some other HSI restoration tasks (e.g., compressive sensing), and report strong results superior to that of existing competitors.

GRLC: Graph Representation Learning With Constraints.

Contrastive learning has been successfully applied in unsupervised representation learning. However, the generalization ability of representation learning is limited by the fact that the loss of downstream tasks (e.g., classification) is rarely taken into account while designing contrastive methods. In this article, we propose a new contrastive-based unsupervised graph representation learning (UGRL) framework by 1) maximizing the mutual information (MI) between the semantic information and the structural information of the data and 2) designing three constraints to simultaneously consider the downstream tasks and the representation learning. As a result, our proposed method outputs robust low-dimensional representations. Experimental results on 11 public datasets demonstrate that our proposed method is superior over recent state-of-the-art methods in terms of different downstream tasks. Our code is available at https://github.com/LarryUESTC/GRLC.

Unsupervised Fusion of Misaligned PAT and MRI Images via Mutually Reinforcing Cross-Modality Image Generation and Registration.

Photoacoustic tomography (PAT) and magnetic resonance imaging (MRI) are two advanced imaging techniques widely used in pre-clinical research. PAT has high optical contrast and deep imaging range but poor soft tissue contrast, whereas MRI provides excellent soft tissue information but poor temporal resolution. Despite recent advances in medical image fusion with pre-aligned multimodal data, PAT-MRI image fusion remains challenging due to misaligned images and spatial distortion. To address these issues, we propose an unsupervised multi-stage deep learning framework called PAMRFuse for misaligned PAT and MRI image fusion. PAMRFuse comprises a multimodal to unimodal registration network to accurately align the input PAT-MRI image pairs and a self-attentive fusion network that selects information-rich features for fusion. We employ an end-to-end mutually reinforcing mode in our registration network, which enables joint optimization of cross-modality image generation and registration. To the best of our knowledge, this is the first attempt at information fusion for misaligned PAT and MRI. Qualitative and quantitative experimental results show the excellent performance of our method in fusing PAT-MRI images of small animals captured from commercial imaging systems.

Deep Variation Prior: Joint Image Denoising and Noise Variance Estimation without Clean Data.

With recent deep learning based approaches showing promising results in removing noise from images, the best denoising performance has been reported in a supervised learning setup that requires a large set of paired noisy images and ground truth data for training. The strong data requirement can be mitigated by unsupervised learning techniques, however, accurate modelling of images or noise variances is still crucial for high-quality solutions. The learning problem is ill-posed for unknown noise distributions. This paper investigates the tasks of image denoising and noise variance estimation in a single, joint learning framework. To address the ill-posedness of the problem, we present deep variation prior (DVP), which states that the variation of a properly learnt denoiser with respect to the change of noise satisfies some smoothness properties, as a key criterion for good denoisers. Building upon DVP and under the assumption that the noise is zero mean and pixel-wise independent conditioned on the image, an unsupervised deep learning framework, that simultaneously learns a denoiser and estimates noise variances, is developed. Our method does not require any clean training images or an external step of noise estimation, and instead, approximates the minimum mean squared error denoisers using only a set of noisy images. With the two underlying tasks being considered in a single framework, we allow them to be optimised for each other. The experimental results show a denoising quality comparable to that of supervised learning and accurate noise variance estimates.