Unsupervised Deep Learning Framework Research Articles

AI-NERD: Elucidation of relaxation dynamics beyond equilibrium through AI-informed X-ray photon correlation spectroscopy

Understanding and interpreting dynamics of functional materials in situ is a grand challenge in physics and materials science due to the difficulty of experimentally probing materials at varied length and time scales. X-ray photon correlation spectroscopy (XPCS) is uniquely well-suited for characterizing materials dynamics over wide-ranging time scales. However, spatial and temporal heterogeneity in material behavior can make interpretation of experimental XPCS data difficult. In this work, we have developed an unsupervised deep learning (DL) framework for automated classification of relaxation dynamics from experimental data without requiring any prior physical knowledge of the system. We demonstrate how this method can be used to accelerate exploration of large datasets to identify samples of interest, and we apply this approach to directly correlate microscopic dynamics with macroscopic properties of a model system. Importantly, this DL framework is material and process agnostic, marking a concrete step towards autonomous materials discovery.

Unsupervised deep-learning framework for 5D seismic denoising and interpolation

We develop an unsupervised framework to reconstruct missing data from noisy and incomplete five-dimensional (5D) seismic data. Our method comprises two main components: a deep-learning network and the projection onto the convex sets (POCS) method. The model works iteratively, passing the data between the two components and splitting the data into a group of patches. The developed deep-learning model is inspired by the transformer architecture, wherein the extracted patches are flattened, and an embedded layer (fully connected layer) is used to transform the 1D flattened vector into the feature latent space. Afterward, an attention network is used to highlight the important information within these features, which improves the denoising performance of the deep-learning model. Furthermore, we use several skip connections between the fully connected layers to enhance the learning capability of the network. Next, the output of the POCS algorithm is considered the input of the deep-learning model for the following iteration. The developed model iteratively works in an unsupervised scheme wherein labeled data are not required. A performance comparison with benchmark methods using several synthetic and field examples shows that our method outperforms the traditional methods.

SUGAR: Spherical ultrafast graph attention framework for cortical surface registration

Cortical surface registration plays a crucial role in aligning cortical functional and anatomical features across individuals. However, conventional registration algorithms are computationally inefficient. Recently, learning-based registration algorithms have emerged as a promising solution, significantly improving processing efficiency. Nonetheless, there remains a gap in the development of a learning-based method that exceeds the state-of-the-art conventional methods simultaneously in computational efficiency, registration accuracy, and distortion control, despite the theoretically greater representational capabilities of deep learning approaches. To address the challenge, we present SUGAR, a unified unsupervised deep-learning framework for both rigid and non-rigid registration. SUGAR incorporates a U-Net-based spherical graph attention network and leverages the Euler angle representation for deformation. In addition to the similarity loss, we introduce fold and multiple distortion losses to preserve topology and minimize various types of distortions. Furthermore, we propose a data augmentation strategy specifically tailored for spherical surface registration to enhance the registration performance. Through extensive evaluation involving over 10,000 scans from 7 diverse datasets, we showed that our framework exhibits comparable or superior registration performance in accuracy, distortion, and test-retest reliability compared to conventional and learning-based methods. Additionally, SUGAR achieves remarkable sub-second processing times, offering a notable speed-up of approximately 12,000 times in registering 9,000 subjects from the UK Biobank dataset in just 32 min. This combination of high registration performance and accelerated processing time may greatly benefit large-scale neuroimaging studies.

Unsupervised deep learning framework for temperature-compensated damage assessment using ultrasonic guided waves on edge device

Fueled by the rapid development of machine learning (ML) and greater access to cloud computing and graphics processing units, various deep learning based models have been proposed for improving performance of ultrasonic guided wave structural health monitoring (GW-SHM) systems, especially to counter complexity and heterogeneity in data due to varying environmental factors (e.g., temperature) and types of damages. Such models typically comprise of millions of trainable parameters, and therefore add to cost of deployment due to requirements of cloud connectivity and processing, thus limiting the scale of deployment of GW-SHM. In this work, we propose an alternative solution that leverages TinyML framework for development of light-weight ML models that could be directly deployed on embedded edge devices. The utility of our solution is illustrated by presenting an unsupervised learning framework for damage detection in honeycomb composite sandwich structure with disbond and delamination type of damages, validated using data generated by finite element simulations and experiments performed at various temperatures in the range 0–90 °C. We demonstrate a fully-integrated solution using a Xilinx Artix-7 FPGA for data acquisition and control, and edge-inference of damage. Despite the limited number of features, the lightweight model shows reasonably high accuracy, thereby enabling detection of small size defects with improved sensitivity on an edge device for online GW-SHM.

Composite encoder–decoder network for rapid bridge damage assessment using long-term monitoring acceleration data

In recent years, the deployment of structural health monitoring (SHM) systems has become paramount for safeguarding critical infrastructures. Notwithstanding, the development of an unsupervised deep learning framework capable of learning from long-term sensor data remains a critical challenge, particularly in accurately assessing the exact damage location. This study addresses this gap by proposing a novel approach for rapid bridge damage assessment. The proposed method employed a deep overcomplete encoder–decoder network (DOEDN) to reconstruct the acceleration data acquired from each sensor node on the bridge. The reconstruction losses generated by the DOEDN framework are then used as damage-sensitive features. Additionally, a damage indicator based on Gaussian processes is introduced to assess the damage location and evaluate its severity. The performance and sensitivity of the proposed DOEDN framework are evaluated through long-term monitoring acceleration data from a numerical highway bridge model and the well-known full-scale Z24 bridge. Furthermore, comparative assessments against the regular deep undercomplete encoder–decoder network are conducted using metrics including mean absolute error, coefficient of determination ( R2), and mean intersection over union. The results show that the proposed DOEDN framework can reasonably assess the damage location and evaluate its severity across various structural scenarios in the bridge, even in the presence of temperature variations, thus providing a practical and effective solution for bridge health monitoring.

An Unsupervised Deep Learning Framework for Retrospective Gating of Catheter-Based Cardiac Imaging

Motion artifacts are a major challenge in the in vivo application of catheter-based cardiac imaging modalities. Gating is a critical tool for suppressing motion artifacts. Electrocardiogram (ECG) gating requires a trigger device or synchronous ECG recordings for retrospective analysis. Existing retrospective software gating methods extract gating signals through separate steps based on changes in vessel morphology or image features, which require a high computational cost and are prone to error accumulation. In this paper, we report on an end-to-end unsupervised learning framework for retrospective image-based gating (IBG) of catheter-based intracoronary images, named IBG Network. It establishes a direct mapping from a continuously acquired image sequence to a gated subsequence. The network was trained on clinical data sets in an unsupervised manner, addressing the difficulty of obtaining the gold standard in deep learning-based motion suppression techniques. Experimental results of in vivo intravascular ultrasound and optical coherence tomography sequences show that the proposed method has better performance in terms of motion artifact suppression and processing efficiency compared with the state-of-the-art nonlearning signal-based and IBG methods.

Deep Variation Prior: Joint Image Denoising and Noise Variance Estimation without Clean Data.

With recent deep learning based approaches showing promising results in removing noise from images, the best denoising performance has been reported in a supervised learning setup that requires a large set of paired noisy images and ground truth data for training. The strong data requirement can be mitigated by unsupervised learning techniques, however, accurate modelling of images or noise variances is still crucial for high-quality solutions. The learning problem is ill-posed for unknown noise distributions. This paper investigates the tasks of image denoising and noise variance estimation in a single, joint learning framework. To address the ill-posedness of the problem, we present deep variation prior (DVP), which states that the variation of a properly learnt denoiser with respect to the change of noise satisfies some smoothness properties, as a key criterion for good denoisers. Building upon DVP and under the assumption that the noise is zero mean and pixel-wise independent conditioned on the image, an unsupervised deep learning framework, that simultaneously learns a denoiser and estimates noise variances, is developed. Our method does not require any clean training images or an external step of noise estimation, and instead, approximates the minimum mean squared error denoisers using only a set of noisy images. With the two underlying tasks being considered in a single framework, we allow them to be optimised for each other. The experimental results show a denoising quality comparable to that of supervised learning and accurate noise variance estimates.

Sensor anomaly detection for small modular reactors utilizing improved autoencoder

The anomaly detection of widely distributed sensors in small modular reactors (SMRs) is critical to the safe operation of systems. Existing methods of sensor anomaly detection for transient conditions are less studied and have some shortcomings, necessitating further research in this context. Therefore, this paper presents an unsupervised deep-learning framework for a gated recurrent autoencoder with the Luong attentional mechanism and the residual connection, which is a multiple-input and multiple-output model. A single model can monitor SMR systems equipped with many sensors, and the training of the model does not require an anomaly label. The general threshold-setting method of the autoencoder for anomaly detection is also improved. The Bayesian estimation algorithm sets a dynamic threshold value in the residual evaluation stage. Finally, simulation tests are performed on experimental data sets, and the residual series obtained from the output of this framework are examined for anomalous sensors using Bayesian estimation methods. The mean squared error (MSE), that is, the average squared difference between the reconstructed value and the actual standard value, is used as the evaluation index. The experimental results demonstrate that the MSE of the improved autoencoder is smaller than that of the unimproved model in both transient and steady-state conditions. In addition, the dynamic threshold method reduces the false alarm rate of anomaly detection.

An unsupervised deep learning framework for large-scale lung CT deformable image registration

Accurate lung CT deformable image registration is especially useful in many medical image analyzing domains. In this paper, we present a novel unsupervised deep learning framework to speed up registration processing with high accuracy. Our approach consists of a convolutional neural network (CNN) model with frequent connections between layers for extracting robust image features, and of well-designed pre-processing and post-processing techniques to handle with large images without losing the precision. Additionally, during training stage, the local cross coefficient (LCC) and L2-norm for gradients of dense displacement fields (DDF) are adopted to form loss function in the model. Experiments on a large-scale lung CT dataset with each image size of over 400 × 400 × 350 show that our method achieves the best performances on Dice score of 0.9245 and mean squared error (MSE) of 0.0046 compared with some traditional and learning-based methods. Besides, our model has been proved to be robust for various deformations. Above all, our method is several orders of magnitude faster than the state-of-the-art non-learning-based algorithms.

OSC-CO2: coattention and cosegmentation framework for plant state change with multiple features.

Cosegmentation and coattention are extensions of traditional segmentation methods aimed at detecting a common object (or objects) in a group of images. Current cosegmentation and coattention methods are ineffective for objects, such as plants, that change their morphological state while being captured in different modalities and views. The Object State Change using Coattention-Cosegmentation (OSC-CO2) is an end-to-end unsupervised deep-learning framework that enhances traditional segmentation techniques, processing, analyzing, selecting, and combining suitable segmentation results that may contain most of our target object's pixels, and then displaying a final segmented image. The framework leverages coattention-based convolutional neural networks (CNNs) and cosegmentation-based dense Conditional Random Fields (CRFs) to address segmentation accuracy in high-dimensional plant imagery with evolving plant objects. The efficacy of OSC-CO2 is demonstrated using plant growth sequences imaged with infrared, visible, and fluorescence cameras in multiple views using a remote sensing, high-throughput phenotyping platform, and is evaluated using Jaccard index and precision measures. We also introduce CosegPP+, a dataset that is structured and can provide quantitative information on the efficacy of our framework. Results show that OSC-CO2 out performed state-of-the art segmentation and cosegmentation methods by improving segementation accuracy by 3% to 45%.

Super-resolution biomedical imaging via reference-free statistical implicit neural representation

Objective. Supervised deep learning for image super-resolution (SR) has limitations in biomedical imaging due to the lack of large amounts of low- and high-resolution image pairs for model training. In this work, we propose a reference-free statistical implicit neural representation (INR) framework, which needs only a single or a few observed low-resolution (LR) image(s), to generate high-quality SR images. Approach. The framework models the statistics of the observed LR images via maximum likelihood estimation and trains the INR network to represent the latent high-resolution (HR) image as a continuous function in the spatial domain. The INR network is constructed as a coordinate-based multi-layer perceptron, whose inputs are image spatial coordinates and outputs are corresponding pixel intensities. The trained INR not only constrains functional smoothness but also allows an arbitrary scale in SR imaging. Main results. We demonstrate the efficacy of the proposed framework on various biomedical images, including computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, and ultrasound images, across different SR magnification scales of 2×, 4×, and 8×. A limited number of LR images were used for each of the SR imaging tasks to show the potential of the proposed statistical INR framework. Significance. The proposed method provides an urgently needed unsupervised deep learning framework for numerous biomedical SR applications that lack HR reference images.

Unsupervised-learning-based calibration method in microscopic fringe projection profilometry.

Microscopic fringe projection profilometry (MFPP) technology is widely used in 3D measurement. The measurement precision performed by the MFPP system is closely related to the calibration accuracy. However, owing to the shallow depth of field, calibration in MFPP is frequently influenced by low-quality target images, which would generate inaccurate features and calibration parameter estimates. To alleviate the problem, this paper proposes an unsupervised-learning-based calibration robust to defocus and noise, which could effectively enhance the image quality and increase calibration accuracy. In this method, first, an unsupervised image deblurring network (UIDNet) is developed to recover a sharp target image from the deteriorated one. Free from capturing strictly paired images by a specific vision system or generating the dataset by simulation, the unsupervised deep learning framework can learn more accurate features from the multi-quality target dataset of convenient image acquisition. Second, multi-perceptual loss and Fourier frequency loss are introduced into the UIDNet to improve the training performance. Third, a robust calibration compensation strategy based on 2D discrete Fourier transform is also developed to evaluate the image quality and improve the detection accuracy of the reference feature centers for fine calibration. The relevant experiments demonstrate that the proposed calibration method can achieve superior performance in terms of calibration accuracy and measurement precision.

Work Together: Correlation-Identity Reconstruction Hashing for Unsupervised Cross-Modal Retrieval

Unsupervised cross-modal hashing has attracted considerable attention to support large-scale cross-modal retrieval. Although promising progresses have been made so far, existing methods still suffer from limited capability on excavating and preserving the intrinsic multi-modal semantics. In this paper, we propose a Correlation-Identity Reconstruction Hashing (CIRH) method to alleviate this challenging problem. We develop a new unsupervised deep cross-modal hash learning framework to model and preserve the heterogeneous multi-modal correlation semantics into both hash codes and functions, and simultaneously, we involve both the hash codes and functions with the descriptive identity semantics. Specifically, we construct a multi-modal collaborated graph to model the heterogeneous multi-modal correlations, and jointly perform the intra-modal and cross-modal semantic aggregation on homogeneous and heterogeneous graph networks to generate a multi-modal complementary representation with correlation reconstruction. Furthermore, an identity semantic reconstruction process is designed to involve the generated representation with identity semantics by reconstructing the input modality representations. Finally, we propose a correlation-identity consistent hash function learning strategy to transfer the modelled multi-modal semantics into the neural networks of modality-specific deep hash functions. Experiments demonstrate the superior performance of the proposed method on both retrieval accuracy and efficiency. We provide our source codes and experimental datasets at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/XizeWu/CIRH</uri> .

Unsupervised deep learning framework for data-driven gating in positron emission tomography.

Physiological motion, such as respiratory motion, has become a limiting factor in the spatial resolution of positron emission tomography (PET) imaging as the resolution of PET detectors continue to improve. Motion-induced misregistration between PET and CT images can also cause attenuation correction artifacts. Respiratory gating can be used to freeze the motion and to reduce motion induced artifacts. In this study, we propose a robust data-driven approach using an unsupervised deep clustering network that employs an autoencoder (AE) to extract latent features for respiratory gating. We first divide list-mode PET data into short-time frames. The short-time frame images are reconstructed without attenuation, scatter, or randoms correction to avoid attenuation mismatch artifacts and to reduce image reconstruction time. The deep AE is then trained using reconstructed short-time frame images to extract latent features for respiratory gating. No additional data are required for the AE training. K-means clustering is subsequently used to perform respiratory gating based on the latent features extracted by the deep AE. The effectiveness of our proposed Deep Clustering method was evaluated using physical phantom and real patient datasets. The performance was compared against phase gating based on an external signal (External) and image based principal component analysis (PCA) with K-means clustering (Image PCA). The proposed method produced gated images with higher contrast and sharper myocardium boundaries than those obtained using the External gating method and Image PCA. Quantitatively, the gated images generated by the proposed Deep Clustering method showed larger center of mass (COM) displacement and higher lesion contrast than those obtained using the other two methods. The effectiveness of our proposed method was validated using physical phantom and real patient data. The results showed our proposed framework could provide superior gating than the conventional External method and Image PCA.

Style transfer generative adversarial networks to harmonize multisite MRI to a single reference image to avoid overcorrection.

Recent work within neuroimaging consortia have aimed to identify reproducible, and often subtle, brain signatures of psychiatric or neurological conditions. To allow for high-powered brain imaging analyses, it is often necessary to pool MR images that were acquired with different protocols across multiple scanners. Current retrospective harmonization techniques have shown promise in removing site-related image variation. However, most statistical approaches may over-correct for technical, scanning-related, variation as they cannot distinguish between confounded image-acquisition based variability and site-related population variability. Such statistical methods often require that datasets contain subjects or patient groups with similar clinical or demographic information to isolate the acquisition-based variability. To overcome this limitation, we consider site-related magnetic resonance (MR) imaging harmonization as a style transfer problem rather than a domain transfer problem. Using a fully unsupervised deep-learning framework based on a generative adversarial network (GAN), we show that MR images can be harmonized by inserting the style information encoded from a single reference image, without knowing their site/scanner labels a priori. We trained our model using data from five large-scale multisite datasets with varied demographics. Results demonstrated that our style-encoding model can harmonize MR images, and match intensity profiles, without relying on traveling subjects. This model also avoids the need to control for clinical, diagnostic, or demographic information. We highlight the effectiveness of our method for clinical research by comparing extracted cortical and subcortical features, brain-age estimates, and case-control effect sizes before and after the harmonization. We showed that our harmonization removed the site-related variances, while preserving the anatomical information and clinical meaningful patterns. We further demonstrated that with a diverse training set, our method successfully harmonized MR images collected from unseen scanners and protocols, suggesting a promising tool for ongoing collaborative studies. Source code is released in USC-IGC/style_transfer_harmonization (github.com).

Unsupervised deep learning framework for ultrasonic-based distributed damage detection in concrete: integration of a deep auto-encoder and Isolation Forest for anomaly detection

This study presented an unsupervised anomaly detection-based framework for distributed damage detection in concrete using ultrasonic response signals. A deep fully connected auto-encoder was employed to reconstruct the ultrasonic response signals. This model was trained on the intact specimen’s responses. The auto-encoder demonstrated a relatively high prediction error encountering the damaged specimen’s responses. Two time-domain features (mean squared error and reconstructed-to-original signal ratio) and one frequency-domain feature (fundamental amplitude ratio) were defined to measure the reconstruction error of the auto-encoder (the damage-sensitive features). Finally, the Isolation Forest algorithm was implemented for anomaly (damage) detection. The beauty of this framework is that it requires a few numbers of data only from the intact specimen for training the auto-encoder and collecting the binary decision trees of the Isolation Forest. The framework was successfully implemented for damage detection in five geopolymer concrete specimens with different mix proportions. Using all three introduced damage-sensitive features, the framework demonstrated an average prediction accuracy of 95.0% and 93.0% for damaged and intact stages, respectively.

Deformable registration of multimodal retinal images using a weakly supervised deep learning approach

There are different retinal vascular imaging modalities widely used in clinical practice to diagnose different retinal pathologies. The joint analysis of these multimodal images is of increasing interest since each of them provides common and complementary visual information. However, if we want to facilitate the comparison of two images, obtained with different techniques and containing the same retinal region of interest, it will be necessary to make a previous registration of both images. Here, we present a weakly supervised deep learning methodology for robust deformable registration of multimodal retinal images, which is applied to implement a method for the registration of fluorescein angiography (FA) and optical coherence tomography angiography (OCTA) images. This methodology is strongly inspired by VoxelMorph, a general unsupervised deep learning framework of the state of the art for deformable registration of unimodal medical images. The method was evaluated in a public dataset with 172 pairs of FA and superficial plexus OCTA images. The degree of alignment of the common information (blood vessels) and preservation of the non-common information (image background) in the transformed image were measured using the Dice coefficient (DC) and zero-normalized cross-correlation (ZNCC), respectively. The average values of the mentioned metrics, including the standard deviations, were DC = 0.72 ± 0.10 and ZNCC = 0.82 ± 0.04. The time required to obtain each pair of registered images was 0.12 s. These results outperform rigid and deformable registration methods with which our method was compared.

Transfer learning framework for low-dose CT reconstruction based on marginal distribution adaptation in multi-scale.

With the increasing use of CT in clinical practice, limiting CT radiation exposure to reduce potential cancer risks has become one of the important directions of medical imaging research. As the dose decreases, the reconstructed CT image will be severely degraded by projection noise. As an important method of image processing, supervised deep learning has been widely used in the restoration of low-dose CT(LDCT) in recent years. However, the normal-dose CT (NDCT) corresponding to a specific LDCT (it is regareded as the label of the LDCT, which is necessary for supervised learning) are very difficult to obtain. So that the application of supervised learning methods in LDCT reconstruction is limited. It is necessary to construct a unsupervised deep learning framework for LDCT reconstruction that does not depend on paired LDCT-NDCTdatasets. We presented an unsupervised learning framework for the transferring from the identity mapping to the low-dose recontruction task, called Marginal Distribution Adaptation in Multi-scale (MDAM). For NDCTs as source domain data, MDAM is an identity map with two parts: firstly, it establishes a dimensionality reduction mapping, which can obtain the same feature distribution from NDCTs and LDCTs; and then NDCTs is retrieved by reconstructing the image overview and details from the low-dimensional features. For the purpose of the feature transfer between source domain and target domain (LDCTs), we introduce the multi-scale feature extraction in the MDAM, and then eliminate differences in probability distributions of these multi-scale features between NDCTs and LDCTs through wavelet decomposition and domain adaptationlearning. Image quality evaluation metrics and subjective quality scores show that, as an unsupervised method, the performance of the MDAM approaches or even surpasses some state-of-the-art supervised methods. Especially, MDAM has been favorably evaluated in terms of noise suppression, structural preservation, and lesiondetection. We demonstrated that, the MDAM framework can reconstruct corresponding NDCTs from LDCTs with high accuracy, and without relying on any labeles. Moreover, it is more suitable for clinical application compared with supervised learningmethods. This article is protected by copyright. All rights reserved.

Intrinsic Decomposition Model-Guided Two-Stream Coupled Autoencoder for Unsupervised Hyperspectral Image Change Detection

Hyperspectral image change detection (HSI-CD) is one of the main research topics in remote sensing. Theoretically, the ground objects can be considered to have changed when their spectral features behave differently. However, in practical scenarios, this presupposition does not hold as the collected features are affected by many factors, such as illumination conditions, atmospheric effects, and topographic changes. Inspired by the intrinsic image decomposition, we propose a novel unsupervised deep learning framework based on two-stream coupled autoencoder (TSCA) to cope with bi-temporal co-registered HSI-CD. The network consists of two symmetric encoders and a decoder, which can jointly decompose bi-temporal images into abundance coefficients corresponding to the same set of spectral bases. As our network separates the component of spectral variation from multiple images, the extracted abundance features with inherent properties of materials can provide better performance for change detection. Moreover, to enforce alignment of the feature space, a reasonable consistency loss is devised to constrain the solution space, by cross-reconstruction in both branches. Experimental results demonstrate its superiority over the recently developed state of the arts.

Unsupervised deep learning for molecular dynamics simulations: a novel analysis of protein-ligand interactions in SARS-CoV-2 Mpro.

Molecular dynamics (MD) simulations, which are central to drug discovery, offer detailed insights into protein-ligand interactions. However, analyzing large MD datasets remains a challenge. Current machine-learning solutions are predominantly supervised and have data labelling and standardisation issues. In this study, we adopted an unsupervised deep-learning framework, previously benchmarked for rigid proteins, to study the more flexible SARS-CoV-2 main protease (Mpro). We ran MD simulations of Mpro with various ligands and refined the data by focusing on binding-site residues and time frames in stable protein conformations. The optimal descriptor chosen was the distance between the residues and the center of the binding pocket. Using this approach, a local dynamic ensemble was generated and fed into our neural network to compute Wasserstein distances across system pairs, revealing ligand-induced conformational differences in Mpro. Dimensionality reduction yielded an embedding map that correlated ligand-induced dynamics and binding affinity. Notably, the high-affinity compounds showed pronounced effects on the protein's conformations. We also identified the key residues that contributed to these differences. Our findings emphasize the potential of combining unsupervised deep learning with MD simulations to extract valuable information and accelerate drug discovery.