Unsupervised Deep Learning Research Articles

Segment-Based Unsupervised Deep Learning for Human Activity Recognition using Accelerometer Data and SBOA based Channel Attention Networks

The deep learning community has increasingly focused on the critical challenges of human activity segmentation and detection based on sensors, which have numerous real-world applications. In most prior efforts, activity segmentation and recognition have been treated as separate processes, relying on pre-segmented sensor streams. This research proposes an unsupervised deep learning approach for Human Activity Recognition (HAR) that is segment-based, with an emphasis on activity continuity. The approach integrates segment-based SimCLR with Segment Feature Decorrelation (SDFD) and a new framework that leverages pairs of segment data for contrastive learning of visual representations. Furthermore, the Secretary Bird Optimization Algorithm (SBOA) and Channel Attention with Spatial Attention Network (CASANet) are utilized to enhance the performance of sensor-based human activity detection. CASANet effectively extracts key features and spatial dependencies in sensor data, while SBOA optimizes the model for greater accuracy and generalization. Evaluations on two publicly available datasets—Mhealth and PAMAP2—demonstrated an average F1 score of 98%, highlighting the approach’s efficacy in improving activity recognition performance.

STAD: Ship trajectory anomaly detection in ocean with dynamic pattern clustering

With the continuous increase of maritime traffic, developing an efficient and accurate model for ship trajectory anomaly detection has become crucial for ensuring maritime transportation safety. The high complexity and variability of the marine environment lead to diverse ship trajectory patterns, making it challenging to learn effective trajectory representations for accurately identifying anomalies. We thus proposed an unsupervised deep learning model called STAD for ship trajectory anomaly detection in ocean to address this challenge. Concretely, STAD leverages offset reconstruction-based representation learning and a deep Gaussian Mixture Model (GMM) estimation network to learn the underlying complex clustering patterns of ship trajectories and utilize the learned patterns to enhance trajectory anomaly detection. Extensive experiments on multiple AIS datasets indicate that our model significantly outperforms existing methods in detecting multiple representative types of ship trajectory anomalies, including shift deviation, abnormal heading, and abnormal speeding. This study could help closely monitor the status of ship movement and detect abnormal behaviors in advance, thus benefiting maritime safety.

Enhanced Collaborative Filtering: Combining Autoencoder and Opposite User Inference to Solve Sparsity and Gray Sheep Issues

In recent years, the study of recommendation systems has become crucial, capturing the interest of scientists and academics worldwide. Music, books, movies, news, conferences, courses, and learning materials are some examples of using the recommender system. Among the various strategies employed, collaborative filtering stands out as one of the most common and effective approaches. This method identifies similar active users to make item recommendations. However, collaborative filtering has two major challenges: sparsity and gray sheep. Inspired by the remarkable success of deep learning across a multitude of application areas, we have integrated deep learning techniques into our proposed method to effectively address the aforementioned challenges. In this paper, we present a new method called Enriched_AE, focused on autoencoder, a well-regarded unsupervised deep learning technique renowned for its superior ability in data dimensionality reduction, feature extraction, and data reconstruction, with an augmented rating matrix. This matrix not only includes real users but also incorporates virtual users inferred from opposing ratings given by real users. By doing so, we aim to enhance the accuracy of predictions, thus enabling more effective recommendation generation. Through experimental analysis of the MovieLens 100K dataset, we observe that our method achieves notable reductions in both RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error), underscoring its superiority over the state-of-the-art collaborative filtering models.

Noise attenuation in distributed acoustic sensing data using a guided unsupervised deep learning network

Distributed acoustic sensing (DAS) is a promising technology introducing a new paradigm in the acquisition of high-resolution seismic data. However, DAS data often show weak signals compared with the background noise, especially in challenging installation environments. In this study, we develop a new approach to denoise DAS data that leverages an unsupervised deep learning (DL) model, eliminating the need for labeled training data. The input DAS data undergo band-pass filtering to eliminate high-frequency content. Subsequently, a continuous wavelet transform (CWT) is performed, and the finest scale is used to guide the DL model in reconstructing the DAS signal. First, we extract 2D patches from the band-pass filtered data and the CWT scale of the data. Then, these patches are converted using an unrolling mechanism into 1D vectors to form the input of the DL model. A self-attention layer is included in each layer to extract the spatial relation between the band-pass filtered data and the CWT scale. Through an iterative process, the DL model tunes its parameters to suppress DAS noise, with the band-pass filtered data serving as the target for the network. The denoising performance of our framework is validated using field examples from the San Andreas Fault Observatory at Depth and Frontier Observatory for Research in Geothermal Energy data sets, where the data are recorded by a fiber-optic cable. Comparative analyses against three benchmark methods reveal the robust denoising performance of our framework.

Unsupervised deep denoising for four-dimensional scanning transmission electron microscopy

By simultaneously achieving high spatial and angular sampling resolution, four dimensional scanning transmission electron microscopy (4D STEM) is enabling analysis techniques that provide great insight into the atomic structure of materials. Applying these techniques to scientifically and technologically significant beam-sensitive materials remains challenging because the low doses needed to minimise beam damage lead to noisy data. We demonstrate an unsupervised deep learning model that leverages the continuity and coupling between the probe position and the electron scattering distribution to denoise 4D STEM data. By restricting the network complexity it can learn the geometric flow present but not the noise. Through experimental and simulated case studies, we demonstrate that denoising as a preprocessing step enables 4D STEM analysis techniques to succeed at lower doses, broadening the range of materials that can be studied using these powerful structure characterization techniques.

A hybrid of supervised and unsupervised deep learning models for multi-vendor kernel conversion of chest CT images

ObjectiveWhen reconstructing a computed tomography (CT) volume, different filter kernels can be used to highlight different structures depending on the medical purpose. The aim of this study was to perform CT conversion for intra-/inter-vendor kernel conversion while preserving image quality. Materials and methodsThis study used CT scans from 632 patients who underwent contrast-enhanced chest CT on either a GE or Siemens scanner. Raw data from each CT scan was reconstructed with Standard and Chest kernels of GE or B10f, B30f, B50f, and B70f kernels of Siemens. In intra-vendor, all images reconstructed with one kernel are paired with another kernel, so the U-Net based supervised method was applied. In the case of inter-vendor where the input and target kernels have each different vendor, Siemens' B30f and GE's Standard kernel were trained through unsupervised image-to-image translation using contrastive learning. ResultsIn the intra-vendor, quantitative evaluation of the image quality of our model showed reasonable performance on the internal test set (structural similarity index measure (SSIM) > 0.96, peak signal-to-noise ratio (PSNR) > 42.55) compared with the SR-block model (SSIM > 0.93, PSNR > 42.92). In the 6-class classification to evaluate the inter-vendor conversion performance, similar accuracy was shown in the converted image (0.977) compared to the original image (0.998). ConclusionsIn this study, we developed a network that can translate a given CT image into a target kernel among multi-vendors. Our model showed clinically acceptable quality in quantitative and qualitative evaluations, including image quality metrics.

Natural gas leakage detection from offshore platform by OGI camera and unsupervised deep learning

Natural gas leak from offshore platform poses a potential to cause explosion disaster and bring significant causalities and economic losses. Existing deep learning-based leak detection approaches are limited by the requirement of a large number of labeled leak datasets, and also has worse performance in the complex and changeable marine environment. This study proposes a detection approach of natural gas leakage from offshore platform by integrating optical gas imaging (OGI) camera and unsupervised deep probability learning. In this approach, unsupervised deep learning is applied to learn the changeable infrared features of offshore platform, and variational Bayesian inference is integrated to provide the larger epistemic uncertainty contour corresponding to the infrared natural gas plume. An epistemic uncertainty-based detection score is proposed as the detection criterion to improve the accuracy of natural gas plume detection and localization. An OGI imaging experiment of natural gas leak from offshore platform is conducted to construct the benchmark dataset. With such datasets, two pre-defined parameters, namely Monte Carlo sampling number m = 100 and dropout probability p=0.1 are determined to guarantee the detection accuracy and efficiency. Comparison between the proposed approach and prevalent unsupervised deep learning approach is also conducted. The results demonstrate that our proposed approach has the higher detection accuracy with AUC = 0.9753, as well as the real-time capability with inference time of 4s/frame. Overall, our proposed approach provides a more accurate and generalized approach of natural gas detection for safety monitoring and detection management of offshore platforms.

Unpaired data training enables super-resolution confocal microscopy from low-resolution acquisitions.

Supervised deep-learning models have enabled super-resolution imaging in several microscopic imaging modalities, increasing the spatial lateral bandwidth of the original input images beyond the diffraction limit. Despite their success, their practical application poses several challenges in terms of the amount of training data and its quality, requiring the experimental acquisition of large, paired databases to generate an accurate generalized model whose performance remains invariant to unseen data. Cycle-consistent generative adversarial networks (cycleGANs) are unsupervised models for image-to-image translation tasks that are trained on unpaired datasets. This paper introduces a cycleGAN framework specifically designed to increase the lateral resolution limit in confocal microscopy by training a cycleGAN model using low- and high-resolution unpaired confocal images of human glioblastoma cells. Training and testing performances of the cycleGAN model have been assessed by measuring specific metrics such as background standard deviation, peak-to-noise ratio, and a customized frequency content measure. Our cycleGAN model has been evaluated in terms of image fidelity and resolution improvement using a paired dataset, showing superior performance than other reported methods. This work highlights the efficacy and promise of cycleGAN models in tackling super-resolution microscopic imaging without paired training, paving the path for turning home-built low-resolution microscopic systems into low-cost super-resolution instruments by means of unsupervised deep learning.

Enhancing Unsupervised Anomaly Detection With Score-Guided Network.

Anomaly detection plays a crucial role in various real-world applications, including healthcare and finance systems. Owing to the limited number of anomaly labels in these complex systems, unsupervised anomaly detection methods have attracted great attention in recent years. Two major challenges faced by the existing unsupervised methods are as follows: 1) distinguishing between normal and abnormal data when they are highly mixed together and 2) defining an effective metric to maximize the gap between normal and abnormal data in a hypothesis space, which is built by a representation learner. To that end, this work proposes a novel scoring network with a score-guided regularization to learn and enlarge the anomaly score disparities between normal and abnormal data, enhancing the capability of anomaly detection. With such score-guided strategy, the representation learner can gradually learn more informative representation during the model training stage, especially for the samples in the transition field. Moreover, the scoring network can be incorporated into most of the deep unsupervised representation learning (URL)-based anomaly detection models and enhances them as a plug-in component. We next integrate the scoring network into an autoencoder (AE) and four state-of-the-art models to demonstrate the effectiveness and transferability of the design. These score-guided models are collectively called SG-Models. Extensive experiments on both synthetic and real-world datasets confirm the state-of-the-art performance of SG-Models.

Sifting the debris: Patterns in the SNR population with unsupervised ML methods

Context. Supernova remnants (SNRs) carry vast amounts of mechanical and radiative energy that heavily influence the structural, dynamical, and chemical evolution of galaxies. To this day, more than 300 SNRs have been discovered in the Milky Way, exhibiting a wide variety of observational features. However, existing classification schemes are mainly based on their radio morphology. Aims. In this work, we introduce a novel unsupervised deep learning pipeline to analyse a representative subsample of the Galactic SNR population (~50% of the total) with the aim of finding a connection between their multi-wavelength features and their physical properties. Methods. The pipeline involves two stages: (1) a representation learning stage, consisting of a convolutional autoencoder that feeds on imagery from infrared and radio continuum surveys (WISE 22 μm, Hi-GAL 70 μm and SMGPS 30 cm) and produces a compact representation in a lower-dimensionality latent space; and (2) a clustering stage that seeks meaningful clusters in the latent space that can be linked to the physical properties of the SNRs and their surroundings. Results. Our results suggest that this approach, when combined with an intermediate uniform manifold approximation and projection (UMAP) reprojection of the autoencoded embeddings into a more clusterable manifold, enables us to find reliable clusters. Despite a large number of sources being classified as outliers, most clusters relate to the presence of distinctive features, such as the distribution of infrared emission, the presence of radio shells and pulsar wind nebulae, and the existence of dust filaments.

An Interference Cancelation and Signal Decoding Method for Non-orthogonal Multiple Access Based on Deep Learning

In wireless communication, spectral efficiency is one of the key performance parameters. The available limited resources must be effectively shared among different applications. Due to the tremendous demand for wireless applications, an effective technique could be used for sharing the resources (time/frequency). Non-orthogonal Multiple Access (NOMA), is a wireless network technique aimed at boosting spectral efficiency. In NOMA, Users in the downlink have received a super-imposed signal from a Base Station (BS) with varying power allocations. Depending on the power allocation, the receiver employs interference cancelation techniques to decode its own data. The effectiveness of interference cancelation hinges on achieving a perfect channel condition, which is challenging to achieve in practical scenarios. This study introduces an unsupervised Deep Learning (DL) approach known as autoencoder, aimed at interference cancelation to ensure signal decoding from super-imposed signal. In contrast to the traditional approach, simulation results of the proposed structure show that the DL-based receiver achieves superior performance in correctly decoding the signal.

Scan-Specific Unsupervised Highly Accelerated Non-Cartesian CEST Imaging Using Implicit Neural Representation and Explicit Sparse Prior.

Chemical exchange saturation transfer (CEST) is a promising magnetic resonance imaging (MRI) technique. CEST imaging usually requires a long scan time, and reducing acquisition time is highly desirable for clinical applications. A novel scan-specific unsupervised deep learning algorithm is proposed to accelerate steady-state pulsed CEST imaging with golden-angle stack-of-stars trajectory using hybrid-feature hash encoding implicit neural representation. Additionally, imaging quality is further improved by using the explicit prior knowledge of low rank and weighted joint sparsity in the spatial and Z-spectral domain of CEST data. In the retrospective acceleration experiment, the proposed method outperforms other state-of-the-art algorithms (TDDIP, LRTES, kt-SLR, NeRP, CRNN, and PBCS) for the in vivo human brain dataset under various acceleration rates. In the prospective acceleration experiment, the proposed algorithm can still obtain results close to the fully-sampled images. The hybrid-feature hash encoding implicit neural representation combined with explicit sparse prior (INRESP) can efficiently accelerate CEST imaging. The proposed algorithm achieves reduced error and improved image quality compared to several state-of-the-art algorithms at relatively high acceleration factors. The superior performance and the training database-free characteristic make the proposed algorithm promising for accelerating CEST imaging in various applications.

Forecasting dominance of SARS-CoV-2 lineages by anomaly detection using deep AutoEncoders.

The COVID-19 pandemic is marked by the successive emergence of new SARS-CoV-2 variants, lineages, and sublineages that outcompete earlier strains, largely due to factors like increased transmissibility and immune escape. We propose DeepAutoCoV, an unsupervised deep learning anomaly detection system, to predict future dominant lineages (FDLs). We define FDLs as viral (sub)lineages that will constitute >10% of all the viral sequences added to the GISAID, a public database supporting viral genetic sequence sharing, in a given week. DeepAutoCoV is trained and validated by assembling global and country-specific data sets from over 16 million Spike protein sequences sampled over a period of ~4years. DeepAutoCoV successfully flags FDLs at very low frequencies (0.01%-3%), with median lead times of 4-17weeks, and predicts FDLs between ~5 and ~25 times better than a baseline approach. For example, the B.1.617.2 vaccine reference strain was flagged as FDL when its frequency was only 0.01%, more than a year before it was considered for an updated COVID-19 vaccine. Furthermore, DeepAutoCoV outputs interpretable results by pinpointing specific mutations potentially linked to increased fitness and may provide significant insights for the optimization of public health 'pre-emptive' intervention strategies.

Unsupervised deep metric learning algorithm for crop disease images based on knowledge distillation networks

Unsupervised deep metric learning algorithm for crop disease images based on knowledge distillation networks

Representing the Information of Multiplayer Online Battle Arena (MOBA) Video Games Using Convolutional Accordion Auto-Encoder (A2E) Enhanced by Attention Mechanisms

In this paper, we propose a representation of the visual information about Multiplayer Online Battle Arena (MOBA) video games using an adapted unsupervised deep learning architecture called Convolutional Accordion Auto-Encoder (Conv_A2E). Our study includes a presentation of current representations of MOBA video game information and why our proposal offers a novel and useful solution to this task. This approach aims to achieve dimensional reduction and refined feature extraction of the visual data. To enhance the model’s performance, we tested several attention mechanisms for computer vision, evaluating algorithms from the channel attention and spatial attention families, and their combination. Through experimentation, we found that the best reconstruction of the visual information with the Conv_A2E was achieved when using a spatial attention mechanism, deformable convolution, as its mean squared error (MSE) during testing was the lowest, reaching a value of 0.003893, which means that its dimensional reduction is the most generalist and representative for this case study. This paper presents one of the first approaches to applying attention mechanisms to the case study of MOBA video games, representing a new horizon of possibilities for research.

Puncturing-Based Resource Allocation for URLLC and eMBB Services via Matching Theory and Unsupervised Deep Learning

Puncturing-Based Resource Allocation for URLLC and eMBB Services via Matching Theory and Unsupervised Deep Learning

Enhanced Deformation Vector Field Generation With Diffusion Models and Mamba‐Based Network for Registration Performance Enhancement

ABSTRACTRecent advancements in deformable image registration (DIR) have seen the emergence of supervised and unsupervised deep learning techniques. However, supervised methods are limited by the quality of deformation vector fields (DVFs), while unsupervised approaches often yield suboptimal results due to their reliance on indirect dissimilarity metrics. Moreover, both methods struggle to effectively model long‐range dependencies. This study proposes a novel DIR method that integrates the advantages of supervised and unsupervised learning and tackle issues related to long‐range dependencies, thereby improving registration results. Specifically, we propose a DVF generation diffusion model to enhance DVFs diversity, which could be used to facilitate the integration of supervised and unsupervised learning approaches. This fusion allows the method to leverage the benefits of both paradigms. Furthermore, a multi‐scale frequency‐weighted denoising module is integrated to enhance DVFs generation quality and improve the registration accuracy. Additionally, we propose a novel MambaReg network that adeptly manages long‐range dependencies, further optimizing registration outcomes. Experimental evaluation of four public data sets demonstrates that our method outperforms several state‐of‐the‐art techniques based on either supervised or unsupervised learning. Qualitative and quantitative comparisons highlight the superior performance of our approach.

Remote sensing framework for geological mapping via stacked autoencoders and clustering

Supervised machine learning methods for geological mapping via remote sensing face limitations due to the scarcity of accurately labelled training data that can be addressed by unsupervised learning, such as dimensionality reduction and clustering. Dimensionality reduction methods have the potential to play a crucial role in improving the accuracy of geological maps. Although conventional dimensionality reduction methods may struggle with nonlinear data, unsupervised deep learning models such as autoencoders can model non-linear relationships. Stacked autoencoders feature multiple interconnected layers to capture hierarchical data representations useful for remote sensing data. We present an unsupervised machine learning-based framework for processing remote sensing data using stacked autoencoders for dimensionality reduction and k-means clustering for mapping geological units. We use Landsat 8, ASTER, and Sentinel-2 datasets to evaluate the framework for geological mapping of the Mutawintji region in Western New South Wales, Australia. We also compare stacked autoencoders with principal component analysis (PCA) and canonical autoencoders. Our results reveal that the framework produces accurate and interpretable geological maps, efficiently discriminating rock units. The results reveal that the combination of stacked autoencoders with Sentinel-2 data yields the best performance accuracy when compared to other combinations. We find that stacked autoencoders enable better extraction of complex and hierarchical representation of the input data when compared to canonical autoencoders and PCA. We also find that the generated maps align with prior geological knowledge of the study area while providing novel insights into geological structures.

Unsupervised adversarial neural network for enhancing vasculature in photoacoustic tomography images using optical coherence tomography angiography

Photoacoustic tomography (PAT) is a powerful imaging modality for visualizing tissue physiology and exogenous contrast agents. However, PAT faces challenges in visualizing deep-seated vascular structures due to light scattering, absorption, and reduced signal intensity with depth. Optical coherence tomography angiography (OCTA) offers high-contrast visualization of vasculature networks, yet its imaging depth is limited to a millimeter scale. Herein, we propose OCPA-Net, a novel unsupervised deep learning method that utilizes the rich vascular feature of OCTA to enhance PAT images. Trained on unpaired OCTA and PAT images, OCPA-Net incorporates a vessel-aware attention module to enhance deep-seated vessel details captured from OCTA. It leverages a domain-adversarial loss function to enforce structural consistency and a novel identity invariant loss to mitigate excessive image content generation. We validate the structural fidelity of OCPA-Net on simulation experiments, and then demonstrate its vascular enhancement performance on in vivo imaging experiments of tumor-bearing mice and contrast-enhanced pregnant mice. The results show the promise of our method for comprehensive vessel-related image analysis in preclinical research applications.

Robust seismic data denoising via self-supervised deep learning

Seismic data denoising is a critical component of seismic data processing, yet effectively removing erratic noise, characterized by its non-Gaussian distribution and high amplitude, remains a substantial challenge for conventional methods and deep-learning (DL) algorithms. Supervised learning frameworks typically outperform others, but they require pairs of noisy data sets alongside corresponding clean ground truth, which is impractical for real-world seismic data sets. In contrast, unsupervised learning (UL) methods, which do not rely on ground truth during training, often fall short in performance when compared with their supervised or traditional denoising counterparts. Moreover, current unsupervised DL methods fail to address the specific challenges posed by erratic seismic noise adequately. This paper introduces a novel zero-shot unsupervised DL framework designed specifically to mitigate random and erratic noise, with a particular emphasis on blended noise. Drawing inspiration from Noise2Noise (N2N) and data augmentation principles, we develop a robust self-supervised denoising network called robust Noiser2Noiser. Our approach eliminates the need for paired noisy and clean data sets as required by supervised methods or paired noisy data sets as in N2N. Instead, our framework relies solely on the original noisy seismic data set. Our methodology generates two independent recorrupted data sets from the original noisy data set, using one as the input and the other as the training target. Subsequently, we use a DL-based denoiser, denoising convolutional neural network, for training purposes. To address various types of random and erratic noise, the original noisy data set is recorrupted with the same noise type. Detailed explanations for generating training input and target data for blended data are provided. We apply our network to synthetic and real marine data examples, demonstrating significantly improved noise attenuation performance compared with traditional denoising methods and state-of-the-art UL Codes are available on https://github.com/Ji-seismic/N2N_deblending .