Unsupervised Deep Learning Approach Research Articles

Segment-Based Unsupervised Deep Learning for Human Activity Recognition using Accelerometer Data and SBOA based Channel Attention Networks

The deep learning community has increasingly focused on the critical challenges of human activity segmentation and detection based on sensors, which have numerous real-world applications. In most prior efforts, activity segmentation and recognition have been treated as separate processes, relying on pre-segmented sensor streams. This research proposes an unsupervised deep learning approach for Human Activity Recognition (HAR) that is segment-based, with an emphasis on activity continuity. The approach integrates segment-based SimCLR with Segment Feature Decorrelation (SDFD) and a new framework that leverages pairs of segment data for contrastive learning of visual representations. Furthermore, the Secretary Bird Optimization Algorithm (SBOA) and Channel Attention with Spatial Attention Network (CASANet) are utilized to enhance the performance of sensor-based human activity detection. CASANet effectively extracts key features and spatial dependencies in sensor data, while SBOA optimizes the model for greater accuracy and generalization. Evaluations on two publicly available datasets—Mhealth and PAMAP2—demonstrated an average F1 score of 98%, highlighting the approach’s efficacy in improving activity recognition performance.

Natural gas leakage detection from offshore platform by OGI camera and unsupervised deep learning

Natural gas leak from offshore platform poses a potential to cause explosion disaster and bring significant causalities and economic losses. Existing deep learning-based leak detection approaches are limited by the requirement of a large number of labeled leak datasets, and also has worse performance in the complex and changeable marine environment. This study proposes a detection approach of natural gas leakage from offshore platform by integrating optical gas imaging (OGI) camera and unsupervised deep probability learning. In this approach, unsupervised deep learning is applied to learn the changeable infrared features of offshore platform, and variational Bayesian inference is integrated to provide the larger epistemic uncertainty contour corresponding to the infrared natural gas plume. An epistemic uncertainty-based detection score is proposed as the detection criterion to improve the accuracy of natural gas plume detection and localization. An OGI imaging experiment of natural gas leak from offshore platform is conducted to construct the benchmark dataset. With such datasets, two pre-defined parameters, namely Monte Carlo sampling number m = 100 and dropout probability p=0.1 are determined to guarantee the detection accuracy and efficiency. Comparison between the proposed approach and prevalent unsupervised deep learning approach is also conducted. The results demonstrate that our proposed approach has the higher detection accuracy with AUC = 0.9753, as well as the real-time capability with inference time of 4s/frame. Overall, our proposed approach provides a more accurate and generalized approach of natural gas detection for safety monitoring and detection management of offshore platforms.

An Interference Cancelation and Signal Decoding Method for Non-orthogonal Multiple Access Based on Deep Learning

In wireless communication, spectral efficiency is one of the key performance parameters. The available limited resources must be effectively shared among different applications. Due to the tremendous demand for wireless applications, an effective technique could be used for sharing the resources (time/frequency). Non-orthogonal Multiple Access (NOMA), is a wireless network technique aimed at boosting spectral efficiency. In NOMA, Users in the downlink have received a super-imposed signal from a Base Station (BS) with varying power allocations. Depending on the power allocation, the receiver employs interference cancelation techniques to decode its own data. The effectiveness of interference cancelation hinges on achieving a perfect channel condition, which is challenging to achieve in practical scenarios. This study introduces an unsupervised Deep Learning (DL) approach known as autoencoder, aimed at interference cancelation to ensure signal decoding from super-imposed signal. In contrast to the traditional approach, simulation results of the proposed structure show that the DL-based receiver achieves superior performance in correctly decoding the signal.

Gene expression clock: an unsupervised deep learning approach for predicting circadian rhythmicity from whole genome expression

Gene expression clock: an unsupervised deep learning approach for predicting circadian rhythmicity from whole genome expression

Application of deep learning for structural health monitoring of a composite overwrapped pressure vessel undergoing cyclic loading

Structural Health Monitoring (SHM) using ultrasonic-guided waves (UGWs) enables continuous monitoring of components with complex geometries and provides extensive information about their structural integrity and their overall condition. Composite Overwrapped Pressure Vessels (COPVs) used for storing hydrogen gases at very high pressures are an example of a critical infrastructure that could benefit significantly from SHM. This can be used to increase the periodic inspection intervals, ensure safe operating conditions by early detection of anomalies, and ultimately estimate the remaining lifetime of COPVs. Therefore, in the Digital Quality Infrastructure Initiative (QI-Digital) in Germany, an SHM system is being developed for COPVs used in a hydrogen refueling station. In this study, the results of a lifetime fatigue test on a Type IV COPV subjected to many thousands of load cycles under different temperatures and pressures are presented to demonstrate the strengths and challenges associated with such an SHM system. During the cyclic testing up to the final material failure of the COPV, a sensor network of fifteen surface-mounted piezoelectric (PZT) wafers was used to collect the UGW data. However, the pressure variations, the aging process of the COPV, the environmental parameters, and possible damages simultaneously have an impact on the recorded signals. This issue and the lack of labeled data make signal processing and analysis even more demanding. Thus, in this study, semi-supervised, and unsupervised deep learning approaches are utilized to separate the influence of different variables on the UGW data with the final aim of detecting and localizing the damage before critical failure.

Insights into traditional Large Deformation Diffeomorphic Metric Mapping and unsupervised deep-learning for diffeomorphic registration and their evaluation

This paper explores the connections between traditional Large Deformation Diffeomorphic Metric Mapping methods and unsupervised deep-learning approaches for non-rigid registration, particularly emphasizing diffeomorphic registration. The study provides useful insights and establishes connections between the methods, thereby facilitating a profound understanding of the methodological landscape. The methods considered in our study are extensively evaluated in T1w MRI images using traditional NIREP and Learn2Reg OASIS evaluation protocols with a focus on fairness, to establish equitable benchmarks and facilitate informed comparisons. Through a comprehensive analysis of the results, we address key questions, including the intricate relationship between accuracy and transformation quality in performance, the disentanglement of the influence of registration ingredients on performance, and the determination of benchmark methods and baselines. We offer valuable insights into the strengths and limitations of both traditional and deep-learning methods, shedding light on their comparative performance and guiding future advancements in the field.

Enhancing voxel-based dosimetry accuracy with an unsupervised deep learning approach for hybrid medical image registration.

Deformable registration is required to generate a time-integrated activity (TIA) map which is essential for voxel-based dosimetry. The conventional iterative registration algorithm using anatomical images (e.g., computed tomography (CT)) could result in registration errors in functional images (e.g., single photon emission computed tomography (SPECT) or positron emission tomography (PET)). Various deep learning-based registration tools have been proposed, but studies specifically focused on the registration of serial hybrid images were not found. In this study, we introduce CoRX-NET, a novel unsupervised deep learning network designed for deformable registration of hybrid medical images. The CoRX-NET structure is based on the Swin-transformer (ST), allowing for the representation of complex spatial connections in images. Its self-attention mechanism aids in the effective exchange and integration of information across diverse image regions. To augment the amalgamation of SPECT and CT features, cross-stitch layers have been integrated into the network. Two different 177 Lu DOTATATE SPECT/CT datasets were acquired at different medical centers. 22 sets from Seoul National University and 14 sets from Sunway Medical Centre are used for training/internal validation and external validation respectively. The CoRX-NET architecture builds upon the ST, enabling the modeling of intricate spatial relationships within images. To further enhance the fusion of SPECT and CT features, cross-stitch layers have been incorporated within the network. The network takes a pair of SPECT/CT images (e.g., fixed and moving images) and generates a deformed SPECT/CT image. The performance of the network was compared with Elastix and TransMorph using L1 loss and structural similarity index measure (SSIM) of CT, SSIM of normalized SPECT, and local normalized cross correlation (LNCC) of SPECT as metrics. The voxel-wise root mean square errors (RMSE) of TIA were compared among the different methods. The ablation study revealed that cross-stitch layers improved SPECT/CT registration performance. The cross-stitch layers notably enhance SSIM (internal validation: 0.9614vs. 0.9653, external validation: 0.9159vs. 0.9189) and LNCC of normalized SPECT images (internal validation: 0.7512vs. 0.7670, external validation: 0.8027vs. 0.8027). CoRX-NET with the cross-stitch layer achieved superior performance metrics compared to Elastix and TransMorph, except for CT SSIM in the external dataset. When qualitatively analyzed for both internal and external validation cases, CoRX-NET consistently demonstrated superior SPECT registration results. In addition, CoRX-NET accomplished SPECT/CT image registration in less than 6s, whereas Elastix required approximately 50s using the same PC's CPU. When employing CoRX-NET, it was observed that the voxel-wise RMSE values for TIA were approximately 27% lower for the kidney and 33% lower for the tumor, compared to when Elastix was used. This study represents a major advancement in achieving precise SPECT/CT registration using an unsupervised deep learning network. It outperforms conventional methods like Elastix and TransMorph, reducing uncertainties in TIA maps for more accurate dose assessments.

Unsupervised deep learning approach for structural anomaly detection using probabilistic features

Civil structures may deteriorate during their service life due to degradation or damage imposed by natural hazards such as earthquakes, wind, and impact. Structural performance anomaly detection is essential to provide an early warning of structural degradation limit states in order to prevent potential catastrophic failure. Data-driven machine learning approaches have been widely used for this, due to their capability in capturing features sensitive to damage-induced anomalies from structural health monitoring (SHM) data, assuming that such data are available. Although machine learning models have been used, many are challenged by the vast operational and environmental variability that can corrupt SHM data and by (typically) strongly correlated information from different sensors in the SHM data. This paper proposes an unsupervised deep learning approach for the detection of structural anomaly based on a deep convolutional variational autoencoder (DCVAE) for feature extraction coupled with support vector data description (SVDD) for anomaly detection. The proposed DCVAE-SVDD method has several appealing strengths. First, the variational latent encoding is used to capture the features of monitoring data through a probability distribution. The integration of the Kullback–Leibler divergence in the loss function provides accurate estimation of the probability distributions. Second, the DCVAE designed with convolutional and deconvolutional operations utilizes the correlation among multisensor data to avoid loss of correlation features and achieve better performance in feature extraction. Third, the SVDD is utilized to create a minimum-volume hypersphere that contains the anomaly-sensitive statistical features of the state. The hypersphere accurately separates anomaly-sensitive statistical features of reference states of structure from the anomalous ones. A computational frame model and a laboratory grandstand model are used to evaluate the performance of the proposed method for detecting structural anomaly. The results demonstrate the superiority of the proposed DCVAE-SVDD in detection accuracy over the other commonly used structural anomaly detection methods (deep autoencoder combined with SVDD autoregressive model with one-class support vector machine, and principal component analysis).

Supervised and Unsupervised Deep Learning Approaches for EEG Seizure Prediction.

Epilepsy affects more than 50 million people worldwide, making it one of the world's most prevalent neurological diseases. The main symptom of epilepsy is seizures, which occur abruptly and can cause serious injury or death. The ability to predict the occurrence of an epileptic seizure could alleviate many risks and stresses people with epilepsy face. We formulate the problem of detecting preictal (or pre-seizure) with reference to normal EEG as a precursor to incoming seizure. To this end, we developed several supervised deep learning approaches model to identify preictal EEG from normal EEG. We further develop novel unsupervised deep learning approaches to train the models on only normal EEG, and detecting pre-seizure EEG as an anomalous event. These deep learning models were trained and evaluated on two large EEG seizure datasets in a person-specific manner. We found that both supervised and unsupervised approaches are feasible; however, their performance varies depending on the patient, approach and architecture. This new line of research has the potential to develop therapeutic interventions and save human lives.

Identification of rare cortical folding patterns using unsupervised deep learning

Abstract Like fingerprints, cortical folding patterns are unique to each brain even though they follow a general species-specific organization. Some folding patterns have been linked with neurodevelopmental disorders. However, due to the high inter-individual variability, the identification of rare folding patterns that could become biomarkers remains a very complex task. This paper proposes a novel unsupervised deep learning approach to identify rare folding patterns and assess the degree of deviations that can be detected. To this end, we preprocess the brain MR images to focus the learning on the folding morphology and train a beta variational auto-encoder (β−VAE) on the inter-individual variability of the folding to identify outliers. We compare the detection power of the latent space and of the reconstruction errors, using synthetic benchmarks and one actual rare configuration related to the central sulcus. Finally, we assess the generalization of our method on a developmental anomaly located in another region and we validate the relevance of our approach on patients suffering from drug-resistant epilepsy. Our results suggest that this method enables encoding relevant folding characteristics that can be enlightened and better interpreted based on the generative power of the β−VAE. The latent space and the reconstruction errors bring complementary information and enable the identification of rare patterns of different nature. This method generalizes well to a different region on another dataset and demonstrates promising results on the epileptic patients. Code is available at https://github.com/neurospin-projects/2022_lguillon_rare_folding_detection.

NPCA: a linear dimensionality reduction method using a multilayer perceptron.

Background: Linear dimensionality reduction techniques are widely used in many applications. The goal of dimensionality reduction is to eliminate the noise of data and extract the main features of data. Several dimension reduction methods have been developed, such as linear-based principal component analysis (PCA), nonlinear-based t-distributed stochastic neighbor embedding (t-SNE), and deep-learning-based autoencoder (AE). However, PCA only determines the projection direction with the highest variance, t-SNE is sometimes only suitable for visualization, and AE and nonlinear methods discard the linear projection. Results: To retain the linear projection of raw data and generate a better result of dimension reduction either for visualization or downstream analysis, we present neural principal component analysis (nPCA), an unsupervised deep learning approach capable of retaining richer information of raw data as a promising improvement to PCA. To evaluate the performance of the nPCA algorithm, we compare the performance of 10 public datasets and 6 single-cell RNA sequencing (scRNA-seq) datasets of the pancreas, benchmarking our method with other classic linear dimensionality reduction methods. Conclusion: We concluded that the nPCA method is a competitive alternative method for dimensionality reduction tasks.

Anomaly Detection in Hobbing Tool Images: Using An Unsupervised Deep Learning Approach in Manufacturing Industry

This study explores the application of the PatchCore algorithm for anomaly classification in hobbing tools, an area of keen interest in industrial artificial intelligence application. Despite utilizing limited training images, the algorithm demonstrates capability in recognizing a variety of anomalies, promising to reduce the time-intensive labeling process traditionally undertaken by domain experts. The algorithm demonstrated an accuracy of 92%, precision of 84%, recall of 100%, and a balanced F1 score of 91%, showcasing its proficiency in identifying anomalies. However, the investigation also highlights that while the algorithm effectively identifies anomalies, it doesn't primarily recognize domain-specific wear issues. Thus, the presented approach is used only for pre-classification, with domain experts subsequently segmenting the images indicating significant wear. The intention is to employ a supervised learning procedure to identify actual wear. This premise will be further investigated in future research studies.

The deep neural network solver for B-spline approximation

This paper introduces a novel unsupervised deep learning approach to address the knot placement problem in the field of B-spline approximation, called deep neural network solvers (DNN-Solvers). Given discrete points, the DNN acts as a solver for calculating knot positions in the case of a fixed knot number. The input can be any initial knots and the output provides the desirable knots. The loss function is based on the approximation error. The DNN-Solver converts the lower-dimensional knot placement problem, characterized as a nonconvex nonlinear optimization problem, into a search for suitable network parameters within a high-dimensional space. Owing to the over-parameterization nature, DNN-Solvers are less likely to be trapped in local minima and robust against initial knots. Moreover, the unsupervised learning paradigm of DNN-Solvers liberates us from constructing high-quality synthetic datasets with labels. Numerical experiments demonstrate that DNN-Solvers are excellent in both approximation results and efficiency under the premise of an appropriate number of knots.

Innovative AI techniques to aid clinical diagnosis of complex lung pathologies in ultrasound images

Lung ultrasound (LUS) has recently gained increasing interest as a reliable method for point of care (POC) diagnostics and management of lung diseases. LUS is easily accessible, has no radiation-related risks and is portable, so it does not require relocating the patient, minimising the risk of further infection. This research aims to develop efficient and automated methods for lung pathology diagnosis using AI to support clinicians. We have introduced a binary classifier based on a state-of-the-art Swin Transformer to discriminate between LUS clips of healthy patients and patients with interstitial lung disease (ILD). Differently from previous approaches, this is better aligned with the current clinical assessments of ILD since it evaluates LUS clips instead of single frames; and does not require any additional annotations from clinicians, since its training is based only on the already available medical report. Furthermore, we propose an unsupervised deep learning approach (based on a generative adversarial network) to convert in real-time US volumes to MRI-like volumes of the thoracic region. This approach can be used to spatially localise the US probe with respect to the MRI in real-time and gather anatomical contextual information of the imaged region, providing thus guidance to clinicians.

MultiNet: Deep unsupervised power control for industrial MU-MIMO networks

This paper presents Multinet, an unsupervised deep learning (DL) approach for power allocation in industrial environments and IIoT applications. Multinet extends the previously proposed singular value decomposition network (SVDNet), which utilizes supervised DL to approximate the performance of the WMMSE algorithm. While SVDNet requires labeled data for training, limiting its scalability and generalization performance, in contrast, Multinet employs unsupervised DL to directly optimize the sum rate maximization objective function, eliminating the need for labeled datasets and improving training efficiency. Simulation studies are conducted to evaluate Multinet’s performance in an industrial environment, utilizing parameters derived from measured large-scale fading characteristics of the industrial radio channel at 5200 MHz. The suitability of Multinet for industrial applications is thus assessed and numerical evaluations demonstrate that Multinet outperforms benchmark supervised and unsupervised DL-based power control schemes in terms of sum rate and energy efficiency.

Multi-resolution Cross-modality Image Registration Using Unsupervised Deep Learning Approach.

Multi-resolution Cross-modality Image Registration Using Unsupervised Deep Learning Approach.

Forged document detection and writer identification through unsupervised deep learning approach

Forged document detection and writer identification through unsupervised deep learning approach

Subject clustering by IF-PCA and several recent methods.

Subject clustering (i.e., the use of measured features to cluster subjects, such as patients or cells, into multiple groups) is a problem of significant interest. In recent years, many approaches have been proposed, among which unsupervised deep learning (UDL) has received much attention. Two interesting questions are 1) how to combine the strengths of UDL and other approaches and 2) how these approaches compare to each other. We combine the variational auto-encoder (VAE), a popular UDL approach, with the recent idea of influential feature-principal component analysis (IF-PCA) and propose IF-VAE as a new method for subject clustering. We study IF-VAE and compare it with several other methods (including IF-PCA, VAE, Seurat, and SC3) on 10 gene microarray data sets and eight single-cell RNA-seq data sets. We find that IF-VAE shows significant improvement over VAE, but still underperforms compared to IF-PCA. We also find that IF-PCA is quite competitive, slightly outperforming Seurat and SC3 over the eight single-cell data sets. IF-PCA is conceptually simple and permits delicate analysis. We demonstrate that IF-PCA is capable of achieving phase transition in a rare/weak model. Comparatively, Seurat and SC3 are more complex and theoretically difficult to analyze (for these reasons, their optimality remains unclear).

Development of a deep learning-based error detection system without error dose maps in the patient-specific quality assurance of volumetric modulated arc therapy.

To detect errors in patient-specific quality assurance (QA) for volumetric modulated arc therapy (VMAT), we proposed an error detection method based on dose distribution analysis using unsupervised deep learning approach and analyzed 161 prostate VMAT beams measured with a cylindrical detector. For performing error simulation, in addition to error-free dose distribution, dose distributions containing nine types of error, including multileaf collimator (MLC) positional errors, gantry rotation errors, radiation output errors and phantom setup errors, were generated. Only error-free data were employed for the model training, and error-free and error data were employed for the tests. As a deep learning model, the variational autoencoder (VAE) was adopted. The anomaly of test data was quantified by calculating Mahalanobis distance based on the feature vectors acquired from a trained encoder. Based on this anomaly, test data were classified as 'error-free' or 'any-error.' For comparison with conventional approaches, gamma (γ)-analysis was performed, and supervised learning convolutional neural network (S-CNN) was constructed. Receiver operating characteristic curves were obtained to evaluate their performance with the area under the curve (AUC). For all error types, except systematic MLC positional and radiation output errors, the performance of the methods was in the order of S-CNN ˃ VAE-based ˃ γ-analysis (only S-CNN required error data for model training). For example, in random MLC positional error simulation, the AUC of our method, S-CNN and γ-analysis were 0.699, 0.921 and 0.669, respectively. Our results showed that the VAE-based method has the potential to detect errors in patient-specific VMAT QA.

Seismic damage identification of high arch dams based on an unsupervised deep learning approach

In actual engineering scenarios of arch dams, the incompleteness and nonstationarity of dynamic monitoring signals limit the accurate cognition of the health state. The effectiveness and robustness of damage characteristics in complex environments restrict the practical application of damage diagnosis theory. In this study, guided by the direct extraction of damage sensitivity features from the acceleration response signals of the arch dam, a seismic damage identification approach of high arch dams based on unsupervised learning is developed. Aiming at the problems of low measurement accuracy and poor identification robustness in the existing artificially designed damage-sensitive features, a denoising contractual sparse deep auto-encoder (DCS-DAE) model is proposed by exploring the mapping relationship between monitoring data and the structural state. This model integrates the advantages of denoising auto-encoder, compressive auto-encoder, and sparse auto-encoder. On this basis, based on the principle of reconstruction error and small probability, combined with box-plot and WKNN algorithm, a damage identification framework based on DCS-DAE is constructed. The effectiveness and noise resistance of the proposed method are verified by an extremely high arch dam. The results demonstrate that the damage identification framework based on multi-objective DCS-DAE constructed in this paper only requires the vibration information of the structure in the intact scenario, which provides a solution with higher stability and robustness for the seismic damage identification of high arch dams under strong noise pollution.