Unsupervised Learning Network Research Articles

Single-Shot 3D Reconstruction via Nonlinear Fringe Transformation: Supervised and Unsupervised Learning Approaches.

The field of computer vision has been focusing on achieving accurate three-dimensional (3D) object representations from a single two-dimensional (2D) image through deep artificial neural networks. Recent advancements in 3D shape reconstruction techniques that combine structured light and deep learning show promise in acquiring high-quality geometric information about object surfaces. This paper introduces a new single-shot 3D shape reconstruction method that uses a nonlinear fringe transformation approach through both supervised and unsupervised learning networks. In this method, a deep learning network learns to convert a grayscale fringe input into multiple phase-shifted fringe outputs with different frequencies, which act as an intermediate result for the subsequent 3D reconstruction process using the structured-light fringe projection profilometry technique. Experiments have been conducted to validate the practicality and robustness of the proposed technique. The experimental results demonstrate that the unsupervised learning approach using a deep convolutional generative adversarial network (DCGAN) is superior to the supervised learning approach using UNet in image-to-image generation. The proposed technique's ability to accurately reconstruct 3D shapes of objects using only a single fringe image opens up vast opportunities for its application across diverse real-world scenarios.

A hybrid model based on unsupervised learning for seismic response feature extraction and gas-bearing reservoir prediction using longitudinal and transverse waves seismic data in low-degree exploration areas: A case study

Predicting gas-bearing reservoirs from seismic data is a major field of research. Recently, deep learning has been applied to achieve better results; however, challenges such as high sample demand remain. Drilling data are often insufficient, and gas-bearing sample data are difficult to acquire in large quantities, particularly for low-level (new) exploration areas, which limits gas-bearing reservoir prediction by supervised learning. Therefore, this study proposes a hybrid model using unsupervised learning to extract seismic response features and predict gas-bearing reservoir distribution. First, seismic attribute analysis was used to extract as many longitudinal and transverse waves seismic attributes as possible to provide richer information about gas-bearing reservoirs. However, abundant seismic attributes contain a large amount of redundant information, thereby increasing computation time. Thus, principal component analysis (PCA) was used to reduce the dimensions of the extracted multi-component seismic attributes, remove redundant information, and obtain the most important characteristics that are sensitive to gas-bearing reservoirs. The sensitive seismic attributes obtained after the PCA were fed into a self-organizing neural network (SOM) for unsupervised learning. Neurons in the SOM compete with each other to optimize their own features and extract reservoir-sensitive seismic response features for prediction. To evaluate the proposed method, it was applied to part of the Marmousi2 model and real seismic data for validation. The following results were obtained: (1)The results of the synthetic data test results not only proved the effectiveness of the proposed method, but also showed that the method has strong noise resistance capability and is suitable for predicting gas-bearing reservoirs with low signal-to-noise ratio seismic data; (2) Based on the drilling data and sedimentary facies characteristics in the test area, the actual 3D seismic data test results showed that there are four wells that match the predicted results, and the distribution boundaries of the gas-bearing reservoirs they depict are relatively clear; (3) In low exploration areas (areas with few or no wells), this method is important as a guide to hydrocarbon exploration.

Unsupervised learning-based image recovery and diagnosis method for blade attachment of marine current turbine

Marine current energy is a clean and renewable resource that is an available energy source with future power generation potential. However, complex subsea conditions can cause blade failures in marine current turbines (MCTs); therefore, the fault diagnosis of MCTs is essential to ensure their good operation. Image-based diagnosis is a nondestructive approach. To improve the efficiency of using human capital, this study uses the unsupervised learning network GANomaly trained only with normal samples to diagnose the leaf attachment of MCTs. Marine biofouling is a lengthy process, and according to the literature, the blades were contaminated by a 1.1-mm-thick layer of lithium lubricating grease, therefore, this study uses ropes to simulate this attachment. First, we address the problem of blurred underwater images collected using a multiscale retinex enhancement algorithm with color recovery to form a new dataset. GANomaly is modified to diagnose MCT blades with different attachment degrees. The experimental results show that the accuracy of the enhanced dataset is improved by 1.75 %, and the accuracy of the improved GANomaly model reaches 91.23 %, thereby enhancing the efficiency of the fault diagnosis of MCTs.

Graph Representation Learning for Large-Scale Neuronal Morphological Analysis.

The analysis of neuronal morphological data is essential to investigate the neuronal properties and brain mechanisms. The complex morphologies, absence of annotations, and sheer volume of these data pose significant challenges in neuronal morphological analysis, such as identifying neuron types and large-scale neuron retrieval, all of which require accurate measuring and efficient matching algorithms. Recently, many studies have been conducted to describe neuronal morphologies quantitatively using predefined measurements. However, hand-crafted features are usually inadequate for distinguishing fine-grained differences among massive neurons. In this article, we propose a novel morphology-aware contrastive graph neural network (MACGNN) for unsupervised neuronal morphological representation learning. To improve the retrieval efficiency in large-scale neuronal morphological datasets, we further propose Hash-MACGNN by introducing an improved deep hash algorithm to train the network end-to-end to learn binary hash representations of neurons. We conduct extensive experiments on the largest dataset, NeuroMorpho, which contains more than 100 000 neurons. The experimental results demonstrate the effectiveness and superiority of our MACGNN and Hash-MACGNN for large-scale neuronal morphological analysis.

Uncovering Critical Causes of Highway Work Zone Accidents Using Unsupervised Machine Learning and Social Network Analysis

Uncovering Critical Causes of Highway Work Zone Accidents Using Unsupervised Machine Learning and Social Network Analysis

Identification of epileptic networks with graph convolutional network incorporating oscillatory activities and evoked synaptic responses

Stereoelectroencephalography (SEEG) offers unique neural data from in-depth brain structures with fine temporal resolutions to better investigate the origin of epileptic brain activities. Although oscillatory patterns from different frequency bands and functional connectivity computed from the SEEG datasets are employed to study the epileptic zones, direct electrical stimulation-evoked electrophysiological recordings of synaptic responses, namely cortical-cortical evoked potentials (CCEPs), from the same SEEG electrodes are not explored for the localization of epileptic zones. Here we proposed a two-stream model with unsupervised learning and graph convolutional network tailored to the SEEG and CCEP datasets in individual patients to perform localization of epileptic zones. We compared our localization results with the clinically marked electrode sites determined for surgical resections. Our model had good classification capability when compared to other state-of-the-art methods. Furthermore, based on our prediction results we performed group-level brain-area mapping analysis for temporal, frontal and parietal epilepsy patients and found that epileptic and non-epileptic brain networks were distinct in patients with different types of focal epilepsy. Our unsupervised data-driven model provides personalized localization analysis for the epileptic zones. The epileptic and non-epileptic brain areas disclosed by the prediction model provide novel insights into the network-level pathological characteristics of epilepsy.

Image Retrieval Using Auto Encoding Features In Deep Learning

The latest technologies and growth in availability of image storage in day to day life has made a vast storage place for the images in the database. Several devices which help in capturing the image contribute to a huge repository of images. Keeping in mind the daily input in the database, one must think of retrieving those images according to certain criteria mentioned. Several techniques such as shape of the object, Discrete Wavelet transform (DWT), texture features etc. were used in determining the type of image and classifying them. Segmentation also plays a vital role in image retrieval but the robustness is lacking in most of the cases. The process of retrieval mainly depends on the special characteristics possessed by an image rather than the whole image. Two types of image retrieval can be seen. One with a general object and the other which may be specific to some type of application. Modern deep neural networks for unsupervised feature learning like Deep Autoencoder (AE) learn embedded representations by stacking layers on top of each other. These learnt embedded-representations, however, may degrade as the AE network deepens due to vanishing gradient, resulting in decreased performance. We have introduced here the ResNet Autoencoder (RAE) and its convolutional version (C-RAE) for unsupervised feature based learning. The proposed model is tested on three distinct databases Corel1K, Cifar-10, Cifar-100 which differ in size. The presented algorithm have significantly reduced computation time and provided very high image retrieval levels of accuracy.

The dynamics of representation learning in shallow, non-linear autoencoders * * This article is an updated version of: Refinetti M and Goldt S 2022 The dynamics of representation learning in shallow, non-linear autoencoders Proc. 39th Int. Conf. on Machine Learning (Proc. Machine Learning Research vol 162) ed K Chaudhuri, S Jegelka, L Song, C Szepesvari, G Niu and S Sabato

Autoencoders are the simplest neural network for unsupervised learning, and thus an ideal framework for studying feature learning. While a detailed understanding of the dynamics of linear autoencoders has recently been obtained, the study of non-linear autoencoders has been hindered by the technical difficulty of handling training data with non-trivial correlations—a fundamental prerequisite for feature extraction. Here, we study the dynamics of feature learning in non-linear, shallow autoencoders. We derive a set of asymptotically exact equations that describe the generalisation dynamics of autoencoders trained with stochastic gradient descent (SGD) in the limit of high-dimensional inputs. These equations reveal that autoencoders learn the leading principal components of their inputs sequentially. An analysis of the long-time dynamics explains the failure of sigmoidal autoencoders to learn with tied weights, and highlights the importance of training the bias in ReLU autoencoders. Building on previous results for linear networks, we analyse a modification of the vanilla SGD algorithm, which allows learning of the exact principal components. Finally, we show that our equations accurately describe the generalisation dynamics of non-linear autoencoders trained on realistic datasets such as CIFAR10, thus establishing shallow autoencoders as an instance of the recently observed Gaussian universality.

A class-level matching unsupervised transfer learning network for rolling bearing fault diagnosis under various working conditions

As an effective method, deep transfer learning is used to solve the problem of unsupervised fault diagnosis of rolling bearings. In the process of obtaining domain invariant features, the feature matching at the domain-level does not consider the distribution of each category in matching the global distribution of source domain features and target domain features. To solve this problem, a class-level matching transfer learning network is proposed. In this method, source domain data and target domain data from different categories are matched first. Meanwhile, domain-level matching and class-level matching are combined based on the maximum classifier discrepancy structure. Then the transfer training process is divided into three stages: domain-level matching, class-level matching, and target domain pseudo-label direct guidance network training. The proposed method utilizes the target domain pseudo-label output by the network to extract the features of the target data by self-directing. Compared with the improved maximum classifier discrepancy on the Paderborn University dataset, the proposed method improves the fault diagnosis average accuracy by 12.95% on the six transfer tasks. It significantly improves the diagnostic accuracy of unsupervised bearing fault diagnosis under various working conditions.

Patch-based camera-aware person-to-group learning and group similarity strategy for unsupervised group re-identification

In real life, people often go together in groups, which extends to an important but less researched task of computer vision: group re-identification. Differing from the related task of person re-identification, group re-identification involves unique challenges: self-occlusion, layout changes, membership changes, and the lack of labeled data. To solve these problems, we propose a patch-based camera-aware person-to-group unsupervised learning network. For the problem of high labeling costs, we train an unsupervised feature extraction network to extract the features of group members. Moreover, the feature extraction network is based on image patches, which solves the problem of group member alignment caused by occlusion within the group. To handle the large intra-ID variance resulting from changes in camera views, we add camera information to the model and propose the multi-grained camera-aware loss. For the problems of layout changes and membership changes, we adopt a person-to-group strategy. Specifically, we propose a novel group distance metric, named Modified Hausdorff Distance, to exploit the information of all the group members. Experimental results on two public datasets demonstrate the superiority of our approach.

Quantum process tomography with unsupervised learning and tensor networks

The impressive pace of advance of quantum technology calls for robust and scalable techniques for the characterization and validation of quantum hardware. Quantum process tomography, the reconstruction of an unknown quantum channel from measurement data, remains the quintessential primitive to completely characterize quantum devices. However, due to the exponential scaling of the required data and classical post-processing, its range of applicability is typically restricted to one- and two-qubit gates. Here, we present a technique for performing quantum process tomography that addresses these issues by combining a tensor network representation of the channel with a data-driven optimization inspired by unsupervised machine learning. We demonstrate our technique through synthetically generated data for ideal one- and two-dimensional random quantum circuits of up to 10 qubits, and a noisy 5-qubit circuit, reaching process fidelities above 0.99 using several orders of magnitude fewer (single-qubit) measurement shots than traditional tomographic techniques. Our results go far beyond state-of-the-art, providing a practical and timely tool for benchmarking quantum circuits in current and near-term quantum computers.

Fully Linear Graph Convolutional Networks for Semi-Supervised and Unsupervised Classification

This article presents FLGC, a simple yet effective fully linear graph convolutional network for semi-supervised and unsupervised learning. Instead of using gradient descent, we train FLGC based on computing a global optimal closed-form solution with a decoupled procedure, resulting in a generalized linear framework and making it easier to implement, train, and apply. We show that (1) FLGC is powerful to deal with both graph-structured data and regular data, (2) training graph convolutional models with closed-form solutions improve computational efficiency without degrading performance, and (3) FLGC acts as a natural generalization of classic linear models in the non-Euclidean domain (e.g., ridge regression and subspace clustering). Furthermore, we implement a semi-supervised FLGC and an unsupervised FLGC by introducing an initial residual strategy, enabling FLGC to aggregate long-range neighborhoods and alleviate over-smoothing. We compare our semi-supervised and unsupervised FLGCs against many state-of-the-art methods on a variety of classification and clustering benchmarks, demonstrating that the proposed FLGC models consistently outperform previous methods in terms of accuracy, robustness, and learning efficiency. The core code of our FLGC is released at https://github.com/AngryCai/FLGC .

Beamspace Channel Estimation for Wideband Millimeter-Wave MIMO: A Model-Driven Unsupervised Learning Approach

Millimeter-wave (mmWave) communications have been one of the promising technologies for future wireless networks that integrate a wide range of data-demanding applications. To compensate for the large channel attenuation in mmWave band and avoid high hardware cost, a lens-based beamspace massive multiple-input multiple-output (MIMO) system is considered. However, the spatial-wideband effect in wideband mmWave systems makes channel estimation very challenging, especially when the receiver is equipped with a limited number of radio-frequency (RF) chains. Furthermore, the real channel data cannot be obtained before the mmWave system is used in a new environment, which makes it impossible to train a deep learning (DL)-based channel estimator using real data set beforehand. To solve the problem, we propose a model-driven unsupervised learning network, named learned denoising-based generalized expectation consistent (LDGEC) signal recovery network. By utilizing the Stein’s unbiased risk estimator loss, the LDGEC network can be trained only with limited measurements corresponding to the pilot symbols, instead of the real channel data. Even if designed for unsupervised learning, the LDGEC network can be supervisingly trained with the real channel via the denoiser-by-denoiser way. The numerical results demonstrate that the LDGEC-based channel estimator significantly outperforms state-of-the-art compressive sensing-based algorithms when the receiver is equipped with a small number of RF chains and low-resolution ADCs.

NILRNN: A Neocortex-Inspired Locally Recurrent Neural Network for Unsupervised Feature Learning in Sequential Data

Unsupervised feature learning refers to the problem of learning useful feature extraction functions from unlabeled data. Despite the great success of deep learning networks in this task in recent years, both for static and for sequential data, these systems can in general still not compete with the high performance of our brain at learning to extract useful representations from its sensory input. We propose the Neocortex-Inspired Locally Recurrent Neural Network: a new neural network for unsupervised feature learning in sequential data that brings ideas from the structure and function of the neocortex to the well-established fields of machine learning and neural networks. By mimicking connection patterns in the feedforward circuits of the neocortex, our system tries to generalize some of the ideas behind the success of convolutional neural networks to types of data other than images. To evaluate the performance of our system at extracting useful features, we have trained different classifiers using those and other learnt features as input and we have compared the obtained accuracies. Our system has shown to outperform other shallow feature learning systems in this task, both in terms of the accuracies achieved and in terms of how fast the classification task is learnt. The results obtained confirm our system as a state-of-the-art shallow feature learning system for sequential data, and suggest that extending it to or integrating it into deep architectures may lead to new successful networks that are competent at dealing with complex sequential tasks.

Tensor networks for unsupervised machine learning.

Modeling the joint distribution of high-dimensional data is a central task in unsupervised machine learning. In recent years, many interests have been attracted to developing learning models based on tensor networks, which have the advantages of a principle understanding of the expressive power using entanglement properties, and as a bridge connecting classical computation and quantum computation. Despite the great potential, however, existing tensor network models for unsupervised machine learning only work as a proof of principle, as their performance is much worse than the standard models such as restricted Boltzmann machines and neural networks. In this Letter, we present autoregressive matrix product states (AMPS), a tensor network model combining matrix product states from quantum many-body physics and autoregressive modeling from machine learning. Our model enjoys the exact calculation of normalized probability and unbiased sampling. We demonstrate the performance of our model using two applications, generative modeling on synthetic and real-world data, and reinforcement learning in statistical physics. Using extensive numerical experiments, we show that the proposed model significantly outperforms the existing tensor network models and the restricted Boltzmann machines, and is competitive with state-of-the-art neural network models.

Autoencoded Elastic Wave-Equation Traveltime Inversion: Toward Reliable Near-Surface Tomogram

Due to unexpected environmental variations and poor consistencies in land data acquisition, complex near-surface seismograms are usually polluted unreasonably with a low signal-to-noise ratio (SNR). These complicated circumstances bring more challenges in identifying accurate first arrivals for the following wave-equation traveltime inversion (WT). Recently, the autoencoder (AE) is a typical unsupervised learning network whose basic principle is to compress the input seismic data for their intrinsic features in the latent space with an encoder and, thereafter, to decipher these features for seismic profiles as the output with a decoder. This process is fully automatic with high stability and is not very sensitive to data quality. In this paper, we propose an elastic WT inversion algorithm based on the AE method (AEWT) to invert the P-velocity model. Compared with the standard WT, the AEWT method automatically extracts the intrinsic features of the refractions with AE as reference data for the misfit functionals. Feature images in the latent space show similar but intensified sensitivity to the traveltimes with respect to velocity perturbations. We present one synthetic and two field data tests for comparing the proposed AEWT and the standard WT tomograms to investigate the locations of a buried fault and the depth of a buried sinkhole. All these experiments demonstrate that the proposed elastic AEWT method can reduce errors caused by low SNR and obtain a more reliable and stable P-velocity tomogram.

A Developmental Approach for Training Deep Belief Networks

Deep belief networks (DBNs) are stochastic neural networks that can extract rich internal representations of the environment from the sensory data. DBNs had a catalytic effect in triggering the deep learning revolution, demonstrating for the very first time the feasibility of unsupervised learning in networks with many layers of hidden neurons. These hierarchical architectures incorporate plausible biological and cognitive properties, making them particularly appealing as computational models of human perception and cognition. However, learning in DBNs is usually carried out in a greedy, layer-wise fashion, which does not allow to simulate the holistic maturation of cortical circuits and prevents from modeling cognitive development. Here we present iDBN, an iterative learning algorithm for DBNs that allows to jointly update the connection weights across all layers of the model. We evaluate the proposed iterative algorithm on two different sets of visual stimuli, measuring the generative capabilities of the learned model and its potential to support supervised downstream tasks. We also track network development in terms of graph theoretical properties and investigate the potential extension of iDBN to continual learning scenarios. DBNs trained using our iterative approach achieve a final performance comparable to that of the greedy counterparts, at the same time allowing to accurately analyze the gradual development of internal representations in the deep network and the progressive improvement in task performance. Our work paves the way to the use of iDBN for modeling neurocognitive development.

Joint Deep Reinforcement Learning and Unsupervised Learning for Channel Selection and Power Control in D2D Networks.

Device-to-device (D2D) technology enables direct communication between devices, which can effectively solve the problem of insufficient spectrum resources in 5G communication technology. Since the channels are shared among multiple D2D user pairs, it may lead to serious interference between D2D user pairs. In order to reduce interference, effectively increase network capacity, and improve wireless spectrum utilization, this paper proposed a distributed resource allocation algorithm with the joint of a deep Q network (DQN) and an unsupervised learning network. Firstly, a DQN algorithm was constructed to solve the channel allocation in the dynamic and unknown environment in a distributed manner. Then, a deep power control neural network with the unsupervised learning strategy was constructed to output an optimized channel power control scheme to maximize the spectrum transmit sum-rate through the corresponding constraint processing. As opposed to traditional centralized approaches that require the collection of instantaneous global network information, the algorithm proposed in this paper used each transmitter as a learning agent to make channel selection and power control through a small amount of state information collected locally. The simulation results showed that the proposed algorithm was more effective in increasing the convergence speed and maximizing the transmit sum-rate than other traditional centralized and distributed algorithms.

A novel single robot image shadow detection method based on convolutional block attention module and unsupervised learning network.

Shadow detection plays a very important role in image processing. Although many algorithms have been proposed in different environments, it is still a challenging task to detect shadows in natural scenes. In this paper, we propose a convolutional block attention module (CBAM) and unsupervised domain adaptation adversarial learning network for single image shadow detection. The new method mainly contains three steps. Firstly, in order to reduce the data deviation between the domains, the hierarchical domain adaptation strategy is adopted to calibrate the feature distribution from low level to high level between the source domain and the target domain. Secondly, in order to enhance the soft shadow detection ability of the model, the boundary adversarial branch is proposed to obtain structured shadow boundary. Meanwhile, a CBAM is added in the model to reduce the correlation between different semantic information. Thirdly, the entropy adversarial branch is combined to further suppress the high uncertainty at the boundary of the prediction results, and it obtains the smooth and accurate shadow boundary. Finally, we conduct abundant experiments on public datasets, the RMSE has the lowest values with 9.6 and BER with 6.6 on ISTD dataset, the results show that the proposed shadow detection method has better edge structure compared with the existing deep learning detection methods.

Attraction and Repulsion: Unsupervised Domain Adaptive Graph Contrastive Learning Network

Graph convolutional networks (GCNs) are important techniques for analytics tasks related to graph data. To date, most GCNs are designed for a single graph domain. They are incapable of transferring knowledge from/to different domains (graphs), due to the limitation in graph representation learning and domain adaptation across graph domains. This paper proposes a novel Graph Contrastive Learning Network (GCLN) for unsupervised domain adaptive graph learning. The key innovation is to enforce attraction and repulsion forces within each single graph domain, and across two graph domains. Within each graph, an attraction force encourages local patch node features to be similar to global representation of the entire graph, whereas a repulsion force will repel node features so they can separate network from its permutations ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> domain-specific graph contrastive learning). Across two graph domains, an attraction force encourages node features from two domains to be largely consistent, whereas a repulsion force ensures features are discriminative to differentiate graph domains ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> cross-domain graph contrastive learning). The within- and cross-domain graph contrastive learning is carried out by optimizing an objective function, which combines source classifier and target classifier loss, domain-specific contrastive loss, and cross-domain contrastive loss. As a result, feature learning from graphs is facilitated using knowledge transferred between graphs. Experiments on real-world datasets demonstrate that GCLN outperforms state-of-the-art graph neural network algorithms.