Unsupervised Neural Network Model Research Articles

LSA-PINN: A new method based on Physics-Informed Neural Network with lightweight self-attention for solving modified Bloch equation

The Spin-Exchange Relaxation-Free (SERF) atomic magnetometers play an increasingly significant roles in cardiac and brain magnetometry fields, etc. For the SERF atomic magnetometer, the evolution and interaction with the magnetic field of atomic spins can be described by the Bloch equation. However, the traditional Bloch equation does not take into account the transport phenomena caused by the atomic density gradient within the vapor cell, which makes it unable to accurately reflect the polarization distribution in the vapor cell, thereby reducing the accuracy of magnetic field measurement. To achieve a more accurate representation of the evolution characteristics of the spatial distribution of the atomic ensemble, a diffusion term for the polarization strength is introduced into the Bloch equation. Furthermore, a new unsupervised Physics-Informed Neural Network (PINN) model with lightweight self-attention (LSA) module is proposed to solve the modified nonlinear equation. The introduction of LSA enhances the adaptive representational capability of PINN, enabling it to more effectively extract global features and consequently obtain more accurate numerical solutions of the Bloch equation. The experimental results show that LSA-PINN achieves a minimum loss value of 3.98×10-2, which is 62 % lower than the traditional PINN. This study provides new insights and methods to address the limitations of traditional Bloch equation and gain a deeper understanding of system behavior.

NeuroDAVIS: A neural network model for data visualization

The task of dimensionality reduction and visualization of high-dimensional datasets remains a challenging problem since long. Modern high-throughput technologies produce large datasets having multiple views with relatively new data types. Visualization of these datasets require efficient algorithms that can uncover hidden patterns in the data without affecting the local and global structures, and bring out the inherent non-linearity within the data. To this end, however, very few such methodologies exist, which can realize this task. In this work, we have introduced a novel unsupervised deep neural network model, called NeuroDAVIS, for data visualization. NeuroDAVIS is capable of extracting important features from the data, without assuming any data distribution, visualize effectively in low dimension, and preserve both local and global structures simultaneously. It has been shown theoretically that neighbourhood relationship of the data in high dimension remains preserved in lower dimension. The performance of NeuroDAVIS has been evaluated on a wide variety of synthetic and real high-dimensional datasets including numeric, textual, image and biological data. NeuroDAVIS has been highly competitive against t-Distributed Stochastic Neighbor Embedding (t-SNE), Uniform Manifold Approximation and Projection (UMAP), Principal Component Analysis (PCA) and Random Projection (RP) with respect to visualization quality, preservation of data size, shape, and both local and global structure. It has outperformed Fast interpolation-based t-SNE (Fit-SNE), a variant of t-SNE, for most of the datasets. For the biological datasets, besides t-SNE, UMAP, Fit-SNE, PCA and RP, NeuroDAVIS has also performed well as compared to other state-of-the-art algorithms, like Potential of Heat-diffusion for Affinity-based Trajectory Embedding (PHATE) and the siamese neural network-based method, called IVIS. Downstream classification and clustering analyses have also revealed favourable results for NeuroDAVIS-generated embeddings.

Spiritual versus Religious: A Natural Language Processing Perspective

The method of Natural Language Processing (NLP) is used to analyze the literature on spirituality and religion. Specifically, the corpus produced in the spirituality/religion related scholarly literatures are used to train unsupervised neural network models (Word2Vec) that learn the extent to which words associate syntactically and semantically with one another. These models provide insights into what scholars mean when they use such terms as spiritual, religious, and spiritual-but-not-religious. For instance, they reveal that in the scholarly literature the term spiritual is used more often in contexts that describe an individual's experiences, emotions, and feelings, whereas the term religious is used more often in contexts that highlight an individual's identity and affiliations. The results also suggest that NLP methods may help scholars to perform reasonably meaningful vector operations (e. g., spiritual minus religious) that can be used to explore quickly and efficiently the syntactic and semantic patterns in a large corpus.

Unsupervised Learning Based on Meibography Enables Subtyping of Dry Eye Disease and Reveals Ocular Surface Features.

This study aimed to establish an image-based classification that can reveal the clinical characteristics of patients with dry eye using unsupervised learning methods. In this study, we analyzed 82,236 meibography images from 20,559 subjects. Using the SimCLR neural network, the images were categorized. Data for each patient were averaged and subjected to mini-batch k-means clustering, and validated through consensus clustering. Statistical metrics determined optimal category numbers. Using a UNet model, images were segmented to identify meibomian gland (MG) areas. Clinical features were assessed, including tear breakup time (BUT), tear meniscus height (TMH), and gland atrophy. A thorough ocular surface evaluation was conducted on 280 cooperative patients. SimCLR neural network achieved clustering patients with dry eye into six image-based subtypes. Patients in different subtypes harbored significantly different noninvasive BUT, significantly correlated with TMH. Subtypes 1 and 5 had the most severe MG atrophy. Subtype 2 had the highest corneal fluorescent staining (CFS). Subtype 4 had the lowest TMH, whereas subtype 5 had the highest. Subtypes 3 and 6 had the largest MG areas, and the upper MG areas of a person's bilateral eyes were highly correlated. Image-based subtypes are related to meibum quality, CFS, and morphological characteristics of MG. In this study, we developed an unsupervised neural network model to cluster patients with dry eye into image-based subtypes using meibography images. We annotated these subtypes with functional and morphological clinical characteristics.

Model metamers reveal divergent invariances between biological and artificial neural networks

Deep neural network models of sensory systems are often proposed to learn representational transformations with invariances like those in the brain. To reveal these invariances, we generated ‘model metamers’, stimuli whose activations within a model stage are matched to those of a natural stimulus. Metamers for state-of-the-art supervised and unsupervised neural network models of vision and audition were often completely unrecognizable to humans when generated from late model stages, suggesting differences between model and human invariances. Targeted model changes improved human recognizability of model metamers but did not eliminate the overall human–model discrepancy. The human recognizability of a model’s metamers was well predicted by their recognizability by other models, suggesting that models contain idiosyncratic invariances in addition to those required by the task. Metamer recognizability dissociated from both traditional brain-based benchmarks and adversarial vulnerability, revealing a distinct failure mode of existing sensory models and providing a complementary benchmark for model assessment.

A theoretical analysis of continuous firing condition for pulse-coupled neural networks with its applications

The pulse-coupled neural network (PCNN) has become a popular biology-inspired model because of its remarkable performance, such as image fusion, segmentation, and recognition. Although the PCNN is an unsupervised neural network model, its parameters are often manually adjusted, which leads to the shortcomings of being time-consuming, verbose, and with bad consistency. Researchers have conducted work on the PCNN's neurodynamic analysis and parameter settings. However, most of these works were proposed for specific application areas, and there are no widely recognized and accepted methods or theories for PCNN parameter setting at present. Thus, how to reduce the difficulty of parameter setting based on neurodynamic analysis is a very important research area for applying PCNN. In this work, we proposed a general formulae of a continuous firing condition based on the dynamic analysis of PCNN neurons to avoid constructing an invalid model and to learn the neurons' firing characteristics. The results obtained from the proposed formulae were used to explore the usability of the PCNN parameter setting theory. We first utilized the pre-existing theory of the PCNN firing period to examine the rationality of our theory, and then we also introduce an image fusion method using our theory and whale optimization algorithm to verify our method and theory, besides, numerical tests and experiments on common images were also performed to verify our theory. Experimental results show that our method and theory is effective.

An unsupervised wavelet neural network model for approximating the solutions of non-linear nervous stomach model governed by tension, food and medicine

The human stomach is a complex organ. Its role is to degrade food particles by using mechanical forces and chemical reactions in order to release nutrients. All ingested items, including our nutrition, should first pass through the stomach, making it arguably the most crucial segment in the gastrointestinal tract. Computational and mathematical modeling of the stomach is an emerging field of biomechanics where several complex phenomena, such as solid mechanics of the gastric wall, gastric electrophysiology, and fluid mechanics of the digesta need to be addressed. Developing a meshfree comprehensive algorithm for solving the nervous stomach model that enables analysing the relationships between these phenomena remains one of the most significant challenges in biomechanics. This research dedicates to study the dynamics of nervous stomach model governed by a mathematical representation depending on three categories viz. Tension (T), Food (F) and Medicine (M), i.e. TFM model. In this regard, a machine learning paradigm, namely POLYnomial WinOwed with Gaussian (PolyWOG) Wavelet Neural Network (PWNN) model has been implemented for handling the non-linear TFM models. We compared the obtained outcomes of present work with results of a well-known numerical computing paradigm and an existing wavelet neural algorithm. Also, we have done statistical assessment studies at different testing points, which reveal that the proposed architecture is effective and accurate.

Unsupervised Graph Representation Learning with Cluster-aware Self-training and Refining

Unsupervised graph representation learning aims to learn low-dimensional node embeddings without supervision while preserving graph topological structures and node attributive features. Previous Graph Neural Networks (GNN) require a large number of labeled nodes, which may not be accessible in real-world applications. To this end, we present a novel unsupervised graph neural network model with Cluster-aware Self-training and Refining ( CLEAR ). Specifically, in the proposed CLEAR model, we perform clustering on the node embeddings and update the model parameters by predicting the cluster assignments. To avoid degenerate solutions of clustering, we formulate the graph clustering problem as an optimal transport problem and leverage a balanced clustering strategy. Moreover, we observe that graphs often contain inter-class edges, which mislead the GNN model to aggregate noisy information from neighborhood nodes. Therefore, we propose to refine the graph topology by strengthening intra-class edges and reducing node connections between different classes based on cluster labels, which better preserves cluster structures in the embedding space. We conduct comprehensive experiments on two benchmark tasks using real-world datasets. The results demonstrate the superior performance of the proposed model over baseline methods. Notably, our model gains over 7% improvements in terms of accuracy on node clustering over state-of-the-arts.

Efficient Non-Sampling Graph Neural Networks

A graph is a widely used and effective data structure in many applications; it describes the relationships among nodes or entities. Currently, most semi-supervised or unsupervised graph neural network models are trained based on a very basic operation called negative sampling. Usually, the purpose of the learning objective is to maximize the similarity between neighboring nodes while minimizing the similarity between nodes that are not close to each other. Negative sampling can reduce the time complexity by sampling a small fraction of the negative nodes instead of using all of the negative nodes when optimizing the objective. However, sampling of the negative nodes may fail to deliver stable model performance due to the uncertainty in the sampling procedure. To avoid such disadvantages, we provide an efficient Non-Sampling Graph Neural Network (NS-GNN) framework. The main idea is to use all the negative samples when optimizing the learning objective to avoid the sampling process. Of course, directly using all of the negative samples may cause a large increase in the model training time. To mitigate this problem, we rearrange the origin loss function into a linear form and take advantage of meticulous mathematical derivation to reduce the complexity of the loss function. Experiments on benchmark datasets show that our framework can provide better efficiency at the same level of prediction accuracy compared with existing negative sampling-based models.

Real-Time Kinematically Synchronous Planning for Cooperative Manipulation of Multi-Arms Robot Using the Self-Organizing Competitive Neural Network.

This paper presents a real-time kinematically synchronous planning method for the collaborative manipulation of a multi-arms robot with physical coupling based on the self-organizing competitive neural network. This method defines the sub-bases for the configuration of multi-arms to obtain the Jacobian matrix of common degrees of freedom so that the sub-base motion converges along the direction for the total pose error of the end-effectors (EEs). Such a consideration ensures the uniformity of the EE motion before the error converges completely and contributes to the collaborative manipulation of multi-arms. An unsupervised competitive neural network model is raised to adaptively increase the convergence ratio of multi-arms via the online learning of the rules of the inner star. Then, combining with the defined sub-bases, the synchronous planning method is established to achieve the synchronous movement of multi-arms robot rapidly for collaborative manipulation. Theory analysis proves the stability of the multi-arms system via the Lyapunov theory. Various simulations and experiments demonstrate that the proposed kinematically synchronous planning method is feasible and applicable to different symmetric and asymmetric cooperative manipulation tasks for a multi-arms system.

Multi-scale asphalt pavement deformation detection and measurement based on machine learning of full field-of-view digital surface data

Asphalt pavement deformation is a common phenomenon due to the material property and traffic loads. Aiming at ensuring traffic comfort and safety, it is essential to constantly monitor the multi-scale pavement deformation with a non-destructive and automatic system. This paper presents a full field-of-view asphalt pavement deformation inspection framework based on multi-dimensional surface data and machine learning, which can detect and measure pavement rutting, roughness and large-span deformation simultaneously. In this integrated system, one-dimensional convolutional neural network (1D CNN) classification and localization models are developed for two-step rutting detection. The quarter-car model with multiple measuring lines is employed for full-lane international roughness index (IRI) measurement and spatial analysis. The unsupervised K-means convolutional neural network (K-CNN) model is proposed for large-span deformation detection. The results show that the overall F1 scores of rutting classification and localization are 99.56% and 97.24%, respectively. The full-lane measurement suggests that IRI presents a bimodal distribution in transverse space due to the concentrated traffic loads on the wheel path. In addition, the unsupervised K-CNN achieves an average consistency index (CI) of 91.61% on large-span sunken and heave deformation segmentation.

Architecture of the health system as an enabler of better wellbeing.

Health systems worldwide have heterogenous capacities and financing characteristics. No clear empirical evidence is available on the possible outcomes of these characteristics for population wellbeing. The study aims to provide empirical insight into health policy alternatives to support the development of health system architecture to improve population wellbeing. We developed an unsupervised neural network model to cluster countries and used the Human Development Index to derive a wellbeing model. The results show that no single health system architecture is associated with a higher level of population wellbeing. Strikingly, high levels of health expenditure and physical health capacity do not guarantee a high level of population wellbeing and different health systems correspond to a certain wellbeing level. Our analysis shows that alternative options exist for some health system characteristics. These can be considered by governments developing health policy priorities.

Outlier Vehicle Trajectory Detection Using Deep Autoencoders in Santiago, Chile.

In the last decade, a large amount of data from vehicle location sensors has been generated due to the massification of GPS systems to track them. This is because these sensors usually include multiple variables such as position, speed, angular position of the vehicle, etc., and, furthermore, they are also usually recorded in very short time intervals. On the other hand, routes are often generated so that they do not correspond to reality, due to artifacts such as buildings, bridges, or sensor failures and where, due to the large amount of data, visual analysis of human expert is unable to detect genuinely anomalous routes. The presence of such abnormalities can lead to faulty sensors being detected which may allow sensor replacement to reliably track the vehicle. However, given the reliability of the available sensors, there are very few examples of such anomalies, which can make it difficult to apply supervised learning techniques. In this work we propose the use of unsupervised deep neural network models based on stacked autoencoders to detect anomalous routes in vehicles within Santiago de Chile. The results show that the proposed model is capable of effectively detecting anomalous paths in real data considering validation given by an expert user, reaching a performance of 82.1% on average. As future work, we propose to incorporate the use of Long Short-Term Memory (LSTM) and attention-based networks in order to improve the detection of anomalous trajectories.

A novel health indicator for intelligent prediction of rolling bearing remaining useful life based on unsupervised learning model

Construction of an intelligent Health Indicator (HI) that can accurately describe the degradation process is a prerequisite to accelerating the development of an automatic remaining useful life (RUL) prediction model for rotating machinery such as bearings. This research aims to develop an intelligent model that can predict the remaining useful life of bearings without physical human intervention. The intelligent HI model, named Multi-Scale-Multi-Head Attention with Automatic Encoder-Decoder (MSMHA-AED), is constructed based on an unsupervised neural network model and can extract multi-scale coded features of bearings from raw vibration signals. The model is fitted with an ensemble health indicator designed to fuse the metrics of healthy and damaged coded features of bearings to create a more reliable health indicator for RUL prediction. The intelligent HI model is subsequently used to develop three neural network-based prognostic models to examine the reliability of the proposed health indicator in RUL predictions. Using the prognostic model, the magnitude of degradation in the bearings is estimated by measuring the similarity between the coded features of healthy and unknown damaged bearings using three measurement methods. It was found that the similarity measured by the Wasserstein distance method offers more suitable results for damage quantification due to its unique capability of measuring health indicators in more monotonic condition. It is also found that the proposed model is less prone to giving false alarms even when used to detect degradation for the first time. All performance indicators of the proposed approach show better and more robust metrics than the state-of-art methods.

Anomaly Detection for Shielded Cable Including Cable Joint Using a Deep Learning Approach

A defect occurring in cable joints is much more severe than a defect on cables due to its high incidence and an intensive electrical stress. In conventional reflectometry, it is hard to distinguish between a reflected signal from normal cable joints and that from faulty cable joints. This paper proposes a novel time–frequency domain reflectometry (TFDR) method based on an unsupervised neural network model combining long short-term memory (LSTM) and variational autoencoder (VAE) that can detect joint defects as well as cable defects. To verify the proposed method, a test bed is constructed with two failure scenarios; (1) defects on cables, and (2) defects in cable joints. In both scenarios, the proposed method successfully detects the failure using an anomaly score that conventional TFDR does not have. The proposed anomaly detection technique is expected to become the cornerstone of systems that can detect anomalies of earlier-stage of defects.

Improving Extractive Text Summarization Performance Using Enhanced Feature Based RBM Method

Text summarization is the process of creating a short, accurate and fluent summary of a longer text document. As plenty of digital data is available online, automatic text summarization methods greatly needed to help and understand the lengthy & complex documents quickly by discovering the relevant information. This paper proposes the text summarization method for short news articles and long scientific papers using unsupervised neural network model. The proposed method works in four main steps: input document pre-processing, feature extraction, feature enhancement and final summary generation. We have extracted combination of various statistical and linguistic features from input document, which helps in improving the quality of sentence selection. Further Restricted Boltzmann Machine (RBM) model is used to capture & enhance the discriminative, abstract features in an unsupervised way to improve the overall performance without losing any significant information. Sentences are scored based on enhanced feature set and top sentences are selected for final extractive summary. Performance of the proposed method is evaluated using Rouge score and compared with TextRank, LexRank, LSA & Luhn baseline methods and the results demonstrates that proposed methodology performs better compared to other methods.

Recursive training based physics-inspired neural network for electric water heater modeling

Aggregating flexibility from residential electric water heaters (EWHs) is fast gaining commercial interest. Flexibility modeling of an EWH involves highly precise and quick simulation of EWH water temperature using the EWH thermal dynamics model for various flexibility control actions. Since EWH tank water temperature data is usually unavailable or costly to obtain, developing an accurate and computationally inexpensive EWH thermal dynamics model with limited sensor data is essential for devising advanced control strategies for EWH flexibility aggregation. In this paper, we present a novel recursive training-based unsupervised physics-informed neural network (PINN) model for predicting tank water temperature which requires only historical EWH power consumption data to train the model. PINN models enable the integration of domain knowledge from traditional physical processes and methods into neural network (NN) models. Single-zone thermal grey-box differential equation model (DEM) is used as the basis to develop and demonstrate proof-of-concept of the proposed approach. Physics from the single-zone model is encoded into the PINN loss function to incorporate domain knowledge and the PINN architecture is structured to mimic the single-zone DEM. The recursive training approach enables the use of previous-step water temperature as an input to the simulation model. Two separate models for EWH ON- and OFF-states are developed and trained with real-world EWH power consumption data. Water temperature prediction results indicate that the proposed approach has similar performance as the traditional single-zone DEM model, thereby demonstrating the ability of the proposed model to learn the underlying physics behind the single-zone model without water temperature data. The proposed model has high accuracy and performs well outside the control set point temperatures indicating its suitability for simulating load shifting and other DR events. Additionally, EWH simulation results for two different scenarios with different water demand compositions are presented to study the effects of propagation errors on temperature prediction. The proposed approach paves the way for developing advanced EWH flexibility modeling tools for the aggregator to precisely control a large portfolio of EWHs considering user comfort and rebound effects.

Visual numerosity perception shows no advantage in real-world scenes compared to artificial displays

While the human visual system is sensitive to numerosity, the mechanisms that allow perception to extract and represent the number of objects in a scene remains unknown. Prominent theoretical approaches posit that numerosity perception emerges from passive experience with visual scenes throughout development, and that unsupervised deep neural network models mirror all characteristic behavioral features observed in participants. Here, we derive and test a novel prediction: if the visual number sense emerges from exposure to real-world scenes, then the closer a stimulus aligns with the natural statistics of the real world, the better number perception should be. But – in contrast to this prediction - we observe no such advantage (and sometimes even a notable impairment) in number perception for natural scenes compared to artificial dot displays in college-aged adults. These findings are not accounted for by the difficulty in object identification, visual clutter, the parsability of objects from the rest of the scene, or increased occlusion. This pattern of results represents a fundamental challenge to recent models of numerosity perception based in experiential learning of statistical regularities, and instead suggests that the visual number sense is attuned to abstract number of objects, independent of their underlying correlation with non-numeric features. We discuss our results in the context of recent proposals that suggest that object complexity and entropy may play a role in number perception.

SFNet: A slow feature extraction network for parallel linear and nonlinear dynamic process monitoring

In a typical industrial process, there may exist both linear and nonlinear relationships among process variables. Besides, the existence of process dynamics poses challenges to process monitoring. Some monitoring methods have been developed for dynamic processes. However, purely linear or nonlinear methods can hardly tackle the hybrid linear and nonlinear relationships among process variables. In this work, a novel unsupervised neural network model, named slow feature network (SFNet), is proposed to monitor dynamic processes. In SFNet, a linear mapping module and a transform gate structure are constructed to realize the parallel extraction of linear and nonlinear characteristics. Moreover, slow constraints on features are designed to capture the velocity characteristics while retaining variability information so that the process dynamic behavior can be directly observed. Various statistics are developed to monitor the process status from both variability and velocity as well as both linear and nonlinear perspectives. Thus, monitoring results corresponding to different statistical information reflect different process behavior and fault characteristics with meaningful interpretations. One numerical example and two real industrial examples are adopted to validate the proposed method. For these experimental examples, the proposed SFNet method achieves an average false alarm rate of 5.17% (min 0.8%, max 8.08%), an average detection accuracy of 87.5% (min 50.2%, max 100%), and an average precision of 98.6% (min 96.3%, max 100%), outperforming counterpart methods in the literature.

Matching sensor ontologies with unsupervised neural network with competitive learning

Sensor ontologies formally model the core concepts in the sensor domain and their relationships, which facilitates the trusted communication and collaboration of Artificial Intelligence of Things (AIoT). However, due to the subjectivity of the ontology building process, sensor ontologies might be defined by different terms, leading to the problem of heterogeneity. In order to integrate the knowledge of two heterogeneous sensor ontologies, it is necessary to determine the correspondence between two heterogeneous concepts, which is the so-called ontology matching. Recently, more and more neural networks have been considered as an effective approach to address the ontology heterogeneity problem, but they require a large number of manually labelled training samples to train the network, which poses an open challenge. In order to improve the quality of the sensor ontology alignment, an unsupervised neural network model is proposed in this work. It first models the ontology matching problem as a binary classification problem, and then uses a competitive learning strategy to efficiently cluster the ontologies to be matched, which does not require the labelled training samples. The experiment utilizes the benchmark track provided by the Ontology Alignment Evaluation Initiative (OAEI) and multiple real sensor ontology alignment tasks to test our proposal’s performance. The experimental results show that the proposed approach is able to determine higher quality alignment results compared to other matching strategies under different domain knowledge such as bibliographic and real sensor ontologies.