Simple Neural Network Research Articles

Pattern Augmented Lightweight Convolutional Neural Network for Intrusion Detection System

As the world increasingly becomes more interconnected, the demand for safety and security is ever-increasing, particularly for industrial networks. This has prompted numerous researchers to investigate different methodologies and techniques suitable for intrusion detection systems (IDS) requirements. Over the years, many studies have proposed various solutions in this regard, including signature-based and machine learning (ML)-based systems. More recently, researchers are considering deep learning (DL)-based anomaly detection approaches. Most proposed works in this research field aim to achieve either one or a combination of high accuracy, considerably low false alarm rates (FARs), high classification specificity and detection sensitivity, lightweight DL models, or other ML and DL-related performance measurement metrics. In this study, we propose a novel method to convert a raw dataset to an image dataset to magnify patterns by utilizing the Short-Term Fourier transform (STFT). The resulting high-quality image dataset allowed us to devise an anomaly detection system for IDS using a simple lightweight convolutional neural network (CNN) that classifies denial of service and distributed denial of service. The proposed methods were evaluated using a modern dataset, CSE-CIC-IDS2018, and a legacy dataset, NSLKDD. We have also applied a combined dataset to assess the generalization of the proposed model across various datasets. Our experimental results have demonstrated that the proposed methods achieved high accuracy and considerably low FARs with high specificity and sensitivity. The resulting loss and accuracy curves have demonstrated the efficacy of our raw dataset to image dataset conversion methodology, which is evident as an excellent generalization of the proposed lightweight CNN model was observed, effectively avoiding overfitting. This holds for both the modern and legacy datasets, including their mixed versions.

Heatwave 1987: the Piraeus versus Athens case.

Heatwaves represent the main indices of climate change, while mortality is one of the established markers of their human effects. For unknown reasons populations adapt to temperature variations/challenges differently. Thus, to allow better precision and prediction, heatwave evaluations should be enriched by historical context and local data. The mortality data for 1987 were collected from the Piraeus municipality registry, whereas data for Athens were obtained from literature retrieved from PUBMED. Ambient characteristics were extracted from the Geronikolou's 1991 BSc thesis and the reports of national organizations. From the death events, the odds ratio and relative risk in Piraeus compared to the Athens were calculated. Finally, a simple neural network proposed the dominant ambient parameter of the heatwave effects in the city residents of each location. The 1987 heatwave was more lethal (seven-fold) in Athens than in Piraeus and dependent on atmospheric nitric oxide (NO) concentration (with probability 0.999). In the case of Piraeus in 1987, ozone characterized the phenomenon (with probability 0.993). The odds of dying due to a heatwave are highly dependent on lifestyle, population sensitivity to preventive measures and public health policy, while the phenomenon was mainly moderated by ozone in Piraeus in 1987, and NO in Athens irrespective of year.

A simple probabilistic neural network for machine understanding

We discuss the concept of probabilistic neural networks with a fixed internal representation being models for machine understanding. Here, ‘understanding’ is interpretted as the ability to map data to an already existing representation which encodes an a priori organisation of the feature space. We derive the internal representation by requiring that it satisfies the principles of maximal relevance and of maximal ignorance about how different features are combined. We show that, when hidden units are binary variables, these two principles identify a unique model—the hierarchical feature model—which is fully solvable and provides a natural interpretation in terms of features. We argue that learning machines with this architecture possess a number of interesting properties, such as the continuity of the representation with respect to changes in parameters and data, the possibility of controlling the level of compression and the ability to support functions that go beyond generalisation. We explore the behaviour of the model with extensive numerical experiments and argue that models in which the internal representation is fixed reproduce a learning modality which is qualitatively different from that of traditional models, such as restricted Boltzmann machines.

Effective time-series Data Augmentation with Analytic Wavelets for bearing fault diagnosis

In the realm of rotary machine maintenance, rolling bearings emerge as crucial yet frequently vulnerable components. Ensuring their operational integrity is pivotal for the overall safety and performance of the machinery. Traditional diagnostic methods have utilized a variety of signal processing techniques, ranging from blind source separation to wavelet analysis, and more recently, deep neural networks (DNNs). However, a significant challenge in this domain is the scarcity of labeled failure data in real-world applications, leading to a reliance on data augmentation (DA) techniques to enhance the quality and quantity of the training data. Compounding this problem is the use of DA techniques tailored for images, which are not suitable for the time–frequency representations. This study proposes an innovative Data Augmentation with Analytic Wavelets (DAAW) approach, tailored specifically for time-series (TS) data classification in bearing fault diagnosis. The essence of DAAW lies in its ability to generate synthetic scalogram samples that closely mirror the properties of original samples. The proposed DAAW is directly applied at the input stage of a learning model by successively adjusting predefined parameters to generate scalograms, without the need for auxiliary DA algorithms. This technique is versatile in generating synthetic time–frequency data samples, and the parameterization enables it to be tailored to different time series datasets. Through experimentation with publicly available datasets, we show that combining a simple Convolutional Neural Network (CNN) with the proposed approach results in improved performance. It outperforms existing methods, achieving a mean classification accuracy of 99.49%. The critical distance for average accuracy, as determined by the Nemenyi post-hoc test, is 0.4619. Our results also show enhanced performance with accuracy rates of 90.14%, 78.94%, and 62.86% even under low signal-to-noise ratio (SNR) conditions, using 80%, 60%, and 50% of the dataset, respectively, for model training. Notably, its efficacy remains robust even under challenging low SNR conditions. The proposed DAAW significantly improves classification accuracy in bearing fault diagnosis, simultaneously obviating the need for proxy DA tasks.

Using a simple radial basis function neural network to predict the glass transition temperature of alkali borate glasses

Artificial Neural Networks (ANN) are powerful machine learning algorithms. In the context of glass science, ANNs have been successfully used to induce composition-property regression models. These models are already influencing the way that we design new glasses, moving away from the traditional “cook and look” approach to computer-aided design. In this work, we induced an ANN model for the glass transition temperature Tg for glasses of the family xM2O⋅(100−x)B2O3, where M = Li, Na, K, Rb, and Cs, using simple radial basis function neural networks (RBF) with only two neurons. Statistically, the results of our methodologies are comparable to other complex models (empirical or physical) available in the literature with more neurons, such as the recent GlassNet and the topological model proposed by Mauro and co-authors (2009). This result supports the idea that a simpler specialized model can perform better than a complex general model for a restricted training domain.

Hybrid convolutional long short‐term memory models for sales forecasting in retail

AbstractThis study proposes novel sales forecasting approaches that merge deep learning methods in a hybrid model. Long short‐term memory (LSTM) is adopted for modeling the temporal characteristics of the data, whereas the convolutional neural network (CNN) focuses on identifying and extracting relevant exogenous information. We propose stacked (S‐CNN‐LSTM) and parallel (P‐CNN‐LSTM) hybrid architectures to understand complex time series data with varying seasonal patterns and multiple products correlations. The performance drivers of both architectures were empirically tested with a real‐world multivariate retail dataset and outperformed when compared with simple neural network architectures and standard autoregressive methods for short and long‐term forecasting horizons. When compared with traditional predictive approaches, the proposed hybrid models reduce the computational complexity while providing flexibility and robustness.

Resolving inter-crystal scatter in a light-sharing depth-encoding PET detector

Objective. Inter-crystal scattering (ICS) in light-sharing positron emission tomography (PET) detectors leads to ambiguity in positioning the initial interaction, which significantly degrades the contrast, quantitative accuracy, and spatial resolution of the resulting image. Here, we attempt to resolve the positioning ambiguity of ICS in a light-sharing depth-encoding detector by exploiting the confined, deterministic light-sharing enabled by the segmented light guide unique to Prism-PET. Approach. We first considered a test case of ICS between two adjacent crystals using an analytical and a neural network approach. The analytical approach used a Bayesian estimation framework constructed from a scatter absorption model—the prior—and a detector response model—the likelihood. A simple neural network was generated for the same scenario, to provide mutual validation for the findings. Finally, we generalized the solution to three-dimensional event positioning that handles all events in the photopeak using a convolutional neural network with unique architecture that separately predicts the identity and depth-of-interaction (DOI) of the crystal containing the first interaction. Main results. The analytical Bayesian method generated an estimation error of 20.5 keV in energy and 3.1 mm in DOI. Further analysis showed that the detector response model was sufficiently robust to achieve adequate performance via maximum likelihood estimation (MLE), without prior information. We then found convergent results using a simple neural network. In the generalized solution using a convolutional neural network, we found crystal identification accuracy of 83% and DOI estimation error of 3.0 mm across all events. Applying this positioning algorithm to simulated data, we demonstrated significant improvements in image quality over the baseline, centroid-based positioning approach, attaining 38.9% improvement in intrinsic spatial resolution and enhanced clarity in hot spots of diameters 0.8 to 2.5 mm. Significance. The accuracy of our findings exceeds those of previous reports in the literature. The Prism-PET light guide, mediating confined and deterministic light-sharing, plays a key role in ICS recovery, as its mathematical embodiment—the detector response model—was the essential driver of accuracy in our results.

Monthly Global Solar Radiation Model Based on Artificial Neural Network, Temperature Data and Geographical and Topographical Parameters: A Case Study in Spain

Solar energy plays an essential role in the current energy context to achieve sustainable development while supplying energy needs, creating jobs, and protecting the environment. Many solar radiation models have provided valid estimates at many different locations, using appropriate input variables for specific climatic conditions, but predictions are less accurate on a regional scale. Since radiometric weather stations are relatively dispersed, even in the most developed countries, it is interesting to develop indirect models based on measurements that are common in secondary network stations. This paper develops a monthly global solar radiation model based on a simple neural network structure, using temperature, geographical, and topographical data from 105 meteorological stations, representative of the whole of peninsular Spain. A hierarchical clustering procedure was employed to select the data used to train and validate the model. To avoid functional dependencies between parameters and variables, which hinder the generality of the model, all input and output variables are dimensionless. The estimates fit the 1260 monthly data with RRMSE values of about 6%, which improves results obtained previously, using regression models, and proves that simplicity is compatible with the generality and accuracy of a model, even in large regions with very varied characteristics.

Coupled HR–HNN Neuron with a Locally Active Memristor

Local activity could be the source for complexity. In this study, a multistable locally active memristor is proposed, whose nonvolatile memory, as well as locally active characteristics, is validated by the power-off plot and DC [Formula: see text]–[Formula: see text] plot. Based on the two-dimensional Hindmarsh–Rose neuron and a one-dimensional Hopfield neuron, a simple neural network is constructed by connecting the two neurons with the locally active memristor. Coexisting multiple firing patterns under different initial conditions are investigated according to the controlled coupling factor. The results suggest that the system exhibits coexisting periodic and chaotic bursting with different firing patterns. Complex firing only occurs in the locally active area of the defined memristor, meanwhile the system shows a periodic oscillation in the passive area. Beyond this, the coupled neurons exhibit the specific phenomenon of attractor growing in the locally active region of the memristor. The circuit simulations by Power Simulation (PSIM) are included confirming the numerical simulations and theoretic analysis.

Abstract 65: Automated Detection of Facial Droop: Ml/AI Based Tool for Early Detection of Stroke

Introduction: Facial droop (FD) is a common symptom (45-60%) of stroke so early detection is critical for timely treatment. FD is a part of almost all stroke scales. Our study is to develop a machine learning (ML)/AI based tool that uses an automated detection of FD as one of the components. Our approach uses a normalized value of minor facial expressions called a blendshape. Methods: Google’s Mediapipe was used to create 52 different blendshape values (e.g., left eyebrow raised, right mouth lowered, etc.) from an input image, which were then used as inputs for a simple deep neural network (DNN). We trained this DNN using the blendshapes from 872 images of people with symptoms of facial palsy (FP) and 872 images of people without symptoms exhibiting multiple emotions (neutral, smiling, frowning, etc.). Results: The proposed approach was evaluated on a dataset of facial images of patients with FD and strokes. Our model showed promising results with over 91% test accuracy. This result suggests that the proposed approach is a promising new method for the early detection of FP with high accuracy. The approach is integrated in our ML/AI based tool to replicate different stroke scales. Conclusions: Our model shows that the use of Blendshapes as features in a ML classifier can produce great results in the detection of FP. For greater accuracy, a better dataset with higher quality images as well as unilateral facial expressions (winking, smirks, etc.) should be used. In a future study, a ML model that uses Blendshapes as inputs could be used to classify FP on a scale.

A self-supervised learning model based on variational autoencoder for limited-sample mammogram classification

Deep learning models have found extensive application in medical imaging analysis, particularly in mammography classification. However, these models encounter challenges associated with limited annotated mammography public datasets. In recent years, self-supervised learning (SSL) has emerged as a noteworthy solution to addressing data scarcity by leveraging pretext and downstream tasks. Nevertheless, we recognize a notable scarcity of self-supervised learning models designed for the classification task in mammography. In this context, we propose a novel self-supervised learning model for limited-sample mammogram classification. Our proposed SSL model comprises two primary networks. The first is a pretext task network designed to learn discriminative features through mammogram reconstruction using a variational autoencoder (VAE). Subsequently, the downstream network, dedicated to the classification of mammograms, uses the encoded space extracted by the VAE as input through a simple convolutional neural network. The performance of the proposed model is assessed on public INbreast and MIAS datasets. Comparative analyzes are conducted for the proposed model against previous studies for the same classification task and dataset. The proposed SSL model demonstrates high performance with an AUC of 0.94 for density, 0.99 for malignant-nonmalignant classifications on INbreast, 0.97 for benign-malignant, 0.99 for density, and 0.99 for normal-benign-malignant classifications on MIAS. Additionally, the proposed model reduces computational costs with only 228 trainable parameters, 204.95K FLOPs, and a depth of 3 in mammogram classification. Overall, the proposed SSL model exhibits a robust network architecture characterized by repeatability, consistency, generalization ability, and transferability among datasets, providing less computational complexity than previous studies.Graphical abstract

Neural network approximation in the Lipkin-Meshkov-Glick model

In this paper we approximate the wave function of the Lipkin-Meshkov-Glick (LMG) model by a simple feed-forward neural network. By using the NetKet toolkit we obtain the ground state energies of LMG systems with different interaction strengths and find out that they are very close to the exact values.

Collective Decision of One-vs-Rest Networks for Open-Set Recognition.

Unknown examples that are unseen during training often appear in real-world pattern recognition tasks, and an intelligent self-learning system should be able to distinguish between known examples and unknown examples. Accordingly, open-set recognition (OSR), which addresses the problem of classifying knowns and identifying unknowns, has recently been highlighted. However, conventional deep neural networks (DNNs) using a softmax layer are vulnerable to overgeneralization, producing high confidence scores for unknowns. In this article, we propose a simple OSR method that is based on the intuition that the OSR performance can be maximized by setting strict and sophisticated decision boundaries that reject unknowns while maintaining satisfactory classification performance for knowns. For this purpose, a novel network structure, in which multiple one-vs-rest networks (OVRNs) follow a convolutional neural network (CNN) feature extractor, is proposed. Here, an OVRN is a simple feedforward neural network that is designed to assign confidence scores that are lower than those in the softmax layer to unknown samples so that unknown samples can be more effectively separated from known classes. Furthermore, the collective decision score is modeled by combining the multiple decisions reached by the OVRNs to alleviate overgeneralization. Extensive experiments were conducted on various datasets, and the experimental results show that the proposed method performs significantly better than the state-of-the-art methods by effectively reducing overgeneralization. The code is available at https://github.com/JaeyeonJang/Openset-collective-decision.

Enhancing Patient Outcome Predictions Through Differential Privacy with Optimized RNN Model Applied to Electronic Health Record

This study introduces a novel method for predicting patient admission and discharge events using Electronic Health Records (EHR) while prioritizing the protection of personal privacy. Our approach utilizes a Simple Recurrent Neural Network (RNN) model enhanced with a Differential Privacy mechanism to strike a balance between high-accuracy outcome predictions and the confidentiality of patient data. At the core of our methodology is the application of a Simple RNN to EHR data, which facilitates the prediction of whether a patient will be admitted to or discharged from a healthcare facility. To strengthen data confidentiality, we integrate Differential Privacy by injecting controlled noise into the dataset, ensuring that our model’s predictions preserve the privacy of individual patient records. We conducted experiments using six different classifiers to implement our privacy-preserving prediction strategy. Among these, the Random Forest classifier emerged as the most accurate, proving the effectiveness of our method in providing reliable predictions without compromising privacy. In contrast, the XG Boost classifier showed the least precision, indicating some limitations in its ability to balance privacy with predictive accuracy. This research significantly advances the field of healthcare informatics by presenting a sophisticated solution that combines cutting-edge predictive models with stringent privacy safeguards. Our findings highlight the critical need to maintain a delicate balance between achieving precise clinical predictions and upholding the moral responsibility to protect patient privacy in the modern landscape of digital health records.

Spectroscopic Distance, Mass, and Age Estimations for APOGEE DR17

We derive distances and masses of stars from the Sloan Digital Sky Survey (SDSS) Apache Point Observatory Galactic Evolution Experiment Data Release 17 using simple neural networks. Training data for distances comes from Gaia EDR3, supplemented by literature distances for star clusters. For masses, the network is trained using asteroseismic masses for evolved stars and isochrone masses for main-sequence stars. The models are trained on effective temperature, surface gravity, metallicity, and carbon and nitrogen abundances. We found that our distance predictions have median fractional errors that range from ≈20% at low log g and ≈10% at higher log g with a standard deviation of ≈11%. The mass predictions have a standard deviation of ±12%. Using the masses, we derive ages for evolved stars based on the correspondence between mass and age for giant stars given by isochrones. The results are compiled into a Value Added Catalog called DistMass that contains distances and masses for 733,901 independent spectra, plus ages for 396,548 evolved stars.

An Embedded Machine Learning Fault Detection System for Electric Fan Drive

Industrial fans are critical components in industrial production, where unexpected damage of important fans can cause serious disruptions and economic costs. One trending market segment in this area is where companies are trying to add value to their products to detect faults and prevent breakdowns, hence saving repair costs before the main product is damaged. This research developed a methodology for early fault detection in a fan system utilizing machine learning techniques to monitor the operational states of the fan. The proposed system monitors the vibration of the fan using an accelerometer and utilizes a machine learning model to assess anomalies. Several of the most widely used algorithms for fault detection were evaluated and their results benchmarked for the vibration monitoring data. It was found that a simple Convolutional Neural Network (CNN) model demonstrated notable accuracy without the need for feature extraction, unlike conventional machine learning (ML)-based models. Additionally, the CNN model achieved optimal accuracy within 30 epochs, demonstrating its efficiency. Evaluating the CNN model performance on a validation dataset, the hyperparameters were updated until the optimal result was achieved. The trained model was then deployed on an embedded system to make real-time predictions. The deployed model demonstrated accuracy rates of 99.8%, 99.9% and 100.0% for Fan-Fault state, Fan-Off state, and Fan-On state, respectively, on the validation data set. Real-time testing further confirmed high accuracy scores ranging from 90% to 100% across all operational states. Challenges addressed in this research include algorithm selection, real-time deployment onto an embedded system, hyperparameter tuning, sensor integration, energy efficiency implementation and practical application considerations. The presented methodology showcases a promising approach for efficient and accurate fan fault detection with implications for broader applications in industrial and smart sensing applications.

Research on optical computing system architecture for simple recurrent neural networks

Moore’s Law, relating to the speed and capabilities of computers is becoming less applicable. In this ‘post-Moore’ era, a cross-disciplinary team based in the Constructive Electronics Laboratory, Kyushu University, Japan, is investigating optical computing system infrastructures, with a view to driving computing technology forward in a way that negates the need to comply with Moore’s Law. Associate Professor Satoshi Kawakami is an expert in electric circuits and computer architecture who is part of the team. The team’s expertise covers materials, devices, circuits, architectures and algorithms and is geared towards pioneering new computing technologies in the post-Moore era. Kawakami believes that the continuous improvement of computer systems with higher performance and lower power consumption/energy consumption will be essential to realise a sustainable advanced information society and wants to maximise the advantages of devices and hide their disadvantages at the system level, which will necessitate collaboration with higher system layers. Another important goal is reducing power consumption by improving the efficiency of computers. In one current project, the researchers are exploring optical computing system infrastructure for simple recurrent neural networks. The team is keen to re-examine the ideal state of optical circuits from the perspective of the entire system, including electrical memory and interfaces.

A simple neural network solar tracker for optimizing conversion efficiency in off-grid solar generators

A new model of neural network and a new type of neural controller are proposed, aiming to reduce cost and complexity without sacrificing efficiency of traditional, more complex neural net-based solar trackers. The solution is derived from Mark Tilden’s neural and nervous networks, using a biologic analogy to seamlessly integrate sensors, artificial neurons and effectors in a single, efficient device. Testing is in progress at the time of the elaboration of this paper but available and relevant preliminary results are shown. The project aims to develop a small pilot tracker – based solar plant for testing purposes and to develop a useable technology for the ever-growing demand for green power.

Training a shallow NN to erase ink seepage in historical manuscripts based on a degradation model

In historical recto–verso manuscripts, very often the text written on the opposite page of the folio penetrates through the fiber of the paper, so that the texts on the two sides appear mixed. This is a very impairing damage that cannot be physically removed, and hinders both the work of philologists and palaeographers and the automatic analysis of linguistic contents. A procedure based on neural networks (NN) is proposed here to clean up the complex background of the manuscripts from this interference. We adopt a very simple shallow NN whose learning phase employs a training set generated from the data itself using a theoretical blending model that takes into account ink diffusion and saturation. By virtue of the parametric nature of the model, various levels of damage can be simulated in the training set, favoring a generalization capability of the NN. More explicitly, the network can be trained without the need for a large class of other similar manuscripts, but is still able, at least to some extent, to classify manuscripts with varying degrees of corruption. We compare the performance of this NN and other methods both qualitatively and quantitatively on a reference dataset and heavily damaged historical manuscripts.

Automatic change-point detection in time series via deep learning

Abstract Detecting change points in data is challenging because of the range of possible types of change and types of behaviour of data when there is no change. Statistically efficient methods for detecting a change will depend on both of these features, and it can be difficult for a practitioner to develop an appropriate detection method for their application of interest. We show how to automatically generate new offline detection methods based on training a neural network. Our approach is motivated by many existing tests for the presence of a change point being representable by a simple neural network, and thus a neural network trained with sufficient data should have performance at least as good as these methods. We present theory that quantifies the error rate for such an approach, and how it depends on the amount of training data. Empirical results show that, even with limited training data, its performance is competitive with the standard cumulative sum (CUSUM) based classifier for detecting a change in mean when the noise is independent and Gaussian, and can substantially outperform it in the presence of auto-correlated or heavy-tailed noise. Our method also shows strong results in detecting and localizing changes in activity based on accelerometer data.