Simple Neural Network Research Articles

Analyzing the heterogenous effects of factors on high-range speeding likelihood of taxi speeders: Does explainable deep learning provides more insights than random parameter approach?

The random parameters Generalized Linear Model (GLM) is frequently used to model speeding characteristics and capture the heterogenous effects of factors. However, this statistical approach is seldom employed for prediction and generalization due to the challenge of transferring its predefined errors. Recently, the emergence of explainable AI techniques has illuminated a new path for analyzing factors associated with risky driving behaviors. Despite this, there remains a gap that comparing results from machine and deep learning (ML/DL) approaches with those from random parameters GLM. This study aims to apply the random parameter GLM and explainable deep learning to evaluate the heterogenous effects of factors on the taxis’ high-range speeding likelihood. Initially, a Beta GLM with random parameters (BGLM-RP) is developed to model the high-range speeding likelihood among taxi drivers. Additionally, XGBoost, a simple convolutional neural network (Simple-CNN), a deeper CNN (DCNN), and a deeper CNN with self-attention (DCNN-SA) are developed. The quantified explanations and illustrations of the factors’ heterogenous effects from ML/DL models are derived from pseudo coefficients by decomposing factors’ SHapley Additive exPlanations (SHAP) values. All the developed statistical, ML, and DL models are compared in terms of mean absolute errors and mean square errors on testing and full data. Results show that DCNN-SA excels in prediction on testing data, indicating its superior generalization capabilities, while BGLM-RP outperforms other models on full data. The DCNN-SA can reveal the heterogenous effects of factors for both in-sample and out-of-sample data, which is not possible for the random parameter GLM. However, BGLM-RP can reveal larger magnitudes of the factors’ heterogenous effects for in-sample data. The signs and significances are identical between the varying coefficients from BGLM-RP and the pseudo coefficients from the ML/DL models, demonstrating the validity and rationale of using the proposed explanation framework to quantify the factors’ effects in ML/DL models. The study also discusses the contributions of various factors to the high-range speeding likelihood of taxi drivers.

A hybrid deep recurrent artificial neural network with a simple exponential smoothing feedback mechanism

Both classical forecasting methods and machine learning approaches are used to solve forecasting problems. Deep artificial neural networks, one of the machine learning methods, are widely used today and give very good results. Recurrent neural networks, a type of deep neural network, are very important in forecasting problems. Simple recurrent artificial neural networks, which are the simplest deep recurrent neural networks, are often preferred in solving forecasting problems due to the small number of parameters they use. Simple exponential smoothing, one of the classical forecasting methods, attracts attention with its performance in solving forecasting problems. The motivation of the study is to create a new forecasting method by combining a classical and simple forecasting method with a deep recurrent artificial neural network in an architecture. In this, a new hybrid deep recurrent artificial neural network with a simple exponential smoothing feedback mechanism is proposed. The architecture of the proposed method is created as a combination of simple recurrent artificial neural networks and simple exponential smoothing methods. In the training of the proposed method, two training algorithms based on sine cosine optimization and particle swarm optimization algorithms are proposed. In these training algorithms, two different solution strategies such as restarting, and early stopping rule are used to avoid overfitting and local optimum problems. The performance of the proposed method is analysed using stock market datasets and compared with both different deep and shallow artificial neural networks and classical forecasting methods. As a result of the analyses, it is concluded that the proposed method is successful in one step ahead of forecasting performance.

Generating Daily High-Resolution Regional XCO2 by Deep Neural Network and Multi-Source Data

CO2 is one of the primary greenhouse gases impacting global climate change, making it crucial to understand the spatiotemporal variations of CO2. Currently, commonly used satellites serve as the primary means of CO2 observation, but they often suffer from striping issues and fail to achieve complete coverage. This paper proposes a method for constructing a comprehensive high-spatiotemporal-resolution XCO2 dataset based on multiple auxiliary data sources and satellite observations, utilizing multiple simple deep neural network (DNN) models. Global validation results against ground-based TCCON data demonstrate the excellent accuracy of the constructed XCO2 dataset (R is 0.94, RMSE is 0.98 ppm). Using this method, we analyze the spatiotemporal variations of CO2 in China and its surroundings (region: 0°–60° N, 70°–140° E) from 2019 to 2020. The gapless and fine-scale CO2 generation method enhances people’s understanding of CO2 spatiotemporal variations, supporting carbon-related research.

MIMO Signal Detection Based on IM-LSTMNet Model

Signal detection is crucial in multi-input multi-output orthogonal frequency division multiplexing (MIMO-OFDM) systems, yet classical detection methods often struggle with nonlinear issues in wireless channels. To handle this challenge, we propose a novel signal detection method for MIMO-OFDM system based on the fractional Fourier transform (FrFT), leveraging the robust time series processing capabilities of long short-term memory (LSTM) networks. Our innovative approach, termed IM-LSTMNet, integrates LSTM with convolutional neural networks (CNNs) and incorporates a Squeeze and Excitation Network to emphasize critical information, enhancing neural network performance. The proposed IM-LSTMNet is applied to the FrFT-based MIMO-OFDM system to improve signal detection performance. We compare the detection results of IM-LSTMNet with zero forcing (ZF), minimum mean square error (MMSE), simple LSTM neural network, and CNN–LSTM network by evaluating the bit error rate. Experimental results demonstrate that IM-LSTMNet outperforms ZF, MMSE, LSTM, and other methods, significantly enhancing system signal detection performance. This work offers a promising advancement in MIMO-OFDM signal detection, presenting a deep learning-based solution that effectively improves the system signal detection performance.

Deep neural networks as variational solutions for correlated open quantum systems

In this work we apply deep neural networks to find the non-equilibrium steady state solution to correlated open quantum many-body systems. Motivated by the ongoing search to find more powerful representations of (mixed) quantum states, we design a simple prototypical convolutional neural network and show that parametrizing the density matrix directly with more powerful models can yield better variational ansatz functions and improve upon results reached by neural density operator based on the restricted Boltzmann machine. Hereby we give up the explicit restriction to positive semi-definite density matrices. However, this is fulfilled again to good approximation by optimizing the parameters. The great advantage of this approach is that it opens up the possibility of exploring more complex network architectures that can be tailored to specific physical properties. We show how translation invariance can be enforced effortlessly and reach better results with fewer parameters. We present results for the dissipative one-dimensional transverse-field Ising model and a two-dimensional dissipative Heisenberg model compared to exact values.

A non-memory-based functional neural framework for animal caching behavior

The brain’s extraordinary abilities are often attributed to its capacity to learn and adapt. But memory has its limitations, especially when faced with tasks such as retrieving thousands of food items—a common behavior in scatter-hoarding animals. Here, we propose a brain mechanism that may facilitate caching and retrieval behaviors, with a focus on hippocampal spatial cells. Rather than memorizing the locations of their caches, as previously hypothesized, we suggest that cache-hoarding animals employ a static mechanism akin to hash functions commonly used in computing. Our mathematical model aligns with the activity of hippocampal spatial cells, which respond to an animal’s positional attention. We know that the region that activates each spatial cell remains consistent across subsequent visits to the same area but not between areas. This remapping, combined with the uniqueness of cognitive maps, produces persistent hash functions that can serve both food caching and retrieval. We present a simple neural network architecture that can generate such a probabilistic hash that is unique to the animal and not sensitive to environmental changes. This mechanism could serve a virtually boundless capacity for the encoding of any structured data.

Hybrid CMOS-Memristor synapse circuits for implementing Ca ion-based plasticity model

Neuromorphic computing research is being actively pursued to address the challenges posed by the need for energy-efficient processing of big data. One of the promising approaches to tackle the challenges is the hardware implementation of spiking neural networks (SNNs) with bio-plausible learning rules. Numerous research works have been done to implement the SNN hardware with different synaptic plasticity rules to emulate human brain operations. While a standard spike-timing-dependent-plasticity (STDP) rule is emulated in many SNN hardware, the various STDP rules found in the biological brain have rarely been implemented in hardware. This study proposes a CMOS-memristor hybrid synapse circuit for the hardware implementation of a Ca ion-based plasticity model to emulate the various STDP curves. The memristor was adopted as a memory device in the CMOS synapse circuit because memristors have been identified as promising candidates for analog non-volatile memory devices in terms of energy efficiency and scalability. The circuit design was divided into four sub-blocks based on biological behavior, exploiting the non-volatile and analog state properties of memristors. The circuit was designed to vary weights using an H-bridge circuit structure and PWM modulation. The various STDP curves have been emulated in one CMOS-memristor hybrid circuit, and furthermore a simple neural network operation was demonstrated for associative learning such as Pavlovian conditioning. The proposed circuit is expected to facilitate large-scale operations for neuromorphic computing through its scale-up.

Neural coding of space by time.

The intertwining of space and time poses a significant scientific challenge, transcending disciplines from philosophy and physics to neuroscience. Deciphering neural coding, marked by its inherent spatial and temporal dimensions, has proven to be a complex task. In this paper, we present insights into temporal and spatial modes of neural coding and their intricate interplay, drawn from neuroscientific findings. We illustrate the conversion of a purely spatial input into the temporal form of a singular spike train, demonstrating storage, transmission to remote locations, and recall through spike bursts corresponding to Sharp Wave Ripples. Moreover, the converted temporal representation can be transformed back into a spatiotemporal pattern. The principles of the transformation process are illustrated using a simple feed-forward spiking neural network. The frequencies and phases of Subthreshold Membrane potential Oscillations play a pivotal role in this framework. The model offers insights into information multiplexing and phenomena such as stretching or compressing time of spike patterns.

CNN-LSTM for MFCC-based Speech Recognition on Smart Mirrors for Edge Computing Command

Smart mirrors are conventional mirrors that are augmented with embedded system capabilities to provide comfort and sophistication for users, including introducing the speech command function. However, existing research still applies the Google Speech API, which utilizes the cloud and provides sub-optimal processing time. Our research aim is to design speech recognition using Mel-frequency cepstral coefficients (MFCC) and convolutional neural network–long short-term memory (CNN-LSTM) to be applied to smart mirror edge devices for optimum processing time. Our first step was to download a synthetic speech recognition dataset consisting of waveform audio files (WAVs) from Kaggle, which included the utterances “left,” “right,” “yes,” “no,” “on,” and “off. ” We then designed speech recognition by involving Fourier transformation and low-pass filtering. We benchmark MFCC with linear predictive coding (LPC) because both are feature extraction methods on speech datasets. Then, we benchmarked CNN-LSTM with LSTM, simple recurrent neural network (RNN), and gated recurrent unit (GRU). Finally, we designed a smart mirror system complete with GUI and functions. The test results show that CNN-LSTM performs better than the three other methods with accuracy, precision, recall, and an f1-score of 0.92. The speech command with the best precision is "no," with a value of 0.940. Meanwhile, the command with the best recall is "off," with a value of 0.963. On the other hand, the speech command with the worst precision and recall is "other," with a value of 0.839. The contribution of this research is a smart mirror whose speech commands are carried out on the edge device with CNN-LSTM.

Design and optimization of two-stage controller for three-phase multi-converter/multi-machine electric vehicle

Abstract Electric vehicles (EVs) cut greenhouse gas emissions and our use of non-renewable resources, making them more attractive. EVs have lower fuel and maintenance expenses than internal combustion engine automobiles. This study proposes a multi-converter/Multi‒Machine system with two induction motors (IM) that drive a pure EV’s rear wheels. EV two-stage controllers using a simple Adaline neural network (NN) regulate Field-Oriented regulate of a three-phase IM. To control IM speed, the first controller level is a hybrid proportional–integral (PI) with a robust integral sign of error (RISE) controller. Injection torque is controlled by PI‒adaline NN in the second controller step. The simple Adaline NN improves two-stage controller performance. The Multi-Verse Optimization algorithm found the ideal RISE parameter to improve EV drive system performance. A plug-in EV’s linear speed is controlled by the Electronic Differential Controller (EDC). It uses the driver’s reference speed and steering angle to set each driving wheel’s reference speed. EDC adjusts wheel speeds to enhance traction and stability during cornering, accelerating, and decelerating. Utilizing this information, the EDC can effectively distribute power and torque to the wheels, thereby enhancing vehicle handling and overall performance. Three distinct road scenarios and the designated driving route topology have been used to act and demonstrate the resistive forces that affected the EV while it was traveling down the road. By using Matlab (Simulink), EV’s roadworthiness and efficiency will be evaluated.

Determining plastic slips in rate-independent crystal plasticity models through machine learning algorithms

Dislocation slip-based crystal plasticity models have been a great success in connecting the fundamental physics with the macroscopic deformation of crystalline materials. Pioneered by Taylor in his work on “plastic strain in metals” (Taylor, 1938), and further advanced by Bishop and Hill (1951a, 1951b), the Taylor–Bishop–Hill theory laid the foundation of today’s constitutive models on crystal plasticity. An intriguing part of those modeling is to determine the active slip systems—which system to be involved in and how much it contributes to the deformation. In this paper, we developed a machine learning-based algorithm to determine accurately and efficiently the active slip systems in crystal plasticity constitutive models. Applications to the common three polycrystalline metals, face-centered cubic (FCC) copper, body-centered cubic (BCC) α-iron, and hexagonal close-packed (HCP) AZ31B, demonstrate that even a simple neural network could give rise to accurate and efficient results in comparing with traditional routines. There seems to be plenty of space for further reducing the computation time and hence scaling up the simulating samples.

3D positioning of Camellia oleifera fruit-grabbing points for robotic harvesting

Camellia oleifera is an oilseed crop with high economic value. The short optimum harvest period and high labour costs of C. oleifera harvesting have prompted research on intelligent robotic harvesting. This study focused on the determination of grabbing points for the robotic harvesting of C. oleifera fruits, providing a basis for the decision making of the fruit-picking robot. A relatively simple 2D convolutional neural network (CNN) and stereoscopic vision replaced the complex 3D CNN to realise the 3D positioning of the fruit. Apple datasets were used for the pretraining of the model and knowledge transfer, which shared a certain degree of similarity to C. oleifera fruit. In addition, a fully automatic coordinate conversion method has been proposed to transform the fruit position information in the image into its 3D position in the robot coordinate system. Results showed that the You Only Look Once (YOLO)v8x model trained using 1012 annotated samples achieved the highest performance for fruit detection, with mAP50 of 0.96 on the testing dataset. With knowledge transfer based on the apple datasets, YOLOv8x using few-shot learning realised a testing mAP50 of 0.95, reducing manual annotation. Moreover, the error in the 3D coordinate calculation was lower than 2.1 cm on the three axes. The proposed method provides the 3D coordinates of the grabbing point for the target fruit in the robot coordinate system, which can be transferred directly to the robot control system to execute fruit-picking actions. This dataset was published online to reproduce the results of this study.

Symmetric silicon microring resonator optical crossbar array for accelerated inference and training in deep learning

Photonic integrated circuits are emerging as a promising platform for accelerating matrix multiplications in deep learning, leveraging the inherent parallel nature of light. Although various schemes have been proposed and demonstrated to realize such photonic matrix accelerators, the in situ training of artificial neural networks using photonic accelerators remains challenging due to the difficulty of direct on-chip backpropagation on a photonic chip. In this work, we propose a silicon microring resonator (MRR) optical crossbar array with a symmetric structure that allows for simple on-chip backpropagation, potentially enabling the acceleration of both the inference and training phases of deep learning. We demonstrate a 4×4 circuit on a Si-on-insulator platform and use it to perform inference tasks of a simple neural network for classifying iris flowers, achieving a classification accuracy of 93.3%. Subsequently, we train the neural network using simulated on-chip backpropagation and achieve an accuracy of 91.1% in the same inference task after training. Furthermore, we simulate a convolutional neural network for handwritten digit recognition, using a 9×9 MRR crossbar array to perform the convolution operations. This work contributes to the realization of compact and energy-efficient photonic accelerators for deep learning.

Retraction Note: Simplified neural network for generalized least absolute deviation

Retraction Note: Simplified neural network for generalized least absolute deviation

Enhanced Indoor Positioning Using RSSI and Time-Distributed Auto Encoder-Gated Recurrent Unit Model.

This study presents a novel approach to indoor positioning leveraging radio frequency identification (RFID) technology based on received signal strength indication (RSSI). The proposed methodology integrates Gaussian Kalman filtering for effective signal preprocessing and a time-distributed auto encoder-gated recurrent unit (TAE-GRU) model for precise location prediction. Addressing the prevalent challenges of low accuracy and extended localization times in current systems, the proposed method significantly enhances the preprocessing of RSSI data and effectively captures the temporal relationships inherent in the data. Experimental validation demonstrates that the proposed approach achieves a 75.9% improvement in localization accuracy over simple neural network methods and markedly enhances the speed of localization, thereby proving its practical applicability in real-world indoor localization scenarios.

Black phosphorus quantum dots functionalized with photochromic poly(vinylspiropyran)-grafted polydopamine for transient digital-type memristors

Abstract The covalent functionalization of black phosphorus quantum dots (BPQDs) with organic species or polymers will inevitably change or damage their electronic structure and intrinsic structure. To address this problem and explore the application of BPQDs in transient digital-type memristors, a polydopamine (PDA) thin film is first synthesized in situ onto the surface of BPQDs to produce a donor–acceptor-type BPQDs@PDA composite that is directly used to react with 2-bromoisobutyryl bromide to give BPQDs@PDA-Br. By using BPQDs@PDA-Br as an atom transfer radical polymerization agent, a large number of polyvinylspiropyran (PSP) chains are in situ grown from the PDA surface to yield BPQDs@PDA-PSP. Upon ultraviolet (UV)–visible light illumination, the 2 isomers of the spiropyran (ring-closed spiropyran form and ring-opened merocyanine) in the PSP moieties will interconvert into each other rapidly. As expected, the as-fabricated indium tin oxide (ITO)/BPQDs@PDA-PSP/ITO device exhibits typical nonvolatile digital-type memristive performance under visible irradiation, with a small turn-on voltage of −1.52 V, a turn-off voltage of +1.16 V, and an ON/OFF ratio current ratio of 1.02 × 104. Upon UV illumination, the information stored in the device is quickly and completely erased within 6 s. By utilizing a simple memristor-based convolutional neural network, one can easily realize handwritten digit recognition. After 10 epochs of training, numeral recognition accuracy can reach up to 96.21%.

Improved DeepSORT-Based Object Tracking in Foggy Weather for AVs Using Sematic Labels and Fused Appearance Feature Network.

The presence of fog in the background can prevent small and distant objects from being detected, let alone tracked. Under safety-critical conditions, multi-object tracking models require faster tracking speed while maintaining high object-tracking accuracy. The original DeepSORT algorithm used YOLOv4 for the detection phase and a simple neural network for the deep appearance descriptor. Consequently, the feature map generated loses relevant details about the track being matched with a given detection in fog. Targets with a high degree of appearance similarity on the detection frame are more likely to be mismatched, resulting in identity switches or track failures in heavy fog. We propose an improved multi-object tracking model based on the DeepSORT algorithm to improve tracking accuracy and speed under foggy weather conditions. First, we employed our camera-radar fusion network (CR-YOLOnet) in the detection phase for faster and more accurate object detection. We proposed an appearance feature network to replace the basic convolutional neural network. We incorporated GhostNet to take the place of the traditional convolutional layers to generate more features and reduce computational complexities and costs. We adopted a segmentation module and fed the semantic labels of the corresponding input frame to add rich semantic information to the low-level appearance feature maps. Our proposed method outperformed YOLOv5 + DeepSORT with a 35.15% increase in multi-object tracking accuracy, a 32.65% increase in multi-object tracking precision, a speed increase by 37.56%, and identity switches decreased by 46.81%.

CRYOMOVE: Cold chain real-time management of vaccine delivery using PCM and deep learning

Vaccines are temperature-sensitive biological products that can become ineffective or even hazardous if exposed to temperatures outside the recommended range. Therefore, it is crucial to maintain the cold chain during the transportation and storage of vaccines in order to ensure that they arrive at their destination in optimal condition. Vaccines are typically transported and stored in containers lined with Phase Change Material (PCM) to maintain a specific temperature range (usually lower than the ambient temperature). Firstly, we have simulated the melting of PCM at three distinct external temperatures using ANSYS Fluent software in order to forecast the Melt Fraction (MF) vs. time curve for each of these temperatures. The external temperature consistently undergoes dynamic changes. Running a simulation each time the temperature shifts is impractical due to its time and data-intensive nature. Consequently, we adopted a more efficient approach by training a simple Artificial Neural Network (ANN) based on simulation data. This ANN is designed to learn and predict the MF versus time curve for any given external temperature. Real-time temperature data from the vaccine box is transmitted to the cloud. Subsequently, the machine learning model operates on the cloud to predict the remaining time for the PCM to melt. This predictive analysis is performed recursively every 10 min to account for dynamic fluctuations in external temperatures, providing continuous updates on the time remaining for PCM meltdown. This is then conveyed to the cold chain managers to take preventive measures if the external conditions are harsh and there is a chance for vaccines to get ruined because of the temperature. This same approach can be extended to other applications beyond vaccine delivery such as food storage and transportation, pharmaceuticals, and more.

Deep learning with convolution neural network detecting mesiodens on panoramic radiographs: comparing four models.

The aim of this study was to develop an optimal, simple, and lightweight deep learning convolutional neural network (CNN) model to detect the presence of mesiodens on panoramic radiographs. A total of 628 panoramic radiographs with and without mesiodens were used as training, validation, and test data. The training, validation, and test dataset were consisted of 218, 51, and 40 images with mesiodens and 203, 55, and 61 without mesiodens, respectively. Unclear panoramic radiographs for which the diagnosis could not be accurately determined and other modalities were required for the final diagnosis were retrospectively identified and employed as the training dataset. Four CNN models provided within software supporting the creation of neural network models for deep learning were modified and developed. The diagnostic performance of the CNNs was evaluated according to accuracy, precision, recall and F1 scores, receiver operating characteristics (ROC) curves, and area under the ROC curve (AUC). In addition, we used SHapley Additive exPlanations (SHAP) to attempt to visualize the image features that were important in the classifications of the model that exhibited the best diagnostic performance. A binary_connect_mnist_LeNet model exhibited the best performance of the four deep learning models. Our results suggest that a simple lightweight model is able to detect mesiodens. It is worth referring to AI-based diagnosis before an additional radiological examination when diagnosis of mesiodens cannot be made on unclear images. However, further revaluation by the specialist would be also necessary for careful consideration because children are more radiosensitive than adults.

A simple and effective deep neural network based QRS complex detection method on ECG signal.

Introduction: The QRS complex is the most prominent waveform within the electrocardiograph (ECG) signal. The accurate detection of the QRS complex is an essential step in the ECG analysis algorithm, which can provide fundamental information for the monitoring and diagnosis of the cardiovascular diseases. Methods: Seven public ECG datasets were used in the experiments. A simple and effective QRS complex detection algorithm based on the deep neural network (DNN) was proposed. The DNN model was composed of two parts: a feature pyramid network (FPN) based backbone with dual input channels to generate the feature maps, and a location head to predict the probability of point belonging to the QRS complex. The depthwise convolution was applied to reduce the parameters of the DNN model. Furthermore, a novel training strategy was developed. The target of the DNN model was generated by using the points within 75 milliseconds and beyond 150 milliseconds from the closest annotated QRS complexes, and artificial simulated ECG segments with high heart rates were generated in the data augmentation. The number of parameters and floating point operations (FLOPs) of our model was 26976 and 9.90M, respectively. Results: The proposed method was evaluated through a cross-dataset test and compared with the sophisticated state-of-the-art methods. On the MITBIH NST, the proposed method demonstrated slightly better sensitivity (95.59% vs. 95.55%) and lower presicion (91.03% vs. 92.93%). On the CPSC 2019, the proposed method have similar sensitivity (95.15% vs.95.13%) and better precision (91.75% vs. 82.03%). Discussion: Experimental results show the proposed algorithm achieved a comparable performance with only a few parameters and FLOPs, which would be useful for the application of ECG analysis on the wearable device.