Neural Network Research Research Articles

Recurrent Neural Network Research on a Flipped Classroom Online Collaborative Teaching Platform

A Recurrent Neural Network (RNN) is a type of artificial neural network designed to effectively process sequential data by maintaining a memory of past inputs. A Collaborative Online Teaching platform is a digital space designed to facilitate interactive and engaging learning experiences between educators and students in virtual environments. It integrates various tools and features to enable real-time communication, content sharing, and collaborative activities. The collaborative features such as breakout rooms, group projects, and interactive whiteboards promote teamwork and active participation among students. Assessment tools within the platform facilitate the evaluation of student progress through quizzes, assignments, and grading functionalities. This research investigates the integration of Problem Based Learning (PBL) and cooperative instructional methods within flipped classrooms, aiming to examine their impact on students' intrinsic motivation to learn. The study endeavors to develop a tutorial strategy aligned with the concept of "flipped schools" to enhance students' self-directed study, problem-solving abilities, academic achievement, and interest in instruction. First-year students from Southern Taiwan College of Science and Technology's School of Information Management participated, focusing on education in C#. The participants were divided into a trial group, experiencing the proposed teaching tactics, and an identical control group with randomly selected groups. Pre- and post-test scores from computing classes were analyzed to assess learning outcomes. Additionally, a survey explored students' perceptions of the flipped classroom approach, project-based schooling, and collaborative learning. Recurrent Neural Network (RNN) processes were employed to analyze sequential data, such as student performance over time and engagement patterns. Results indicate that the recommended instructional strategies led to improved comprehension levels, with the experimental group showing significantly greater progress compared to the control.

Advanced Heart Disease Prediction: Deep Learning-Enhanced Convolutional Neural Network in the Internet of Medical Things Environment

Using the large dataset from the UCI Machine Learning Repository, this study presents a state-of-the-art Hybrid Convolutional Neural Network (HCNN) for predicting heart disease. The HCNN design has convolutional layers, leftover blocks, and attention methods. These help pull more features from cardiovascular health data, even when the patterns are very complex. Utilizing these deep learning parts, the HCNN shows better prediction skills, getting higher accuracy, strong classification (AUC), and a fair F1 Score. Because the model can change to find complicated connections in the information, it could be used to make medical diagnosis better. The HCNN is unique because it can instantly learn hierarchical structures from raw data. This lets it find hidden features that are important for accurately predicting heart disease. The convolutional layers help the model find local patterns, and the residue blocks stop problems with disappearing gradients, which makes training for deep designs more efficient. Attention processes make the network even more focused on important traits, which adds to its amazing ability to tell them apart. This work opens the door for researchers, doctors, and data scientists to use deep learning, and especially HCNN, to improve cardiovascular health analytics. This study takes a big step toward more accurate and efficient heart disease prognosis by giving a complete overview of the model's architecture and performance metrics. This opens the door for more research and use of advanced neural networks in the field of predictive medical diagnostics.

Hypergraph construction using Multi-Sensor for helicopter Tail-Drive system fault diagnosis

Recently, neural network research in the field of mechanical fault diagnosis often requires a large number of data samples, while graph neural networks constructed using a single sensor under limited samples of the graph topological relationship is difficult to cover the coupling information, and the problem of over-smoothing of the graph can lead to failure of the fault diagnosis. As a result, Multi-Channel Hypergraph Convolutional Network (McHGCN) for limited-sample fault diagnosis of helicopter tail drive system is proposed. Features are extracted from multi-sensor data and hyper edges are constructed by k-nearest neighbor method thus exploiting the similarity. Nodal feature representation of multi-sensor data is done with 1D convolutional network, which prevents over-smoothing while preserving the learning depth. Finally, fault diagnosis is accomplished by semi-supervised learning of the hypergraph convolutional layer. Limited-sample fault diagnosis experiments are conducted on a helicopter tail drive system to verify the effectiveness of the method.

Different Types of Neural Networks and Applications: Evidence from Feedforward, Convolutional and Recurrent Neural Networks

Neural networks have achieved great process in the 90 years since they were officially introduced in 1943. Because of its wide application and huge research and development potential, this technology attracts more and more scientific and technological workers to the research of neural networks. Neural network technology is an essential component of AI development, and it is a significant indicator of a country's overall strength. In this paper, this study will demonstrate Feedforward Neural Network, Convolution Neural Network and Recurrent Neural networks and evaluate them through datasets from kaggle.com. and CSDN (China IT community). Through this paper, readers can have a better outlook and understanding of the operating principles of each type of neural network as well as their specific jobs (what kind of jobs they specialized in) and each application of these neural networks. So that this paper can promote readers' thoughts and help them start learning neural networks or be a supplement or reference for future scholars. In the end, this paper will present the outcome, which is the evaluation of the accuracy, loss curve, and accuracy curve of neural networks.

Review of multi‐scale mechanical behavior research on composite solid propellants based on data‐driven approach

AbstractComposite solid propellant is a kind of viscoelastic composite with high filling ratio and multi‐scale composition characteristics, and its macroscopic mechanical properties strongly depend on the microstructure of the propellant materia. However, with the increasing complexity of composition, structure and properties of composite solid propellants, the traditional research paradigm based on experimental observation, theoretical modeling and numerical simulation has encountered new scientific challenges and technical bottlenecks in the mechanical behavior analysis, charge design and manufacturing of composite solid propellants. Among them, the problems such as insufficient experimental observation, lack of theoretical model, limited numerical analysis and difficult verification of results restrict the development of composite solid propellants in future‐oriented engineering applications to a certain extent. The data‐driven computational mechanics method can directly establish complex relationships between variables from high‐dimensional and high‐throughput data, which can capture trends that are difficult to be found by traditional mechanics research methods, and has inherent advantages in the analysis, prediction and optimization of complex systems. This paper mainly reviews and evaluates the research of neural network based modeling, model‐free data‐driven calculation and data‐driven multi‐scale calculation, which provides the correct direction for the subsequent research of multi‐scale mechanical behavior of composite solid propellants based on data‐driven.

Mapping the knowledge structure of artificial neural network research in the stock market: a bibliometric analysis and future research pathways

PurposeIn past decades, artificial neural network (ANN) models have revolutionised various stock market operations due to their superior ability to deal with nonlinear data and garnered considerable attention from researchers worldwide. The present study aims to synthesize the research field concerning ANN applications in the stock market to a) systematically map the research trends, key contributors, scientific collaborations, and knowledge structure, and b) uncover the challenges and future research areas in the field.Design/methodology/approachTo provide a comprehensive appraisal of the extant literature, the study adopted the mixed approach of quantitative (bibliometric analysis) and qualitative (intensive review of influential articles) assessment to analyse 1,483 articles published in the Scopus and Web of Science indexed journals during 1992–2022. The bibliographic data was processed and analysed using VOSviewer and R software.FindingsThe results revealed the proliferation of articles since 2018, with China as the dominant country, Wang J as the most prolific author, “Expert Systems with Applications” as the leading journal, “computer science” as the dominant subject area, and “stock price forecasting” as the predominantly explored research theme in the field. Furthermore, “portfolio optimization”, “sentiment analysis”, “algorithmic trading”, and “crisis prediction” are found as recently emerged research areas.Originality/valueTo the best of the authors’ knowledge, the current study is a novel attempt that holistically assesses the existing literature on ANN applications throughout the entire domain of stock market. The main contribution of the current study lies in discussing the challenges along with the viable methodological solutions and providing application area-wise knowledge gaps for future studies.

Accelerating spiking neural network simulations with PymoNNto and PymoNNtorch.

Spiking neural network simulations are a central tool in Computational Neuroscience, Artificial Intelligence, and Neuromorphic Engineering research. A broad range of simulators and software frameworks for such simulations exist with different target application areas. Among these, PymoNNto is a recent Python-based toolbox for spiking neural network simulations that emphasizes the embedding of custom code in a modular and flexible way. While PymoNNto already supports GPU implementations, its backend relies on NumPy operations. Here we introduce PymoNNtorch, which is natively implemented with PyTorch while retaining PymoNNto's modular design. Furthermore, we demonstrate how changes to the implementations of common network operations in combination with PymoNNtorch's native GPU support can offer speed-up over conventional simulators like NEST, ANNarchy, and Brian 2 in certain situations. Overall, we show how PymoNNto's modular and flexible design in combination with PymoNNtorch's GPU acceleration and optimized indexing operations facilitate research and development of spiking neural networks in the Python programming language.

An Overview of the Application of Convolutional Neural Networks inSentiment Analysis

The field of natural language processing, or NLP, uses its understanding of human language to find practical solutions to issues. It mainly includes two parts: the core task and the application. The core task represents the common problem that needs to be solved in various natural language application directions. It includes language models, morphology, grammar analysis, semantic analysis, etc. At the same time, the application section focuses on specific natural language processing tasks such as machine translation, information retrieval, question-answering systems, dialogue systems, etc. Natural language processing has made a significant contribution to the development of human society and the economy and provides strong support for all aspects of research work. Opinion mining, or sentiment analysis, is a subfield of natural language processing that develops systems for identifying and extracting ideas from text. Sentiment analysis is a hot topic since it has many practical applications. Many opinion-expressing texts are available on review sites, forums, blogs, and social media as the amount of publicly available information on the Internet grows. This unstructured information can then be automatically transformed into structured data about products, services, brands, politics, or other topics on which people can express their opinions using sentiment analysis systems. This information can be used for marketing analytics, public relations, product reviews, network sponsor ratings, product feedback, and customer service. With the rapid growth of labeled sample data sets and the notable enhancement in graphics processor (GPU) performance, convolutional neural network research has advanced rapidly and achieved remarkable leads to various computer vision tasks. By reviewing the application of CNN, we see that convolutional operations are naturally suitable for some text processing and, thus, naturally suitable for the background of sentiment analysis.

Guest Editorial: Foreword to the Special Issue on Advanced Research and Applications of Deep Learning and Neural Network in Image Recognition

Over the last two decades, the realm of image recognition has undergone a remarkable transformation, characterized by an astonishing pace of advancement [...]

Dynamic multi-granularity spatial-temporal graph attention network for traffic forecasting

Traffic forecasting, as the cornerstone of the development of intelligent transportation systems, plays a crucial role in facilitating accurate control and management of urban traffic. By treating sensors as nodes in a road network, recent research on modeling complex spatial-temporal graph structures has achieved notable advancements in traffic forecasting. However, limited by the increasing number of sensors and recorded data points, most of the recent studies on spatial-temporal graph neural network (STGNN) research concentrate on aggregating short-term (e.g. recent one-hour) traffic history to predict future data. Furthermore, almost all previous STGNNs neglect to incorporate the cyclical patterns that appear in the traffic historical data. For example, the cyclical patterns of traffic on the same day or hour of each week can help improve the accuracy of future traffic predictions. In this paper, we propose a novel Dynamic Multi-Granularity Spatial-Temporal Graph Attention Network (DmgSTGAT) framework for traffic forecasting, which leverages multi-granularity spatial-temporal correlations across different time-scales and variables to efficiently consider cyclical patterns in traffic data. We also design effective temporal encoding and transformer encoding layers to produce meaningful multi-granularity sensor-level, day-level, hour-level, and point-level representations. The multi-granularity spatial-temporal graph attention network can use the produced representations to extract useful but sparsely distributed patterns accurately, which also avoids the influence of extra noise from the long-term history. Experimental results on four real-world traffic datasets show that DmgSTGAT can achieve state-of-the-art performance with the help of multi-granularity cyclical patterns compared with various recent baselines.

Comprehensive Review of Backpropagation Neural Networks

The Backpropagation Neural Network (BPNN) is a deep learning model inspired by the biological neural network. Introduced in the 1980s, the BPNN quickly became a focal point in neural network research due to its outstanding learning capability and adaptability. The network structure consists of input, hidden, and output layers, and it optimizes weights through the backpropagation algorithm, widely applied in image recognition, speech processing, natural language processing, and more. The mathematical model of neurons describes the relationship between input and output, and the training process involves adjusting weights and biases using optimization algorithms like gradient descent. In applications, BPNN excels in image recognition, speech processing, natural language processing, and financial forecasting. Researchers continuously experiment with optimization algorithms, including the Grey Wolf Algorithm, Genetic Algorithm, Particle Swarm Algorithm, Simulated Annealing Algorithm, as well as comprehensive strategies and improved gradient descent algorithms. In the future, with the ongoing development of deep learning, BPNN is poised to play a crucial role in tasks such as image recognition and speech processing.

Interpretable convolutional neural network for analyzing precipitation in the pre-rainy season of South China

Abstract The research and application of convolutional neural networks (CNNs) on statistical downscaling have been hampered by the fact that deep learning is highly dependent on sample size and is considered to be a black-box model. Therefore, a CNN model with transfer learning (CNN-TL) is proposed to study the pre-rainy season precipitation of South China. First, an augmented monthly dataset is created by sliding a fixed-length window over the daily circulation field and precipitation data for the entire year. Next, a base CNN network is pretrained on the augmented dataset, and then the network parameters are tuned on the actual monthly dataset from South China. Finally, guided backpropagation is conducted to obtain the distribution regions of the key features and explain the net. The coefficient of determination (R2) and root mean square error (RMSE) show that the CNN-TL model has higher explanatory power and better fitting performance than the feature extraction-based random forest (RF). Compared with the base CNN, the transfer learning approach can improve the explanatory power of the model by 10.29% and reduce the average RMSE by 6.82%. In addition, the interpretation results of the model show that the critical regions are primarily South China and its surrounding areas, including the Indochinese Peninsula, the Bay of Bengal, and the South China Sea. Furthermore, the ablation experiments and composite analysis illustrate that these regions are very important.

Research the adaptive neural networks using possibility in control systems for cycle chemistry at thermal power plants

The study is devoted to the analysis of the learning possibilities of adaptive neural networks in the context of water-chemical regime management systems. During a series of experiments on a laboratory stand with a disturbing effect, transient characteristics describing changes in the controlled value were obtained. The obtained mathematical models using the TunePID program served as the basis for the analysis and research of adaptive neural networks. Adaptive neural networks are a powerful tool capable of learning from the data obtained and dynamically changing their structure and parameters to accurately predict and control the behavior of the system. Unlike classical mathematical models, adaptive neural networks have the ability to process nonlinear dependencies and adapt to changing conditions without the need for manual reconfiguration. The stages of identification of disturbances and adaptation of compensator settings were described and illustrated. The study included an analysis of the work of a neural network with a different number of hidden neurons and various activation functions. The results showed that neural networks trained on data using an adaptive approach are able to predict and control the system with an accuracy comparable to the calculated parameters of the control coefficients calculated manually.

Multi-Spatial Pyramid Feature and Optimizing Focal Loss Function for Object Detection

Previous deep convolutional neural network research has made significant progress toward improving the speed and accuracy of object detection. However, despite these advancements, the inaccurate detection of multi-object (small objects) remains challenging in the traffic environments. In this paper, we propose a new architecture called YOLOM, which is specifically designed to achieve enhanced multi-object (small objects) detection precision. YOLOM incorporates several innovative features: a multi-spatial pyramid (MSP), an optimized focal loss (OFLoss) function, and an objectness loss that incorporates effective intersection over union (EIoU) calculations. These features collectively yield enhanced accuracy and reduce the miss rate of small objects, particularly in the multi-object cases. According to the sizes of receptive field features with different spatial scales with pooling layers, we propose the MSP module. We optimize the focal loss as a classification function instead of the cross-entropy loss, which solves some class imbalance problems caused by anchor-free detection when encountering disparate datasets. Due to the superior performance of EIoU in confidence scoring, we use EIoU to participate in the objectness loss calculation of our work. Therefore, our method substitutes EIoU for YOLOX's objectness loss. The experimental results demonstrate that our strategies significantly outperform some end-to-end object detection methods.

Topological equivalence classification and enumeration of n-input linearly separable Boolean functions

A perceptron, as an “artificial neuron”, plays a crucial role in the research of neural networks. As a mathematical model of linearly separable Boolean functions, perceptron neurons can realize any nonlinearly separable Boolean function. Linearly separable Boolean functions can be divided into a smaller number of the same functional classes. These functional classes are equivalent to the core basic units (AND, NOT, OR, and other components) that make up the circuit. Therefore, the research on the classification problem of linearly separable Boolean functions is very important. In this study, a novel algorithm, called a topological equivalence classification algorithm, is proposed for classifying balanced linearly separable Boolean functions. By the proposed algorithm, the total number of the topological equivalence classes and the number of balanced linearly separable Boolean functions in each of the topological equivalence classes are obtained. According to the one-to-one correspondence between n-input balanced linearly separable Boolean functions and n-1-input linearly separable Boolean functions, we get the topological classification of linearly separable Boolean functions. As for n=9, the accurate results of classification and counting the number of linearly separable Boolean functions are obtained: There are 53,063,448 topological equivalence classes, and the total number of linearly separable Boolean functions is 144,380,202,286,068,288.

The memristor: Principle, mechanism, and application

The memristor is an electronic device and the only fundamental circuit element that can directly implement memory and logic computation between electronic devices. The main characteristic of the memristor is that its resistance value varies with the direction and magnitude of the current flowing through it, and this variation is reversible. It can store charge, similar to RAM in traditional computers, but with more energy-efficient, faster, and higher-density storage. Therefore, the memristor is attracting attention in the research of artificial intelligence and neural networks. This paper summarizes the mathematical principles and application scenarios of memristive systems, including ideal memristors. two theoretically mature resistance change mechanisms are summarized, namely the con-ductive filament model (mainly for RRAM, Resistive random-access Finally, the design of memristor-related circuit components and the current state of research in artificial intelli-gence, especially in the field of neural networks, are reviewed. The authors are optimistic about the future development of memristors for medium and low power edge computing platforms and believe that memristors will become one of the computing platforms for ar-tificial intelligence.

Review of diffractive deep neural networks

In 2018, a UCLA research group published an important paper on optical neural network (ONN) research in the journal Science. It developed the world’s first all-optical diffraction deep neural network (DNN) system, which can perform MNIST dataset classification tasks at near-light-speed. To be specific, the UCLA research group adopted a terahertz light source as the input, established the all-optical diffractive DNN (D2NN) model using the Rayleigh-Sommerfeld diffraction theory, optimized the model parameters using the stochastic gradient descent algorithm, and then used 3D printing technology to make the diffraction grating and built the D2NN system. This research opened a new ONN research direction. Here, we first review and analyze the development history and basic theory of artificial neural networks (ANNs) and ONNs. Second, we elaborate D2NN as holographic optical elements (HOEs) interconnected by free space light and describe the theory of D2NN. Then we cover the nonlinear research and application scenarios for D2NN. Finally, the future directions and challenges of D2NN are briefly discussed. Hopefully, our work can provide support and help to researchers who study the theory and application of D2NN in the future.

Spiking neural network research based on brain-inspired computing

With the shortcomings of deep learning in training cost, generalization ability, interpretability, and reliability, increasing attention was concentrated on the novel neuromorphic devices. A spiking neural network can better simulate the information transmission mode of biological neurons, and has the characteristics of strong computing power and low power consumption. Therefore, this study provides an overview of the pulse neural network from the perspective of its fundamental structure and operating principle. In terms of structure optimization, there is a summary of the aspects of the encoding mode of a spiking neural network and topology structure. In addition, the deficiency and development of the spiking neural network are analyzed. Pulsed neurons work by integrating the mechanism of firing, exchanging information with the time of the signal, and transmitting signals to adjacent neurons only when the membrane potential reaches a threshold. Spiking neural network (SNN) still lags behind traditional Artificial neural network (ANN) in accuracy, has large computational capacity and is inefficiently trained due to the complexity of pulsed neuron models. Future developments lie in bionics, deeper studies of the complex dynamics of the human brain, and synaptic connections in the human brain.

Naive Prediction of Protein Backbone Phi and Psi Dihedral Angles Using Deep Learning.

Protein structure prediction represents a significant challenge in the field of bioinformatics, with the prediction of protein structures using backbone dihedral angles recently achieving significant progress due to the rise of deep neural network research. However, there is a trend in protein structure prediction research to employ increasingly complex neural networks and contributions from multiple models. This study, on the other hand, explores how a single model transparently behaves using sequence data only and what can be expected from the predicted angles. To this end, the current paper presents data acquisition, deep learning model definition, and training toward the final protein backbone angle prediction. The method applies a simple fully connected neural network (FCNN) model that takes only the primary structure of the protein with a sliding window of size 21 as input to predict protein backbone ϕ and ψ dihedral angles. Despite its simplicity, the model shows surprising accuracy for the ϕ angle prediction and somewhat lower accuracy for the ψ angle prediction. Moreover, this study demonstrates that protein secondary structure prediction is also possible with simple neural networks that take in only the protein amino-acid residue sequence, but more complex models are required for higher accuracies.

SpinalNet: Deep Neural Network With Gradual Input

Deep neural networks (DNNs) have achieved the state of the art performance in numerous fields. However, DNNs need high computation times, and people always expect better performance in a lower computation. Therefore, we study the human somatosensory system and design a neural network (SpinalNet) to achieve higher accuracy with fewer computations. Hidden layers in traditional NNs receive inputs in the previous layer, apply activation function, and then transfer the outcomes to the next layer. In the proposed SpinalNet, each layer is split into three splits: 1) input split, 2) intermediate split, and 3) output split. Input split of each layer receives a part of the inputs. The intermediate split of each layer receives outputs of the intermediate split of the previous layer and outputs of the input split of the current layer. The number of incoming weights becomes significantly lower than traditional DNNs. The SpinalNet can also be used as the fully connected or classification layer of DNN and supports both traditional learning and transfer learning. We observe significant error reductions with lower computational costs in most of the DNNs. Traditional learning on the VGG-5 network with SpinalNet classification layers provided the state-of-the-art (SOTA) performance on QMNIST, Kuzushiji-MNIST, and EMNIST (Letters, Digits, and Balanced) datasets. Traditional learning with ImageNet pre-trained initial weights and SpinalNet classification layers provided the SOTA performance on STL-10, Fruits 360, Bird225, and Caltech-101 datasets. The scripts of the proposed SpinalNet training are available at the following link: <uri>https://github.com/dipuk0506/SpinalNet</uri> <i>Impact Statement</i>&#x2014;Research in deep neural networks (DNNs) has gained significant attention from industries and academia due to their eye-catching performance. DNNs have enabled machines to perform myriad tasks with high accuracy that once only humans could do. Several researchers have recently proposed different types of NNs and have achieved high accuracy. The recent success of biologically inspired convolutional neural networks and the miraculous spinal architecture of humans has motivated us to develop a neural network with gradual inputs. We have achieved superior performance in several datasets. After the first online appearance of the initial version of this paper, several researchers have applied the proposed neural network in several new datasets and reported promising results. We may observe numerous novel applications of SpinalNet in upcoming years.