Neural Network Theory Research Articles

Neural Abstraction-Based Controller Synthesis and Deployment

Abstraction-based techniques are an attractive approach for synthesizing correct-by-construction controllers to satisfy high-level temporal requirements. A main bottleneck for successful application of these techniques is the memory requirement, both during controller synthesis (to store the abstract transition relation) and in controller deployment (to store the control map). We propose memory-efficient methods for mitigating the high memory demands of the abstraction-based techniques using neural network representations . To perform synthesis for reach-avoid specifications, we propose an on-the-fly algorithm that relies on compressed neural network representations of the forward and backward dynamics of the system. In contrast to usual applications of neural representations, our technique maintains soundness of the end-to-end process. To ensure this, we correct the output of the trained neural network such that the corrected output representations are sound with respect to the finite abstraction. For deployment, we provide a novel training algorithm to find a neural network representation of the synthesized controller and experimentally show that the controller can be correctly represented as a combination of a neural network and a look-up table that requires a substantially smaller memory. We demonstrate experimentally that our approach significantly reduces the memory requirements of abstraction-based methods. We compare the performance of our approach with the standard abstraction-based synthesis on several models. For the selected benchmarks, our approach reduces the memory requirements respectively for the synthesis and deployment by a factor of 1.31× 10 5 and 7.13× 10 3 on average, and up to 7.54× 10 5 and 3.18× 10 4 . Although this reduction is at the cost of increased off-line computations to train the neural networks, all the steps of our approach are parallelizable and can be implemented on machines with higher number of processing units to reduce the required computational time.

Enhancing Recommender Systems with Semantic User Profiling through Frequent Subgraph Mining on Knowledge Graphs

Recommender systems play a crucial role in personalizing online user experiences by creating user profiles based on user–item interactions and preferences. Knowledge graphs (KGs) are intricate data structures that encapsulate semantic information, expressing users and items in a meaningful way. Although recent deep learning-based recommendation algorithms that embed KGs have demonstrated impressive performance, the richness of semantics and explainability embedded in the KGs are often lost due to the opaque nature of vector representations in deep neural networks. To address this issue, we propose a novel user profiling method for recommender systems that can encapsulate user preferences while preserving the original semantics of the KGs, using frequent subgraph mining. Our approach involves creating user profile vectors from a set of frequent subgraphs that contain information about user preferences and the strength of those preferences, measured by frequency. Subsequently, we trained a deep neural network model to learn the relationship between users and items, thereby facilitating effective recommendations using the neural network’s approximation ability. We evaluated our user profiling methodology on movie data and found that it demonstrated competitive performance, indicating that our approach can accurately represent user preferences while maintaining the semantics of the KGs. This work, therefore, presents a significant step towards creating more transparent and effective recommender systems that can be beneficial for a wide range of applications and readers interested in this field.

A traffic anomaly detection approach based on unsupervised learning for industrial cyber–physical system

In Industrial Cyber–Physical Systems (ICPSs), the attacker can intrude into the cyber system through many penetration tools and attack the physical system. Payload-based traffic anomaly detection is a popular technique against these attacks. Due to the imbalanced distribution of normal and attack samples in ICPS, existing payload-based detection methods are mostly implemented based on unsupervised learning, typically comprising a word segmentation model and an unsupervised classifier. However, existing methods may disrupt semantic correlations and face challenges in extracting complex payload dependence relationships. To address these issues, this paper proposes a traffic anomaly detection approach, which consists of a data preprocessing model, an unsupervised word segmentation model, and an unsupervised classification model based on autoencoder. The unsupervised word segmentation model utilizes Long Short-Term Memory (LSTM) to calculate the probability of each word segmentation combination, effectively addressing the issue of inaccurate segmentation results in existing payload segmentation models. The unsupervised classification model, which combines 1D-Convolutional Neural Network (1D-CNN) and Bidirectional Encoder Representation from Transformers (BERT), addresses the challenge of extracting complex payload dependence relationships in existing classification models. The proposed detection approach is evaluated using a Cyber–Physical Attack Dataset (CPAD). Compared with the state-of-the-art detection approaches, the proposed approach has shown a significant improvement in Precision, with an increase of 18.83%. Additionally, the Recall has also been substantially enhanced, with a gain of 22.3%. Overall, the F1 has demonstrated a comprehensive improvement of 20.60%.

Towards a phenomenological understanding of neural networks: data

A theory of neural networks (NNs) built upon collective variables would provide scientists with the tools to better understand the learning process at every stage. In this work, we introduce two such variables, the entropy and the trace of the empirical neural tangent kernel (NTK) built on the training data passed to the model. We empirically analyze the NN performance in the context of these variables and find that there exists correlation between the starting entropy, the trace of the NTK, and the generalization of the model computed after training is complete. This framework is then applied to the problem of optimal data selection for the training of NNs. To this end, random network distillation (RND) is used as a means of selecting training data which is then compared with random selection of data. It is shown that not only does RND select data-sets capable of outperforming random selection, but that the collective variables associated with the RND data-sets are larger than those of the randomly selected sets. The results of this investigation provide a stable ground from which the selection of data for NN training can be driven by this phenomenological framework.

Unsupervised learning spectral functions with neural networks

Reconstructing spectral functions from Euclidean Green’s functions is an ill-posed inverse problem that is crucial for understanding the properties of many-body systems. In this proceeding, we propose an automatic differentiation (AD) framework utilizing neural network representations for spectral reconstruction from propagator observables. We construct spectral functions using neural networks and optimize the network parameters unsupervisedly based on the reconstruction error of the propagator. Compared to the maximum entropy method, the AD framework demonstrates better performance in situations with high noise levels. It is noteworthy that neural network representations provide non-local regularization, which has the potential to significantly improve the solution of inverse problems.

Safe and Stable Secondary Voltage Control of Microgrids Based on Explicit Neural Networks

This paper proposes a novel safety-critical secondary voltage control method based on explicit neural networks (NNs) for islanded microgrids (MGs) that can guarantee any state inside the desired safety bound even during the transient. Firstly, an integrator is introduced in the feedback loop to fully eliminate the steady-state error caused by primary control. Then, considering the impact of secondary control on the stability of the whole system, a set of transient stability and safety constraints is developed. In order to achieve online implementation that requires fast computation, an explicit NN-based secondary voltage controller is designed to cast the time-consuming constrained optimization in the offline NN training phase, by leveraging the local Lipschitzness of activation functions. Specially, instead of using the NN as a black box, the explicit representation of NN is substituted into the closed-loop MG for transferring the stability and safety constraints. Finally, the NN is trained by safe imitation learning, where an optimization problem is formulated by maximizing the imitation accuracy and volume of the stable region while satisfying the stability and safety constraints. Thus, the safe and stable region is approximated that any trajectory initiates within will converge to the equilibrium while bounded by safety conditions. The effectiveness of the proposed method is verified on a prototype MG with detailed dynamics.

Neural network representations for the inter- and intra-class common vector classifiers

Common Vector Approach (CVA) is a known linear regression-based classifier, which also enables an extension to inter-class discrimination, known as the Discriminative Common Vector Approach (DCVA). The characteristics of linear regression classifiers (LRCs) enable the possibility of a schematic implementation that is similar to the neuron model of artificial neural networks (ANNs). In this work, we explore this schematic similarity to come up with an ANN representation for both CVA and DCVA. The new representation eliminates the need for projection matrices in its implementation, hence significantly reduces the memory requirements and computational complexities of the processes. Furthermore, since the new representation is in a neural style, it is expected to provide a solid and intriguing extension of CVA (and DCVA) by further incorporating adaptation or activation processes to the already successful CVA-based classifiers.

A combined deep neural network and semi-supervised clustering method for sports injury risk prediction

Sports injury risk prediction plays an important role in evaluating the stability of the body in the process of sports. Based on deep neural network and semi-supervised clustering theory, this paper constructs a sports injury risk prediction model, and designs a recognition algorithm based on feature sequence classification combined with Bayesian filtering, which solves the problems of false alarms and missed alarms caused by relying on local spatial attribute features. The FMS (functional movement screen) kit was used to test seven movements (squat, hurdle step, straight leg squat, shoulder flexibility, active straight leg lift, trunk stability push-up, and body rotation) of 29 athletes. In the simulation process, Pareto analysis method was used to evaluate the overall risk of athletes' after-school physical exercise, and the FMS score, receiver operating characteristic (ROC) curve and Youden index were calculated. The results showed that the best cut-off points of the total score of FMS for athletes as a whole, male athletes and female athletes were 15.5, 15.5 and 16.5, respectively, which effectively verified the generalization performance of the above sports prediction model.

Estimation of intelligent-based quality of service in terrestrial FSO links: A case of South Africa propagation paths

Free Space Optical (FSO) communication is one of the popular technologies’ dominants in the optical communication domain. Despite all the advantages, FSO links are grossly affected by severe atmospheric conditions such as fog, rain, snow, smoke, and dust or aerosol particles suspended in the air. Fog is one of the major challenges for free-space optics to achieve carrier-class availability and causes extreme attenuation. A 10-year (2010––2019) visibility data were obtained from the South African Weather Services (SAWS) with a special focus on Cape Town (–33.925° S, 18.424° E). This extracted and processed Cape Town data was used to estimate the needed fog-induced attenuation. The intelligent algorithm is rooted in the theory of Artificial Neural Networks (ANN) systems which intelligently control the sub-tropical climate conditions of the chosen region. The adopted network-based technique is capable to interact with ever-changing weather by generating appropriate improvements to the free space optical communication link quality of service for efficient services to the customers. In this study, an Intelligent Algorithm (IA) based on an Artificial Neural Network (ANN) has been utilized to improve the reliability of QoS that can satisfy customer SLAs and improve received signal over FSO communications. The results obtained by implementing the IA system indicates that the power transmitted range from 8 to 100 dB and the adjusted Signal to Noise Ratio (SNR) between 390–––420 dB at 650 nm optical wavelength, whereas the same trend could be observed when the optical wavelength of 1550 nm was applied. Subsequently, the model effectively estimated the automatic transmit power control for enhancement in SNR and resulting in better QoS at the receiving end, regardless of the related weather changes along with the transmission link.

Multiple-instance ensemble for construction of deep heterogeneous committees for high-dimensional low-sample-size data

Deep ensemble learning, where we combine knowledge learned from multiple individual neural networks, has been widely adopted to improve the performance of neural networks in deep learning. This field can be encompassed by committee learning, which includes the construction of neural network cascades. This study focuses on the high-dimensional low-sample-size (HDLS) domain and introduces multiple instance ensemble (MIE) as a novel stacking method for ensembles and cascades. In this study, our proposed approach reformulates the ensemble learning process as a multiple-instance learning problem. We utilise the multiple-instance learning solution of pooling operations to associate feature representations of base neural networks into joint representations as a method of stacking. This study explores various attention mechanisms and proposes two novel committee learning strategies with MIE. In addition, we utilise the capability of MIE to generate pseudo-base neural networks to provide a proof-of-concept for a “growing” neural network cascade that is unbounded by the number of base neural networks. We have shown that our approach provides (1) a class of alternative ensemble methods that performs comparably with various stacking ensemble methods and (2) a novel method for the generation of high-performing “growing” cascades. The approach has also been verified across multiple HDLS datasets, achieving high performance for binary classification tasks in the low-sample size regime.

A heterogeneous graph neural recommendation model with hierarchical social trust

In traditional social recommendation algorithms, methods based on Graph Neural Networks (GNNs) representation of social networks are widely used, but the increasing sparsity of data and the tendency of users’ social influence to be complex make it difficult for the algorithms to improve the recommendation quality efficiently. To address the issues, a heterogeneous graph neural recommendation model with hierarchical social trust (GraphNON) is proposed. The algorithm is improved by fully exploiting user-friend relationships and modeling different propagation channels of social influence in multiple ways respectively. Firstly, different types of potential friends are mined based on user-user social graph and user-item rating graph respectively. Different dynamic influences of direct friends and indirectly friends among users are modeled through hierarchical influence propagation. Secondly, associations between items are mined from user-item rating graph, and influence transmission among items is modeled through hierarchical influence propagation. Finally, updated user characteristics and item characteristics are used to predict whether a user is interested in an item or not. It is shown on both datasets that the model has significant improvement in both hit rate (HR) and normalized discounted cumulative gain (NDCG) metrics compared to the optimal benchmark model. For a potential feature dimension of 64, the HR of the Yelp dataset improved by 18.3% and the NDCG by 29.3%, while the HR of the Flickr dataset improved by 31.4% and the NDCG by 37.1%.

TopoBERT: a plug and play toponym recognition module harnessing fine-tuned BERT

ABSTRACT Extracting precise geographical information from the textual content, referred to as toponym recognition, is fundamental in geographical information retrieval and crucial in a plethora of spatial analyses, e.g. mining location-based information from social media, news reports, and surveys for various applications. However, the performance of existing toponym recognition methods and tools is deficient in supporting tasks that rely on extracting fine-grained geographic information from texts, e.g. locating people sending help requests with addresses through social media during disasters. The emerging pretrained language models have revolutionized natural language processing and understanding by machines, offering a promising pathway to optimize toponym recognition to underpin practical applications. In this paper, TopoBERT, a uniquely designed toponym recognition module based on a one-dimensional Convolutional Neural Network (CNN1D) and Bidirectional Encoder Representation from Transformers (BERT), is proposed and fine-tuned. Three datasets are leveraged to tune the hyperparameters and discover the best strategy to train the model. Another seven datasets are used to evaluate the performance. TopoBERT achieves state-of-the-art performance (average f1-score = 0.854) compared to the seven baseline models. It is encapsulated into easy-to-use python scripts and can be seamlessly applied to diverse toponym recognition tasks without additional training.

Research on the construction and sustainable development of online teaching model for universities based on deep learning

Abstract The worldwide outbreak and pandemic of the New Crown Pneumonia epidemic in 2020 spawned changes in higher education, the most obvious of which was the shift from offline to online teaching models. In this paper, we view online teaching practice activities during the epidemic as a complete dissemination process and systematically study online teaching models in higher education based on convolutional neural network theory. By interpreting the interview records of 100 university teachers interviewed, we can get that 34% of teachers consider themselves as the role of sharers, 29% consider themselves as the leader of students on the learning path, and 24% consider themselves as the role of nurturer for students. Meanwhile, dozens more teachers positioned themselves in various roles that distinguished them from their usual educator roles. From the above data, we can see that the online teaching model has promoted the transformation of teachers’ roles, which is conducive to the development of online teaching and more conducive to promoting the sustainable development of online teaching.

A Fault Detection Algorithm Based on Artificial Neural Network Threshold Selection in Multi-Terminal DC Grids

The DC fault is one of the critical challenges in the High Voltage Multi-Terminal Direct Current (HV-MTDC) grid. The fixed value threshold used in some implementations may be affected by different grid conditions and thus causes errors in the fault detection procedure. This paper offers a two-stage fault detection algorithm based on the integrated Artificial Neural Network (ANN) and Discrete Wavelet Transform (DWT) theory. The main innovations of the proposed method are: 1) Increasing the reliability of the fault detection operation by solving the fixed value threshold problems; 2) Offering high robustness against the wide range of fault resistances; 3) Providing bus protection capability and 4) Setting relays locally, and avoiding the requirement to high-speed communications links. Also, it offers DC fault detection in a short period of time. The performance of the proposed protection scheme has been ascertained via simulations in the MATLAB/Simulink environment. The results prove the robustness of the proposed method against noise disturbance, high resistance fault, and grid parameters variations.

Simulation analysis of visual perception model based on pulse coupled neural network

Pulse-coupled neural networks perform well in many fields such as information retrieval, depth estimation and object detection. Based on pulse coupled neural network (PCNN) theory, this paper constructs a visual perception model framework and builds a real image reproduction platform. The model firstly analyzes the structure and generalization ability of neural network multi-class classifier, uses the minimax criterion of feature space as the splitting criterion of visual perception decision node, which solves the generalization problem of neural network learning algorithm. In the simulation process, the initial threshold is optimized by the two-dimensional maximum inter-class variance method, and in order to improve the real-time performance of the algorithm, the fast recurrence formula of neural network is derived and given. The PCNN image segmentation method based on genetic algorithm is analyzed. The genetic algorithm improves the loop termination condition and the adaptive setting of model parameters of PCNN image segmentation algorithm, but the PCNN image segmentation algorithm still has the problem of complexity. In order to solve this problem, this paper proposed an IGA-PCNN image segmentation method combining the improved algorithm and PCNN model. Firstly, it used the improved immune genetic algorithm to adaptively obtain the optimal threshold, and then replaced the dynamic threshold in PCNN model with the optimal threshold, and finally used the pulse coupling characteristics of PCNN model to complete the image segmentation. From the coupling characteristics of PCNN, junction close space of image and gray level characteristics, it determined the local gray mean square error of image connection strength coefficient. The feature extraction and object segmentation properties of PCNN come from the spike frequency of neurons, and the number of neurons in PCNN is equal to the number of pixels in the input image. In addition, the spatial and gray value differences of pixels should be considered comprehensively to determine their connection matrix. Digital experiments show that the multi-scale multi-task pulse coupled neural network model can shorten the total training time by 17 h, improve the comprehensive accuracy of the task test data set by 1.04%, and shorten the detection time of each image by 4.8 s compared with the series network model of multiple single tasks. Compared with the traditional PCNN algorithm, it has the advantages of fast visual perception and clear target contour segmentation, and effectively improves the anti-interference performance of the model.

A Novel Electric Vehicle Battery Management System Using an Artificial Neural Network-Based Adaptive Droop Control Theory

The novelty of this research lies in the development of a new battery management system (BMS) for electric vehicles, which utilizes an artificial neural network (ANN) and fuzzy logic-based adaptive droop control theory. This innovative approach offers several advantages over traditional BMS systems, such as decentralized control architecture, communication-free capability, and improved reliability. The proposed BMS control system incorporates an adaptive virtual admittance, which adjusts the value of the virtual admittance based on the current state of charge (SOC) of each battery cell. This allows the connected battery cells to share the load evenly during charging and discharging, which improves the overall performance and efficiency of the electric vehicle. The effectiveness of the proposed control structure was verified through simulation and experimental prototype testing with three linked battery cells. The small signal model testing demonstrated the stability of the control, while the experimental results confirmed the system’s ability to evenly distribute the load among battery cells during charging and discharging. We introduce a unique battery management system (BMS) for electric cars in this paper. Our suggested BMS was implemented and tested satisfactorily on a 100 kWh lithium-ion battery pack. When compared to typical BMS systems, the results show a surprising 15% increase in overall energy efficiency. Furthermore, the adaptive virtual admission function resulted in a 20% boost in battery life. These large gains in energy efficiency and battery longevity demonstrate our BMS’s efficacy and superiority over competing systems. Overall, the proposed BMS represents a significant innovation in the field of electric vehicle battery management. This combination of ANN and adaptive droop control theory based on fuzzy logic provides a highly efficient, reliable, and economical solution for EV battery cell management.

Characterization of Dynamic Mechanical Properties of Viscoelastic Damper Based on Physics-Constrained Data-Driven Approach

Traditional physics-based mathematical models of viscoelastic (VE) damper often exhibit different degrees of prediction errors under complex working conditions, whereas data-driven approaches have shown significant potential for overcoming such problems, and have not been introduced into any study to describe the dynamic mechanical properties of VE damper. In this paper, the architecture of microsphere model is adopted to represent the spatial distribution of molecular chain structure of VE damping material, which is categorized into free chains and elastic chains. Then, the physics-informed neural network surrogates are introduced to describe the statistical stress–strain behaviors of free chains and elastic chains. Finally, by constructing the physics-constrained loss function based on the combination of physics-based model and data-driven model, a physics-constrained data-driven model is proposed to characterize the dynamic mechanical behaviors of VE damper used for structure vibration control. Based on the existing experimental data, the prediction ability and robustness of the proposed model are comparatively verified, the results reveal that the proposed model can accurately reflect the stiffness and energy dissipation of VE damper under different ambient temperatures, excitation frequencies, and deformation conditions. This study proves the potential of physics-informed neural network theory in characterizing the complex mechanical behaviors of VE damper.

Machine Learning Methodology for Identifying Vehicles Using Image Processing

Using computer decision-making rather than human decision-making is one of the top priorities of prosperous nations around the world. The reduction of traffic infractions is one area that requires this field. Identifying the type of vehicle will significantly reduce traffic infractions. The aim of using image processing in the context of driving violations is to minimize time wastage, reduce human errors and optimize the use of resources. However, there is still a certain error rate associated with capturing images of offending vehicles and reading their plates, whether done manually or automatically. One solution to this problem is to use image processing and learning algorithms to accurately determine the type of vehicle involved. This approach can be particularly useful in scenarios where it is necessary to identify the traffic volume or count the number of specific types of vehicles passing through a particular area, such as a street or highway. Images of the automobiles, which serve as the paper’s data source, were compiled from a variety of locations with uniform camera-to-vehicle distances. The image’s background is then eliminated by removing it, and the image’s features, including its morphological features, are retrieved and provided to four classifiers so they may carry out the classification procedures. This report focuses on seven different categories of Iranian vehicles. Three sets of raw, filtered and noisy images were subjected to four machine classifiers, including support vector, k-nearest neighbor, perceptron neural network and Bayesian decision theory. The images were subjected to low-pass Gaussian filtering at various frequencies, after which salt-and-pepper noise and Gaussian noise were used to make them noisy. The results demonstrate that our proposed method with the SVM algorithm has provided acceptable results with an accuracy of 97.1, which is not only more accurate than most methods up to this point but also allows the correct classification of Iranian cars.

Can Deep CNNs Avoid Infinite Regress/Circularity in Content Constitution?

The representations of deep convolutional neural networks (CNNs) are formed from generalizing similarities and abstracting from differences in the manner of the empiricist theory of abstraction (Buckner, Synthese 195:5339–5372, 2018). The empiricist theory of abstraction is well understood to entail infinite regress and circularity in content constitution (Husserl, Logical Investigations. Routledge, 2001). This paper argues these entailments hold a fortiori for deep CNNs. Two theses result: deep CNNs require supplementation by Quine’s “apparatus of identity and quantification” in order to (1) achieve concepts, and (2) represent objects, as opposed to “half-entities” corresponding to similarity amalgams (Quine, Quintessence, Cambridge, 2004, p. 107). Similarity amalgams are also called “approximate meaning[s]” (Marcus & Davis, Rebooting AI, Pantheon, 2019, p. 132). Although Husserl inferred the “complete abandonment of the empiricist theory of abstraction” (a fortiori deep CNNs) due to the infinite regress and circularity arguments examined in this paper, I argue that the statistical learning of deep CNNs may be incorporated into a Fodorian hybrid account that supports Quine’s “sortal predicates, negation, plurals, identity, pronouns, and quantifiers” which are representationally necessary to overcome the regress/circularity in content constitution and achieve objective (as opposed to similarity-subjective) representation (Burge, Origins of Objectivity. Oxford, 2010, p. 238). I base myself initially on Yoshimi’s (New Frontiers in Psychology, 2011) attempt to explain Husserlian phenomenology with neural networks but depart from him due to the arguments and consequently propose a two-system view which converges with Weiskopf’s proposal (“Observational Concepts.” The Conceptual Mind. MIT, 2015. 223–248).

Network traffic anomaly detection method based on chaotic neural network

Network abnormal traffic detection is a hot topic in network security. Based on the theory of chaotic neural network, this paper constructs a network traffic anomaly detection model to solve the problems of high dimension of abnormal traffic and overfitting of classification model caused by outliers. The model firstly calculates mutual information according to the correlation degree between features and class tags, and selects excellent feature subset to extract suitable features for online traffic anomaly detection. Then, chaotic neural network algorithm is used to search the best feature subset from the original feature set, and an adaptive strategy is adopted to add and delete the iterated features. The chaotic neural network is used to specify the solution strategy for the problem that it is difficult to select the candidate feature subset as the information gain of each feature in the traffic record, which contributes to improving the quality of data. The results show that this method can reduce the training time by more than 12% without losing the performance indexes of the original classification model, and significantly improve the classification performance and prediction accuracy.