Neural Network Paradigm Research Articles

Enhancing Efficacy in Breast Cancer Screening with Nesterov Momentum Optimization Techniques

In the contemporary landscape of healthcare, machine learning models are pivotal in facilitating precise predictions, particularly in the nuanced diagnosis of complex ailments such as breast cancer. Traditional diagnostic methodologies grapple with inherent challenges, including excessive complexity, elevated costs, and reliance on subjective interpretation, which frequently culminate in inaccuracies. The urgency of early detection cannot be overstated, as it markedly broadens treatment modalities and significantly enhances survival rates. This paper delineates an innovative optimization framework designed to augment diagnostic accuracy by amalgamating momentum-based optimization techniques within a neural network paradigm. Conventional machine learning approaches are often encumbered by issues of overfitting, data imbalance, and the inadequacy of capturing intricate patterns in high-dimensional datasets. To counter these limitations, we propose a sophisticated framework that integrates an adaptive threshold mechanism across an array of gradient-based optimizers, including SGD, RMSprop, adam, adagrad, adamax, adadelta, nadam and Nesterov momentum. This novel approach effectively mitigates oscillatory behavior, refines parameter updates, and accelerates convergence. A salient feature of our methodology is the incorporation of a momentum threshold for early stopping, which ceases training upon the stabilization of momentum below a pre-defined threshold, thereby pre-emptively preventing overfitting. Leveraging the Wisconsin Breast Cancer Dataset, our model achieved a remarkable 99.72% accuracy and 100% sensitivity, significantly curtailing misclassification rates compared to traditional methodologies. This framework stands as a robust solution for early breast cancer diagnosis, thereby enhancing clinical decision making and improving patient outcomes.

"Synthetic Map": A Graphic Organizer Inspired by Artificial Neural Network Paradigms for Learning Organic Synthesis.

Organic Chemistry is widely recognized as a challenging subject, with the design of syntheses and retrosyntheses identified as particularly difficult tasks. Inspired by the success of artificial neural networks in machine learning, we propose a framework that leverages similar principles to enhance the teaching and learning of organic synthesis. In this paper, we introduce a novel teaching tool, the "Synthetic Map", that attempts to visually recreate an expert's mental map and conceptual understanding of organic synthesis built over years of experience. The educational benefits of the Synthetic Map were evaluated through its implementation in an Organic Chemistry course of a Pharmacy degree over two years. The new tool promoted students' learning by providing a mental organizer fostering a deeper understanding of the subject and empowering students to design and execute effective synthetic strategies.

Toward Explainable Multiparty Learning: A Contrastive Knowledge Sharing Framework.

Multiparty learning provides solutions for training joint models with decentralized data under legal and practical constraints. However, traditional multiparty learning approaches are confronted with obstacles, such as system heterogeneity, statistical heterogeneity, and incentive design. Determining how to deal with these challenges and further improve the efficiency and performance of multiparty learning has become an urgent problem to be solved. In this article, we propose a novel contrastive multiparty learning framework for knowledge refinement and sharing with an accountable incentive mechanism. Since the existing parameter averaging method is contradictory to the learning paradigm of neural networks, we simulate the process of human cognition and communication and analogize multiparty learning as a many-to-one knowledge-sharing problem. The approach is capable of integrating the acquired explicit knowledge of each client in a transparent manner without privacy disclosure, and it reduces the dependence on data distribution and communication environments. The proposed scheme achieves significant improvement in model performance in a variety of scenarios, as we demonstrated through experiments on several real-world datasets.

Predictions in biometric models

One of the domains of genetic enhancement that has extensively employed both simulation and authentic data is Biometrics. Selecting efficient models for the Genome-Wide Selection (GWS) process using molecular markers (SNPs) presents several challenges. Among these challenges is the effective identification of the optimal model for fitting a given dataset. To contribute to this endeavor, this paper's primary objective is to assess the predictive accuracy of nine (9) distinct models, each following different paradigms within the realm of Biometrics. The data employed in this study were generated through simulation, encompassing the primary issues encountered in this field of research, including high dimensionality, nonlinearity, and multicollinearity. As the primary findings, notable observations include the enhancement of predictive efficiency as data noise decreases, the predominance of the tree paradigm (for low noise levels, BOO), and the efficacy of the neural network paradigm (for high noise levels, RBF).

A two-timescale neurodynamic approach to robust distributed model predictive control for nonlinear systems

In this paper, a robust distributed model predictive control strategy is proposed for nonlinear dynamical systems affected by bounded disturbances. Grounded on sequential quadratic programming, a two-timescale recurrent neural network paradigm is deployed for solving minimax optimization problems. Sufficient criteria for the two-timescale recurrent neural network are delineated, which specifically guarantee its stability and optimality. Under these conditions, it is demonstrated that the recurrent neural network converges to optimal solutions, thereby enhancing the robustness of the control strategy. The simulation results exhibit significantly enhanced convergence in comparison to the mono-timescale recurrent neural network.

IFNN: Enhanced interpretability and optimization in FNN via Adam algorithm

This investigation seeks to reconcile the interpretability of artificial intelligence with the imperative of maintaining optimal performance, through the introduction of the IFNN model. This model represents a sophisticated iteration of the Fuzzy Neural Network (FNN) paradigm, engineered for enhanced interpretability. At the heart of this innovation lies the application of the Adam optimization algorithm, integrated into a tri-layered architectural construct. This approach is deliberately designed to elevate the models' accuracy across an array of datasets, thereby positioning the IFNN model as an example of an interpretable AI system that does not compromise on precision.A seminal aspect of this model is its capacity to transmute logical neurons within the intermediary layer into clear fuzzy rules. This transformative process propels the model beyond the confines of traditional AI frameworks, ushering in a new era of transparency in the AI decision-making arena. Such progress is achieved through a meticulous fuzzy rule-based examination, anchored by an exhaustive appraisal of interpretability metrics, including but not limited to sensitivity, completeness, and the analysis of fuzzy rule consequents. These logical neurons, the progenitors of the aforementioned fuzzy rules, endow the model with the ability to engage in deep, interpretable analyses of data.The IFNN model has been tested through statistical analysis, interpretability assessments, and empirical validation against real-world datasets about sepsis identification, showcasing its unparalleled ability to unlock and articulate the complex knowledge embedded within data. This model represents a significant evolution in AI methodologies, providing a clear window into the rationale underpinning its decisions, achieved through an advanced fuzzy rule-based methodology and a full spectrum of interpretability metrics.

Variable Temporal Length Training for Action Recognition CNNs.

Most current deep learning models are suboptimal in terms of the flexibility of their input shape. Usually, computer vision models only work on one fixed shape used during training, otherwise their performance degrades significantly. For video-related tasks, the length of each video (i.e., number of video frames) can vary widely; therefore, sampling of video frames is employed to ensure that every video has the same temporal length. This training method brings about drawbacks in both the training and testing phases. For instance, a universal temporal length can damage the features in longer videos, preventing the model from flexibly adapting to variable lengths for the purposes of on-demand inference. To address this, we propose a simple yet effective training paradigm for 3D convolutional neural networks (3D-CNN) which enables them to process videos with inputs having variable temporal length, i.e., variable length training (VLT). Compared with the standard video training paradigm, our method introduces three extra operations during training: sampling twice, temporal packing, and subvideo-independent 3D convolution. These operations are efficient and can be integrated into any 3D-CNN. In addition, we introduce a consistency loss to regularize the representation space. After training, the model can successfully process video with varying temporal length without any modification in the inference phase. Our experiments on various popular action recognition datasets demonstrate the superior performance of the proposed method compared to conventional training paradigm and other state-of-the-art training paradigms.

Machine unlearning in brain-inspired neural network paradigms.

Machine unlearning, which is crucial for data privacy and regulatory compliance, involves the selective removal of specific information from a machine learning model. This study focuses on implementing machine unlearning in Spiking Neuron Models (SNMs) that closely mimic biological neural network behaviors, aiming to enhance both flexibility and ethical compliance of AI models. We introduce a novel hybrid approach for machine unlearning in SNMs, which combines selective synaptic retraining, synaptic pruning, and adaptive neuron thresholding. This methodology is designed to effectively eliminate targeted information while preserving the overall integrity and performance of the neural network. Extensive experiments were conducted on various computer vision datasets to assess the impact of machine unlearning on critical performance metrics such as accuracy, precision, recall, and ROC AUC. Our findings indicate that the hybrid approach not only maintains but in some cases enhances the neural network's performance post-unlearning. The results confirm the practicality and efficiency of our approach, underscoring its applicability in real-world AI systems.

Viewing resolution and depth-of-field enhancement for a digital 3D display based on neural network-enabled multilayer view perspective fitting

The traditional digital three-dimensional (3D) display suffers from low resolution and a narrow depth of field (DoF) due to the lack of planar pixels transformed into view perspectives and the limitation of the diffraction effect of the lens, respectively, which are the main drawbacks to restrict the commercial application of this display technology. Here, the neural network-enabled multilayer view perspective fitting between the reconstructed and original view perspectives across the desired viewing depth range is proposed to render the optimal elemental image array (EIA) for enhancing the viewing resolution as well as the DoF of the digital 3D display. Actually, it is an end-to-end result-oriented coding method to render the fusion EIA with optimal multidepth fusion and resolution enhancement with high registration accuracies for both view perspective and depth reconstructions by using a depth-distributed fitting neural network paradigm. The 3D images presented in the simulations and optical experiments with improved viewing resolution and extended viewing depth range are demonstrated, verifying the feasibility of the proposed method.

Positional embeddings for solving PDEs with evolutional deep neural networks

This work extends the paradigm of evolutional deep neural networks (EDNNs) to solving parametric time-dependent partial differential equations (PDEs) on domains with geometric structure. By introducing positional embeddings based on eigenfunctions of the Laplace-Beltrami operator, geometric properties are encoded intrinsically and Dirichlet, Neumann and periodic boundary conditions of the PDE solution are enforced directly through the neural network architecture. The proposed embeddings lead to improved error convergence for static PDEs and extend EDNNs towards computational domains of realistic complexity.Several steps are taken to improve performance of EDNNs: Solving the EDNN update equation with a Krylov solver avoids the explicit assembly of Jacobians and enables scaling to larger neural networks. Computational efficiency is further improved by an ad-hoc active sampling scheme that uses the PDE dynamics to effectively sample collocation points. A modified linearly implicit Rosenbrock method is proposed to alleviate the time step requirements of stiff PDEs. Lastly, a completely training-free approach, which automatically enforces initial conditions and only requires time integration, is compared against EDNNs that are trained on the initial conditions. We report results for the Korteweg-de Vries equation, a nonlinear heat equation and (nonlinear) advection-diffusion problems on domains with and without holes and various boundary conditions, to demonstrate the effectiveness of the method. The numerical results highlight EDNNs as a promising surrogate model for parametrized PDEs with slow decaying Kolmogorov n-width.

Two-step interpretable modeling of ICU-AIs

We present a novel methodology for integrating high resolution longitudinal data with the dynamic prediction capabilities of survival models. The aim is two-fold: to improve the predictive power while maintaining the interpretability of the models. To go beyond the black box paradigm of artificial neural networks, we propose a parsimonious and robust semi-parametric approach (i.e., a landmarking competing risks model) that combines routinely collected low-resolution data with predictive features extracted from a convolutional neural network, that was trained on high resolution time-dependent information. We then use saliency maps to analyze and explain the extra predictive power of this model. To illustrate our methodology, we focus on healthcare-associated infections in patients admitted to an intensive care unit.

Measuring Task Similarity and Its Implication in Fine-Tuning Graph Neural Networks

The paradigm of pre-training and fine-tuning graph neural networks has attracted wide research attention. In previous studies, the pre-trained models are viewed as universally versatile, and applied for a diverse range of downstream tasks. In many situations, however, this practice results in limited or even negative transfer. This paper, for the first time, emphasizes the specific application scope of graph pre-trained models: not all downstream tasks can effectively benefit from a graph pre-trained model. In light of this, we introduce the measure task consistency to quantify the similarity between graph pre-training and downstream tasks. This measure assesses the extent to which downstream tasks can benefit from specific pre-training tasks. Moreover, a novel fine-tuning strategy, Bridge-Tune, is proposed to further diminish the impact of the difference between pre-training and downstream tasks. The key innovation in Bridge-Tune is an intermediate step that bridges pre-training and downstream tasks. This step takes into account the task differences and further refines the pre-trained model. The superiority of the presented fine-tuning strategy is validated via numerous experiments with different pre-trained models and downstream tasks.

Fine-Tuning Graph Neural Networks by Preserving Graph Generative Patterns

Recently, the paradigm of pre-training and fine-tuning graph neural networks has been intensively studied and applied in a wide range of graph mining tasks. Its success is generally attributed to the structural consistency between pre-training and downstream datasets, which, however, does not hold in many real-world scenarios. Existing works have shown that the structural divergence between pre-training and downstream graphs significantly limits the transferability when using the vanilla fine-tuning strategy. This divergence leads to model overfitting on pre-training graphs and causes difficulties in capturing the structural properties of the downstream graphs. In this paper, we identify the fundamental cause of structural divergence as the discrepancy of generative patterns between the pre-training and downstream graphs. Furthermore, we propose G-Tuning to preserve the generative patterns of downstream graphs. Given a downstream graph G, the core idea is to tune the pre-trained GNN so that it can reconstruct the generative patterns of G, the graphon W. However, the exact reconstruction of a graphon is known to be computationally expensive. To overcome this challenge, we provide a theoretical analysis that establishes the existence of a set of alternative graphons called graphon bases for any given graphon. By utilizing a linear combination of these graphon bases, we can efficiently approximate W. This theoretical finding forms the basis of our model, as it enables effective learning of the graphon bases and their associated coefficients. Compared with existing algorithms, G-Tuning demonstrates consistent performance improvement in 7 in-domain and 7 out-of-domain transfer learning experiments.

A Broadly Applicable Coarse Alignment Framework for the Point Cloud of Hull Blocks

_ The terrestrial laser scanning (TLS) technology, which has been widely used in recent years, provides a new approach to control the construction quality of hull blocks in shipyards. As the cloud-model registration problem plays the most significant role in the data processing stage of utilizing the TLS devices, this paper concentrates on developing a broadly applicable coarse alignment framework for the point cloud of hull blocks. This framework involves two steps: the recognition of the connection regions of the point clouds and the coarse registration method according to the classification results. To detect the connection regions automatically, a hand-crafted simple model of the connection regions of hull blocks is built and its supporting detection method is proposed, besides, considering the recent overwhelming success of deep learning methods, a deep learning network suitable for large hull block datasets is constructed according to the Point-Net, and convolutional neural networks paradigms. Then, the coarse alignment method based on the detected connection regions is proposed. Experimental results illustrate the good performance of the proposed framework. Keywords computers in construction; shipbuilding; automation

Multi-head neural networks for simulating particle breakage dynamics

The breakage of brittle particulate materials into smaller particles under compressive or impact loads can be modelled as an instantiation of the population balance integro-differential equation. In this paper, the emerging computational science paradigm of physics-informed neural networks is studied for the first time for solving both linear and nonlinear variants of the governing dynamics. Unlike conventional methods, the proposed neural network provides rapid simulations of arbitrarily high resolution in particle size, predicting values on arbitrarily fine grids without the need for model retraining. The network is assigned a simple multi-head architecture tailored to uphold monotonicity of the modelled cumulative distribution function over particle sizes. The method is theoretically analyzed and validated against analytical results before being applied to real-world data of a batch grinding mill. The agreement between laboratory data and numerical simulation encourages the use of physics-informed neural nets for optimal planning and control of industrial comminution processes.

Privacy preserved and decentralized thermal comfort prediction model for smart buildings using federated learning.

Thermal comfort is a crucial element of smart buildings that assists in improving, analyzing, and realizing intelligent structures. Energy consumption forecasts for such smart buildings are crucial owing to the intricate decision-making processes surrounding resource efficiency. Machine learning (ML) techniques are employed to estimate energy consumption. ML algorithms, however, require a large amount of data to be adequate. There may be privacy violations due to collecting this data. To tackle this problem, this study proposes a federated deep learning (FDL) architecture developed around a deep neural network (DNN) paradigm. The study employs the ASHRAE RP-884 standard dataset for experimentation and analysis, which is available to the general public. The data is normalized using the min-max normalization approach, and the Synthetic Minority Over-sampling Technique (SMOTE) is used to enhance the minority class's interpretation. The DNN model is trained separately on the dataset after obtaining modifications from two clients. Each client assesses the data greatly to reduce the over-fitting impact. The test result demonstrates the efficiency of the proposed FDL by reaching 82.40% accuracy while securing the data.

HITS-based Propagation Paradigm for Graph Neural Networks

In this article, we present a new propagation paradigm based on the principle of Hyperlink-Induced Topic Search (HITS) algorithm. The HITS algorithm utilizes the concept of a “self-reinforcing” relationship of authority-hub. Using HITS, the centrality of nodes is determined via repeated updates of authority-hub scores that converge to a stationary distribution. Unlike PageRank-based propagation methods, which rely solely on the idea of authorities (in-links), HITS considers the relevance of both authorities (in-links) and hubs (out-links), thereby allowing for a more informative graph learning process. To segregate node prediction and propagation, we use a Multilayer Perceptron in combination with a HITS-based propagation approach and propose two models: HITS-GNN and HITS-GNN+. We provided additional validation of our models’ efficacy by performing an ablation study to assess the performance of authority-hub in independent models. Moreover, the effect of the main hyper-parameters and normalization is also analyzed to uncover how these techniques influence the performance of our models. Extensive experimental results indicate that the proposed approach significantly improves baseline methods on the graph (citation network) benchmark datasets by a decent margin for semi-supervised node classification, which can aid in predicting the categories (labels) of scientific articles not exclusively based on their content but also based on the type of articles they cite.

Hebbian dreaming for small datasets

The dreaming Hopfield model constitutes a generalization of the Hebbian paradigm for neural networks, that is able to perform on-line learning when “awake” and also to account for off-line “sleeping” mechanisms. The latter have been shown to enhance storing in such a way that, in the long sleep-time limit, this model can reach the maximal storage capacity achievable by networks equipped with symmetric pairwise interactions. In this paper, we inspect the minimal amount of information that must be supplied to such a network to guarantee a successful generalization, and we test it both on random synthetic and on standard structured datasets (i.e., MNIST, Fashion-MNIST and Olivetti). By comparing these minimal thresholds of information with those required by the standard (i.e., always “awake”) Hopfield model, we prove that the present network can save up to ∼90% of the dataset size, yet preserving the same performance of the standard counterpart. This suggests that sleep may play a pivotal role in explaining the gap between the large volumes of data required to train artificial neural networks and the relatively small volumes needed by their biological counterparts. Further, we prove that the model Cost function (typically used in statistical mechanics) admits a representation in terms of a standard Loss function (typically used in machine learning) and this allows us to analyze its emergent computational skills both theoretically and computationally: a quantitative picture of its capabilities as a function of its control parameters is achieved and consistency between the two approaches is highlighted.The resulting network is an associative memory for pattern recognition tasks that learns from examples on-line, generalizes correctly (in suitable regions of its control parameters) and optimizes its storage capacity by off-line sleeping: such a reduction of the training cost can be inspiring toward sustainable AI and in situations where data are relatively sparse.

Early energy performance analysis of smart buildings by consolidated artificial neural network paradigms

The assessment of energy performance in smart buildings has emerged as a prominent area of research driven by the increasing energy consumption trends worldwide. Analyzing the attributes of buildings using optimized machine learning models has been a highly effective approach for estimating the cooling load (CL) and heating load (HL) of the buildings. In this study, an artificial neural network (ANN) is used as the basic predictor that undergoes optimization using five metaheuristic algorithms, namely coati optimization algorithm (COA), gazelle optimization algorithm (GOA), incomprehensible but intelligible-in-time logics (IbIL), osprey optimization algorithm (OOA), and sooty tern optimization algorithm (STOA) to predict the CL and HL of a residential building. The models are trained and tested via an Energy Efficiency dataset (downloaded from UCI Repository). A score-based ranking system is built upon three accuracy evaluators including mean absolute percentage error (MAPE), root mean square error (RMSE), and percentage-Pearson correlation coefficient (PPCC) to compare the prediction accuracy of the models. Referring to the results, all models demonstrated high accuracy (e.g., PPCCs >89%) for predicting both CL and HL. However, the calculated final scores of the models (43, 20, 39, 38, and 10 in HL prediction and 36, 20, 42, 42, and 10 in CL prediction for the STOA, OOA, IbIL, GOA, and COA, respectively) indicated that the GOA, IbIL, and STOA perform better than COA and OOA. Moreover, a comparison with various algorithms used in earlier literature showed that the GOA, IbIL, and STOA provide a more accurate solution. Therefore, the use of ANN optimized by these three algorithms is recommended for practical early forecast of energy performance in buildings and optimizing the design of energy systems.

Predicting the flowability of UHPC and identifying its significant influencing factors using an accurate ANN model

In this research, a one-hidden layer artificial neural network paradigm (ANN) was created to forecast the slump flow of ultra-high-performance concrete (UHPC). To achieve this goal, 3,200 ANNs were evaluated to estimate the fresh UHPC’s slump flow utilizing 793 observations. The performance metrics measured on training and test data subsets were in the same order of magnitude, thereby pointing out the proper work of the k-fold validation procedure. The results of the connection weight approach analysis (CWA) indicated that water dosage had the highest positive importance in slump flow, preceding the superplasticizer volume ratio. Other factors that positively influenced slump flow were the water-to-powder ratio, the dosage of high-alkali glass powder, the water-to-binder ratio, and limestone concentration. The most negative influences on rheology were the high-alumina FC3R and metakaolin. The ANN accurately predicted the slump flow of UHPC, while the results of the CWA analysis were well-correlated with previous research.