Layer Neural Network Research Articles

A new fusion neural network model and credit card fraud identification.

Credit card fraud identification is an important issue in risk prevention and control for banks and financial institutions. In order to establish an efficient credit card fraud identification model, this article studied the relevant factors that affect fraud identification. A credit card fraud identification model based on neural networks was constructed, and in-depth discussions and research were conducted. First, the layers of neural networks were deepened to improve the prediction accuracy of the model; second, this paper increase the hidden layer width of the neural network to improve the prediction accuracy of the model. This article proposes a new fusion neural network model by combining deep neural networks and wide neural networks, and applies the model to credit card fraud identification. The characteristic of this model is that the accuracy of prediction and F1 score are relatively high. Finally, use the random gradient descent method to train the model. On the test set, the proposed method has an accuracy of 96.44% and an F1 value of 96.17%, demonstrating good fraud recognition performance. After comparison, the method proposed in this paper is superior to machine learning models, ensemble learning models, and deep learning models.

Numerical investigation of the effective receptive field and its relationship with convolutional kernels and layers in convolutional neural network

The receptive field (RF) plays a crucial role in convolutional neural networks (CNNs) because it determines the amount of input information that each neuron in a CNN can perceive, which directly affects the feature extraction ability. As the number of convolutional layers in CNNs increases, there is a decay of the RF according to the two-dimensional Gaussian distribution. Thus, an effective receptive field (ERF) can be used to characterize the available part of the RF. The ERF is calculated by the kernel size and layer number within the neural network architecture. Currently, ERF calculation methods are typically applied to single-channel input data that are both independent and identically distributed. However, such methods may result in a loss of effective information if they are applied to more general (i.e., multi-channel) datasets. Therefore, we proposed a multi-channel ERF calculation method. By conducting a series of numerical experiments, we determined the relationship between the ERF and the convolutional kernel size in conjunction with the layer number. To validate the new method, we used the recently published global wave surrogate model for climate simulation (GWSM4C) and its accompanying dataset. According to the newly established relationship, we refined the kernel size and layer number in each neural network of the GWSM4C to produce the same ERF but lower RF attenuation rates than those of the original version. By visualizing the gradient map at several points in West African and East Pacific areas, the high gradient value regions confirmed the known swell sources, which indicated effective feature extraction in these areas. Furthermore, the new version of the GWSM4C yielded better prediction accuracy for significant wave height in global swell pools. The root mean square errors in the West African and East Pacific regions reduced from approximately 0.3 m, in the original model to about 0.15 m, in the new model. Moreover, these improvements were attributed to the higher efficiency of the newly modified neural network structure that allows the inclusion of more historical winds while maintaining acceptable computational consumption.

Utilizing Attention-Enhanced Deep Neural Networks for Large-Scale Preliminary Diabetes Screening in Population Health Data

Early screening for diabetes can promptly identify potential early stage patients, possibly delaying complications and reducing mortality rates. This paper presents a novel technique for early diabetes screening and prediction, called the Attention-Enhanced Deep Neural Network (AEDNN). The proposed AEDNN model incorporates an Attention-based Feature Weighting Layer combined with deep neural network layers to achieve precise diabetes prediction. In this study, we utilized the Diabetes-NHANES dataset and the Pima Indians Diabetes dataset. To handle significant missing values and outliers, group median imputation was applied. Oversampling techniques were used to balance the diabetes and non-diabetes groups. The data were processed through an Attention-based Feature Weighting Layer for feature extraction, producing a feature matrix. This matrix was subjected to Hadamard product operations with the raw data to obtain weighted data, which were subsequently input into deep neural network layers for training. The parameters were fine-tuned and the L2 regularization and dropout layers were added to enhance the generalization performance of the model. The model’s reliability was thoroughly assessed through various metrics, including the accuracy, precision, recall, F1 score, mean squared error (MSE), and R2 score, as well as the ROC and AUC curves. The proposed model achieved a prediction accuracy of 98.4% in the Pima Indians Diabetes dataset. When the test dataset was expanded to the large-scale Diabetes-NHANES dataset, which contains 52,390 samples, the test precision of the model improved further to 99.82%, with an AUC of 0.9995. A comparative analysis was conducted using multiple models, including logistic regression with L1 regularization, support vector machine (SVM), random forest, K-nearest neighbors (KNNs), AdaBoost, XGBoost, and the latest semi-supervised XGBoost. The feature extraction method using attention mechanisms was compared with the classical feature selection methods, Lasso and Ridge. The experiments were performed on the same dataset, and the conclusion was that the Attention-based Ensemble Deep Neural Network (AEDNN) outperformed all the aforementioned methods. These results indicate that the model not only performs well on smaller datasets but also fully leverages its advantages on larger datasets, demonstrating strong generalization ability and robustness. The proposed model can effectively assist clinicians in the early screening of diabetes patients. This is particularly beneficial for the preliminary screening of high-risk individuals in large-scale, extensive healthcare datasets, followed by detailed examination and diagnosis. Compared to the existing methods, our AEDNN model showed an overall performance improvement of 1.75%.

Predicting Bacterial Antibiotic Resistance using MALDI-TOF Mass Spectrometry Databases with ELM Applications.

Early detection of antibiotic resistance is a crucial task, especially for vulnerable patients under prolonged treatments with a single antibiotic. To solve this, machine learning approaches have been reported in the state of art. Researchers have used MALDI-TOF MS in order to predict antibiotic resistance and/or susceptibility in bacterial samples. Weis, et al. implemented LR, LightGBM and ANN to study the antibiotic resistance on bacterial strains of Escherichia Coli, Staphylococcus Aureus, and Klebsiella Pneumoniae. Despite promising results, the models have not achieved perfect accuracy, specifically when the classes are unbalanced. On the other hand, Extreme Learning Machine (ELM) is a training algorithm for forward propagation of single hidden layer neural networks, which converges much faster than traditional methods and offers promising performance along with less programmer intervention. In this way, this study introduced improved ELMs, including two weighted ELMs proposed by Zong, and the SMOTE technique in order to create new synthetic samples of the minority class. After heuristic optimization of ELM hiper-parameters, results demonstrated 85% in accuracy and 85% in geometric mean for the classification problem in the case of weighted ELM 1 subject to the SMOTE technique of oversampling.

Deep-time neural networks: An efficient approach for solving high-dimensional PDEs

This paper presents the Deep-Time Neural Network (DTNN), an efficient and novel deep-learning approach for solving partial differential equations (PDEs). DTNN leverages the power of deep neural networks to approximate the solution for a class of quasi-linear parabolic PDEs. We demonstrate that DTNN significantly reduces the computational cost and speeds up the training process compared to other models in the literature. The results of our study indicate that DTNN architecture is promising for the fast and accurate solution of time-dependent PDEs in various scientific and engineering applications. The DTNN architecture addresses the pressing need for enhanced time considerations in the deeper layers of Artificial Neural Networks (ANNs), thereby improving convergence time for high-dimensional PDE solutions. This is achieved by integrating time into the hidden layers of the DTNN, demonstrating a marked improvement over existing ANN-based solutions regarding efficiency and speed.

Investigating 10 Yr of Volcanoacoustic Activity at Tungurahua Volcano, Ecuador, Aided by Machine Learning

Abstract Here, we present the results of applying diverse data processing and machine learning tools to investigate a very large dataset obtained from single station infrasonic recordings from the last 10 yr of the most recent period of explosive activity at Tungurahua volcano, Ecuador. To increase the quality and quantity of information extracted from the large data set and enhance pattern recognition, we combined traditional techniques with more recent ones. We divided the investigation into sequential steps: detection, discrimination, cleaning, and clustering. For the detection step, we tested the classical short-term average/long-term average algorithm and an algorithm specific for explosions detection called “Volcanic INfrasound Explosions Detector Algorithm (VINEDA)” and detected 118,516 events. To clean the detected signals from potential false positives, we used supervised classification that reduced the events to 75,483, and a catalog cleaning procedure using shallow learners including support vector machines, random forests, and a single layer neural network, trained using data from a manual catalog, to a final number of 36,359 events. This led to a sixfold increase in detected explosions compared to the manual catalog. Then, we applied hierarchical clustering to a well-studied time window of activity using two independent difference metrics: dynamic time warping and waveform cross correlation and showed the insights and drawbacks from this approach. We showed that the different techniques were able to reveal repeating and striving events between selected different eruptive phases and associated them to possible changes in eruptive dynamics. Finally, to analyze the whole dataset at once we used a convolutional autoencoder network and obtained similar results to the classical clustering in a fraction of the time. We identified different families of explosions that appeared, sometimes intermittently, and revealed various potentially competing eruptive processes during the whole time period.

Distance preserving machine learning for uncertainty aware accelerator capacitance predictions

Abstract Accurate uncertainty estimations are essential for producing reliable machine learning models, especially in safety-critical applications such as accelerator systems. Gaussian process models are generally regarded as the gold standard for this task; however, they can struggle with large, high-dimensional datasets. Combining deep neural networks with Gaussian process approximation techniques has shown promising results, but dimensionality reduction through standard deep neural network layers is not guaranteed to maintain the distance information necessary for Gaussian process models. We build on previous work by comparing the use of the singular value decomposition against a spectral-normalized dense layer as a feature extractor for a deep neural Gaussian process approximation model and apply it to a capacitance prediction problem for the High Voltage Converter Modulators in the Oak Ridge Spallation Neutron Source. Our model shows improved distance preservation and predicts in-distribution capacitance values with less than 1% error.

AWDP-FL: An Adaptive Differential Privacy Federated Learning Framework

Data security and user privacy concerns are receiving increasing attention. Federated learning models based on differential privacy offer a distributed machine learning framework that protects data privacy. However, the noise introduced by the differential privacy mechanism may affect the model’s usability, especially when reasonable gradient clipping is absent. Fluctuations in the gradients can lead to issues like gradient explosion, compromising training stability and potentially leaking privacy. Therefore, gradient clipping has become a crucial method for protecting both model performance and data privacy. To balance privacy protection and model performance, we propose the Adaptive Weight-Based Differential Privacy Federated Learning (AWDP-FL) framework, which processes model gradient parameters at the neural network layer level. First, by designing and recording the change trends of two-layer historical gradient sequences, we analyze and predict gradient variations in the current iteration and calculate the corresponding weight values. Then, based on these weights, we perform adaptive gradient clipping for each data point in each training batch, which is followed by gradient momentum updates based on the third moment. Before uploading the parameters, Gaussian noise is added to protect privacy while maintaining model accuracy. Theoretical analysis and experimental results validate the effectiveness of this framework under strong privacy constraints.

The topology and geometry of neural representations

A central question for neuroscience is how to characterize brain representations of perceptual and cognitive content. An ideal characterization should distinguish different functional regions with robustness to noise and idiosyncrasies of individual brains that do not correspond to computational differences. Previous studies have characterized brain representations by their representational geometry, which is defined by the representational dissimilarity matrix (RDM), a summary statistic that abstracts from the roles of individual neurons (or responses channels) and characterizes the discriminability of stimuli. Here, we explore a further step of abstraction: from the geometry to the topology of brain representations. We propose topological representational similarity analysis, an extension of representational similarity analysis that uses a family of geotopological summary statistics that generalizes the RDM to characterize the topology while de-emphasizing the geometry. We evaluate this family of statistics in terms of the sensitivity and specificity for model selection using both simulations and functional MRI (fMRI) data. In the simulations, the ground truth is a data-generating layer representation in a neural network model and the models are the same and other layers in different model instances (trained from different random seeds). In fMRI, the ground truth is a visual area and the models are the same and other areas measured in different subjects. Results show that topology-sensitive characterizations of population codes are robust to noise and interindividual variability and maintain excellent sensitivity to the unique representational signatures of different neural network layers and brain regions.

Brain-body-task co-adaptation can improve autonomous learning and speed of bipedal walking.

Inspired by animals that co-adapt their brain and body to interact with the environment, we present a tendon-driven and over-actuated (i.e., n joint, n+1 actuators) bipedal robot that (i) exploits its backdrivable mechanical properties to manage body-environment interactions without explicit control, and (ii) uses a simple 3-layer neural network to learn to walk after only 2 minutes of 'natural' motor babbling (i.e., an exploration strategy that is compatible with leg and task dynamics; akin to childsplay). This brain-body collaboration first learns to produce feet cyclical movements 'in air' and, without further tuning, can produce locomotion when the biped is lowered to be in slight contact with the ground. In contrast, training with 2 minutes of 'naive' motor babbling (i.e., an exploration strategy that ignores leg task dynamics), does not produce consistent cyclical movements 'in air', and produces erratic movements and no locomotion when in slight contact with the ground. When further lowering the biped and making the desired leg trajectories reach 1cm below ground (causing the desired-vs-obtained trajectories error to be unavoidable), cyclical movements based on either natural or naive babbling presented almost equally persistent trends, and locomotion emerged with naive babbling. Therefore, we show how continual learning of walking in unforeseen circumstances can be driven by continual physical adaptation rooted in the backdrivable properties of the plant and enhanced by exploration strategies that exploit plant dynamics. Our studies also demonstrate that the bio-inspired co-design and co-adaptations of limbs and control strategies can produce locomotion without explicit control of trajectory errors.

Trainable signal encoders that are robust against noise

Within the deep learning paradigm, finite impulse response (FIR) filters are often used to encode audio signals, yielding flexible and adaptive feature representations. We show that a stabilization of FIR filterbanks with fixed filter lengths (convolutional layers with 1-D filters) by means of their condition number leads to encoders that are optimally robust against noise and can be inverted with perfect reconstruction by their transposes. To maintain their flexibility as regular neural network layers we implement the stabilization via a computationally efficient regularizing term in the objective function of the learning problem. In this way, the encoder keeps its expressive power and is optimally stable and noise-robust throughout the whole learning procedure. We show in a denoising task where noise is additionally present in the encoder representation of the signals, that the proposed stabilization of the trainable filterbank encodings is decisive for increasing the signal-to-noise ratio of the denoised signals significantly compared to a model with a naively trained encoder.

Validation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports

AimsThe primary goal of this study is to evaluate the capabilities of Large Language Models (LLMs) in understanding and processing complex medical documentation. We chose to focus on the identification of pathologic complete response (pCR) in narrative pathology reports. This approach aims to contribute to the advancement of comprehensive reporting, health research, and public health surveillance, thereby enhancing patient care and breast cancer management strategies.MethodsThe study utilized two analytical pipelines, developed with open-source LLMs within the healthcare system’s computing environment. First, we extracted embeddings from pathology reports using 15 different transformer-based models and then employed logistic regression on these embeddings to classify the presence or absence of pCR. Secondly, we fine-tuned the Generative Pre-trained Transformer-2 (GPT-2) model by attaching a simple feed-forward neural network (FFNN) layer to improve the detection performance of pCR from pathology reports.ResultsIn a cohort of 351 female breast cancer patients who underwent neoadjuvant chemotherapy (NAC) and subsequent surgery between 2010 and 2017 in Calgary, the optimized method displayed a sensitivity of 95.3% (95%CI: 84.0–100.0%), a positive predictive value of 90.9% (95%CI: 76.5–100.0%), and an F1 score of 93.0% (95%CI: 83.7–100.0%). The results, achieved through diverse LLM integration, surpassed traditional machine learning models, underscoring the potential of LLMs in clinical pathology information extraction.ConclusionsThe study successfully demonstrates the efficacy of LLMs in interpreting and processing digital pathology data, particularly for determining pCR in breast cancer patients post-NAC. The superior performance of LLM-based pipelines over traditional models highlights their significant potential in extracting and analyzing key clinical data from narrative reports. While promising, these findings highlight the need for future external validation to confirm the reliability and broader applicability of these methods.

A 1000FPS@360,000pixels mixed-signal sensing with computing macro featuring analog compression and maximum parallelism for objective detection tasks

Ubiquitous vision sensors continuously collect data from the environment, imposing a significant processing burden. Especially, high-frame-rate convolution processing for large-scale pixel arrays presents substantial challenges. This paper proposes a mixed-signal sensing with computing macro featuring analog compression and maximum parallelism for high-frame-rate object detection tasks. A customized neural network (NN) architecture is designed specifically for object detection. Key circuitry modules, including signal readout, dark current offset, normalization, first-layer convolution, and pooling, are implemented in the analog domain, leading to high-quality information extraction and substantial data compression. Maximum parallelism in the first-layer convolution is realized in our proposed processing flow, ensuring high-frame-rate convolution for large-scale pixel arrays. Additionally, the subsequent NN layers are efficiently deployed on SRAM-based in-memory digital computing. The macro was designed using a 55-nm foundry process. Results show that our macro can achieve 1000 fps@360,000 pixels with a 63.8 TOPS throughput under 4/8-bit hybrid precision, outperforming the state-of-the-art in frame rate by 2–10 times.

Industrial robot energy consumption model identification: A coupling model-driven and data-driven paradigm

Due to wide distribution and low energy efficiency, the energy-saving in industrial robots (IRs) is attracting extensive attention. Accurate energy consumption (EC) models of IRs lay the foundation for energy-saving. However, most dynamic and electrical parameters of IRs are not disclosed by manufacturers, which leads to the invalidity of most model-based EC prediction methods. To bridge this gap, a mechanism-data hybrid-driven method is proposed to predict the EC of IRs in this paper. First, a joint torque prediction model integrating a hybrid-driven parameter identification is developed based on deep reinforcement learning (DRL). The framework for DRL-based parameter identification is constructed through tailored design of interfaces and training mechanisms, wherein the DRL agent can learn to identify the dynamic parameters from the trajectory database. And a deep neural network based on long short-term memory (LSTM) is proposed to predict the EC of IRs according to the joint torques and velocities. The nonlinear item, which is not modeled in the robot dynamic equation, are also encapsulated in the deep neural network with one-dimensional convolutional neural network (1D-CNN) layers to improve the prediction accuracy. To validate the accuracy and efficacy of the proposed method, experiments are conducted on a KUKA KR60-3 industrial robot with different loads. The results demonstrate that the proposed method can predict EC with a mean absolute percentage error of less than 2% under a fixed load and less than 3% under loads not used for agent training.

Improved JPEG Lossless Compression for Compression of Intermediate Layers in Neural Networks Based on Compute-In-Memory

With the development of Convolutional Neural Networks (CNNs), there is a growing requirement for their deployment on edge devices. At the same time, Compute-In-Memory (CIM) technology has gained significant attention in edge CNN applications due to its ability to minimize data movement between memory and computing units. However, the deployment of complex deep neural network models on edge devices with restricted hardware resources continues to be challenged by a lack of adequate storage for intermediate layer data. In this article, we propose an optimized JPEG Lossless Compression (JPEG-LS) algorithm that implements serial context parameter updating alongside parallel encoding. This method is designed for the global prediction and efficient compression of intermediate data layers in neural networks employing CIM techniques. The results indicate average compression ratios of 6.44× for VGG16, 3.62× for ResNet34, 1.67× for MobileNetV2, and 2.31× for InceptionV3. Moreover, the implementation achieves a data throughput of 32 bits per cycle at 600 MHz on the TSMC 28 nm, with a hardware cost of 122 K Gate Count.

Optimal layer selection for latent data augmentation

While data augmentation (DA) is generally applied to input data, several studies have reported that applying DA to hidden layers in neural networks, i.e., feature augmentation, can improve performance. However, in previous studies, the layers to which DA is applied have not been carefully considered, often being applied randomly and uniformly or only to a specific layer, leaving room for arbitrariness. Thus, in this study, we investigated the trends of suitable layers for applying DA in various experimental configurations, e.g., training from scratch, transfer learning, various dataset settings, and different models. In addition, to adjust the suitable layers for DA automatically, we propose the adaptive layer selection (AdaLASE) method, which updates the ratio to perform DA for each layer based on the gradient descent method during training. The experimental results obtained on several image classification datasets indicate that the proposed AdaLASE method altered the ratio as expected and achieved high overall test accuracy.

Evaluating the Impact of Convolutional Neural Network Layer Depth on the Enhancement of Inertial Navigation System Solutions

Secure navigation is pivotal for several applications including autonomous vehicles, robotics, and aviation. The inertial navigation system estimates position, velocity, and attitude through dead reckoning especially when external references like GPS are unavailable. However, the three accelerometers and three gyroscopes that compose the system are exposed to various types of errors including bias errors, scale factor errors, and noise, which can significantly degrade the accuracy of navigation constituting also a key vulnerability of this system. This work aims to adopt a supervised convolutional neural network (ConvNet) to address this vulnerability inherent in inertial navigation systems. In addition to this, this paper evaluates the impact of the ConvNet layer's depth on the accuracy of these corrections. This evaluation aims to determine the optimal layer configuration maximizing the effectiveness of error correction in INS (Inertial Navigation System) leading to precise navigation solutions.

Creep Lifetime Prediction of Alloy 617 using Black Box Machine Learning Approach

Abstract This study explored the application of black box machine learning (ML) to build high throughput models that predict the creep response of Alloy 617. Black Box ML refers to highly complex machine learning algorithms that generate outputs from inputs without an interpretable internal process. The extended qualification process involving chemistry-processing-structure-performance assessment impeded the rapid implementation of suitable heat of alloys into targeted service. There is a critical need to shorten the development-to-deployment timeline for heats of an alloy at specific applications. To that end, black box ML is employed to expedite the HEAT qualification of Alloy 617 and predict performance from chemistry out to an unprecedented timescale. This study gathers creep data for Ni-based alloy 617 – a solid solution strengthened material from a wide range of data sources. Service conditions, mechanical properties, chemistry, and chemical ratios are provided as potential input features. The Pearson Correlation Coefficient with a 95% confidence bound is employed for input feature screening. In the ML algorithm, the selected input features are regarded as predictors, and rupture is regarded as the response. The 3-layered Neural Network (NN) was observed to be the best algorithm for the given data based on statistical rationale. The NN accurately predicts rupture across a range of isotherms and data sources. Post-audit validation confirms the applicability of NN algorithms to unseen data. The black box ML provides a pathway to predict the performance directly from chemistry and opens avenues to rapid heat qualification.

Q-Deformed and delta-parametrized A-generalized logistic function induced Banach space valued multivariate multi layer neural network approximations

Abstract. Here we research the multivariate quantitative approximation of Banach space valued continuous multivariate functions on a box or RN, N ∈ N, by the multivariate normalized, quasi-interpolation, Kantorovich type and quadrature type neural network operators. We investigate also the case of approximation by iterated multilayer neural network operators of the last four types. These approximations are achieved by establishing multidimensional Jackson type inequalities involving the multivariate modulus of continuity of the engaged function or its partial derivatives. Our multivariate operators are defined by using a multidimensional density function induced by a q-deformed and λ-parametrized A-generalized logistic function, which is a sigmoid function. The approximations are pointwise and uniform. The related feed-forward neural network are with one or multi hidden layers. Mathematics Subject Classification (2010): 41A17, 41A25, 41A30, 41A36. Keywords: Multi-layer approximation, q-deformed and λ-parametrized A- generalized logistic function, multivariate neural network approximation, quasi-interpolation operator, Kantorovich type operator, quadrature type operator, multivariate modulus of continuity, iterated approximation.

Development and validation of machine learning models for predicting cancer-related fatigue in lymphoma survivors

BackgroundNew cases of lymphoma are rising, and the symptom burden, like cancer-related fatigue (CRF), severely impacts the quality of life of lymphoma survivors. However, clinical diagnosis and treatment of CRF are inadequate and require enhancement. ObjectiveThe main objective of this study is to construct machine learning-based CRF prediction models for lymphoma survivors to help healthcare professionals accurately identify the CRF population and better personalize treatment and care for patients. MethodsA cross-sectional study in China recruited lymphoma patients from June 2023 to March 2024, dividing them into two datasets for model construction and external validation. Six machine learning algorithms were used in this study: Logistic Regression (LR), Random Forest, Single Hidden Layer Neural Network, Support Vector Machine, eXtreme Gradient Boosting, and Light Gradient Boosting Machine (LightGBM). Performance metrics like the area under the receiver operating characteristic (AUROC) and calibration curves were compared. The clinical applicability was assessed by decision curve, and Shapley additive explanations was employed to explain variable significance. ResultsCRF incidence was 40.7 % (dataset I) and 44.8 % (dataset II). LightGBM showed strong performance in training and internal validation. LR excelled in external validation with the highest AUROC and best calibration. Pain, total protein, physical function, and sleep disturbance were important predictors of CRF. ConclusionThe study presents a machine learning-based CRF prediction model for lymphoma patients, offering dynamic, data-driven assessments that could enhance the development of automated CRF screening tools for personalized management in clinical practice.