Neural Net Model Research Articles

PIDNODEs: Neural ordinary differential equations inspired by a proportional–integral–derivative controller

Neural Ordinary Differential Equations (NODEs) are a novel family of infinite-depth neural-net models through solving ODEs and their adjoint equations. In this paper, we present a strategy to enhance the training and inference of NODEs by integrating a Proportional–Integral–Derivative (PID) controller into the framework of Heavy Ball NODE, resulting in the proposed PIDNODEs and its generalized version, GPIDNODEs. By leveraging the advantages of control, PIDNODEs and GPIDNODEs can address the stiff ODE challenges by adjusting the parameters (i.e., Kp, Ki and Kd) in the PID module. The experiments confirm the superiority of PIDNODEs/GPIDNODEs over other NODE baselines on different computer vision and pattern recognition tasks, including image classification, point cloud separation and learning long-term dependencies from irregular time-series data for a physical dynamic system. These experiments demonstrate that the proposed models have higher accuracy and fewer function evaluations while alleviating the dilemma of exploding and vanishing gradients, particularly when learning long-term dependencies from a large amount of data.

Deep convolutional neural network object net model based cognitive digital twin for trust in human–robot collaborative manufacturing

Deep convolutional neural network object net model based cognitive digital twin for trust in human–robot collaborative manufacturing

Exploring NWS Forecasters’ Assessment of AI Guidance Trustworthiness

Abstract As artificial intelligence (AI) methods are increasingly used to develop new guidance intended for operational use by forecasters, it is critical to evaluate whether forecasters deem the guidance trustworthy. Past trust-related AI research suggests that certain attributes (e.g., understanding how the AI was trained, interactivity, and performance) contribute to users perceiving the AI as trustworthy. However, little research has been done to examine the role of these and other attributes for weather forecasters. In this study, we conducted 16 online interviews with National Weather Service (NWS) forecasters to examine (i) how they make guidance use decisions and (ii) how the AI model technique used, training, input variables, performance, and developers as well as interacting with the model output influenced their assessments of trustworthiness of new guidance. The interviews pertained to either a random forest model predicting the probability of severe hail or a 2D convolutional neural network model predicting the probability of storm mode. When taken as a whole, our findings illustrate how forecasters’ assessment of AI guidance trustworthiness is a process that occurs over time rather than automatically or at first introduction. We recommend developers center end users when creating new AI guidance tools, making end users integral to their thinking and efforts. This approach is essential for the development of useful and used tools. The details of these findings can help AI developers understand how forecasters perceive AI guidance and inform AI development and refinement efforts. Significance Statement We used a mixed-methods quantitative and qualitative approach to understand how National Weather Service (NWS) forecasters 1) make guidance use decisions within their operational forecasting process and 2) assess the trustworthiness of prototype guidance developed using artificial intelligence (AI). When taken as a whole, our findings illustrate that forecasters’ assessment of AI guidance trustworthiness is a process that occurs over time rather than automatically and suggest that developers must center the end user when creating new AI guidance tools to ensure that the developed tools are useful and used.

Neural waves and computation in a neural net model II: Data-like structures and the dynamics of episodic memory.

The computational resources of a neuromorphic network model introduced earlier were investigated in the first paper of this series. It was argued that a form of ubiquitous spontaneous local convolution enabled logical gate-like neural motifs to form into hierarchical feed-forward structures of the Hubel-Wiesel type. Here we investigate concomitant data-like structures and their dynamic rôle in memory formation, retrieval, and replay. The mechanisms give rise to the need for general inhibitory sculpting, and the simulation of the replay of episodic memories, well known in humans and recently observed in rats. Other consequences include explanations of such findings as the directional flows of neural waves in memory formation and retrieval, visual anomalies and memory deficits in schizophrenia, and the operation of GABA agonist drugs in suppressing episodic memories. We put forward the hypothesis that all neural logical operations and feature extractions are of the convolutional hierarchical type described here and in the earlier paper, and exemplified by the Hubel-Wiesel model of the visual cortex, but that in more general cases the precise geometric layering might be obscured and so far undetected.

An Optimization Study of Circumferential Groove Casing Treatment in a High-Speed Axial Flow Compressor

In this paper, a numerical optimization study of single-groove casing treatment was conducted on a high-speed axial compressor. One of the aims is to find the optimal structure of a single groove that can improve compressor stability with minimal loss in efficiency. Another aim is to explore suitable parameters for rapidly evaluating the compressor stall margin. A design optimization platform has been constructed in this paper, which utilizes NSGA-II and a Radial Basis Function (RBF) neural net model to carry out the optimization. The stall margin of the compressor with A single groove was accurately determined by calculating its entire overall performance line. A Pareto front is obtained through optimization, and the optimal design can be selected from the Pareto front. By considering both stall margin and efficiency loss, one of the optimal designs was found to achieve a 7.49% improvement in stall margin with a 0.24% improvement in peak efficiency. Based on the database, the effect of design parameters of a single groove on compressor stability and performance is analyzed. A series of evaluation parameters of stall margin were compared to their degree of correlation with the real stall margin calculated by the entire overall performance line. As a result, tip blockage and momentum ratio can be used as efficient parameters for quickly evaluating the compressor stall margin without the need to calculate the entire performance curve of the compressor.

UNNT: A novel Utility for comparing Neural Net and Tree-based models.

The use of deep learning (DL) is steadily gaining traction in scientific challenges such as cancer research. Advances in enhanced data generation, machine learning algorithms, and compute infrastructure have led to an acceleration in the use of deep learning in various domains of cancer research such as drug response problems. In our study, we explored tree-based models to improve the accuracy of a single drug response model and demonstrate that tree-based models such as XGBoost (eXtreme Gradient Boosting) have advantages over deep learning models, such as a convolutional neural network (CNN), for single drug response problems. However, comparing models is not a trivial task. To make training and comparing CNNs and XGBoost more accessible to users, we developed an open-source library called UNNT (A novel Utility for comparing Neural Net and Tree-based models). The case studies, in this manuscript, focus on cancer drug response datasets however the application can be used on datasets from other domains, such as chemistry.

Seasonally adjusted laboratory reference intervals to improve the performance of machine learning models for classification of cardiovascular diseases

BackgroundVariation in laboratory healthcare data due to seasonal changes is a widely accepted phenomenon. Seasonal variation is generally not systematically accounted for in healthcare settings. This study applies a newly developed adjustment method for seasonal variation to analyze the effect seasonality has on machine learning model classification of diagnoses.MethodsMachine learning methods were trained and tested on ~ 22 million unique records from ~ 575,000 unique patients admitted to Danish hospitals. Four machine learning models (adaBoost, decision tree, neural net, and random forest) classifying 35 diseases of the circulatory system (ICD-10 diagnosis codes, chapter IX) were run before and after seasonal adjustment of 23 laboratory reference intervals (RIs). The effect of the adjustment was benchmarked via its contribution to machine learning models trained using hyperparameter optimization and assessed quantitatively using performance metrics (AUROC and AUPRC).ResultsSeasonally adjusted RIs significantly improved cardiovascular disease classification in 24 of the 35 tested cases when using neural net models. Features with the highest average feature importance (via SHAP explainability) across all disease models were sex, C- reactive protein, and estimated glomerular filtration. Classification of diseases of the vessels, such as thrombotic diseases and other atherosclerotic diseases consistently improved after seasonal adjustment.ConclusionsAs data volumes increase and data-driven methods are becoming more advanced, it is essential to improve data quality at the pre-processing level. This study presents a method that makes it feasible to introduce seasonally adjusted RIs into the clinical research space in any disease domain. Seasonally adjusted RIs generally improve diagnoses classification and thus, ought to be considered and adjusted for in clinical decision support methods.

Neural waves and computation in a neural net model I: Convolutional hierarchies.

The computational resources of a neuromorphic network model introduced earlier are investigated in the context of such hierarchical systems as the mammalian visual cortex. It is argued that a form of ubiquitous spontaneous local convolution, driven by spontaneously arising wave-like activity-which itself promotes local Hebbian modulation-enables logical gate-like neural motifs to form into hierarchical feed-forward structures of the Hubel-Wiesel type. Extra-synaptic effects are shown to play a significant rôle in these processes. The type of logic that emerges is not Boolean, confirming and extending earlier findings on the logic of schizophrenia.

Convolutional Neural Networks for Determining the Ion Beam Impact on a Space Debris Object

Introduction. Space debris is a serious problem that significantly complicates space activity. This problem can be mitigated by active space debris removal. The ion beam shepherd (IBS) concept assumes the contactless removal of a space debris object (SDO) by the plume of an ion thruster (IT). Techniques for determining the force impact from the IT to the SDO are of crucial importance for implementing the IBS concept.Problem Statement. A launcher’s upper stage, approximated by a cylinder, is considered an SDO deorbited by the plume of the IT. The SDO can change its orientation and position relative to the shepherd satellite. The shepherd satellite shall be able to determine the force transmitted to the SDO by the IT, using only SDO’s images as the input information.Purpose. The study aims to develop a neural net model that can map an SDO image to the force transmitted by an IT plume to this object and estimate the accuracy of such models.Material and Methods. Plasma physics methods are used to obtain ground truth values of the ion beam force. The deep learning methodology is applied to create neural net models.Results. Three different approaches for end-to-end ion force determination have been investigated. The first model uses a single convolutional neural net (CNN). The second model is an ensemble network consisting of four sub-models, and a classifier is used to pick the correct sub-model. The last model is similar to the first one but is trained on all images used for the second model. After training, all three models’ accuracy and computational complexity are estimated. These estimates demonstrate the acceptable performance of CNN-based models.Conclusions. This paper demonstrates that CNNs can be used to determine the force impact without knowledge about the SDO position and orientation and significantly faster than the previous methods.

Evaluation of machine learning method in genomic selection for growth traits of Pacific white shrimp

The Pacific white shrimp is one of the most important species in the aquaculture industry worldwide, and the growth is regarded as primary trait for selective breeding programmes. In this study, the heritability and genetic correlation of two growth traits, including body length (BL) and the ratio of abdomen length to cephalothorax length (AL/CL) were analyzed, and the genomic prediction based on different genomic selection models including machine learning method were evaluated. The heritabilities of BL and AL/CL were 0.25 ± 0.04 and 0.07 ± 0.03, respectively. The two phenotypes showed moderate negative correlations (−0.70 ± 0.14). Comparison of the different prediction models showed that NeuralNet had the highest prediction accuracy. The prediction accuracy of NeuralNet increased by about 10% compared to GBLUP. Furthermore, NeuralNet presented the highest prediction accuracy under different marker densities, and the prediction accuracy using 1000 SNPs was similar to that estimated by total SNPs. When comparing multi-trait models (MTM) and single-trait models (STM), NeuralNet outperformed the other methods, which increased prediction accuracy by around 30%. Overall, the NeuralNet model may have better application prospects for genomic selection breeding in shrimp. These results provide a strong basis for accelerating the application of genomic selection breeding in shrimp improvement programmes.

External Validation of Models for Predicting Disability in Community-Dwelling Older People in the Netherlands: A Comparative Study.

Advanced statistical modeling techniques may help predict health outcomes. However, it is not the case that these modeling techniques always outperform traditional techniques such as regression techniques. In this study, external validation was carried out for five modeling strategies for the prediction of the disability of community-dwelling older people in the Netherlands. We analyzed data from five studies consisting of community-dwelling older people in the Netherlands. For the prediction of the total disability score as measured with the Groningen Activity Restriction Scale (GARS), we used fourteen predictors as measured with the Tilburg Frailty Indicator (TFI). Both the TFI and the GARS are self-report questionnaires. For the modeling, five statistical modeling techniques were evaluated: general linear model (GLM), support vector machine (SVM), neural net (NN), recursive partitioning (RP), and random forest (RF). Each model was developed on one of the five data sets and then applied to each of the four remaining data sets. We assessed the performance of the models with calibration characteristics, the correlation coefficient, and the root of the mean squared error. The models GLM, SVM, RP, and RF showed satisfactory performance characteristics when validated on the validation data sets. All models showed poor performance characteristics for the deviating data set both for development and validation due to the deviating baseline characteristics compared to those of the other data sets. The performance of four models (GLM, SVM, RP, RF) on the development data sets was satisfactory. This was also the case for the validation data sets, except when these models were developed on the deviating data set. The NN models showed a much worse performance on the validation data sets than on the development data sets.

Similarity Regression Of Functions In Different Compiled Forms With Neural Attentions On Dual Control-Flow Graphs

Abstract Detecting if two functions in different compiled forms are similar has a wide range of applications in software security. We present a method that leverages both semantic and structural features of functions, learned by a neural-net model on the underlying control-flow graphs (CFGs). In particular, we devise a neural function-similarity regressor (NFSR) with attentions on dual CFGs. We train and evaluate NFSR on a dataset consisting of nearly 4 million functions from over 14 900 binary files. Experiments show that NFSR is superior to the SOTA models of SAFE, Gemini and GMN, especially for binary functions with large CFGs. An ablation study shows that attention on dual CFGs plays a significant role in detecting function similarities.

A hybrid deep neural net learning model for predicting Coronary Heart Disease using Randomized Search Cross-Validation Optimization

Coronary Heart Disease (CHD) is a life-threatening public health problem. Many chronic CHDs and health risks can be avoided, reversed, and reduced with proper risk assessment. Medical professionals find it challenging to anticipate heart attacks and heart failures since it is a complex process requiring knowledge, experience, and medical resource facilitation. Although healthcare is generally information-savvy, not all available data are analysed to find hidden patterns and make informed and timely decisions since heart disease prediction relies heavily on clinical data processing. This study proposes a hybrid deep neural net learning model for predicting CHD using the BRFSS-2015 Dataset. The best features subset is chosen based on the co-relation score and dataset classes are balanced using the cluster-abundant data class approach. Bi-direction Long Short-Term Memory (BiLSTM) and Gated Recurrent Unit (GRU) hyper-parameter tuning is accomplished using Randomized Search Cross-Validation Optimization (RSCV). In comparison to GRU, LSTM, and BiLSTM-GRU, this suggested model obtains a classification accuracy of 98.28% which outperforms existing models.

Integrated Disaster Risk Management for Flood Detection on Remote Sensing Images using Deep Learning techniques

<p>Floods are one of the leading causes of damage, prompting mortality and substantial destruction to the structure and total economy of the affected nations. Remote sensing, satellite imagery, global positioning system, and geographic information system (GIS) are widely employed for flood identification to examine flood-related losses. Recently, accurate and automated flood detection models using remote sensing images have become effective for flood disaster management, risk manager, infrastructure planning, disaster rescue management, etc. Computer vision and deep learning (DL) models provide prompt and rapid flood detection in remote sensing images. In this aspect, this paper presents a multiverse optimization with a deep transfer learning-enabled flood detection (MVODTL-FD) technique for disaster risk management. In the proposed MVODTL-FD technique, remote sensing images are investigated for the effectual detection of floods. To accomplish this, the presented MVODTL-FD technique applies a guided normal filter (GNF) based image preprocessing approach to eliminate the noise. In addition, the proposed MVODTL-FD technique uses a deep convolutional neural network-based Squeeze Net model for feature extraction, and the hyperparameter process is performed using the MVO algorithm. At last, the flood detection process is performed using support vector machine (SVM) classification. For establishing the improved version of the MVODTL-FD method, a wide-ranging experimental analysis is performed. The MVODTL-FD model is rated higher in the comparative analysis than other DL models.</p>

Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations

Evidence-based medicine, the practice in which healthcare professionals refer to the best available evidence when making decisions, forms the foundation of modern healthcare. However, it relies on labour-intensive systematic reviews, where domain specialists must aggregate and extract information from thousands of publications, primarily of randomised controlled trial (RCT) results, into evidence tables. This paper investigates automating evidence table generation by decomposing the problem across two language processing tasks: named entity recognition, which identifies key entities within text, such as drug names, and relation extraction, which maps their relationships for separating them into ordered tuples. We focus on the automatic tabulation of sentences from published RCT abstracts that report the results of the study outcomes. Two deep neural net models were developed as part of a joint extraction pipeline, using the principles of transfer learning and transformer-based language representations. To train and test these models, a new gold-standard corpus was developed, comprising over 550 result sentences from six disease areas. This approach demonstrated significant advantages, with our system performing well across multiple natural language processing tasks and disease areas, as well as in generalising to disease domains unseen during training. Furthermore, we show these results were achievable through training our models on as few as 170 example sentences. The final system is a proof of concept that the generation of evidence tables can be semi-automated, representing a step towards fully automating systematic reviews.

Predicting risk of suicidal behaviour after initiation of selective serotonin reuptake inhibitors in children, adolescents and young adults: protocol for development and validation of clinical prediction models

IntroductionThere is concern regarding suicidal behaviour risk during selective serotonin reuptake inhibitor (SSRI) treatment among the young. A clinically useful model for predicting suicidal behaviour risk should have high predictive...

Hierarchical distinctions in the production and perception of nuclear tunes in American English

In Autosegmental-Metrical models of intonational phonology, different types of pitch accents, phrase accents, and boundary tones concatenate to create a set of phonologically distinct phrase-final nuclear tunes. This study asks if an eight-way distinction in nuclear tune shape in American English, predicted from the combination of two (monotonal) pitch accents, two phrase accents, and two boundary tones, is evident in speech production and in speech perception. F0 trajectories from a large-scale imitative speech production experiment were analyzed using bottom-up(k-means) clustering, neural net classification, GAMM modeling, and modeling of turning point alignment. Listeners’ perception of the same tunes is tested in a perceptual discrimination task and related to the imitation results. Emergent grouping of tunes in the clustering analysis, and related classification accuracy from the neural net, show a merging of some of the predicted distinctions among tunes whereby tune shapes that vary primarily in the scaling of final f0 are not reliably distinguished. Within five emergent clusters, subtler distinctions among tunes are evident in GAMMs and f0 turning point modeling. Clustering of individual participants’ production data shows a range of partitions of the data, with nearly all participants making a primary distinction between a class of High-Rising and Non-High-Rising tunes, and with up to four secondary distinctions among the non-Rising class. Perception results show a similar pattern, with poor pairwise discrimination for tunes that differ primarily, but by a small degree, in final f0, and highly accurate discrimination when just one member of a pair is in the High-Rising tune class. Together, the results suggest a hierarchy of distinctiveness among nuclear tunes, with a robust distinction based on holistic tune shape and poorly differentiated distinctions between tunes with the same holistic shape but small differences in final f0. The observed distinctions from clustering, classification, and perception analyses align with the tonal specification of a binary pitch accent contrast {H*, L*} and a maximally ternary {H%, M%, L%} boundary tone contrast; the findings do not support distinct tonal specifications for the phrase accent and boundary tone from the AM model. 

Attention-guided multi-scale context aggregation network for multi-modal brain glioma segmentation.

Accurate segmentation of brain glioma is a critical prerequisite for clinical diagnosis, surgical planning and treatment evaluation. In current clinical workflow, physicians typically perform delineation of brain tumor subregions slice-by-slice, which is more susceptible to variabilities in raters and also time-consuming. Besides, even though convolutional neural networks (CNNs) are driving progress, the performance of standard models still have some room for furtherimprovement. To deal with these issues, this paper proposes an attention-guided multi-scale context aggregation network (AMCA-Net) for the accurate segmentation of brain glioma in the magnetic resonance imaging (MRI) images withmulti-modalities. AMCA-Net extracts the multi-scale features from the MRI images and fuses the extracted discriminative features via a self-attention mechanism for brain glioma segmentation. The extraction is performed via a series of down-sampling, convolution layers, and the global context information guidance (GCIG) modules are developed to fuse the features extracted for contextual features. At the end of the down-sampling, a multi-scale fusion (MSF) module is designed to exploit and combine all the extracted multi-scale features. Each of the GCIG and MSF modules contain a channel attention (CA) module that can adaptively calibrate feature responses and emphasize the most relevant features. Finally, multiple predictions with different resolutions are fused through different weightings given by a multi-resolution adaptation (MRA) module instead of the use of averaging or max-pooling to improve the final segmentationresults. Datasets used in this paper are publicly accessible, that is, the Multimodal Brain Tumor Segmentation Challenges 2018 (BraTS2018) and 2019 (BraTS2019). BraTS2018 contains 285 patient cases and BraTS2019 contains 335 cases. Simulations show that the AMCA-Net has better or comparable performance against that of the other state-of-the-art models. In terms of the Dice score and Hausdorff 95 for the BraTS2018 dataset, 90.4% and 10.2 mm for the whole tumor region (WT), 83.9% and 7.4 mm for the tumor core region (TC), 80.2% and 4.3 mm for the enhancing tumor region (ET), whereas the Dice score and Hausdorff 95 for the BraTS2019 dataset, 91.0% and 10.7 mm for the WT, 84.2% and 8.4 mm for the TC, 80.1% and 4.8 mm for the ET. The proposed AMCA-Net performs comparably well in comparison to several state-of-the-art neural net models in identifying the areas involving the peritumoral edema, enhancing tumor, and necrotic and non-enhancing tumor core of brain glioma, which has great potential for clinical practice. In future research, we will further explore the feasibility of applying AMCA-Net to other similar segmentationtasks.

Real‐Time Optimization Using Machine Learning Models Applied to the 4,4′‐Diphenylmethane Diisocyanate Production Process

AbstractIn this work, the optimal time‐varying allocation of steam in a large‐scale industrial isocyanate production process is addressed. This is a problem that falls into the category of real‐time optimization (RTO). The application of RTO in practice faces two problems: First the available rigorous process models may not be suitable for use in real‐time connected to the process. Second, there is always a mismatch between the predictions of the model and the behavior of the real plant. We address the first problem by training a neural net model as a surrogate to data generated by a rigorous simulation model so that the model is simple to implement and short execution times result. The second problem is tackled by adapting the optimization problem based on measured data such that convergence to the optimal operating conditions for the real plant is achieved.

Short-term Load Prediction of Electric Vehicle Charging Stations built on GA-BPNN Model

In response to the rapid growth of electric vehicles (EVs), the expanding charging demand significantly impacts the distribution network, and load prediction for EV charging stations is required. We combine the genetic algorithm and BPNN to achieve a more accurate and efficient prediction effect for electric vehicle charging stations. It adopts GA to optimize the weights and thresholds of BPNN to obtain the optimal solution of the BP neural net prediction model, thus establishing a BPNN prediction model built on GA. We apply this model to the load prediction of actual EV charging stations. The MATLAB simulation study shows that the prediction model optimized by GA acquires higher accuracy and better practicability than the unoptimized prediction model.