Deep Neural Network Prediction Research Articles

CLSSATP: Contrastive learning and self-supervised learning model for aquatic toxicity prediction.

As compound concentrations in aquatic environments increase, the habitat degradation of aquatic organisms underscores the growing importance of studying the impact of chemicals on diverse aquatic populations. Understanding the potential impacts of different chemical substances on different species is a necessary requirement for protecting the environment and ensuring sustainable human development. In this regard, deep learning methods offer significant advantages over traditional experimental approaches in terms of cost, accuracy, and generalization ability. This research introduces CLSSATP, an efficient contrastive self-supervised learning deep neural network prediction model for organic toxicity. The model integrates two modules, a self-supervised learning module using molecular fingerprints for representation, and a contrastive learning module utilizing molecular graphs. Through dual-perspective learning, the model gains clear insights into the structural and property relationships of molecules. The experiment results indicate that our model outperforms comparative methods, demonstrating the effectiveness of our proposed architecture. Moreover, ablation experiments show that the self-supervised module and contrastive learning module respectively provide average performance improvements of 9.43 % and 10.98 % to CLSSATP. Furthermore, by visualizing the representations of our model, we observe that it correctly identifies the substructures that determine the molecular properties, granting itself with interpretability. In conclusion, CLSSATP offers a novel and effective perspective for future research in aquatic toxicity assessment. All of codes and datasets are freely available online at https://github.com/zhaoqi106/CLSSATP.

A Typical Infrared Background Radiation Prediction Model Based on RF-VMD and Optimized Hybrid Neural Network

ABSTRACT The short-term prediction and adjustment of a target’s infrared radiation hold significant value in military camouflage applications. Existing radiation prediction models generally require real-time environmental and meteorological data support, resulting in lag in active camouflage. To meet the demand for active camouflage of background infrared (IR) radiation, a short-term background IR radiation prediction method based on historical data is proposed. First, a random forest (RF) is used to filter the collected multidimensional meteorological parameters. Variational mode decomposition (VMD) is applied for time-frequency analysis on these parameters, optimizing them with Bayesian algorithms and decomposing them into multivariate intrinsic mode functions (IMFs) with similar frequencies to reduce the impact of nonlinearity in the data. Based on the superimposed IMFs as inputs, a hybrid deep neural network prediction model is established. The model optimizes the CNN-LSTM network with residual connections and introduces a multi-head self-attention mechanism to enhance spatiotemporal feature extraction of the multidimensional meteorological parameters, focusing on key temporal feature regions. According to the experimental results, the constructed model demonstrates high prediction accuracy and adaptability across different background environments, with a low parameter count and fast prediction capability, meeting the practical application needs for various complex backgrounds.

Uncertainty-aware genomic deep learning with knowledge distillation.

Deep neural networks (DNNs) have advanced predictive modeling for regulatory genomics, but challenges remain in ensuring the reliability of their predictions and understanding the key factors behind their decision making. Here we introduce DEGU (Distilling Ensembles for Genomic Uncertainty-aware models), a method that integrates ensemble learning and knowledge distillation to improve the robustness and explainability of DNN predictions. DEGU distills the predictions of an ensemble of DNNs into a single model, capturing both the average of the ensemble's predictions and the variability across them, with the latter representing epistemic (or model-based) uncertainty. DEGU also includes an optional auxiliary task to estimate aleatoric, or data-based, uncertainty by modeling variability across experimental replicates. By applying DEGU across various functional genomic prediction tasks, we demonstrate that DEGU-trained models inherit the performance benefits of ensembles in a single model, with improved generalization to out-of-distribution sequences and more consistent explanations of cis-regulatory mechanisms through attribution analysis. Moreover, DEGU-trained models provide calibrated uncertainty estimates, with conformal prediction offering coverage guarantees under minimal assumptions. Overall, DEGU paves the way for robust and trustworthy applications of deep learning in genomics research.

Repairs and Breaks Prediction for Deep Neural Networks

With the increasing prevalence of software incorporating deep neural networks (DNNs), quality assurance for these software systems has become a crucial concern. To this end, various methods have been proposed to repair the misbehavior of DNNs by modifying their weights. However, these repair methods may not meet the developer's needs for a given dataset and model. In this study, we build prediction models for repair outcomes (i.e., repairs and breaks) to help determine whether the repair method is likely to work. By using our prediction models, developers and operators of DNNs can decide whether or not to apply a repair method, and if so, which method to use. Our prediction models utilize four metrics as explanatory metrics that represent the confidence or ambiguity in the DNN predictions. We experimented with four repair methods and 10 datasets. The experimental results demonstrate that our prediction models successfully select a repair method that meets developers’ needs in 16 out of 24 cases, resulting in an average time saving of 16.29% compared to the naive method. Based on these results, our prediction models can reduce costs for developers and operators when deciding whether to employ repair methods for real-world applications of DNNs.

Automatic Hardy and Clapham’s classification of hallux sesamoid position on foot radiographs using deep neural network

BackgroundThere is currently no deep neural network (DNN) capable of automatically classifying tibial sesamoid position (TSP) on foot radiographs. MethodsA DNN was developed to predict TSP according to the Hardy and Clapham’s classification. A total of 1554 foot radiographs were used for model development. The validation of the model was conducted using radiographs obtained from 113 consecutive first-visit patients of our foot and ankle clinic. On these 113 radiographs, TSP was independently classified by three foot and ankle surgeons and the DNN, and their results were compared. The weighted kappa value of TSP between the DNN prediction and the median of the three surgeons (KAI) was calculated. ResultsThe KAI was 0.95 (95 %CI: 0.85- 1.00), indicating sufficient consistency between the surgeons and the DNN. ConclusionsWe have developed a DNN for automated TSP classification that demonstrates sufficient consistency with foot and ankle surgeons. Levels of evidenceLevel 3 - Retrospective Cohort Study.

A rapid prediction method for water droplet collection coefficients of multiple airfoils based on incremental learning and multi-modal dynamic fusion

The calculation of the water droplet collection coefficient (WDCC) is a crucial step in the numerical study of aircraft icing and the iterative design of anti-icing and deicing systems. Rapid and efficient methods for predicting WDCC are essential for enhancing the efficiency of icing numerical calculations and accelerating the design cycle of these systems. The existing prediction methods are inefficient and fail to meet the real-time requirements of engineering applications. This paper proposes a rapid prediction method for the WDCC for multiple airfoils utilizing deep learning techniques. The method takes enhanced airfoil section images and icing condition parameters as inputs and WDCC as output. A deep neural network prediction model, IncDynamicFusion, for sustainable learning is established by integrating a multimodal dynamic fusion method with an improved iCaRL method (incremental classifier and representation learning). Numerical experimental results demonstrate that the proposed model can quickly and effectively predict the WDCC of multiple airfoils.

Interpreting cis-regulatory interactions from large-scale deep neural networks.

The rise of large-scale, sequence-based deep neural networks (DNNs) for predicting gene expression has introduced challenges in their evaluation and interpretation. Current evaluations align DNN predictions with orthogonal experimental data, providing insights into generalization but offering limited insights into their decision-making process. Existing model explainability tools focus mainly on motif analysis, which becomes complex when interpreting longer sequences. Here we present cis-regulatory element model explanations (CREME), an in silico perturbation toolkit that interprets the rules of gene regulation learned by a genomic DNN. Applying CREME to Enformer, a state-of-the-art DNN, we identify cis-regulatory elements that enhance or silence gene expression and characterize their complex interactions. CREME can provide interpretations across multiple scales of genomic organization, from cis-regulatory elements to fine-mapped functional sequence elements within them, offering high-resolution insights into the regulatory architecture of the genome. CREME provides a powerful toolkit for translating the predictions of genomic DNNs into mechanistic insights of gene regulation.

Partial density of states representation for accurate deep neural network predictions of X-ray spectra.

The performance of a machine learning (ML) algorithm for chemistry is highly contingent upon the architect's choice of input representation. This work introduces the partial density of states (p-DOS) descriptor: a novel, quantum-inspired structural representation which encodes relevant electronic information for machine learning models seeking to simulate X-ray spectroscopy. p-DOS uses a minimal basis set in conjunction with a guess (non-optimised) electronic configuration to extract and then discretise the density of states (DOS) of the absorbing atom to form the input vector. We demonstrate that while the electronically-focused p-DOS performs well in isolation, optimal performance is achieved when supplemented with nuclear structural information imparted via a geometric representation. p-DOS provides a description of the key electronic properties of a system which is not only concise and computationally efficient, but also independent of molecular size or choice of basis set. It can be rapidly generated, facilitating its application with large training sets. Its performance is demonstrated using a wide variety of examples at the sulphur K-edge, including the prediction of ultrafast X-ray spectroscopic signal associated with photoexcited 2(5H)-thiophenone. These results highlight the potential for ML models developed using p-DOS to contribute to the interpretation and prediction of experimental results e.g. in operando measurements of batteries and/or catalysts and femtosecond time-resolved studies, especially those made possible by emergent cutting-edge technologies, especially X-ray free electron lasers.

OCIE: Augmenting model interpretability via Deconfounded Explanation-Guided Learning

Deep neural networks (DNNs) often encounter significant challenges related to opacity, inherent biases, and shortcut learning, which undermine their practical reliability. In this study, we address these issues by constructing a causal graph to model the unbiased learning process of DNNs. This model reveals that recurrent background information in training samples acts as a confounder, leading to spurious correlations between model inputs and outputs, causing biased predictions. To mitigate these problems and promote unbiased feature learning, we propose the Object-guided Consistency Interpretation Enhancement (OCIE) methodology. OCIE enhances DNN interpretability by integrating explicit objects and explanations into the model’s learning process. Initially, OCIE employs a graph-based algorithm to identify explicit objects within self-supervised vision transformer-learned features. Subsequently, it constructs class prototypes to eliminate invalid detected objects. Finally, OCIE aligns explanations with explicit objects, directing the model’s attention towards the most distinctive classification features rather than irrelevant backgrounds. Extensive experiments on different image classification datasets, including general (ImageNet), fine-grained (Stanford Cars and CUB-200), and medical (HAM) datasets, using two prevailing network architectures, demonstrate that OCIE significantly enhances explanation consistency across all datasets. Furthermore, OCIE proves particularly advantageous for fine-grained classification, especially in few-shot scenarios, by improving both interpretability and classification performance. Additionally, our findings highlight the impact of centralized explanations on the sufficiency of model decisions, suggesting that focusing explanations on explicit objects improves the reliability of DNN predictions. Our code is available at: https://github.com/DLAIResearch/OCIE.

A deep neural network prediction method for diabetes based on Kendall's correlation coefficient and attention mechanism.

Diabetes is a chronic disease, which is characterized by abnormally high blood sugar levels. It may affect various organs and tissues, and even lead to life-threatening complications. Accurate prediction of diabetes can significantly reduce its incidence. However, the current prediction methods struggle to accurately capture the essential characteristics of nonlinear data, and the black-box nature of these methods hampers its clinical application. To address these challenges, we propose KCCAM_DNN, a diabetes prediction method that integrates Kendall's correlation coefficient and an attention mechanism within a deep neural network. In the KCCAM_DNN, Kendall's correlation coefficient is initially employed for feature selection, which effectively filters out key features influencing diabetes prediction. For missing values in the data, polynomial regression is utilized for imputation, ensuring data completeness. Subsequently, we construct a deep neural network (KCCAM_DNN) based on the self-attention mechanism, which assigns greater weight to crucial features affecting diabetes and enhances the model's predictive performance. Finally, we employ the SHAP model to analyze the impact of each feature on diabetes prediction, augmenting the model's interpretability. Experimental results show that KCCAM_DNN exhibits superior performance on both PIMA Indian and LMCH diabetes datasets, achieving test accuracies of 99.090% and 99.333%, respectively, approximately 2% higher than the best existing method. These results suggest that KCCAM_DNN is proficient in diabetes prediction, providing a foundation for informed decision-making in the diagnosis and prevention of diabetes.

Physics-driven deep neural networks for damage identification using guided wave damage interaction coefficients

In this paper, recent advancements in the development of a guided wave-based damage identification approach using wave damage interaction coefficients (WDICs) and deep neural networks (DNNs) are presented. These WDICs uniquely describe the complex scattering of guided waves around possible damages and depend on the properties of the damage itself. Hence, they are utilized as physics-based and highly sensitive damage features herein. It is demonstrated, that DNNs can effectively learn intricate relationships between damage characteristics and complex-shaped WDIC patterns from a compact sized training dataset. In this study, two training datasets are created by numerical finite element simulations and experimental scanning laser Doppler vibrometer measurements using a pseudo-damage approach. Therefore, the orientation and thickness of surface-bonded artificial damages are varied to generate the training data of 12 selected damage scenarios. The generalization capabilities of the fully trained DNNs allow to accurately predict WDICs even for damage scenarios unseen during training. The presented damage identification method leverages this powerful ability to characterize properties of unknown damages. Once trained, the accurate DNN predictions become available promptly and can be compared with measured WDICs from an unknown damage for selected sensor positions. The performance of both simulation-based and experiment-based structural health monitoring approaches are assessed, while also addressing current limitations and possible enhancements. For example, the great potential of incorporating the phase information of the complex-valued WDICs in the identification procedure is discussed. Hence, this study highlights the latest advancements in the development of the presented damage identification method with promising results and provides valuable insights in the application of WDICs as physics-based features for DNNs.

ImageDTA: A Simple Model for Drug-Target Binding Affinity Prediction.

Predicting the drug-target binding affinity (DTA) is crucial in drug discovery, and an increasing number of researchers are using artificial intelligence techniques to make such predictions. Many effective deep neural network prediction models have been proposed. However, current methods need improvement in accuracy, complexity, and efficiency. In this study, we propose a method based on a multiscale 2-dimensional convolutional neural network (CNN), namely ImageDTA. Many studies have shown that CNN achieves good learning effects with limited data. Therefore, we take a unique perspective by treating the word vector encoded with a simplified molecular input line entry system (SMILES) string as an "image" and processing it like handling images, fully leveraging the efficient processing capabilities of CNN for image data. Furthermore, we show that ImageDTA has higher training and inference efficiency than pretrained large models and outperforms attention-based graph neural network models in accuracy and interpretability. We also use visualization techniques to select appropriate convolutional kernel sizes, thereby increasing the network's interpretability.

Thermal buckling of the concrete structures reinforced by nanocomposites: Introducing advanced deep neural networks for thermal buckling problem

The thermal buckling behavior of concrete annular sector plates reinforced with nanocomposites is investigated in this work. The thermal buckling issue is stated using mathematical modeling, taking into account the geometric linearity and material heterogeneity caused by the nanocomposite reinforcement. The governing differential equations that are obtained from the theoretical model are solved using the discrete singular convolution (DSC) technique. To generate accurate buckling load predictions, the DSC method’s great accuracy in handling complicated boundary conditions and discontinuities is used. Furthermore, using mathematical modeling to create extensive datasets, the research aims to train deep neural networks (DNNs) to anticipate thermal buckling reactions. These datasets provide a strong basis for DNN training as they include a wide range of factors, such as geometric configurations, loading situations, and material properties. By combining DNN predictions with DSC solutions, this unique technique for solving this difficult issue seeks to improve the accuracy and efficiency of thermal buckling analysis. The findings show how well sophisticated mathematical methods and machine learning models work together, opening the door to creative solutions for the design and analysis of concrete structures reinforced with nanocomposite under complicated stress.

Multi-task aquatic toxicity prediction model based on multi-level features fusion

IntroductionWith the escalating menace of organic compounds in environmental pollution imperiling the survival of aquatic organisms, the investigation of organic compound toxicity across diverse aquatic species assumes paramount significance for environmental protection. Understanding how different species respond to these compounds helps assess the potential ecological impact of pollution on aquatic ecosystems as a whole. Compared with traditional experimental methods, deep learning methods have higher accuracy in predicting aquatic toxicity, faster data processing speed and better generalization ability. ObjectivesThis article presents ATFPGT-multi, an advanced multi-task deep neural network prediction model for organic toxicity. MethodsThe model integrates molecular fingerprints and molecule graphs to characterize molecules, enabling the simultaneous prediction of acute toxicity for the same organic compound across four distinct fish species. Furthermore, to validate the advantages of multi-task learning, we independently construct prediction models, named ATFPGT-single, for each fish species. We employ cross-validation in our experiments to assess the performance and generalization ability of ATFPGT-multi. ResultsThe experimental results indicate, first, that ATFPGT-multi outperforms ATFPGT-single on four fish datasets with AUC improvements of 9.8%, 4%, 4.8%, and 8.2%, respectively, demonstrating the superiority of multi-task learning over single-task learning. Furthermore, in comparison with previous algorithms, ATFPGT-multi outperforms comparative methods, emphasizing that our approach exhibits higher accuracy and reliability in predicting aquatic toxicity. Moreover, ATFPGT-multi utilizes attention scores to identify molecular fragments associated with fish toxicity in organic molecules, as demonstrated by two organic molecule examples in the main text, demonstrating the interpretability of ATFPGT-multi. ConclusionIn summary, ATFPGT-multi provides important support and reference for the further development of aquatic toxicity assessment. All of codes and datasets are freely available online at https://github.com/zhaoqi106/ATFPGT-multi.

Deep learning–based urban energy forecasting model for residential building energy efficiency

Sustainable and inventive city design is becoming more and more dependent on the use of cutting-edge technology as smart cities develop further. Energy efficiency optimization in residential structures is an essential part of the puzzle as it helps conserve resources and keeps the planet habitable. An enhanced Deep Neural Network (DNN) model for household energy efficiency predictions is presented in this research. Our model uses a large dataset of building features, weather, occupancy patterns and energy usage histories. Data is preprocessed, features are engineered and hyperparameters are tweaked to improve DNN prediction. Scalable, easy-to-understand models are essential, as are shifting urban areas and energy landscapes. In this work, the authors have evaluated the proposed model with basic model with different optimizers. Initially, the Stochastic Gradient Descent optimizer applied that gained 91.02% Recall, 93.47% Precision, 93.28% F1-Score, 0.0153 MSE, 0.0166 RMSE and 0.0165 MAE. The proposed model gained 99.52% Recall, 98.91% Precision, 99.09% F1-Score, 0.0140 MSE, 0.0137 RMSE and 0.0139 MAE. By monitoring, analyzing and making decisions in real time, smart city systems can help planners understand energy usage trends. The optimized DNN model advances smart city development by promoting sustainability and resource optimization. Predicting residential buildings’ energy efficiency provides proactive energy savings, cost reduction and environmental impact mitigation. The suggested DNN model shows how smart cities use cutting-edge urban planning to become more sustainable, efficient and resilient.

A hybrid network with DNN and WGAN for supercontinum prediction

In this paper, a Hybrid Network is reported for predicting the output pulse shape parameters of the supercontinuum in optical fibers. The network incorporates the Wasserstein generative adversarial network (WGAN) and the deep neural network (DNN) with a TCN layer. The WGAN augments a limited set of original data, while the designed DNN accurately predicts supercontinuum. The simulation results show the spectra agreement (Agm) of the predictions by individual DNN is 89% and the root mean square error (RMSE) is 0.1092. After retraining with WGAN's extended dataset, the Agm of DNN predictions is improved to 98% and the RMSE is reduced to 0.0513. The input pump power corresponding to the appearance of the fourth soliton peak can be predicted by the retrained DNN. Numerical analysis demonstrates the effectiveness and generalization ability of a Hybrid Network in predicting pulse parameters. The network fits the propagation dynamics in the spectral domain of SC, which provides a promising tool for further research in this field.

Survival Analysis Based Qos Recommendation for Bus Transportation using Deep Learning

Public transportation play an important role in metropolitan cities.Bus services are an important part of public transportation systems, which are frequently chosen because they are convenient, economical, environmentally beneficial, and less taxing on the infrastructure. The temporal patterns of bus transportation incidents are examined in this study using survival analysis, with a focus on the amount of time until certain events like delays, malfunctions, or safe arrivals at destinations. High-dimensional data is being generated at an increasing rate due to technological advancements. As a result, it is quite difficult to analyze such a dataset adequately. Algorithms for machine learning (ML) have become popular tools for modeling complex and nonlinear interactions in a variety of real-world applications, including as the analysis of high-dimensional survival data. Multilayer Deep Neural Network (DNN) models have achieved impressive results recently. Thus, using Keras and TensorFlow, a Cox-based DNN prediction survival model (DNNSurv model) was created. Its findings, however, were limited to survival datasets with large sample sizes or high dimensionality. Using high dimensional survival data, the proposed work will evaluate the DNNSurv model's prediction ability and compare it to a well-known machine learning survival prediction model (random survival forest). The Proposed Work also offers the best settings for a number of hyperparameters, including tuning parameter selection, for this reason. The suggested approach showed through data analysis that, when compared to the ML model, the DNNSurv model performed better overall in terms of the two primary survival prediction evaluation measures (i.e., the concordance index and the time-dependent AUC).

Learning Accurate Pseudo-Labels via Feature Similarity in the Presence of Label Noise

Due to the exceptional learning capabilities of deep neural networks (DNNs), they continue to struggle to handle label noise. To address this challenge, the pseudo-label approach has emerged as a preferred solution. Recent works have achieved significant improvements by exploring the information involved in DNN predictions and designing a straightforward method to incorporate model predictions into the training process by using a convex combination of original labels and predictions as the training targets. However, these methods overlook the feature-level information contained in the sample, which significantly impacts the accuracy of the pseudo label. This study introduces a straightforward yet potent technique named FPL (feature pseudo-label), which leverages information from model predictions as well as feature similarity. Additionally, we utilize an exponential moving average scheme to bolster the stability of corrected labels while upholding the stability of pseudo-labels. Extensive experiments were carried out on synthetic and real datasets across different noise types. The CIFAR10 dataset yielded the highest accuracy of 94.13% (Top1), while Clothing1M achieved 73.54%. The impressive outcomes showcased the efficacy and robustness of learning amid label noise.

SAMP-VGG16: Force-field assisted image-based deep neural network prediction model for short antimicrobial peptides.

During the last three decades, antimicrobial peptides (AMPs) have emerged as a promising therapeutic alternative to antibiotics. The approaches for designing AMPs span from experimental trial-and-error methods to synthetic hybrid peptide libraries. To overcome the exceedingly expensive and time-consuming process of designing effective AMPs, many computational and machine-learning tools for AMP prediction have been recently developed. In general, to encode the peptide sequences, featurization relies on approaches based on (a) amino acid (AA) composition, (b) physicochemical properties, (c) sequence similarity, and (d) structural properties. In this work, we present an image-based deep neural network model to predict AMPs, where we are using feature encoding based on Drude polarizable force-field atom types, which can capture the peptide properties more efficiently compared to conventional feature vectors. The proposed prediction model identifies short AMPs (≤30 AA) with promising accuracy and efficiency and can be used as a next-generation screening method for predicting new AMPs. The source code is publicly available at the Figshare server sAMP-VGG16.

Self-Checking Deep Neural Networks for Anomalies and Adversaries in Deployment

Deep Neural Networks (DNNs) have been widely adopted, yet DNN models are surprisingly unreliable, which raises significant concerns about their use in critical domains. In this work, we propose that runtime DNN mistakes can be quickly detected and properly dealt with <i>in deployment</i>, especially in settings like self-driving vehicles. Just as software engineering (SE) community has developed effective mechanisms and techniques to monitor and check programmed components, our previous work, SelfChecker, is designed to monitor and correct DNN predictions given unintended abnormal test data. SelfChecker triggers an alarm if the decisions given by the internal layer features of the model are inconsistent with the final prediction and provides <i>advice</i> in the form of an alternative prediction. In this paper, we extend SelfChecker to the security domain. Specifically, we describe SelfChecker++, which we designed to target both <i>unintended</i> abnormal test data and <i>intended</i> adversarial samples. Technically, we develop a technique which can transform any runtime inputs triggering alarms into semantically equivalent inputs, then we feed those transformed inputs to the model. Such runtime transformation nullifies any intended crafted samples, making the model immune to adversarial attacks that craft adversarial samples. We evaluated the alarm accuracy of SelfChecker++ on three DNN models and four popular image datasets, and found that SelfChecker++ triggers correct alarms on 63.10% of wrong DNN predictions, and triggers false alarms on 5.77% of correct DNN predictions. We also evaluated the effectiveness of SelfChecker++ in detecting adversarial examples and found it detects on average 70.09% of such samples with advice accuracy that is 20.89% higher than the original DNN model and 18.37% higher than SelfChecker.