Neural Network Architecture Research Articles

Enhanced ResNet50 Model for Aqueous Solubility Prediction of Drug Compounds Using Deep Learning Techniques

Background: Aqueous Solubility (AS) is a critical factor in drug discovery (DD), directly influencing a drug’s bioavailability and overall efficacy. Accurate prediction of AS remains a challenge despite the advancement in machine learning techniques, which are essential for improving the pharmacokinetics and formulation of new compounds. Methods: This study determines an enhanced ResNet50 deep learning architecture for predicting AS in drug compounds. Deep-net models with 8, 16, and 20-layer ResNet50 Convolutional Neural Network (CNN) architectures were developed. A dataset of 9,532 drug compounds, represented by molecular footprints, was used to train the models. The training process utilized a ten-fold cross-validation technique to optimize the model's predictive performance. Results: The 20-layer ResNet50 model outperformed human experts and shallower models, achieving an R² value of 0.423 and an RMSE of 0.678. The model also demonstrated an impressive ASP accuracy rate of 90.6%, significantly surpassing the predictions made by human experts and simpler neural network models. Conclusion: This study demonstrates that deeper-net architectures, particularly the 20-layer ResNet50 model, offer superior performance in predicting AS. These deep learning models provide a reliable and efficient solution for improving solubility predictions, crucial for advancing drug discovery efforts.

Evaluating the impact of downsampling on 3D MRI images segmentation results based on similarity metrics

Medical imaging plays a crucial role in diagnosing patient conditions, with magnetic resonance imaging (MRI) standing as a significant modality for numerous years. However, leveraging convolutional neural network (CNN) architectures like U-Net and its variations for anatomical segmentation demands considerable memory, particularly when working with full 3D image sets. Therefore, downsampling 3D MRIs proves advantageous in reducing memory consumption. Nevertheless, downsampling leads to a reduction in voxel count, potentially impacting the performance of commonly used segmentation metrics. The jaccard similarity index (JSI), dice similarity coefficient (DSC), and structural similarity index (SSIM) are extensively employed in image segmentation contexts. Hence, this study employs all three metrics to assess downsampled images and evaluate the robustness of the metrics when used to evaluate the downsampled 3D MRI images. The results show that JSI and DSC are more robust than SSIM when handling the downsampled data.

Gegenbauer Graph Neural Networks for Time-Varying Signal Reconstruction.

Reconstructing time-varying graph signals (or graph time-series imputation) is a critical problem in machine learning and signal processing with broad applications, ranging from missing data imputation in sensor networks to time-series forecasting. Accurately capturing the spatio-temporal information inherent in these signals is crucial for effectively addressing these tasks. However, existing approaches relying on smoothness assumptions of temporal differences and simple convex optimization techniques that have inherent limitations. To address these challenges, we propose a novel approach that incorporates a learning module to enhance the accuracy of the downstream task. To this end, we introduce the Gegenbauer-based graph convolutional (GegenConv) operator, which is a generalization of the conventional Chebyshev graph convolution by leveraging the theory of Gegenbauer polynomials. By deviating from traditional convex problems, we expand the complexity of the model and offer a more accurate solution for recovering time-varying graph signals. Building upon GegenConv, we design the Gegenbauer-based time graph neural network (GegenGNN) architecture, which adopts an encoder-decoder structure. Likewise, our approach also uses a dedicated loss function that incorporates a mean squared error (MSE) component alongside Sobolev smoothness regularization. This combination enables GegenGNN to capture both the fidelity to ground truth and the underlying smoothness properties of the signals, enhancing the reconstruction performance. We conduct extensive experiments on real datasets to evaluate the effectiveness of our proposed approach. The experimental results demonstrate that GegenGNN outperforms state-of-the-art methods, showcasing its superior capability in recovering time-varying graph signals.

Exploring the Combination of Computer Vision and Surgical Neuroanatomy: A Workflow Involving Artificial Intelligence for the Identification of Skull Base Foramina

The skull base is a complex region in neurosurgery, featuring numerous foramina. Accurate identification of these foramina is imperative to avoid intraoperative complications and to facilitate educational progress in neurosurgical trainees. The intricate landscape of the skull base often challenges both clinicians and learners, necessitating innovative identification solutions. We aimed to develop a computer vision model that automates the identification and labeling of the skull base foramina from various image formats, enhancing surgical planning and educational outcomes. We employed a deep learning methodology, specifically using a convolutional neural network architecture. Our model was trained on a dataset comprising of 3560 high-resolution, annotated images of the skull base, taken from various perspectives and lighting conditions to ensure model generalizability. Model performance was quantitatively assessed using precision and recall metrics. The convolutional neural network model demonstrated strong performance, achieving an average precision of 0.77. At a confidence threshold of 0.28, the model reached an optimal precision of 90.4% and a recall of 89.6%. Validation on an independent test set of images corroborated the model's capability to consistently and accurately identify and label multiple skull base foramina across diverse imaging scenarios. This study successfully introduces a highly accurate computer vision model tailored for the identification of skull base foramina, illustrating the model's potential as a transformative tool in anatomical education and intraoperative structure visualization. The findings suggest promising avenues for future research into automated anatomical recognition models, suggesting a trajectory toward increasingly sophisticated aids in neurosurgical operations and education.

A mobile-optimized convolutional neural network approach for real-time batik pattern recognition

This research focuses on preserving and sharing knowledge about Indonesian batik, a blend of art and technology symbolizing the nation's creativity. To address declining awareness of batik types, a mobile application is introduced for real-time recognition and classification of batik motifs. The goal is to maintain appreciation and understanding of this cultural heritage. Using the EfficientNet convolutional neural network (CNN) architecture, the study enhances model accuracy with effective scaling. A dataset of 1350 images representing 15 batik types supports robust model training and evaluation. Results demonstrate successful implementation, yielding an Android app capable of deep learning-based real-time recognition with an 83% accuracy rate. This innovation aims to empower users to identify and appreciate distinct batik types, ensuring cultural preservation for current and future generations.

Equivariant tensor network potentials

Abstract Machine-learning interatomic potentials (MLIPs) have made a significant contribution to the recent progress in the fields of computational materials and chemistry due to the MLIPs’ ability of accurately approximating energy landscapes of quantum-mechanical models while being orders of magnitude more computationally efficient. However, the computational cost and number of parameters of many state-of-the-art MLIPs increases exponentially with the number of atomic features. Tensor (non-neural) networks, based on low-rank representations of high-dimensional tensors, have been a way to reduce the number of parameters in approximating multidimensional functions, however, it is often not easy to encode the model symmetries into them. In this work we develop a formalism for rank-efficient equivariant tensor networks (ETNs), i.e. tensor networks that remain invariant under actions of SO(3) upon contraction. All the key algorithms of tensor networks like orthogonalization of cores and DMRG-based algorithms carry over to our equivariant case. Moreover, we show that many elements of modern neural network architectures like message passing, pulling, or attention mechanisms, can in some form be implemented into the ETNs. Based on ETNs, we develop a new class of polynomial-based MLIPs that demonstrate superior performance over existing MLIPs for multicomponent systems.

ADP-based online compensation hierarchical sliding-mode control for partially unknown switched nonlinear systems with actuator failures

This article investigates an adaptive dynamic programming-based online compensation hierarchical sliding-mode control problem for a class of partially unknown switched nonlinear systems with actuator failures and uncertain perturbations under an identifier-critic neural networks architecture. Firstly, by introducing a cost function related to hierarchical sliding-mode surfaces for the nominal system, the original control problem is equivalently converted into an optimal control problem. To obtain this optimal control policy, the Hamilton–Jacobi–Bellman equation is solved through an adaptive dynamic programming method. Compared with conventional adaptive dynamic programming methods, the identifier-critic network architecture not only overcomes the limitation on the unknown internal dynamic but also eliminates the approximation error arising from the actor network. The weights in the critic network are tuned via the gradient descent approach and the experience replay technology, such that the persistence of excitation condition can be relaxed. Then, a compensation term containing hierarchical sliding-mode surfaces is used to offset uncertain actuator failures without the fault detection and isolation unit. Based on the Lyapunov stability theory, all states of the closed-loop nonlinear system are stable in the sense of uniformly ultimately boundedness. Finally, numerical and practical examples are given to demonstrate the effectiveness of our presented online compensation control strategy.

A possible late-time transition of M B inferred via neural networks

The strengthening of tensions in the cosmological parameters has led to reconsidering the fundamental aspects of standard cosmology. The tension in the Hubble constant can also be viewed as a tension between local and early Universe constraints on the absolute magnitude M B of Type Ia supernova. In this work, we reconsider the possibility of a variation of this parameter in a model-independent way. We employ neural networks to agnostically constrain the value of the absolute magnitude as well as assess the impact and statistical significance of a variation in M B with redshift from the Pantheon+ compilation, together with a thorough analysis of the neural network architecture. We find an indication for a possible transition redshift at the z ≈ 1 region.

Classification of Histamine Content in Fish Using Near-Infrared Spectroscopy and Machine Learning Techniques

Near-infrared (NIR) spectroscopy has emerged as a popular technique for assessing food quality due to its advantages over complex chemical analysis methods. However, the application of NIR spectroscopy for evaluating fish quality based on histamine content has not been extensively explored. This study investigates the use of NIR spectroscopy in combination with machine learning (ML) techniques to classify fish samples into two safety classes, Safe and Unsafe, based on their histamine content. A comprehensive NIR dataset comprising 11,360 spectra collected at eight distinct positions within the fish body was obtained from 284 fish samples of mackerel, tuna, and pompano species. ML experiments were conducted to classify fish samples based on whether their histamine content exceeded the permissible limit of 100 ppm. To address class imbalance and optimize ML models, various data pre-processing and feature extraction techniques as well as ML algorithms were explored. The results demonstrated that utilizing NIR data specifically obtained from the tail’s flesh, a specific location within the fish, yielded superior models for fish safety classification. A feature extraction method employing pre-processed NIR spectra and their second derivatives, combined with an optimized convolutional neural network architecture, outperformed traditional ML classifiers with an accuracy of approximately 93%.

Hybrid CNN-LSTM for Predicting Diabetes: A Review.

Diabetes is a common and deadly chronic disease caused by high blood glucose levels that can cause heart problems, neurological damage, and other illnesses. Through the early detection of diabetes, patients can live healthier lives. Many machine learning and deep learning techniques have been applied for noninvasive diabetes prediction. The results of some studies have shown that the CNN-LSTM method, a combination of CNN and LSTM, has good performance for predicting diabetes compared to other deep learning methods. This paper reviews CNN-LSTM-based studies for diabetes prediction. In the CNNLSTM model, the CNN includes convolution and max pooling layers and is applied for feature extraction. The output of the max-pooling layer was fed into the LSTM layer for classification. The CNN-LSTM model performed well in extracting hidden features and correlations between physiological variables. Thus, it can be used to predict diabetes. The CNNLSTM model, like other deep neural network architectures, faces challenges such as training on large datasets and biological factors. Using large datasets can further improve the accuracy of detection. The CNN-LSTM model is a promising method for diabetes prediction, and compared with other deep-learning models, it is a reliable method.

Evaluation of deep neural network architectures for authorship obfuscation of Portuguese texts

Preserving authorship anonymity is paramount to protect activists, freedom of expression, and critical journalism. Although there are several mechanisms to provide anonymity on the Internet, one can still identify anonymous authors through their writing style. With the advances in neural network and natural language processing research, the success of a classifier when identifying the author of a text is growing. On the other hand, new approaches that use recurrent neural networks for automatic generation of obfuscated texts have also arisen to fight anonymity adversaries. In this work, we evaluate two approaches that use neural networks to generate obfuscated texts. The first approach uses Generative Adversarial Networks to train an encoder–decoder to transform sentences from an input style into a target style. The second one trains an auto encoder with Gradient Reversal Layer to learn invariant representations. In our experiments, we compared the efficiency of both techniques when removing the stylistic attributes of a text and preserving its original semantics. Our evaluation on real texts clarifies each technique’s trade-offs for Portuguese texts and provides guidance on practical deployment.

Wasserstein Auto-Encoders of Merge Trees (and Persistence Diagrams).

This article presents a computational framework for the Wasserstein auto-encoding of merge trees (MT-WAE), a novel extension of the classical auto-encoder neural network architecture to the Wasserstein metric space of merge trees. In contrast to traditional auto-encoders which operate on vectorized data, our formulation explicitly manipulates merge trees on their associated metric space at each layer of the network, resulting in superior accuracy and interpretability. Our novel neural network approach can be interpreted as a non-linear generalization of previous linear attempts (Pont et al. 2023) at merge tree encoding. It also trivially extends to persistence diagrams. Extensive experiments on public ensembles demonstrate the efficiency of our algorithms, with MT-WAE computations in the orders of minutes on average. We show the utility of our contributions in two applications adapted from previous work on merge tree encoding (Pont et al. 2023). First, we apply MT-WAE to merge tree compression, by concisely representing them with their coordinates in the final layer of our auto-encoder. Second, we document an application to dimensionality reduction, by exploiting the latent space of our auto-encoder, for the visual analysis of ensemble data. We illustrate the versatility of our framework by introducing two penalty terms, to help preserve in the latent space both the Wasserstein distances between merge trees, as well as their clusters. In both applications, quantitative experiments assess the relevance of our framework. Finally, we provide a C++ implementation that can be used for reproducibility.

New insight in cervical cancer diagnosis using convolution neural network architecture

<p>The Pap smear is a screening method for early cervical cancer diagnosis. The selection of the right optimizer in the convolutional neural network (CNN) model is key to the success of the CNN in image classification, including the classification of cervical cancer Pap smear images. In this study, stochastic gradient descent (SGD), root mean square propagation (RMSprop), Adam, AdaGrad, AdaDelta, Adamax, and Nadam optimizers were used to classify cervical cancer Pap smear images from the SipakMed dataset. Resnet-18, Resnet-34, and VGG-16 are the CNN architectures used in this study, and each architecture uses a transfer-learning model. Based on the test results, we conclude that the transfer learning model performs better on all CNNs and optimization techniques and that in the transfer learning model, the optimization has little influence on the training of the model. Adamax, with accuracy values of 72.8% and 66.8%, had the best accuracy for the VGG-16 and Resnet-18 architectures, respectively. Resnet-34 had 54.0%. This is 0.034% lower than Nadam. Overall, Adamax is a suitable optimizer for CNN in cervical cancer classification on Resnet-18, Resnet-34, and VGG-16 architectures. This study provides new insights into the configuration of CNN models for Pap smear image analysis.</p>

Predicting the skin friction’s evolution in a forced turbulent channel flow

The present paper reports on the ability of neural networks (NN) and linear stochastic estimation (LSE) tools to predict the evolution of skin friction in a minimal turbulent channel (Reτ=165) after applying an actuation near the wall that is localized in space and time. Two different NN architectures are compared, namely multilayer perceptrons (MLP) and convolutional neural networks (CNN). The paper describes the effect that the predictive horizon and the type/size/number of wall-based sensors have on the performance of each estimator. The performance of MLPs and LSEs is very similar, and becomes independent of the sensor’s size when they are smaller than 60 wall units. For sufficiently small sensors, the CNN outperforms MLPs and LSEs, suggesting that CNNs are able incorporate some of the non-linearities of the near-wall cycle in their prediction of the skin friction evolution after the actuation. Indeed, the CNN is the only architecture able to achieve reasonable predictive capabilities using pressure sensors only. The predictive horizon has a strong effect on the predictive capacity of both NN and LSE, with a Pearson correlation coefficient that varies from 0.95 for short times (i.e., of the order of the actuation time) to less than 0.4 for times of the order of an eddy turn-over time. The analysis of the weights and filters in the LSE and NNs show that all estimators are targeting wall-signatures consistent with streaks, which is interpreted as the streak being the most causal feature in the near-wall cycle for the present forcing.

Classifying Anemia Diseases based on VGG-16 and CBAM Attention Mechanism Models

Anemia is a major public health concern and has a significant impact on women and children worldwide. This condition is caused by a lack of sufficient Red Blood Cells (RBCs). Traditionally, anemia diagnosis has relied on invasive techniques that require blood samples, leading to pain and discomfort. This study explores deep learning techniques for automating and enhancing anemia detection accuracy using the Blood Cell Count and Detection (BCCD) dataset. We employ the VGG16 convolutional neural network architecture augmented with the Convolutional Block Attention Module (CBAM) to boost feature representation and performance. The BCCD dataset, consisting of 364 images with 4888 labeled blood cells, was used for training and evaluation. Our enhanced VGG16 model achieved accuracy for RBCs, WBCs, and platelets with values of 0.84, 0.93, and 0.92, respectively, effectively identifying various blood cell types and detecting anemia. Precision-recall analysis and confusion matrix metrics confirmed the robustness of the model, with high precision and recall rates and minimal false positives and negatives. These results suggest that advanced deep learning models with attention mechanisms, such as CBAM, can significantly improve medical diagnostics by providing reliable, efficient, and accurate early disease detection tools. Future development will be directed towards fine-tuning and validating the proposed model on other medical imaging datasets for clinical applications.

Unidirectional and hierarchical on-chip interconnected architecture for large-scale hardware spiking neural networks

Spiking Neural Networks (SNNs) exhibit the strong capability to address spatiotemporal dynamic problems. Recent research has explored the hardware SNN systems to solve the spatiotemporal problems in real-time. The Network-on-Chip (NoC) is an effective scheme for building large-scale hardware SNNs. However, for the existing NoC-based hardware SNNs, large area overhead and hardware power are consumed by their interconnections, because of complex topologies and router structures. Therefore, in this work a novel Unidirectional and Hierarchical on-Chip Interconnected Architecture (UHCIA) is proposed to address this problem. The proposed UHCIA mainly combines the novel hybrid topology of unidirectional multiple loops and rings, and uses a deflection router technique. Experimental results show that compared to other works, the UHCIA achieves ∼23.6X of area reduction and ∼6.4X of power reduction, with high system throughput and biological real-time computations.

Detection and classification of electrocardiography using hybrid deep learning models

ObjectiveElectrocardiography (ECGs) has been a vital tool for cardiovascular disease (CVD) diagnosis, which visually depicts the heart's electrical activity. To enhance automatic classification between normal and diseased ECG, it is essential to extract consistent and qualitative features. MethodsPrecision of ECG classification through a hybrid Deep Learning (DL) approach leverages both Convolutional Neural Network (CNN) architecture and Variational Autoencoder (VAE) techniques. By combining these methods, we aim to achieve more accurate and robust ECG interpretation. The method is trained and tested over the PTB-XL dataset, which contains 21,799 with 12-lead ECGs from 18,869 patients, each spanning 10 s. The classification evaluation of five super-classes and 23 sub-classes of CVD, with the proposed CNN-VAE model is compared. ResultsThe classification of various CVDs resulted in the highest accuracy of 98.51%, specificity of 98.12%, sensitivity of 97.9%, and F1-score of 97.95%. We have also achieved the minimum false positive and false negative rates of 2.07 and 1.87, respectively, during validation. The results are validated upon the annotations given by individual cardiologists, who assigned potentially multiple ECG statements to each record. ConclusionWhen compared to other deep learning methods, our suggested CNN-VAE model performs significantly better in the testing phase. This study proposes a new architecture of combining CNN-VAE for CVD classification from ECG data, this can help clinicians to identify the disease earlier and carry out further treatment. The CNN-VAE model can better characterize input signals due to its hybrid architecture.

Machine vision-based detection of forbidden elements in the high-speed automatic scrap sorting line

Highly efficient industrial sorting lines require fast and reliable classification methods. Various types of sensors are used to measure the features of an object to determine which output class it belongs to. One technique involves the use of an RGB camera and a machine learning classifier. The paper is focused on protecting the sorting process against prohibited and dangerous items potentially present in the sorted material that pose a threat to the sorting process or the subsequent metallurgical process. To achieve this, a convolutional neural network classifier was applied under real-life conditions to detect forbidden elements in copper-based metal scrap. A laboratory stand simulating the working conditions in a high-speed scrap sorting line was prepared. Using this custom stand, training and test sets for machine learning were gathered and labeled. An image preprocessing algorithm was designed to increase the robustness of the resulting forbidden element detector system. The performance of multiple neural network architectures and data set augmentations was analyzed. The highest accuracy of 98.03% and F1-score of 97.16% were achieved with a DenseNet-based classifier. The results of this paper show the feasibility of using the presented solution on a high-speed industrial line.

Comparing neural network architectures for non-intrusive speech quality prediction

Non-intrusive speech quality predictors evaluate speech quality without the use of a reference signal, making them useful in many practical applications. Recently, neural networks have shown the best performance for this task. Two such models in the literature are the convolutional neural network based DNSMOS and the bi-directional long short-term memory based Quality-Net, which were originally trained to predict subjective targets and intrusive PESQ scores, respectively. In this paper, these two architectures are trained on a single dataset, and used to predict the intrusive ViSQOL score. The evaluation is done on a number of test sets with a variety of mismatch conditions, including unseen speech and noise corpora, and common voice over IP distortions. The experiments show that the models achieve similar predictive ability on the training distribution, and overall good generalization to new noise and speech corpora. Unseen distortions are identified as an area where both models generalize poorly, especially DNSMOS. Our results also suggest that a pervasiveness of ambient noise in the training set can cause problems when generalizing to certain types of noise. Finally, we detail how the ViSQOL score can have undesirable dependencies on the reference pressure level and the voice activity level.

Deciphering the Language of Protein-DNA Interactions: A Deep Learning Approach Combining Contextual Embeddings and Multi-Scale Sequence Modeling

Deciphering the mechanisms governing protein-DNA interactions is crucial for understanding key cellular processes and disease pathways. In this work, we present a powerful deep learning approach that significantly advances the computational prediction of DNA-interacting residues from protein sequences.Our method leverages the rich contextual representations learned by pre-trained protein language models, such as ProtTrans, to capture intrinsic biochemical properties and sequence motifs indicative of DNA binding sites. We then integrate these contextual embeddings with a multi-window convolutional neural network architecture, which scans across the sequence at varying window sizes to effectively identify both local and global binding patterns.Comprehensive evaluation on curated benchmark datasets demonstrates the remarkable performance of our approach, achieving an area under the ROC curve (AUC) of 0.89 – a substantial improvement over previous state-of-the-art sequence-based predictors. This showcases the immense potential of pairing advanced representation learning and deep neural network designs for uncovering the complex syntax governing protein-DNA interactions directly from primary sequences.Our work not only provides a robust computational tool for characterizing DNA-binding mechanisms, but also highlights the transformative opportunities at the intersection of language modeling, deep learning, and protein sequence analysis. The publicly available code and data further facilitate broader adoption and continued development of these techniques for accelerating mechanistic insights into vital biological processes and disease pathways.In addition, the code and data for this work are available at https://github.com/B1607/DIRP.