Deep Learning Architecture Research Articles

Enabling Multi-Step Forecasting with Structured State Space Learning Module

Data-driven soft sensor incorporated with the model predictive control (MPC) algorithms facilitating product quality and cost control are of imperative importance in industrial processes. However, the widely used one-step forecasting method can not incorporate with MPC and therefore restricts the practical usage of soft sensor. Multi-step forecasting introduces long-term dependencies problems yet has not been effectively resolved within traditional model structure. To address this problem, this paper proposes the deep learning network architecture named Extended State Space Learning Module (ESSLM). ESSLM extends the nonlinear mapping architecture of deep learning based on state space and retains state transfer matrices to characterize the dynamics of the system. ESSLM distinguishes itself from explicit network architectures such as gated RNNs by addressing the long-term dependencies problems through an implicit initialization method, and the MLP and RNN algorithms can be regarded as the manifestation of ESSLM in special cases. ESSLM characterizes the latent space as the coefficients of the orthogonal basis functions so that the input data can be encoded into a high-dimensional feature space with minimal information loss which efficiently achieves multi-step forecasting and give greater utility and practical significance.

Deep learning architectures for location and identification in storage systems

Deep learning architectures for location and identification in storage systems

Discovery of anticancer peptides from natural and generated sequences using deep learning.

Anticancer peptides (ACPs) demonstrate significant potential in clinical cancer treatment due to their ability to selectively target and kill cancer cells. In recent years, numerous artificial intelligence (AI) algorithms have been developed. However, many predictive methods lack sufficient wet lab validation, thereby constraining the progress of models and impeding the discovery of novel ACPs. This study proposes a comprehensive research strategy by introducing CNBT-ACPred, an ACP prediction model based on a three-channel deep learning architecture, supported by extensive in vitro and in vivo experiments. CNBT-ACPred achieved an accuracy of 0.9554 and a Matthews Correlation Coefficient (MCC) of 0.8602. Compared to existing excellent models, CNBT-ACPred increased accuracy by at least 5% and improved MCC by 15%. Predictions were conducted on over 3.8 million sequences from Uniprot, along with 100,000 sequences generated by a deep generative model, ultimately identifying 37 out of 41 candidate peptides from >30 species that exhibited effective in vitro tumor inhibitory activity. Among these, tPep14 demonstrated significant anticancer effects in two mouse xenograft models without detectable toxicity. Finally, the study revealed correlations between the amino acid composition, structure, and function of the identified ACP candidates.

STCNet: Spatio-Temporal Cross Network with subject-aware contrastive learning for hand gesture recognition in surface EMG.

This paper introduces the Spatio-Temporal Cross Network (STCNet), a novel deep learning architecture tailored for robust hand gesture recognition in surface electromyography (sEMG) across multiple subjects. We address the challenges associated with the inter-subject variability and environmental factors such as electrode shift and muscle fatigue, which traditionally undermine the robustness of gesture recognition systems. STCNet integrates a convolutional-recurrent architecture with a spatio-temporal block that extracts features over segmented time intervals, enhancing both spatial and temporal analysis. Additionally, a rolling convolution technique designed to reflect the circular band structure of the sEMG measurement device is incorporated, thus capturing the inherent spatial relationships more effectively. We further propose a subject-aware contrastive learning framework that utilizes both subject and gesture label information to align the representation of vector space. Our comprehensive experimental evaluations demonstrate the superiority of STCNet under aggregated conditions, achieving state-of-the-art performance on benchmark datasets and effectively managing the variability among different subjects. The implemented code can be found at https://github.com/KNU-BrainAI/STCNet.

PLM4CPPs: Protein Language Model-Based Predictor for Cell Penetrating Peptides.

Cell-penetrating peptides (CPPs) are short peptides capable of penetrating cell membranes, making them valuable for drug delivery and intracellular targeting. Accurate prediction of CPPs can streamline experimental validation in the lab. This study aims to assess pretrained protein language models (pLMs) for their effectiveness in representing CPPs and develop a reliable model for CPP classification. We evaluated peptide embeddings generated from BEPLER, CPCProt, SeqVec, various ESM variants (ESM, ESM-2 with expanded feature set, ESM-1b, and ESM-1v), ProtT5-XL UniRef50, ProtT5-XL BFD, and ProtBERT. We developed pLM4CCPs, a novel deep learning architecture using convolutional neural networks (CNNs) as the classifier for binary classification of CPPs. pLM4CCPs demonstrated superior performance over existing state-of-the-art CPP prediction models, achieving improvements in accuracy (ACC) by 4.9-5.5%, Matthews correlation coefficient (MCC) by 9.3-10.2%, and sensitivity (Sn) by 14.1-19.6%. Among all the tested models, ESM-1280 and ProtT5-XL BFD demonstrated the highest overall performance on the kelm data set. ESM-1280 achieved an ACC of 0.896, an MCC of 0.796, a Sn of 0.844, and a specificity (Sp) of 0.978. ProtT5-XL BFD exhibited superior performance with an ACC of 0.901, an MCC of 0.802, an Sn of 0.885, and an Sp of 0.917. pLM4CCPs combine predictions from multiple models to provide a consensus on whether a given peptide sequence is classified as a CPP or non-CPP. This approach will enhance prediction reliability by leveraging the strengths of each individual model. A user-friendly web server for bioactivity predictions, along with data sets, is available at https://ry2acnp6ep.us-east-1.awsapprunner.com. The source code and protocol for adapting pLM4CPPs can be accessed on GitHub at https://github.com/drkumarnandan/pLM4CPPs. This platform aims to advance CPP prediction and peptide functionality modeling, aiding researchers in exploring peptide functionality effectively.

Improved lightweight deep learning architecture for household waste detection

Improved lightweight deep learning architecture for household waste detection

An Online Evaluation Method for Random Number Entropy Sources Based on Time-Frequency Feature Fusion

Traditional entropy source evaluation methods rely on statistical analysis and are hard to deploy on-chip or online. However, online detection of entropy source quality is necessary in some applications with high encryption levels. To address these issues, our experimental results demonstrate a significant negative correlation between minimum entropy values and prediction accuracy, with a Pearson correlation coefficient of −0.925 (p-value = 1.07 × 10−7). This finding offers a novel approach for assessing entropy source quality, achieving an accurate rate in predicting the next bit of a random sequence using neural networks. To further improve prediction capabilities, we also propose a novel deep learning architecture, Fast Fourier Transform-Attention Mechanism-Long Short-Term Memory Network (FFT-ATT-LSTM), that integrates a simplified soft attention mechanism with Fast Fourier Transform (FFT), enabling effective fusion of time-domain and frequency-domain features. The FFT-ATT-LSTM improves prediction accuracy by 4.46% and 8% over baseline networks when predicting random numbers. Additionally, FFT-ATT-LSTM maintains a compact parameter size of 33.90 KB, significantly smaller than Temporal Convolutional Networks (TCN) at 41.51 KB and Transformers at 61.51 KB, while retaining comparable prediction performance. This optimal balance between accuracy and resource efficiency makes FFT-ATT-LSTM suitable for online deployment, demonstrating considerable application potential.

Thoracic Aortic Three-Dimensional Geometry

Background: Aortic structure impacts cardiovascular health through multiple mechanisms. Aortic structural degeneration occurs with aging, increasing left ventricular afterload and promoting increased arterial pulsatility and target organ damage. Despite the impact of aortic structure on cardiovascular health, three-dimensional (3D) aortic geometry has not been comprehensively characterized in large populations. Methods: We segmented the complete thoracic aorta using a deep learning architecture and used morphological image operations to extract multiple aortic geometric phenotypes (AGPs, including diameter, length, curvature, and tortuosity) across various subsegments of the thoracic aorta. We deployed our segmentation approach on imaging scans from 54,241 participants in the UK Biobank and 8,456 participants in the Penn Medicine Biobank. Conclusion: Our method provides a fully automated approach towards quantifying the three-dimensional structural parameters of the aorta. This approach expands the available phenotypes in two large representative biobanks and will allow large-scale studies to elucidate the biology and clinical consequences of aortic degeneration related to aging and disease states.

A Survey on WiFi-based Human Identification: Scenarios, Challenges, and Current Solutions

With the evolution of wireless sensing technology, WiFi-based human identification has demonstrated tremendous potential in human-computer interaction and home security. However, most existing research operates in controlled environments, overlooking the complexities of real-world scenarios, such as signal fading, signal interference and uncertainty, and the diversity of application requirements. This article presents a comprehensive analysis of a series of representative research articles on WiFi-based human identification and summarizes five major challenges in achieving high-precision identification in complex application scenarios. Non-line-of-sight (NLOS) user sensing, coexistence user sensing, dynamic group user sensing, cross-domain user sensing, and multi-task sensing. Additionally, this article proposes a series of current solutions, including improving the signal-to-noise ratio (SNR) of NLOS sensing from the perspective of communication signals and sensing models, addressing coexistence sensing issues, designing adaptable tasks for dynamic user groups, leveraging techniques like adversarial learning and transfer learning for cross-domain problems, and employing modular deep learning architectures. By providing a comprehensive overview of WiFi-based human identification, this survey not only offers insights into current research but also charts a roadmap for future investigations. It is anticipated that this survey will stimulate innovative research endeavors and foster the expansion of wireless sensing technology across diverse application domains.

Reliable, efficient, and scalable photonic inverse design empowered by physics-inspired deep learning

Abstract On-chip computing metasystems composed of multilayer metamaterials have the potential to become the next-generation computing hardware endowed with light-speed processing ability and low power consumption but are hindered by current design paradigms. To date, neither numerical nor analytical methods can balance efficiency and accuracy of the design process. To address the issue, a physics-inspired deep learning architecture termed electromagnetic neural network (EMNN) is proposed to enable an efficient, reliable, and flexible paradigm of inverse design. EMNN consists of two parts: EMNN Netlet serves as a local electromagnetic field solver; Huygens–Fresnel Stitch is used for concatenating local predictions. It can make direct, rapid, and accurate predictions of full-wave field based on input fields of arbitrary variations and structures of nonfixed size. With the aid of EMNN, we design computing metasystems that can perform handwritten digit recognition and speech command recognition. EMNN increases the design speed by 17,000 times than that of the analytical model and reduces the modeling error by two orders of magnitude compared to the numerical model. By integrating deep learning techniques with fundamental physical principle, EMNN manifests great interpretability and generalization ability beyond conventional networks. Additionally, it innovates a design paradigm that guarantees both high efficiency and high fidelity. Furthermore, the flexible paradigm can be applicable to the unprecedentedly challenging design of large-scale, high-degree-of-freedom, and functionally complex devices embodied by on-chip optical diffractive networks, so as to further promote the development of computing metasystems.

Interpretability of fingerprint presentation attack detection systems: a look at the “representativeness” of samples against never-seen-before attacks

Nowadays, fingerprint Presentation Attack Detection systems (PADs) are primarily based on deep learning architectures subjected to massive training. However, their performance decreases to never-seen-before attacks. With the goal of contributing to explaining this issue, we hypothesized that this limited ability to generalize is due to the lack of "representativeness" of the samples available for the PAD training. "Representativeness" is treated here from a geometrical perspective: the spread of samples into the feature space, especially near the decision boundaries. In particular, we explored the possibility of adopting three-dimensionality reduction methods to make the problem affordable through visual inspection. These methods enable visual inspection and interpretation by projecting data into two-dimensional spaces, facilitating the identification of weak areas in the decision regions estimated after the training phase. Our analysis delineates the benefits and drawbacks of each dimensionality reduction method and leads us to make substantial recommendations in the crucial phase of the training design.

CERVIXNET: An Efficient Approach for the Detection and Classifications of the Cervigram Images Using Modified Deep Learning Architecture.

The earlier detection of cervical cancer in women patients can save human life. This article proposes a novel methodology for detecting abnormal cervigram images from healthy cervigram images and segments the cancer regions in the abnormal cervigram images using the deep learning method. The conventional deep learning architecture has been modified into the proposed CervixNet architecture to improve the cervical cancer detection rate. This methodology is constituted of a training and testing process, where the training process generates the training sequences individually for healthy cervigram images and the cancer case cervigram images. The testing process tests the cervigram images into either a healthy or cancer cases using the training sequences generated through the training process. During the testing process of the proposed system, the cancer segmentation algorithm was applied on the abnormal cervigram image to detect and segment the pixels belonging to cancer. Finally, the performance has been carried out on the segmented cancer cervical images for the ground truth images. This proposed methodology has been evaluated on the cervigrams on IMODT and Guanacaste databases. Its performance has been analyzed concerning cancer pixel sensitivity, cancer pixel specificity and cancer pixel accuracy. This research work obtains 98.69% Cancer Pixel Sensitivity (CPS), 98.76% Cancer Pixel Specificity (CPSP), and 99.27% Cancer Pixel Accuracy (CPA) for the set of cervigram images in the IMODT database. This research work obtains 99.22% CPS, 99.03% CPSP, and 99.01% CPA for the set of cervigram images in Guanacaste database. These experimental results of the proposed work have been significantly compared with the state-of-the-art methods and show the significance and novelty of the proposed works.

Segmentation of ADPKD Computed Tomography Images with Deep Learning Approach for Predicting Total Kidney Volume

Background: Total Kidney Volume (TKV) is widely used globally to predict the progressive loss of renal function in patients with Autosomal Dominant Polycystic Kidney Disease (ADPKD). Typically, TKV is calculated using Computed Tomography (CT) images by manually locating, delineating, and segmenting the ADPKD kidneys. However, manual localization and segmentation are tedious, time-consuming tasks and are prone to human error. Specifically, there is a lack of studies that focus on CT modality variation. Methods: In contrast, our work develops a step-by-step framework, which robustly handles both Non-enhanced Computed Tomography (NCCT) and Contrast-enhanced Computed Tomography (CCT) images, ensuring balanced sample utilization and consistent performance across modalities. To achieve this, Artificial Intelligence (AI)-enabled localization and segmentation models are proposed for estimating TKV, which is designed to work robustly on both NCCT and Contrast-Computed Tomography (CCT) images. These AI-based models incorporate various image preprocessing techniques, including dilation and global thresholding, combined with Deep Learning (DL) approaches such as the adapted Single Shot Detector (SSD), Inception V2, and DeepLab V3+. Results: The experimental results demonstrate that the proposed AI-based models outperform other DL architectures, achieving a mean Average Precision (mAP) of 95% for automatic localization, a mean Intersection over Union (mIoU) of 92% for segmentation, and a mean R2 score of 97% for TKV estimation. Conclusions: These results clearly indicate that the proposed AI-based models can robustly localize and segment ADPKD kidneys and estimate TKV using both NCCT and CCT images.

Investigating the intrinsic top-down dynamics of deep generative models

Hierarchical generative models can produce data samples based on the statistical structure of their training distribution. This capability can be linked to current theories in computational neuroscience, which propose that spontaneous brain activity at rest is the manifestation of top-down dynamics of generative models detached from action-perception cycles. A popular class of hierarchical generative models is that of Deep Belief Networks (DBNs), which are energy-based deep learning architectures that can learn multiple levels of representations in a completely unsupervised way exploiting Hebbian-like learning mechanisms. In this work, we study the generative dynamics of a recent extension of the DBN, the iterative DBN (iDBN), which more faithfully simulates neurocognitive development by jointly tuning the connection weights across all layers of the hierarchy. We characterize the number of states visited during top-down sampling and investigate whether the heterogeneity of visited attractors could be increased by initiating the generation process from biased hidden states. To this end, we train iDBN models on well-known datasets containing handwritten digits and pictures of human faces, and show that the ability to generate diverse data prototypes can be enhanced by initializing top-down sampling from “chimera states”, which represent high-level features combining multiple abstract representations of the sensory data. Although the models are not always able to transition between all potential target states within a single-generation trajectory, the iDBN shows richer top-down dynamics in comparison to a shallow generative model (a single-layer Restricted Bolzamann Machine). We further show that the generated samples can be used to support continual learning through generative replay mechanisms. Our findings suggest that the top-down dynamics of hierarchical generative models is significantly influenced by the shape of the energy function, which depends both on the depth of the processing architecture and on the statistical structure of the sensory data.

A Survey of Deep Learning Methods for Noise Classification and Detection in Historical Image Analysis

Historical images often suffer from various types of noise due to aging, degradation, and other environmental factors, which can significantly impact their quality and usability. The advent of deep learning has revolutionized the field of image processing, offering robust methods for noise classification and detection. This survey provides a comprehensive overview of the current state-of-the-art deep learning techniques employed in historical image analysis for identifying and mitigating different types of noise. We review various deep learning architectures, including convolutional neural networks (CNNs), generative adversarial networks (GANs), and auto encoders, that have been applied to this problem. The paper discusses the strengths and limitations of each approach, highlights key challenges such as data scarcity and variability in noise types, and explores future directions in this field. The survey aims to serve as a resource for researchers and practitioners by summarizing the most effective methods, evaluating their performance on different datasets, and outlining potential avenues for further research

Rice Plant Disease Detection Using Efficient Net V2

Rice is a staple crop feeding billions worldwide, yet its production is severely impacted by plant diseases, leading to significant economic losses and food insecurity. This project proposes an advanced Rice Plant Disease Detection System leveraging EfficientNetV2, a state-of the-art deep learning architecture, to achieve high accuracy in identifying and classifying rice diseases. The system incorporates geo-specific tagging during image acquisition, enabling location-based disease mapping and tailored crop recommendations. Key features include real-time disease detection, severity analysis, and actionable insights through a user-friendly dashboard with multilingual support. By addressing the limitations of traditional methods and existing automated solutions—such as overfitting, lack of scalability, and real-world adaptability—this project aims to provide an accurate, scalable, and accessible solution for farmers, ultimately promoting sustainable agriculture and enhancing rice crop yield.

NuFold: end-to-end approach for RNA tertiary structure prediction with flexible nucleobase center representation

RNA plays a crucial role not only in information transfer as messenger RNA during gene expression but also in various biological functions as non-coding RNAs. Understanding mechanical mechanisms of function needs tertiary structure information; however, experimental determination of three-dimensional RNA structures is costly and time-consuming, leading to a substantial gap between RNA sequence and structural data. To address this challenge, we developed NuFold, a novel computational approach that leverages state-of-the-art deep learning architecture to accurately predict RNA tertiary structures. NuFold is a deep neural network trained end-to-end for the output structure from the input sequence. NuFold incorporates a nucleobase center representation, which enables flexible conformation of ribose rings. Benchmark study showed that NuFold clearly outperformed energy-based methods and demonstrated comparable results with existing state-of-the-art deep-learning-based methods. NuFold exhibited a particular advantage in building correct local geometries of RNA. Analyses of individual components in the NuFold pipeline indicated that the performance improved by utilizing metagenome sequences for multiple sequence alignment and increasing the number of recycling. NuFold is also capable of predicting multimer complex structures of RNA by linking the input sequences.

Pneumonia and COVID-19 Classification and Detection Based on Convolutional Neural Network: A Review

Pneumonia diseases are considered one of the most pandemic diseases by the WHO, claiming the lives of millions across the world. Therefore, the necessity of having mechanisms for early diagnosis and detection of these epidemic diseases to preserve people's lives. On the other hand, the increase in cases requires not relying on traditional means of detecting diseases due to these tests' limitations and high costs like RT-PCR and errors in interpretation by humans. Among the available methods for diagnosing Pneumonia diseases are X-rays and CT scans. For accurate and highly efficient diagnosis, computer-aided diagnosis (CAD) is required. Deep learning has been suggested as the best solution to enhance the prognosis for many lung diseases using CAD systems. The ability of machine learning algorithms was not sufficient to diagnose with high accuracy due to several challenges, like: limited dataset, high computational, and less robustness because of huge numbers of features in the images. The aspiration to deep learning to solve the problems of diagnosis more accurate, especially convolution neural networks, showed impressive results in the classification of images. The convolutions layer in the network with filters automatically discover the critical spatial and temporal features in an image. Hierarchical representations were used in human brains in designing the learning process of CNN. Several deep learning architectures have been used to detect pneumonia diseases, the most popular such as AlexNet, VGG-16, Inception-v1, Inception-v3, ResNet-50, Inception-ResNet-V2, and ResNet201. The strength of CNNs lies in the size of data to be trained. Previous algorithms have been used to pre-train on a vast dataset. An augmentation strategy has been used in the model design to increase the dataset's size and quality artificially. This review aims to pave the way for better, more accurate, and efficient models for early detection and classification of lung diseases to improve patient survival.

Multi-modal deep learning architecture for enhanced feature extraction and classification of imagined speech words

Abstract Imagined speech prediction is a challenging task with significant implications for brain-computer interfaces (BCIs) and assistive communication technologies. EEG signals are highly non-stationary and thus it is extremely difficult to find any relevant information from these electroencephalogram (EEG) signals just by seeing them in time domain. Imagined speech-based BCIs can be integrated into assistive technology devices that are specifically designed for individuals with severe physical disabilities and can also play a role in rehabilitation programs for individuals recovering from conditions that affect speech and mobility. This paper proposes a novel hybrid model that combines convolutional neural network (CNN) and recurrent neural networks (RNNs) to predict imagined speech patterns from electroencephalography (EEG) signals. The integration of CNNs allows the model to capture spatial information from multi-channel EEG data, while RNNs excel at capturing temporal dependencies inherent in sequential EEG signals. The proposed model architecture starts with a 1-D CNN layer to extract spatial features from raw EEG data. The output is then fed into RNNs to capture temporal patterns and relationships over time. It achieved competitive accuracy, averaging approximately 85.25, 87.50, 81.58 and 93.92 percent for short words, long words, vowels and short-long words respectively.

Application of Convolutional Neural Networks for Watermelon Detection in UAV Aerial Images: A Case Study

Although human dependence on agriculture decreases with developing technology, it continues. As many resources are increasingly restricted due to various climatic reasons, the importance of studies in this field increases. Applications using deep learning models are frequently encountered in the agricultural field. In particular, there are applications where deep learning models are used as a tool for optimum planting, land use, yield improvement, production/disease/pest control, and other activities.In this study, watermelons in an aerial view of a watermelon field were detected by utilizing the Alexnet deep learning architecture. To obtain yield, watermelons in watermelon fields should be specified and then counted. Aerial images are used for this application. The field image was divided into 50% overlapping sub-images, and each was classified as watermelon, leaf, and soil. Consequently, watermelon regions on the field image were specified. After training the Alexnet and Vgg19 network structure with the dataset, watermelons were to be identified by segmenting the images. It was observed that the Vgg19 network achieved 97.78% accuracy. The results of the experimental applications show that the Vgg19 can be applied for watermelon fruit and yield detection applications.