Advanced Deep Learning Models Research Articles

Prediction of PM2.5 Concentration Based on Deep Learning, Multi-Objective Optimization, and Ensemble Forecast

Accurate and stable prediction of atmospheric PM2.5 concentrations is crucial for air pollution prevention and control. Existing studies usually rely on a single model or use a single evaluation criterion in multi-model ensemble weighted forecasts, neglecting the dual needs for accuracy and stability in PM2.5 forecast. In this study, a novel ensemble forecast model is proposed that overcomes these drawbacks by simultaneously taking into account both forecast accuracy and stability. Specifically, four advanced deep learning models—Long Short-Term Memory Network (LSTM), Graph Convolutional Network (GCN), Transformer, and Graph Sample and Aggregation Network (GraphSAGE)—are firstly introduced. And then, two combined models are constructed as predictors, namely LSTM–GCN and Transformer–GraphSAGE. Finally, a combined weighting strategy is adopted to assign weights to these two combined models using a multi-objective optimization algorithm (MOO), so as to carry out more accurate and stable predictions. The experiments are conducted on the dataset from 36 air quality monitoring stations in Beijing, and results show that the proposed model achieves more accurate and stable predictions than other benchmark models. It is hoped that this proposed ensemble forecast model will provide effective support for PM2.5 pollution forecast and early warning in the future.

Real-time segmentation of short videos under VR technology in dynamic scenes

Abstract This work addresses the challenges of scene segmentation and low segmentation accuracy in short videos by employing virtual reality (VR) technology alongside a 3D DenseNet model for real-time segmentation in dynamic scenes. First, this work extracted short videos by frame and removed redundant background information. Then, the volume rendering algorithm in VR technology was used to reconstruct short videos in dynamic scenes in 3D. It enriched the detailed information of short videos, and finally used the 3D DenseNet model for real-time segmentation of short videos in dynamic scenes, improving the accuracy of segmentation. The experiment compared the performance of High resolution network, Mask region based convolutional neural network, 3D U-Net, Efficient neural network models on the Densely annotation video segmentation dataset. The experimental results showed that the segmentation accuracy of the 3D DenseNet model has reached 99.03%, which was 15.11% higher than that of the ENet model. The precision rate reached 98.33%, and the average segmentation time reached 0.64 s, improving the segmentation accuracy and precision rate. It can adapt to various scene situations and has strong robustness. The significance of this research lies in its innovative approach in tackling these issues. By integrating VR technology with advanced deep learning models, we can achieve more precise segmentation of dynamic scenes in short videos, enabling real-time processing. This has significant practical implications for fields such as video editing, VR applications, and intelligent surveillance. Furthermore, the outcomes of this research contribute to advancing computer vision in video processing, providing valuable insights for the development of future intelligent video processing systems.

Developing explainable models for lncRNA-Targeted drug discovery using graph autoencoders

The application of explainable artificial intelligence technology is crucial in lncRNA-targeted drug discovery, given lncRNA’s role in regulating a broad spectrum of diseases, including cancer. Current advanced deep learning models efficiently predict the association between lncRNA and disease, facilitating the discovery of potential lncRNA-targeted drug. However, these models have notable limitations, including a tendency to overfit due to their focus on integrating various similarity networks for feature extraction of lncRNA and diseases. Additionally, they employ complex classifiers to predict unknown lncRNA-disease associations (LDAs), resulting in reduced interpretability. We designed an explainable model named EM-LTDD, utilizing graph autoencoders (GAE) and self-supervised learning strategies, aimed at identifying potential drugs targeting lncRNA. The EM-LTDD model improves self-supervised training by employing a novel strategy that masks parts of the known lncRNA-disease network. This approach not only enhances the model’s interpretability but also mitigates the impact of noise. Furthermore, the EM-LTDD model reconstructs the lncRNA-disease network with a robust collaborative decoder, minimizing information loss caused by occlusion and improving node representation. Experimental results demonstrate the effectiveness of our model compared to existing leading models, promising to significantly aid in the discovery and design of lncRNA-targeted drugs. Our code and data are available at: https://github.com/huiyingliulevy/EM-LTDD.

Deep Learning for Multi-Source Data-Driven Crop Yield Prediction in Northeast China

The accurate prediction of crop yields is crucial for enhancing agricultural efficiency and ensuring food security. This study assesses the performance of the CNN-LSTM-Attention model in predicting the yields of maize, rice, and soybeans in Northeast China and compares its effectiveness with traditional models such as RF, XGBoost, and CNN. Utilizing multi-source data from 2014 to 2020, which include vegetation indices, environmental variables, and photosynthetically active parameters, our research examines the model’s capacity to capture essential spatial and temporal variations. The CNN-LSTM-Attention model integrates Convolutional Neural Networks, Long Short-Term Memory, and an attention mechanism to effectively process complex datasets and manage non-linear relationships within agricultural data. Notably, the study explores the potential of using kNDVI for predicting yields of multiple crops, highlighting its effectiveness. Our findings demonstrate that advanced deep-learning models significantly enhance yield prediction accuracy over traditional methods. We advocate for the incorporation of sophisticated deep-learning technologies in agricultural practices, which can substantially improve yield prediction accuracy and food production strategies.

A Comparative Sentiment Analysis of Greek Clinical Conversations Using BERT, RoBERTa, GPT-2, and XLNet.

In addressing the critical role of emotional context in patient-clinician conversations, this study conducted a comprehensive sentiment analysis using BERT, RoBERTa, GPT-2, and XLNet. Our dataset includes 185 h of Greek conversations focused on hematologic malignancies. The methodology involved data collection, data annotation, model training, and performance evaluation using metrics such as accuracy, precision, recall, F1-score, and specificity. BERT outperformed the other methods across all sentiment categories, demonstrating its effectiveness in capturing the emotional context in clinical interactions. RoBERTa showed a strong performance, particularly in identifying neutral sentiments. GPT-2 showed promising results in neutral sentiments but exhibited a lower precision and recall for negatives. XLNet showed a moderate performance, with variations across categories. Overall, our findings highlight the complexities of sentiment analysis in clinical contexts, especially in underrepresented languages like Greek. These insights highlight the potential of advanced deep-learning models in enhancing communication and patient care in healthcare settings. The integration of sentiment analysis in healthcare could provide insights into the emotional states of patients, resulting in more effective and empathetic patient support. Our study aims to address the gap and limitations of sentiment analysis in a Greek clinical context, an area where resources are scarce and its application remains underexplored.

Enhancing E-Learning Adaptability with Automated Learning Style Identification and Sentiment Analysis: A Hybrid Deep Learning Approach for Smart Education

In smart education, adaptive e-learning systems personalize the educational process by tailoring it to individual learning styles. Traditionally, identifying these styles relies on learners completing surveys and questionnaires, which can be tedious and may not reflect their true preferences. Additionally, this approach assumes that learning styles are fixed, leading to a cold-start problem when automatically identifying styles based on e-learning platform behaviors. To address these challenges, we propose a novel approach that annotates unlabeled student feedback using multi-layer topic modeling and implements the Felder–Silverman Learning Style Model (FSLSM) to identify learning styles automatically. Our method involves learners answering four FSLSM-based questions upon logging into the e-learning platform and providing personal information like age, gender, and cognitive characteristics, which are weighted using fuzzy logic. We then analyze learners’ behaviors and activities using web usage mining techniques, classifying their learning sequences into specific styles with an advanced deep learning model. Additionally, we analyze textual feedback using latent Dirichlet allocation (LDA) for sentiment analysis to enhance the learning experience further. The experimental results demonstrate that our approach outperforms existing models in accurately detecting learning styles and improves the overall quality of personalized content delivery.

Synthetic Media Analysis Using Deep Learning

In an era characterized by the rapid evolution of digital content creation, synthetic media, particularly deepfake videos, present a formidable challenge to the veracity and integrity of online information. Addressing this challenge requires sophisticated analytical techniques capable of discerning between authentic and manipulated media. This research paper presents a comprehensive study on synthetic media analysis leveraging deep learning methodologies. The suggested method integrates advanced deep learning models, including Convolutional Neural Networks (CNNs) like VGG (Visual Geometry Group), alongside recurrent structures such as LSTM (Long Short-Term Memory). These models are trained and evaluated on a meticulously curated dataset, ensuring diversity and relevance in the synthetic media samples. To facilitate experimentation and reproducibility, the dataset is securely hosted on a reliable platform such as Google Drive. Prior to model training, preprocessing steps including frame extraction are employed to extract essential visual features from the video data. The VGG model serves as a feature extractor, capturing high-level representations of visual content, while the LSTM model learns temporal dependencies and contextual information across frames. Following comprehensive experimentation, the proposed method's ability to detect synthetic media is thoroughly assessed, utilizing metrics such as accuracy. This research contributes to the ongoing discourse on digital media forensics by providing insights into the efficacy of deep learning techniques for synthetic media analysis. The findings underscore the importance of continuous research and development in combating the proliferation of synthetic media, thereby safeguarding the authenticity and trustworthiness of online content. Index Terms—CNN, LSTM, VGG

Enhanced Multi-Task Traffic Forecasting in Beyond 5G Networks: Leveraging Transformer Technology and Multi-Source Data Fusion

Managing cellular networks in the Beyond 5G (B5G) era is a complex and challenging task requiring advanced deep learning approaches. Traditional models focusing on internet traffic (INT) analysis often fail to capture the rich temporal and spatial contexts essential for accurate INT predictions. Furthermore, these models do not account for the influence of external factors such as weather, news, and social trends. This study proposes a multi-source CNN-RNN (MSCR) model that leverages a rich dataset, including periodic, weather, news, and social data to address these limitations. This model enables the capture and fusion of diverse data sources for improved INT prediction accuracy. An advanced deep learning model, the transformer-enhanced CNN-RNN (TE-CNN-RNN), has been introduced. This model is specifically designed to predict INT data only. This model demonstrates the effectiveness of transformers in extracting detailed temporal-spatial features, outperforming conventional CNN-RNN models. The experimental results demonstrate that the proposed MSCR and TE-CNN-RNN models outperform existing state-of-the-art models for traffic forecasting. These findings underscore the transformative power of transformers for capturing intricate temporal-spatial features and the importance of multi-source data and deep learning techniques for optimizing cell site management in the B5G era.

Harnessing large language models over transformer models for detecting Bengali depressive social media text: A comprehensive study

In an era where the silent struggle of underdiagnosed depression pervades globally, our research delves into the crucial link between mental health and social media. This work focuses on early detection of depression, particularly in extroverted social media users, using LLMs such as GPT 3.5, GPT 4 and our proposed GPT 3.5 fine-tuned model DepGPT, as well as advanced Deep learning models(LSTM, Bi-LSTM, GRU, BiGRU) and Transformer models(BERT, BanglaBERT, SahajBERT, BanglaBERT-Base). The study categorized Reddit and X datasets into “Depressive” and “Non-Depressive” segments, translated into Bengali by native speakers with expertise in mental health, resulting in the creation of the Bengali Social Media Depressive Dataset (BSMDD). Our work provides full architecture details for each model and a methodical way to assess their performance in Bengali depressive text categorization using zero-shot and few-shot learning techniques. Our work demonstrates the superiority of SahajBERT and Bi-LSTM with FastText embeddings in their respective domains also tackles explainability issues with transformer models and emphasizes the effectiveness of LLMs, especially DepGPT (GPT 3.5 fine-tuned), demonstrating flexibility and competence in a range of learning contexts. According to the experiment results, the proposed model, DepGPT, outperformed not only Alpaca Lora 7B in zero-shot and few-shot scenarios but also every other model, achieving a near-perfect accuracy of 0.9796 and an F1-score of 0.9804, high recall, and exceptional precision. Although competitive, GPT-3.5 Turbo and Alpaca Lora 7B show relatively poorer effectiveness in zero-shot and few-shot situations. The work emphasizes the effectiveness and flexibility of LLMs in a variety of linguistic circumstances, providing insightful information about the complex field of depression detection models.

Forward Head Posture Classification Using Deep Learning Models on Facial Recognition Data

Forward Head Posture (FHP) refers to a condition where the head protrudes forward, significantly contributing to neck pain and being associated with decreased productivity and psychological distress. This study investigates the nuanced classification of FHP and proposes a universally applicable methodology for its identification and analysis using deep learning. Leveraging the Korean Facial Image (K-FACE) dataset, rigorous image preprocessing with the Yolo-v8 model was conducted to facilitate accurate measurement of the CranioVertebral Angle (CVA) from various perspectives. The study meticulously evaluated the classification effectiveness of three advanced deep learning models: EfficientNet-B7, NFNet-F7, and ResNet-152. Among these, EfficientNet-B7 demonstrated superior performance with an accuracy of 0.69 and a recall score of 0.69 compared to other models. Additionally, comparisons based on camera angles within the EfficientNet-B7 model highlighted its excellence, particularly at the ±75° angle. The importance of image regions in EfficientNet-B7 was confirmed through Grad-CAM analysis, emphasizing the critical role of the neck region in accurately classifying FHP images. This comprehensive performance comparison and the proposed detailed classification methodology underscore the potential for generalization in FHP classification. Furthermore, by leveraging a unique dataset and employing state-of-the-art classification techniques, this research offers a novel perspective on the discourse surrounding FHP. Future research could integrate expanded facial image datasets and apply transfer learning techniques to further enhance the precision of FHP classification, thereby improving diagnostic accuracy and offering targeted interventions for individuals experiencing neck pain associated with FHP.

Optimizing Hyperspectral Image Classification Through Advanced Deep Learning Models Enhanced by Adam

Optimizing Hyperspectral Image Classification Through Advanced Deep Learning Models Enhanced by Adam

Assessing the quality of experience in wireless networks for multimedia applications: A comprehensive analysis utilizing deep learning-based techniques

In the context of the burgeoning progression of wireless network technology and the corresponding escalation in the demand for mobile Internet-based multimedia transmission services, the task of preserving and augmenting user satisfaction has emerged as an imperative concern. This necessitates a sophisticated and accurate evaluation of multimedia service quality within the sphere of wireless networks. To systematically address the nuanced issue of user experience quality, the present study introduces a novel method for evaluating multimedia Quality of Experience (QoE) in wireless networks, employing an advanced deep learning model as the underlying analytical framework. Initially, the research undertakes the task of modeling the video session process, giving due consideration to the status of each temporal interval within the session's architecture. Subsequently, the challenge of QoE prediction is dissected and investigated through the lens of recurrent neural networks (RNNs), culminating in the proposition of an all-encompassing QoE prediction model that harmoniously integrates video information, Quality of Service (QoS) data, user behavior analytics, and facial expression analysis. The empirical segment of this research serves to validate the efficacy of the suggested video QoE evaluation method, engaging both quantitative and qualitative comparison metrics with contemporaneous state-of-the-art QoE models, employing the RTVCQoE dataset as the empirical foundation. The experimental findings illuminate that the QoE model elucidated in this study transcends competing models in performance metrics such as PLCC, SRCC, and KRCC. Consequently, this investigation stands as a seminal contribution to academic literature, furnishing an exacting and dependable QoE evaluation methodology. Such a contribution augments the user experience landscape in multimedia services within wireless networks, and instigates further scholarly exploration and technological innovation in the mobile Internet domain.

PV Identifier: Extraction of small-scale distributed photovoltaics in complex environments from high spatial resolution remote sensing images

The precise location and size of distributed photovoltaics (PVs) is critical to infer the actual installed capacity and assess the remaining PV generation potential, and is therefore the cornerstone of strategic planning for distributed PV deployment. However, identifying small-scale distributed PVs in complex contexts from high spatial resolution remote sensing (HSRRS) images to obtain their information remains an issue. In this study, we propose an advanced deep learning model, called PV Identifier, to enhance the identification accuracy of small-scale PV systems from HSRRS images. PV Identifier uses a fine-grained feature layer (FFL) compatible with the size of PVs to improve the detection capability of the small-scale distributed PVs. At the same time, it effectively distinguishes between PVs and similar background using a novel semantic constraint module (SCM). We test PV Identifier on a distributed PV dataset in California. Experiments show that the inclusion of the FFL positively affects the model's sensitivity to small distributed PVs. Specifically, the PV Identifier with the FFL increases the Recall of identifying residential rooftop PVs by 1.9% compared to the model without the FFL. In addition, the integration of the SCM effectively improves the model's ability to locate residential rooftop PVs in complex environments, resulting in a 1.8% increase in the corresponding Precision. Compared to the four commonly used segmentation models, PV Identifier exhibits superior identification performance for residential rooftop PVs and commercial and industrial PVs, with an Intersection over Union (IoU) of 74.1% and 89.3%, respectively, which is at least 4.1% and 1.8% higher than other models. Overall, PV Identifier provides a viable solution to the problem of identifying small-scale distributed PV in complex backgrounds from HSRRS images.

Talk2Data: A Natural Language Interface for Exploratory Visual Analysis via Question Decomposition

Through a natural language interface (NLI) for exploratory visual analysis, users can directly “ask” analytical questions about the given tabular data. This process greatly improves user experience and lowers the technical barriers of data analysis. Existing techniques focus on generating a visualization from a concrete question. However, complex questions, requiring multiple data queries and visualizations to answer, are frequently asked in data exploration and analysis, which cannot be easily solved with the existing techniques. To address this issue, in this article, we introduce Talk2Data, a natural language interface for exploratory visual analysis that supports answering complex questions. It leverages an advanced deep-learning model to resolve complex questions into a series of simple questions that could gradually elaborate on the users’ requirements. To present answers, we design a set of annotated and captioned visualizations to represent the answers in a form that supports interpretation and narration. We conducted an ablation study and a controlled user study to evaluate the Talk2Data’s effectiveness and usefulness.

A comprehensive review on zero-shot-learning techniques

Advancements in computational capabilities have enabled the implementation of advanced deep learning models across various domains of knowledge, yet the increasing complexity and scarcity of data in specialized areas pose significant challenges. Zero-shot learning (ZSL), a subset of transfer learning, has emerged as an innovative solution to these challenges, focusing on classifying unseen categories present in the test set but absent during training. Unlike traditional methods, ZSL utilizes semantic descriptions, like attribute lists or natural language phrases, to map intermediate features from the training data to unseen categories effectively, enhancing the model’s applicability across diverse and complex domains. This review provides a concise synthesis of the advancements, methodologies, and applications in the field of zero-shot learning, highlighting the milestones achieved and possible future directions. We aim to offer insights into the contemporary developments in ZSL, serving as a comprehensive reference for researchers exploring the potentials and challenges of implementing ZSL-based methodologies in real-world scenarios.

Evolution of Natural Language Processing: A Review

Over the years, Natural Language Processing (NLP) has evolved dramatically, moving from early rule based systems to the current era dominated by advanced deep learning models. An overview of the significant turning points and patterns that have influenced the development of NLP is given in this study. In the early days of Natural Language Processing (NLP), rule based methods were the main focus. Linguists would manually create rules to analyze and comprehend human language. Although these systems were somewhat successful, they were unable to handle the complexity and unpredictability of spoken language. A major change was brought about by the introduction of probabilistic models and machine learning techniques with the emergence of statistical approaches. The development of methods like n gram models and hidden Markov models during this time allowed computers to handle linguistic patterns. Large scale linguistic resources like word embeddings and annotated corpora started to appear, which further accelerated the development of NLP. Machine learning algorithms have led to notable advancements in tasks such as machine translation, named entity recognition, and part of speech tagging. NLP has seen a revolution in recent years thanks to deep learning, which uses neural networks to learn intricate language representations. Sequential dependencies in language can now be better understood because of models like Long Short Term Memory Networks (LSTMs) and Recurrent Neural Networks (RNNs). The addition of attention mechanisms, as demonstrated by Transformer and other models, improves the model's ability to manage long range dependencies and perform better on a variety of NLP tasks. In the future, NLP will develop in ways that go beyond performance measurements, exploring interpretability, ethical issues, and the incorporation of multimodal data. As the area develops, it becomes increasingly important to eliminate biases, ensure ethical AI deployment, and improve user centric experiences. This introduction lays the groundwork for an in depth examination of the development of NLP, highlighting significant turning points, difficulties, and potential future directions in this vibrant and quickly developing subject.

Enhancing Plant Disease Detection using Advanced Deep Learning Models

Objective: The aim of this paper is to analyze the efficacy of employing multiple advanced convolutional neural networks (CNNs) for the purpose of enhancing the accuracy in detecting and classifying various plant diseases. Methods: The research involves the analysis of 7623 training images as well as 1906 validation images of different plant diseases and employed advanced deep learning models like DenseNet169, Xception, InceptionV3, MobileNetV2, and ResNet50V2 to classify them. At first the RGB images are converted to Grayscale and later to enhance their quality, few techniques have been used such as Otsu thresholding, noise removal, distance transform, and watershed techniques. Subsequently, contour features are extracted by calculating morphological values to obtain the necessary region that correspond to diseased areas in plant images. Findings: On evaluating the performance of the applied models on the basis of various metrics, MobileNetV2 and ResNetV2 achieved the highest validation accuracy scores of 99.42% each, with their respective loss values of 0.19 and 0.49. In terms of recall, precision, and F1 score, all models, except MobileNetV2 and InceptionV3, attained optimal scores of 0.99 each. Novelty: The novelty of this paper resides in the incorporation of multiple image segmentation techniques with fine tuning the parameters of advanced Convolutional Neural Network (CNN) models on the basis of various factors such as the number of images, size, channel, classes etc to generate the optimal results. Keywords: Agriculture, Plant diseases, Artificial Intelligence, Advanced CNN models, Watershed Technique

DEEP-EP: Identification of epigenetic protein by ensemble residual convolutional neural network for drug discovery

Epigenetic proteins (EP) play a role in the progression of a wide range of diseases, including autoimmune disorders, neurological disorders, and cancer. Recognizing their different functions has prompted researchers to investigate them as potential therapeutic targets and pharmacological targets. This paper proposes a novel deep learning-based model that accurately predicts EP. This study introduces a novel deep learning-based model that accurately predicts EP. Our approach entails generating two distinct datasets for training and evaluating the model. We then use three distinct strategies to transform protein sequences to numerical representations: Dipeptide Deviation from Expected Mean (DDE), Dipeptide Composition (DPC), and Group Amino Acid (GAAC). Following that, we train and compare the performance of four advanced deep learning models algorithms: Ensemble Residual Convolutional Neural Network (ERCNN), Generative Adversarial Network (GAN), Convolutional Neural Network (CNN), and Gated Recurrent Unit (GRU). The DDE encoding combined with the ERCNN model demonstrates the best performance on both datasets. This study demonstrates deep learning's potential for precisely predicting EP, which can considerably accelerate research and streamline drug discovery efforts. This analytical method has the potential to find new therapeutic targets and advance our understanding of EP activities in disease.

Optimised ResNet50 for Multi-Class Classification of Brain Tumors

Categorizing brain cancers, including glioma, meningioma, and pituitary tumors, based on magnetic resonance imaging (MRI) images presents a significant challenge. Deep learning and machine learning techniques have shown promise in enhancing image categorization. To address this challenge, we leverage the power of the optimized ResNet50 model. Our approach involves classifying medical images using Convolutional Neural Network (CNN) features, which are then compared with the ResNet50 model. The primary goal is to detect brain tumors at an early stage using an advanced deep-learning model. We utilize an accessible dataset from Figshare, containing MRI images of the three distinct categories of brain tumors. Existing brain tumor models face limitations in handling multi-class problems and early-stage diagnosis. Therefore, we propose a fully automated approach employing Convolutional Neural Networks (CNN) to extract diverse properties from brain MRI scans. This method aims to provide accurate tumor diagnosis, even with a high number of classes and limited information in MRI data. Our proposed model involves the creation of identification blocks within a four-layered primary architecture, followed by testing and assessment of the interconnected layers. The results demonstrate that our model outperforms existing methods, achieving an impressive overall classification accuracy of 99.03%,

Enhancing hydrological modeling with transformers: a case study for 24-h streamflow prediction.

In this paper, we address the critical task of 24-h streamflow forecasting using advanced deep-learning models, with a primary focus on the transformer architecture which has seen limited application in this specific task. We compare the performance of five different models, including persistence, long short-term memory (LSTM), Seq2Seq, GRU, and transformer, across four distinct regions. The evaluation is based on three performance metrics: Nash-Sutcliffe Efficiency (NSE), Pearson's r, and normalized root mean square error (NRMSE). Additionally, we investigate the impact of two data extension methods: zero-padding and persistence, on the model's predictive capabilities. Our findings highlight the transformer's superiority in capturing complex temporal dependencies and patterns in the streamflow data, outperforming all other models in terms of both accuracy and reliability. Specifically, the transformer model demonstrated a substantial improvement in NSE scores by up to 20% compared to other models. The study's insights emphasize the significance of leveraging advanced deep learning techniques, such as the transformer, in hydrological modeling and streamflow forecasting for effective water resource management and flood prediction.