Deep Learning Models Research Articles

Enhancing Intrusion Detection Systems with Dimensionality Reduction and Multi-Stacking Ensemble Techniques

The deployment of intrusion detection systems (IDSs) is essential for protecting network resources and infrastructure against malicious threats. Despite the wide use of various machine learning methods in IDSs, such systems often struggle to achieve optimal performance. The key challenges include the curse of dimensionality, which significantly impacts IDS efficacy, and the limited effectiveness of singular learning classifiers in handling complex, imbalanced, and multi-categorical traffic datasets. To overcome these limitations, this paper presents an innovative approach that integrates dimensionality reduction and stacking ensemble techniques. We employ the LogitBoost algorithm with XGBRegressor for feature selection, complemented by a Residual Network (ResNet) deep learning model for feature extraction. Furthermore, we introduce multi-stacking ensemble (MSE), a novel ensemble method, to enhance attack prediction capabilities. The evaluation on benchmark datasets such as CICIDS2017 and UNSW-NB15 demonstrates that our IDS surpasses current models across various performance metrics.

Predicting the Performance of Students Using Deep Ensemble Learning

Universities and schools rely heavily on the ability to forecast student performance, as it enables them to develop efficient strategies for enhancing academic results and averting student attrition. The automation of processes and the management of large datasets generated by technology-enhanced learning tools can facilitate the analysis and processing of these data, which provides crucial insights into the knowledge of students and their engagement with academic endeavors. The method under consideration aims to forecast the academic achievement of students through an ensemble of deep neural networks. The proposed method presents a new feature-ranking mechanism based on existing approaches. This mechanism is effective in identifying the most pertinent features and their correlation with the academic performance of students. The proposed method employs an optimization strategy to concurrently configure and train the deep neural networks within our ensemble system. Furthermore, the proposed ensemble model uses weighted voting among its learning components for more accurate prediction. Put simply, the suggested approach enhances the accuracy of academic performance predictions for students not only by employing weighted ensemble techniques, but also by optimizing the parameters of deep learning models. These experimental outcomes provide evidence that the proposed method outperformed the alternative approaches, accurately predicting student performance with a root-mean-square error (RMSE) value of 1.66, a Mean Absolute Percentage Error (MAPE) value of 9.75, and an R-squared value of 0.7430. These results show a significant improvement compared to the null model (RMSE = 4.05, MAPE = 24.89, and R-squared = 0.2897) and prove the efficiency of the techniques employed in the proposed method.

DLLabelsCT: Annotation tool using deep transfer learning to assist in creating new datasets from abdominal computed tomography scans, case study: Pancreas.

The utilization of artificial intelligence (AI) is expanding significantly within medical research and, to some extent, in clinical practice. Deep learning (DL) applications, which use large convolutional neural networks (CNN), hold considerable potential, especially in optimizing radiological evaluations. However, training DL algorithms to clinical standards requires extensive datasets, and their processing is labor-intensive. In this study, we developed an annotation tool named DLLabelsCT that utilizes CNN models to accelerate the image analysis process. To validate DLLabelsCT, we trained a CNN model with a ResNet34 encoder and a UNet decoder to segment the pancreas on an open-access dataset and used the DL model to assist in annotating a local dataset, which was further used to refine the model. DLLabelsCT was also tested on two external testing datasets. The tool accelerates annotation by 3.4 times compared to a completely manual annotation method. Out of 3,715 CT scan slices in the testing datasets, 50% did not require editing when reviewing the segmentations made by the ResNet34-UNet model, and the mean and standard deviation of the Dice similarity coefficient was 0.82±0.24. DLLabelsCT is highly accurate and significantly saves time and resources. Furthermore, it can be easily modified to support other deep learning models for other organs, making it an efficient tool for future research involving larger datasets.

The CWA Benchmark: A Seismic Dataset from Taiwan for Seismic Research

Abstract In recent years, numerous studies have employed deep learning in seismology, with data-driven neural network models increasingly becoming the norm in emerging research paradigms. Alongside advancements in model architectures, several benchmark datasets have also been introduced. However, many of these datasets suffer from imbalanced distributions, particularly a scarcity of large-magnitude events, which can impair the performance of deep learning-based models. Taiwan, situated in a seismically active region experiences a high frequency of earthquakes annually. The Central Weather Administration (CWA) in Taiwan maintains comprehensive records of these seismic events. This study introduces a benchmark dataset, named CWA, specifically compiled from this extensive database. The CWA benchmark includes 331 events with magnitudes greater than five, collected from a high-density seismic network spanning from 2011 to 2021, making it well-suited for training deep learning models. In addition, the CWA benchmark features over 40 attributes and ∼500,000 seismograms, providing valuable data labels for various seismology-related tasks.

Monitoring Substance Use with Fitbit Biosignals: A Case Study on Training Deep Learning Models Using Ecological Momentary Assessments and Passive Sensing

Monitoring Substance Use with Fitbit Biosignals: A Case Study on Training Deep Learning Models Using Ecological Momentary Assessments and Passive Sensing

Real-Time On-Site Ground-Motion Prediction Using ResNet

Abstract On-site warnings can decrease the range of late alert zone during earthquakes. This study develops a deep learning model to predict whether the maximum peak ground acceleration at a station exceeds 25 Gal based on the waveform of the initial P-wave. A ResNet architecture model is developed to address the degradation problem in multilayered models, thereby enhancing performance. The model exhibits high performance for a test dataset, achieving F1-scores higher than 90% over various time windows, and outperforms the traditional progressive displacement value (Pdv) method and newer convolutional neural network (CNN) methods. Ground motion of two actual earthquakes are applied to test the model and validate its prediction capabilities, and the results indicate that the proposed model has higher accuracy and speed than Pdv and CNN methods. In addition, the model is integrated with the Earthworm earthquake early warning system for real-time waveform analysis and prediction, extending the model’s applicability beyond experimental stages to online systems. The proposed method is also tested using data from earthquakes that occurred in March 2023.

DETECTION OF DUST ON SOLAR PANELS WITH DEEP LEARNING

Solar energy is an environmentally friendly, clean, and sustainable alternative. The widespread use of this energy source offers excellent environmental and economic benefits. However, some factors affect the efficiency of solar panels. One of these factors is dust. When dust accumulates on the surface of solar panels, it can significantly reduce the efficiency of energy production. Therefore, detecting and quickly removing dust from solar panels is crucial. Managing this process with unmanned artificial intelligence systems, especially in large areas, will provide significant advantages in terms of time and cost. In recent years, convolutional neural networks have achieved significant success in image classification. In particular, transfer learning methods have proven their success in this field. In this study, we aim to solve a new task with limited data using pre-trained deep learning models (EfficientNetB3, ResNet50, MobileNet, VGG19, Xception, InceptionResNetV2, VGG16, ResNet101, DenseNet201, EfficientNetB7) to classify dirty and clean solar panels. These models were chosen because they each have different strengths and have performed well on various tasks. The models with the best performance among these models are combined to improve classification prediction. The proposed ensemble learning approach achieved 99.31% classification accuracy by considering the prediction results of the models with a voting approach. As a result, this approach aims to optimize the maintenance processes of solar energy systems, improve energy efficiency, and support sustainable energy use in the long term.

A text-based, generative deep learning model for soil reflectance spectrum simulation in the solar range (400–2499 nm)

Soil spectral reflectance is a necessary input for land surface and radiative transfer models, and can be used to infer soil properties. Numerous soil reflectance inversion models have been developed based on mechanistic approaches, each with their own limitations. Mechanistic models based on radiative transfer theory are usually based on only a few input soil properties, whereas data-driven approaches are limited by high non-uniformity of available published datasets that severely limits the amount of data usable for model calibration. To address these limitations, a fully data-driven soil optics generative model (SOGM) for simulation of soil reflectance spectra from soil property inputs was developed based on the denoising diffusion probabilistic model (DDPM). The model was trained on an extensive dataset comprising nearly 180,000 soil spectra-property set pairs from 17 published datasets. The model generates soil reflectance spectra from text-based inputs describing soil properties and their values rather than only numerical values and labels in binary vector format, which means the model can handle variable formats for property reporting. Because the model is generative, it can simulate reasonable output spectra based on an incomplete set of available input properties, which becomes more reliable as the input property set becomes more complete. Two additional sub-models were also built to complement the SOGM: a spectral padding model that can fill in the gaps for spectra shorter than the target solar range (400 to 2499 nm), and a wet soil spectra model that can estimate the effects of water content on soil reflectance spectra given the dry spectrum predicted by the SOGM. It can also be easily integrated with other soil–plant radiation models used for remote sensing research such as PROSAIL and Helios 3D plant modeling software. The testing results of the SOGM on new datasets not included in model training demonstrated that the model can generate reasonable soil reflectance spectra based on available property inputs. Results also show soil clay/sand/silt fraction, organic carbon content, nitrogen content, and iron content tended to be important properties for spectra simulation. Inclusion of some trace minerals like nickel as model inputs decreased model performance because of their low concentrations and large propensity for ground-truth measurement error.

Application of deep learning models with spectral data augmentation and Denoising for predicting total phosphorus concentration in water pollution

BackgroundWith the increasing severity of global water pollution, accurate prediction models of water pollution content are critical for effective environmental management. However, traditional methods often exhibit low prediction accuracy for pollutant concentrations when data samples are limited and do not adequately address data noise. This study focuses on predicting total phosphorus (TP) concentrations in the Yangtze River Basin by integrating data augmentation and denoising methods with spectral technology and deep learning, using water samples collected from Wuhan to Anhui, China. MethodThe study utilized an improved Conditional Generative Adversarial Networks (CGAN) for data augmentation, increasing dataset diversity and training effectiveness. Adaptive threshold wavelet denoising is applied to reduce noise and improve data quality. A Convolutional Neural Network (CNN) with a coordinate attention (CA) mechanism is used to extract key spectral features linked to TP concentration prediction. Significant FindingsThis study introduces an innovative approach that combines advanced CGAN-based data augmentation, adaptive threshold wavelet denoising, and a CNN model incorporating a CA mechanism, achieving high accuracy in TP concentration prediction. The proposed model outperforms traditional methods, achieving R² = 0.9805, RMSE = 0.0019, and MAE = 0.0009. This novel method significantly enhances prediction performance, providing an effective solution particularly in scenarios with limited data samples.

Enhanced Landslide Spatial Prediction Using Hybrid Deep Learning Model and SHAP Analysis: A Case Study of the Tuyen Quang-Ha Giang Expressway, Vietnam

Enhanced Landslide Spatial Prediction Using Hybrid Deep Learning Model and SHAP Analysis: A Case Study of the Tuyen Quang-Ha Giang Expressway, Vietnam

Advances and clinical Implications of neuroanatomical biomarkers in schizophrenia by optimizing brain tissue segmentation

Schizophrenia (SZ) is a complex mental disorder marked by structural abnormalities in brain regions such as gray matter (GM), white matter (WM), ventricles and the cerebellum. Accurate segmentation of these brain subregions is crucial for diagnosing SZ, but traditional methods often struggle with the complexity of the disorder. To address this, the study employs advanced optimization techniques, specifically Cuckoo Search (CS), Grey Wolf Optimization (GWO) and Improved Grey Wolf Optimization (IGWO), to segment regions extracted from T1-weighted MR images of SZ patients and HC/NC. The IGWO method consistently outperforms others, achieving Structural Similarity Index Measures (SSIM) of 0.98 for ventricles and 0.96 for the cerebellum, with segmentation accuracies of 99% for both regions. Overlapping measures also reflect the effectiveness of proposed method, with DICE coefficients of 0.89 (SZ) and 0.94 (healthy controls) for ventricles and JACCARD indices of 0.80 and 0.95, respectively. For the cerebellum, DICE coefficients were 0.89 (SZ) and 0.93 (healthy controls), with JACCARD indices of 0.79 and 0.88. Subsequent classification using the ResNet50 deep learning model achieved high accuracy rates, particularly for the cerebellum (94%) and ventricles (93%). Performance metrics including recall, specificity, precision and F1-score further validate the robustness of the proposed method. These results highlight the significant role of the cerebellum and ventricles in SZ progression and offer an enhanced approach to improving diagnostic accuracy.

Classifying cognitive impairment based on FDG-PET and combined T1-MRI and rs-fMRI: An ADNI study.

Mild cognitive impairment (MCI) refers to a memory impairment among non-demented adults. It is a condition that increases the risk of dementia, notably due to Alzheimer's disease (AD). MCI is heterogeneous and there is a need for novel diagnostic approaches. Fluorodeoxyglucose positron emission tomography (FDG-PET) imaging provides robust AD biomarker characteristics, while anatomical and functional magnetic resonance imaging (MRI) offer complementary information. Classify MCI and cognitively normal (CN) adults using FDG-PET images; predict individuals with MCI that convert to AD dementia; determine if MRI can achieve comparable performance to FDG-PET classification. Four ADNI cohorts were created. Cohort 1: 805 participants (MCI n = 455; CN n = 350) that underwent FDG-PET. FDG-PET images were inputs to a one-channel 3-dimensional (3D) DenseNet deep learning model. Cohort 2: 348 participants (MCI n = 174; CN n = 174) with MRI and functional MRI. Cohort 3: overlapping cases from cohorts 1 and 2 (MCI n = 70; CN n = 70). Cohort 4: 336 participants (MCI-converters n = 168; MCI-stable n = 168) with FDG-PET from cohort 1. The one/two-channel models' inputs were T1-weighted MRI and/or amplitude of low-frequency fluctuations images, with classification metrics evaluated through 10-fold cross-validation. The FDG-PET model achieved 88.02%±3.82 accuracy for MCI versus CN classification, with 88.70%±4.70 sensitivity and 87.14%±5.03 specificity. Neither MRI model outperformed the FDG-PET model, as the highest MRI-based accuracy was 76.86%±1.95. The FDG-PET model achieved 63.23%±4.68 accuracy in classifying MCI-converters versus MCI-stable. FDG-PET images produced the highest accuracy in classifying MCI versus CN. While MRI-based approaches were inferior to FDG-PET, multi-contrast MRI still offers value for neurodegeneration classification.

An Efficient Model for Lung Cancer Detection through the Integration of Genetic Algorithm and Machine Learning

Prompt lung cancer detection is essential for patient health. Deep Learning (DL) models have been intensively used for lung cancer screening, as they provide high accuracy in diagnoses. However, DL models require significant computational power, which may not be accessible in all settings. Conventional Machine Learning (ML) models may not produce high prediction accuracy, especially with large data. This study uses a Genetic Algorithm (GA) approach to select optimal features from lung cancer images and reduce their dimensionality. This allows conventional ML models to achieve a high prediction accuracy when classifying medical images while using lower computational power compared with DL models. The proposed model integrates GA along with ML for lung cancer detection. The experimental results show that using GA with a feed-forward neural network classifier achieved high performance, reaching 99.70% classification accuracy.

Wayang’s Images Recognition using Vision Transformer

Due to its complex nature and outdated perception, Wayang is a traditional Indonesian art form influenced by Hindu-Buddhism. However, it is difficult for the younger generation to recognize the various types of Wayang. In an effort to preserve Wayang culture, this study evaluates the performance of four deep learning models in recognizing types of Wayang namely, Vision Transformer (ViT), ResNet34, YOLOv5-cls, and YOLOv8-cls. These models were trained and assessed using a dataset of 232 images representing six Wayang types and using matrix such as accuracy, recall, precision, and F1 score. ViT demonstrated efficiency and adaptability despite high computational requirements, achieving the best accuracy (91.3%), showing high adaptability despite substantial computational requirements. Meanwhile, YOLOv5-cls and YOLOv8-cls offered a good balance betwwen accuracy and efficiency. This study suggest that deep learning models can play an essentialrole in Wayang by enhancing recognition accessibility, thus helping younger generations appreciate this tradisional art form.

An intrinsic evaluator for embedding methods in malicious URL detection

Nowadays, machine learning is used in many fields. Not only in fields such as image recognition, machine learning is also used for malicious detection. Especially in recent years, there have been many studies using machine learning for malicious URL detection to replace traditional blacklists. In order to compare the performance of the malicious URLs detection method, researches used the F-score or other detection accuracy to evaluate, but there are some difficulties in evaluating the URL embedding method used in malicious URLs detection because the detection accuracy is also effect by machine learning or deep learning models and data sets. An evaluation method of URL embedding method that is not affected by other factors is particularly important. In this paper, we proposed an intrinsic evaluation method for URL embedding method that is not affected by machine learning models or deep learning models and data sets. Besides, We analyse some URL embedding methods according to intrinsic and extrinsic methods and offer a guidance in selecting suitable embedding methods in URL by analysing the results.

Deep-learning surrogate models for the stability of a wide rectangular tunnel

The stability of a wide rectangular tunnel in spatially variable soils is investigated using deep-learning surrogate models combined with a mixed formulation of limit analysis. The mixed formulation is employed to perform stochastic simulations of tunnels in spatially random soils, generating the data to train the surrogate models. Two types of deep-learning models are tested: Fully Connected Neural Networks (FCNs) and Convolutional Neural Networks (CNNs). To enhance model performance, the study introduces two novel encoding techniques to enhance the identification of tunnel regions in the input data. The first, Mask-FCN, directly incorporates spatial information into the model by adding a binary mask layer to distinguish tunnel regions from the surrounding soil. The second, Grad-FCN, modifies training gradients to indirectly highlight these differences. Additionally, a Mask-CNN model is developed, leveraging spatial filtering to better capture local patterns and correlations. Our results show that the surrogate models, trained on data from stochastic reliability analyses, demonstrate robustness and accuracy across various evaluation metrics. The encoding neural networks outperform the baseline model in accuracy and overall regression performance on training datasets. Importantly, both encoding networks achieve acceptable accuracy on unseen datasets with new tunnel geometric features, a task the baseline model fails to accomplish. The FCN-based encoding neural network achieves prediction capabilities comparable to ordinary CNNs, while the CNN-based encoding neural network delivers high prediction accuracy for unknown datasets with new tunnel geometric characteristics. The study concludes that the generated surrogate models offer a computationally efficient alternative for predicting tunnel stability in spatially variable soils.

Architecture-Aware Augmentation: A Hybrid Deep Learning and Machine Learning Approach for Enhanced Parkinson’s Disease Detection

Parkinson’s Disease (PD) is a progressive neurodegenerative disorder affecting millions worldwide. Early detection is crucial for improving patient outcomes. Spiral drawing analysis has emerged as a non-invasive tool to detect early motor impairments associated with PD. This study examines the performance of hybrid deep learning and machine learning models in detecting PD using spiral drawings, with a focus on the impact of data augmentation techniques. We compare the accuracy of Vision Transformer (ViT) with K-Nearest Neighbors (KNN), Convolutional Neural Networks (CNN) with Support Vector Machines (SVM), and Residual Neural Networks (ResNet-50) with Logistic Regression, evaluating their performance on both augmented and non-augmented data. Our findings reveal that ViT with KNN, initially achieving 96.77% accuracy on unaugmented data, experienced a notable decline across all augmentation techniques, suggesting it relies heavily on global patterns in spiral drawings. In contrast, ResNet-50 with Logistic Regression showed consistent improvement with data augmentation, reaching 93.55% accuracy when rotation and flipping techniques were applied. These results highlight that hybrid models respond differently to augmentation, and careful selection of augmentation strategies is necessary for optimizing model performance. Our study provides important insights into the development of reliable diagnostic tools for early PD detection, emphasizing the need for appropriate augmentation techniques in medical image analysis.

ResNet-based Gender Recognition on Hand Images

The use of biometric features for the surveillance and recognition of certain classes, such as gender, age, and race, is widespread and popular among researchers. Various studies have focused on gender recognition using facial, gait, or audial features. This study aimed to recognize people's gender by analyzing their hand images using a deep learning model. Before training, the images were subjected to several preprocessing stages. In the first stage, the joint points on either side of the hand were detected using the MediaPipe framework. Using the detected points, the orientation of the hands was corrected and rotated so that the fingers pointed upwards. In the last preprocessing stage, the images were smoothened while the edges were preserved by a guided filter. The processed images were used to train and test different versions of the ResNet model. The results were compared with those of some other studies on the same dataset. The proposed method achieved 96.67% recognition accuracy.

To what extent has machine learning achieved in predicting online at-risk students? Evidence from quantitative meta-analysis

Machine learning and data mining techniques hold promise in predicting at-risk students in online learning. This meta-analysis aimed to provide quantitative evidence to validate whether and to what extent machine learning techniques have been achieved in identifying online at-risk students. Meta-regressions examined the impacts of predictor data types, classical or deep learning approaches, and prediction stages on performance. A random-effects meta-analysis of 47 studies with 309 models showed good classification accuracy, higher in summative than early predictions. Deep learning models and diverse predictors can significantly enhance model performance. No publication bias was detected. Implications and recommendations for practice are discussed.

Improving the Security of the LCB Block Cipher Against Deep Learning-Based Attacks

This study presents a robust version of Light Cipher Block (LCB) by addressing the vulnerabilities identified in previous versions. The vulnerabilities in LCB, including a linear S-Box, improper bit shuffling, and subkey reusability, were thoroughly examined. To overcome these weaknesses, a modified version called Secure LCB is introduced, incorporating changes to the P-Box and key generation algorithm. Motivated by Gohr’s work at CRYPTO’19, this paper investigates the use of a neural distinguisher built upon a 1-dimensional convolutional neural network (1-d CNN). The deep learning model is tasked with identifying ciphertexts that have a specific, controlled difference in their inputs, as opposed to those with random input differences. The evaluation of the proposed Secure LCB using the neural distinguisher suggests that the modifications made to LCB have effectively enhanced its resistance against the neural distinguisher’s classification. This highlights the importance of addressing vulnerabilities in cryptographic systems and showcases the potential of machine learning techniques in cryptanalysis.