Deep Learning Components Research Articles

Investigating neural networks with groundwater flow equation loss

Physics-Informed Neural Networks (PINNs) are considered a powerful tool for solving partial differential equations (PDEs), particularly for the groundwater flow (GF) problem. In this paper, we investigate how the deep learning (DL) architecture, within the PINN framework, is connected to the ability to compute a more or less accurate numerical GF solution, so the link ‘PINN architecture - numerical performance’ is explored. Specifically, this paper explores the effect of various DL components, such as different activation functions and neural network structures, on the computational framework. Through numerical results and on the basis of some theoretical foundations of PINNs, this research aims to improve the explicability of PINNs to resolve, in this case, the one-dimensional GF equation. Moreover, our problem involves source terms described by a Dirac delta function, providing insights into the role of DL architecture in solving complex PDEs.

Simulation-based inference for parameter estimation of complex watershed simulators

Abstract. High-resolution, spatially distributed process-based (PB) simulators are widely employed in the study of complex catchment processes and their responses to a changing climate. However, calibrating these PB simulators using observed data remains a significant challenge due to several persistent issues, including the following: (1) intractability stemming from the computational demands and complex responses of simulators, which renders infeasible calculation of the conditional probability of parameters and data, and (2) uncertainty stemming from the choice of simplified representations of complex natural hydrologic processes. Here, we demonstrate how simulation-based inference (SBI) can help address both of these challenges with respect to parameter estimation. SBI uses a learned mapping between the parameter space and observed data to estimate parameters for the generation of calibrated simulations. To demonstrate the potential of SBI in hydrologic modeling, we conduct a set of synthetic experiments to infer two common physical parameters – Manning's coefficient and hydraulic conductivity – using a representation of a snowmelt-dominated catchment in Colorado, USA. We introduce novel deep-learning (DL) components to the SBI approach, including an “emulator” as a surrogate for the PB simulator to rapidly explore parameter responses. We also employ a density-based neural network to represent the joint probability of parameters and data without strong assumptions about its functional form. While addressing intractability, we also show that, if the simulator does not represent the system under study well enough, SBI can yield unreliable parameter estimates. Approaches to adopting the SBI framework for cases in which multiple simulator(s) may be adequate are introduced using a performance-weighting approach. The synthetic experiments presented here test the performance of SBI, using the relationship between the surrogate and PB simulators as a proxy for the real case.

Integrated Genetic Algorithm and Deep Learning Approach for Effective Cyber-Attack Detection and Classification in Industrial Internet of Things (IIoT) Environments

AbstractCyber-attack detection within Industrial Internet of Things (IIoT) environments presents unique challenges due to the complex, resource-constrained, and real-time nature of these networks. Traditional detection techniques often struggle to adapt to the dynamic environment of IIoT. For instance, many existing methods rely on signature-based detection, which fails to identify evolving threats. Other approaches, such as anomaly-based detection, can generate a high rate of False Positives, leading to inefficiencies in threat management. To address these challenges, we propose a novel detection and classification model specifically tailored for IIoT environments. The proposed model integrates Genetic Algorithms (GA) and Deep Learning (DL) to enhance cyber-attack detection within IIoT environments. The GA component optimises feature selection from raw network data, ensuring the extraction of meaningful and relevant features. Leveraging these selected features, the DL component constructs a robust model capable of accurately detecting and classifying various cyber-attack patterns across IIoT devices. Through experimentation on real-world IIoT network traffic (UNSW-NB 15 dataset), the proposed approach demonstrates its efficacy in improving attack detection accuracy and adaptability. The integration of GA and DL offers a synergistic solution that addresses the complexities of IIoT cybersecurity, contributing to a more secure and resilient IIoT ecosystem. The integrated GA–DL classification model developed in this work achieved 98% precision, 96% accuracy, 94% recall, and 12% losses with only less than 50% of the features of the UNSW-NB 15 dataset. The reduction in features required for the identification and classification of cyber-attacks reduces the processing time by 50%.

Conformations of KRAS4B Affected by Its Partner Binding and G12C Mutation: Insights from GaMD Trajectory-Image Transformation-Based Deep Learning.

Binding of partners and mutations highly affects the conformational dynamics of KRAS4B, which is of significance for deeply understanding its function. Gaussian accelerated molecular dynamics (GaMD) simulations followed by deep learning (DL) and principal component analysis (PCA) were carried out to probe the effect of G12C and binding of three partners NF1, RAF1, and SOS1 on the conformation alterations of KRAS4B. DL reveals that G12C and binding of partners result in alterations in the contacts of key structure domains, such as the switch domains SW1 and SW2 together with the loops L4, L5, and P-loop. Binding of NF1, RAF1, and SOS1 constrains the structural fluctuation of SW1, SW2, L4, and L5; on the contrary, G12C leads to the instability of these four structure domains. The analyses of free energy landscapes (FELs) and PCA also show that binding of partners maintains the stability of the conformational states of KRAS4B while G12C induces greater mobility of the switch domains SW1 and SW2, which produces significant impacts on the interactions of GTP with SW1, L4, and L5. Our findings suggest that partner binding and G12C play important roles in the activity and allosteric regulation of KRAS4B, which may theoretically aid in further understanding the function of KRAS4B.

Three‐layer deep learning network random trees for fault detection in chemical production process

AbstractWith the development of technology, the chemical production process is becoming increasingly complex and large‐scale, making fault detection particularly important. However, current detection methods struggle to address the complexities of large‐scale production processes. In this paper, we integrate the strengths of deep learning and machine learning technologies, combining the advantages of bidirectional long‐ and short‐term memory neural networks, fully connected neural networks, and the extra trees algorithm to propose a novel fault detection model named three‐layer deep learning network random trees (TDLN‐trees). First, the deep learning component extracts temporal features from industrial data, combining and transforming them into a higher‐level data representation. Second, the machine learning component processes and classifies the features extracted in the first step. An experimental analysis based on the Tennessee Eastman process verifies the superiority of the proposed method.

Deep Learning Methods Utilization in Mechanical Property of Medium‐Mn Steel

This study presents an innovative method to predict the mechanical properties of medium‐Mn steel by deep learning (DL). Based on datasets, an artificial neural network (ANN) model serves as a crucial component of DL, demonstrating a coefficient of determination of 0.996, which indicates high accuracy between experimental and predicted values. Meanwhile, the contents of Mn and C, as well as Al and intercritical annealing (IA) conditions, have higher permutation feature importance (PFI) scores, which are 22.37% and 49.22%, respectively. An ANN model predicts that the experimental steel has good mechanical properties with IA at 710 °C for 60 min, with predicted values of ultimate tensile strength (UTS) and total elongation (TE) being 925 MPa and 45.3%, respectively. The experimental values for UTS (937 MPa) and TE (44.8%) closely correspond to the predicted results. The absolute errors between the experimental and predicted UTS and TE are 1.2% and 1.1%, respectively.

Study of Deep Learning in Medical Education: Opportunities, Achievements and Future Challenges.

In this era of progress, interest has developed regarding advancing deep learning (DL) in medicine. However, there has been reluctance to use deep learning, particularly among medical educators. The limitations of previous research were examined in this study, along with the extent to which DL can be used in medical education and its potential impact on educational quality. We were interested in discussing DL's prospects, and determining whether we could benefit from it in medical education. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol was used to manage this review procedure. Six databases were searched carefully to obtain relevant studies. Our search identified 981 articles from the database based on our standards. After filtering the duplicated articles, 11 studies were included in the systematic review. The results showed that DL applications attracted researchers' attention in the medical and education technology owing to their effectiveness to provide the personalized assistance and feedback. Furthermore, the majority of research concentrated on teaching medical students how to utilize DL applications in the classroom, and all of them tried to improve medical students' proficiency with DL instruments in practical applications. Deep learning components in medical learning environments have two segments-in the educational settings like speech recognition or Video content analysis for affecting students' learning, and in the medical settings, applying deep learning from diagnosis to prevention. An integration of them can work better in medical education. Medical education uses DL to improve the students' education. DL is a powerful instrument which has become more famous in terms of superb outcomes. Besides, using DL in medical education is likely to continue as a hotly debated area of research and a well-known classroom strategy.

A novel approach for credit card fraud transaction detection using deep reinforcement learning scheme.

Online transactions are still the backbone of the financial industry worldwide today. Millions of consumers use credit cards for their daily transactions, which has led to an exponential rise in credit card fraud. Over time, many variations and schemes of fraudulent transactions have been reported. Nevertheless, it remains a difficult task to detect credit card fraud in real-time. It can be assumed that each person has a unique transaction pattern that may change over time. The work in this article aims to (1) understand how deep reinforcement learning can play an important role in detecting credit card fraud with changing human patterns, and (2) develop a solution architecture for real-time fraud detection. Our proposed model utilizes the Deep Q network for real-time detection. The Kaggle dataset available online was used to train and test the model. As a result, a validation performance of 97.10% was achieved with the proposed deep learning component. In addition, the reinforcement learning component has a learning rate of 80%. The proposed model was able to learn patterns autonomously based on previous events. It adapts to the pattern changes over time and can take them into account without further manual training.

Detecting Cyber-attacks in the Industrial Internet of Things using a Hybrid Deep Random Neural Network

Critical infrastructure now faces greater vulnerabilities and a higher risk of cyberattacks as a result of the (IIoT) quick expansion. The security and dependability of industrial systems must be ensured by identifying and thwarting these threats. In this paper, we use a hybrid approach of deep learning and RNN called hybrid deep random neural network (HDRNN) to offer a novel method of identifying cyber-attacks in the IIoT.The proposed HDRNN model combines the benefits of random neural networks with deep learning to improve the detection of IIoT cyberattacks. The deep learning component makes use of deep neural networks' capacity to extract intricate features from unstructured data, while the random neural network component offers robustness and adaptability to manage changing attack patterns.Realistic threats and benchmark datasets such as UNSW-NB15and DS2OS are used in experimental evaluations. High accuracy, precision, and recall rates are attained by the model, which successfully detects a variety of assaults including infiltration, data manipulation, and denial of service.The suggested HDRNN model offers a promising approach for improving the security of IIoT systems by precisely identifying cyber-attacks in real-time. The model's hybrid nature enables enhanced detection capabilities, adaptability to changing attack patterns, and a reduction in false positives, enabling efficient threat mitigation and protecting crucial infrastructure in the IIoT context.

Artificial-Intelligence-Enhanced Analysis of In Vivo Confocal Microscopy in Corneal Diseases: A Review.

Artificial intelligence (AI) has seen significant progress in medical diagnostics, particularly in image and video analysis. This review focuses on the application of AI in analyzing in vivo confocal microscopy (IVCM) images for corneal diseases. The cornea, as an exposed and delicate part of the body, necessitates the precise diagnoses of various conditions. Convolutional neural networks (CNNs), a key component of deep learning, are a powerful tool for image data analysis. This review highlights AI applications in diagnosing keratitis, dry eye disease, and diabetic corneal neuropathy. It discusses the potential of AI in detecting infectious agents, analyzing corneal nerve morphology, and identifying the subtle changes in nerve fiber characteristics in diabetic corneal neuropathy. However, challenges still remain, including limited datasets, overfitting, low-quality images, and unrepresentative training datasets. This review explores augmentation techniques and the importance of feature engineering to address these challenges. Despite the progress made, challenges are still present, such as the "black-box" nature of AI models and the need for explainable AI (XAI). Expanding datasets, fostering collaborative efforts, and developing user-friendly AI tools are crucial for enhancing the acceptance and integration of AI into clinical practice.

A Multi-View learning approach to enhance automatic 12-Lead ECG diagnosis performance

ObjectivesThe electrocardiogram (ECG) has important clinical value for the early diagnosis of cardiovascular diseases. Recently, the performance of existing diagnosis models based on ECG data has improved with the introduction of deep learning (DL). However, the impact of various combinations of multiple DL components and/or the role of augmentation techniques on the diagnosis has not been sufficiently investigated in this field. In this sense, this study aims to design an integrated model consisting of diverse DL-based modules. MethodsHere, an ensemble-based multi-view learning approach with an ECG augmentation technique is proposed to achieve higher performance than traditional automatic 12-lead ECG diagnosis methods. ResultsAccordingly, several experiments have been conducted with CPSC2018 dataset for evaluation. The proposed model reports an F1 score of 0.840, which outperforms existing state-of-the-art methods in the literature. ConclusionThus, this study provides quantitative evidence demonstrating that the multi-view learning approach can be used as a unified algorithmic method in the field of automatic ECG diagnosis. SignificanceIn this work, we reviewed ensembles of models consisting of multiple DL components for automatic ECG diagnosis and proposed the ECG-specific augmentation technique.

Contribution Analysis of Large Language Models and Data Augmentations for Person Names in Solving Legal Bar Examination at COLIEE 2023

This paper describes our system for COLIEE 2023 Task 4, which automatically answers Japanese legal bar exam problems. We propose an extension to our previous system in COLIEE 2022, which achieved the highest accuracy among all submissions using data augmentation. We focus on problems that include mentions of person names. In this paper, we present two main contributions. First, we incorporate LUKE as our deep learning component, which is a named entity recognition model trained on RoBERTa. Second, we fine-tune the pretrained LUKE model in multiple ways, comparing fine-tuning on training datasets that include alphabetical person names and ensembling different fine-tuning models. We confirmed that LUKE and its fine-tuned model on person type problems improve their accuracies. Our formal run results show that LUKE and our fine-tuning approach using alphabetical person names were effective, achieving an accuracy of 0.69 in the COLIEE 2023 Task 4 formal run.

Research on facial recognition system based on deep learning

This essay explores the integration of deep learning techniques in facial recognition, delving into the advancements, challenges, and prospects in future of this transformative technology. Deep learning, a subset of artificial intelligence, has revolutionized facial recognition by enabling automatic feature learning and robust representations from raw facial data. This paper examines the key components of deep learning in facial recognition, including feature extraction, facial alignment, and face recognition. It highlights the advantages of using deep learning over traditional methods by analysing related research. It also addresses the challenges deep learning faces in facial recognition, such as data bias, privacy concerns, and adversarial attacks. The essay also explores the promising future of deep learning in facial recognition, with ongoing research aiming to improve robustness, fairness, and privacy preservation.

The Improved U-STFM: A Deep Learning-Based Nonlinear Spatial-Temporal Fusion Model for Land Surface Temperature Downscaling

The thermal band of a satellite platform enables the measurement of land surface temperature (LST), which captures the spatial-temporal distribution of energy exchange between the Earth and the atmosphere. LST plays a critical role in simulation models, enhancing our understanding of physical and biochemical processes in nature. However, the limitations in swath width and orbit altitude prevent a single sensor from providing LST data with both high spatial and high temporal resolution. To tackle this challenge, the unmixing-based spatiotemporal fusion model (STFM) offers a promising solution by integrating data from multiple sensors. In these models, the surface reflectance is decomposed from coarse pixels to fine pixels using the linear unmixing function combined with fractional coverage. However, when downsizing LST through STFM, the linear mixing hypothesis fails to adequately represent the nonlinear energy mixing process of LST. Additionally, the original weighting function is sensitive to noise, leading to unreliable predictions of the final LST due to small errors in the unmixing function. To overcome these issues, we selected the U-STFM as the baseline model and introduced an updated version called the nonlinear U-STFM. This new model incorporates two deep learning components: the Dynamic Net (DyNet) and the Chang Ratio Net (RatioNet). The utilization of these components enables easy training with a small dataset while maintaining a high generalization capability over time. The MODIS Terra daytime LST products were employed to downscale from 1000 m to 30 m, in comparison with the Landsat7 LST products. Our results demonstrate that the new model surpasses STARFM, ESTARFM, and the original U-STFM in terms of prediction accuracy and anti-noise capability. To further enhance other STFMs, these two deep-learning components can replace the linear unmixing and weighting functions with minor modifications. As a deep learning-based model, it can be pretrained and deployed for online prediction.

A Multi-level ensemble approach for skin lesion classification using Customized Transfer Learning with Triple Attention.

Skin lesions encompass a variety of skin abnormalities, including skin diseases that affect structure and function, and skin cancer, which can be fatal and arise from abnormal cell growth. Early detection of lesions and automated prediction is crucial, yet accurately identifying responsible regions post-dominance dispersion remains a challenge in current studies. Thus, we propose a Convolutional Neural Network (CNN)-based approach employing a Customized Transfer Learning (CTL) model and Triple Attention (TA) modules in conjunction with Ensemble Learning (EL). While Ensemble Learning has become an integral component of both Machine Learning (ML) and Deep Learning (DL) methodologies, a specific technique ensuring optimal allocation of weights for each model's prediction is currently lacking. Consequently, the primary objective of this study is to introduce a novel method for determining optimal weights to aggregate the contributions of models for achieving desired outcomes. We term this approach "Information Gain Proportioned Averaging (IGPA)," further refining it to "Multi-Level Information Gain Proportioned Averaging (ML-IGPA)," which specifically involves the utilization of IGPA at multiple levels. Empirical evaluation of the HAM1000 dataset demonstrates that our approach achieves 94.93% accuracy with ML-IGPA, surpassing state-of-the-art methods. Given previous studies' failure to elucidate the exact focus of black-box models on specific regions, we utilize the Gradient Class Activation Map (GradCAM) to identify responsible regions and enhance explainability. Our study enhances both accuracy and interpretability, facilitating early diagnosis and preventing the consequences of neglecting skin lesion detection, thereby addressing issues related to time, accessibility, and costs.

The hybrid of BERT and deep learning models for Indonesian sentiment analysis

<span>Artificial intelligence (AI) is one example of how data science innovation has advanced quickly in recent years and has greatly improved human existence. Neural networks, which are a type of machine learning model, are a fundamental component of deep learning in the field of AI. Deep learning models can carry out feature extraction and classification tasks in a single design because of their numerous neural network layers. Modern machine learning algorithms have been shown to perform worse than this model on tasks including text classification, audio recognition, imaginary, and pattern recognition. Deep learning models have outperformed AI-based methods in sentiment analysis and other text categorization tasks. Text data can originate from a number of places, including social media. Sentiment analysis is the computational examination of textual expressions of ideas and feelings. This study employs the convolutional neural network (CNN), long-short term memory (LSTM), CNN-LSTM, and LSTM-CNN models in a deep learning framework using bidirectional encoder representations from transformers (BERT) data representation to assess the performance of machine learning. The implementation of the model utilises YouTube discussion data pertaining to political films associated with the Indonesian presidential election of 2024. Confusion metrics, including as accuracy, precision, and recall, are then used to analyse the model’s performance.</span>

Analysis of Optimization Techniques in 6D Pose Estimation Approaches using RGB Images on Multiple Objects with Occlusion

6D pose estimation is very important for supporting future smart technologies. The previous methods show optimal performance on RGB-D images or single objects. However, the problem still occurs in RGB images or multiple objects with occlusion. This study focuses on solving the problem using a deep learning approach. One of the key components of deep learning is the optimization process, which we research to determine its effect on solving the problem. The research methodology includes implementing the optimization techniques in the methods, measuring loss value, measuring performance, observing experimental results, analyzing statistical significance, and comparing the performance of optimizers. We implement Adam, RMSprop, Adagrad, Adadelta, and SGD optimizers and analyze their effects on the EfficientPose and DPOD methods. We use the LineMod-Occluded dataset to measure the performance of the methods using the ADD metric. According to the experiment, the loss value is low and stable in the experimental scenarios with a number of epochs between 200 and 500. The performance is relatively high in those scenarios, where Adadelta's performance outperforms other optimizers on both methods. Based on the analysis of variance, the effect of optimizers on the performance of the methods is low, but the slight performance increase is significant in this case.

IoT-based automatic deep learning model generation and the application on Empty-dish Recycling Robots

As a modern computer vision technology, deep learning (DL) revolutionizes our lives and reshapes our world with its high performance. Usually, to accomplish each task, researchers need to collect the dataset and modify the basic DL model to optimize the model. In addition, collecting the dataset is time-consuming, and building separate Artificial Intelligence (AI) models for each task is significantly ineffective Therefore, AI research faces challenges in realizing effective data collection and DL model generation. To solve these problems, we design an Internet of Things (IoT)-based automatic DL model generation strategy which also includes an IoT-based data collection method. In detail, we design an Android application that captures images and sends them to a cloud server to create a dataset. Additionally, we apply a two-level checking function to filter out anomalous data and ensure the dataset’s correctness. Next, the cloud server uses the dataset to train and generate a DL model automatically. We have prepared the state-of-the-art DL components in the cloud server and propose an automatic DL model creation process for model generation. We have applied the proposal to Empty-dish Recycling Robots for demonstration and evaluation purposes to easily understand the proposal and measure the performance. The experimental results show that the system successfully collects the dataset and automatically generates the DL model. Furthermore, the checking function deployed on Android devices requires only 0.84 MB and achieves 99.86% accuracy. During training, the time spent on each automatic DL model generation is evidently decreased by about 9.00% to 28.00%.

A Domain Knowledge Transformer Model for Occupation Profiling

Occupation profiling is a subtask of authorship profiling that is broadly defined as an analysis of individuals’ writing styles. Although the problem has been widely explored, no previous studies have attempted to identify Chinese classical poetry. Inspired by Trudgill’s seminal work on stylistic variation as a function of occupation, we present a novel Domain-Knowledge Transformer model to predict a poet’s occupation through their poems’ writing styles. Different from other Indo-European languages, Chinese has rarely used characters and two types of writing forms: traditional Chinese and simplified Chinese. To tackle these problems, we use the language-related component to standardize our input. We also use alphabetization to satisfy the restrictions on rhyming rules and tonal styles. As a special literal form, traditional domain knowledge, for example, named entities, themes, ages and the official career path, is valuable for poet occupation profiling. However, due to the lack of appropriate annotation datasets, it is difficult to recognize these features. Therefore, we proposed the domain knowledge component employing the latent Dirichletal location model to capture the extra theme information and establish named entity dictionaries to recognize the named entity of the datasets in this study. Finally, in the deep learning component, we combine Transformer with a convolutional neural network (CNN) model to perform occupation profiling. The experimental results suggest that our model is effective in this task. Moreover, the results demonstrate an account of other social attribution features of poetry style that are predictive of occupation in this domain.

Forecasting Bitcoin prices using artificial intelligence: Combination of ML, SARIMA, and Facebook Prophet models

In recent years, investors, corporations, and enterprises have shown great interest in the Bitcoin network; thus, promoting its products and services is crucial. This study utilizes an empirical analysis for financial time series and machine learning to perform prediction of bitcoin price and Garman-Klass (GK) volatility using Long Short-Term Memory (LSTM), Seasonal Autoregressive Integrated Moving Average (SARIMA), and Facebook prophet models. The performance findings show that the LTSM boost has a noticeable improvement compared to SARIMA and Facebook Prophet in terms of MSE (Mean Squared Error) and MAE (Mean Average Error). Unlike Long Short-Term Memory (LSTM), a component of Deep Learning (DL), the finding explains why the bitcoin and its volatility forecasting difficulty has been partially met by traditional time series forecasting (SARIMA) and auto-machine-learning technique (Fb-Prophet). Furthermore, the finding confirmed that Bitcoin values are extremely seasonally volatile and random and are frequently influenced by external variables (or news) such as cryptocurrency laws, investments, or social media rumors. Additionally, results show a robust optimistic trend, and the days when most people commute are Monday and Saturday and an annual seasonality. The trend of the price and volatility of bitcoin using SARIMA and FB-Prophet is more predictable. The Fb-Prophet cannot easily fit within the Russian-Ukrainian conflict period, and in some COVID-19 periods, its performance will suffer during the turbulent era. Moreover, Garman-Klass (GK) forecasting seems more effective than the squared returns price measure, which has implications for investors and fund managers. The research presents innovative insights pertaining to forthcoming cryptocurrency regulations, stock market dynamics, and global resource allocation.