Data-driven Deep Learning Model Research Articles

A novel transfer learning strategy for wind power prediction based on TimesNet-GRU architecture

Currently, data-driven deep learning models are widely applied in the field of wind power prediction. However, when historical data are insufficient, deep learning models struggle to exhibit satisfactory predictive performance. In order to overcome the issue of limited training data for new wind farms, this study proposes a novel transfer learning strategy to address the challenge of less-sample learning in short-term wind power prediction. The research is conducted in two stages. In the pre-training stage, the TimesNet-GRU prediction model is established using data from a source wind farm. Parallel TimesNet modules are employed to extract multi-period features from various input feature sequences, followed by the extraction of long- and short-term features from the time series through gate recurrent unit (GRU). In the transfer learning stage, an effective transfer strategy is designed to freeze and retrain certain parameters of the TimesNet-GRU, thereby constructing a prediction model for the target wind farm. To validate the effectiveness of this approach, the results from testing with actual data from five wind farms in northwest China demonstrate that the proposed method exhibits significant advantages over models without transfer learning as explored in this study.

A comparative assessment of CFD based LSTM and GRU for hydrodynamic predictions of gas-solid fluidized bed

Numerical simulation of large-scale fluidized beds presents significant challenges for both scaling up the simulations and evaluating their performance. However, full-scale computational fluid dynamics (CFD) simulations of these reactors incur substantial computational time and cost. To overcome these challenges, we propose a novel approach that combines CFD simulations with data-driven deep-learning models to predict complex hydrodynamics. In this study, we trained two machine learning (ML) algorithms, namely the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), using spatiotemporal data obtained from CFD simulations. Impressively, the trained models accurately predicted future events at successive time steps, exhibiting excellent agreement for mean voidage, gas velocity, and particle velocity. Among the models utilized, LSTM outperformed the other. This demonstrated its superior ability to capture spatiotemporal from CFD data while requiring less computational time and power. The trained models provide a reliable means to extrapolate desirable mean particle velocity and voidage values within fluidized beds. These findings hold promising opportunities for enhancing the design and operation of gas-solid fluidized bed reactors, potentially leading to a significant reduction of up to 57–68% in simulation time.

Assessments of Data-Driven Deep Learning Models on One-Month Predictions of Pan-Arctic Sea Ice Thickness

Assessments of Data-Driven Deep Learning Models on One-Month Predictions of Pan-Arctic Sea Ice Thickness

An investigation of multivariate data-driven deep learning models for predicting COVID-19 variants

The Coronavirus Disease 2019 (COVID-19) pandemic has swept almost all parts of the world. With the increasing number of COVID-19 cases worldwide, Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) has mutated into various variants. Given the increasingly dangerous conditions of the pandemic, it is crucial to predict the number of COVID-19 cases. Deep Learning and Long Short-Term Memory (LSTM) have predicted disease progress with reasonable accuracy and small errors. LSTM training is used to predict confirmed cases of COVID-19 based on variants identified using the European Centre for Disease Prevention and Control (ECDC) COVID-19 dataset containing confirmed cases identified from 30 European countries. Tests were conducted using the LSTM and Bidirectional LSTM (BiLSTM) models with the addition of Recurrent Neural Network (RNN) as comparisons on hidden size and layer size. The obtained result showed that in testing hidden sizes 25, 50, 75, and 100, the RNN model provided better results, with the minimum Mean Squared Error (MSE) value of 0.01 and the Root Mean Square Error (RMSE) value of 0.012 for B.1.427/B.1.429 variant with a hidden size of 100. Further testing layer sizes 2, 3, 4, and 5 shows that the BiLSTM model provided better results, with a minimum MSE value of 0.01 and an RMSE of 0.01 for the B.1.427/B.1.429 variant with a hidden size of 100 and layer size of 2.

Probabilistic state estimation in district heating grids using deep neural network

Probabilistic state estimation is critical for operating and controlling district heating grids efficiently. However, computational bottlenecks of traditional solvers limit the feasibility of uncertainty-aware Bayesian estimation. This paper proposes using deep neural networks (DNNs) to enable fast and accurate posterior estimation. Fully-connected neural networks (FCNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs) are evaluated as candidate approximators of the physical model. Markov chain Monte Carlo sampling in the heat exchange space is leveraged to generate posterior samples. Experiments on a benchmark heating grid demonstrate FCNNs can efficiently learn the mapping from heat exchanges to network states. A FCNN trained on 20 training epochs after hyperparameter optimization provides the best approximation accuracy and uncertainty estimates, outperforming prior methods based on Deep Neural Networks. The results highlight the potential of data-driven deep learning models for probabilistic state estimation. The proposed framework could enable real-time uncertainty-aware control and decision-making for future intelligent district heating grids.

Morning Glory Flower Detection in Aerial Images Using Semi-Supervised Segmentation with Gaussian Mixture Models

The invasive morning glory, Ipomoea purpurea (Convolvulaceae), poses a mounting challenge in vineyards by hindering grape harvest and as a secondary host of disease pathogens, necessitating advanced detection and control strategies. This study introduces a novel automated image analysis framework using aerial images obtained from a small fixed-wing unmanned aircraft system (UAS) and an RGB camera for the large-scale detection of I. purpurea flowers. This study aimed to assess the sampling fidelity of aerial detection in comparison with the actual infestation measured by ground validation surveys. The UAS was systematically operated over 16 vineyard plots infested with I. purpurea and another 16 plots without I. purpurea infestation. We used a semi-supervised segmentation model incorporating a Gaussian Mixture Model (GMM) with the Expectation-Maximization algorithm to detect and count I. purpurea flowers. The flower detectability of the GMM was compared with that of conventional K-means methods. The results of this study showed that the GMM detected the presence of I. purpurea flowers in all 16 infested plots with 0% for both type I and type II errors, while the K-means method had 0% and 6.3% for type I and type II errors, respectively. The GMM and K-means methods detected 76% and 65% of the flowers, respectively. These results underscore the effectiveness of the GMM-based segmentation model in accurately detecting and quantifying I. purpurea flowers compared with a conventional approach. This study demonstrated the efficiency of a fixed-wing UAS coupled with automated image analysis for I. purpurea flower detection in vineyards, achieving success without relying on data-driven deep-learning models.

A Comprehensive Framework for Long-Tailed Learning via Pretraining and Normalization.

Data in the visual world often present long-tailed distributions. However, learning high-quality representations and classifiers for imbalanced data is still challenging for data-driven deep learning models. In this work, we aim at improving the feature extractor and classifier for long-tailed recognition via contrastive pretraining and feature normalization, respectively. First, we carefully study the influence of contrastive pretraining under different conditions, showing that current self-supervised pretraining for long-tailed learning is still suboptimal in both performance and speed. We thus propose a new balanced contrastive loss and a fast contrastive initialization scheme to improve previous long-tailed pretraining. Second, based on the motivative analysis on the normalization for classifier, we propose a novel generalized normalization classifier that consists of generalized normalization and grouped learnable scaling. It outperforms traditional inner product classifier as well as cosine classifier. Both the two components proposed can improve recognition ability on tail classes without the expense of head classes. We finally build a unified framework that achieves competitive performance compared with state of the arts on several long-tailed recognition benchmarks and maintains high efficiency.

Dynamic optimal control of coal tar chemical looping gasification based on process modelling and intelligent screening

Aiming to solve the problem of low utilization and poor economic efficiency of coal tar, this study proposes a new method to convert coal tar into syngas that can be used for coal-to-oil production through chemical looping gasification reactions, thus reducing the cost of coal-to-oil production. This proposal converts tar to syngas for Fischer-Tropsch reaction through chemical looping gasification cleaning process, which is realized by process simulation and artificial intelligence prediction to ensure the stable products quality under feed jump abnormal conditions. The proposed process model is first built and the optimal operating parameters are determined by sensitivity analysis after the optimization sequence is determined from correlation analysis. A dynamic model based on the optimal steady-state model is second generated to design an optimal control scheme to efficiently deal with the unusual conditions of the feed step. The data-driven deep learning model LSTM is used to intelligently screen the optimal proportional control scheme with the advantage of low time and cost consumption. Finally, the optimal control scheme is obtained by combining the prediction results of the LSTM model and the controller additions of other important parameters, and the control effect is tested by simulating the tar feed leap conditions proves that the control6 scheme is highly resistant to disturbances. The method achieves 84.1 % high carbon conversion rate of medium-low temperature tar and 88.14 % excellent selectivity of syngas. At the same time, the adjusting of feed step condition with long-term and short-term memory network promotes the development of process design with the low time and cost consumption.

Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics

Traditional data-driven deep learning models often struggle with high training costs, error accumulation, and poor generalizability in complex physical processes. Physics-informed deep learning (PiDL) addresses these challenges by incorporating physical principles into the model. Most PiDL approaches regularize training by embedding governing equations into the loss function, yet this depends heavily on extensive hyperparameter tuning to weigh each loss term. To this end, we propose to leverage physics prior knowledge by “baking” the discretized governing equations into the neural network architecture via the connection between the partial differential equations (PDE) operators and network structures, resulting in a PDE-preserved neural network (PPNN). This method, embedding discretized PDEs through convolutional residual networks in a multi-resolution setting, largely improves the generalizability and long-term prediction accuracy, outperforming conventional black-box models. The effectiveness and merit of the proposed methods have been demonstrated across various spatiotemporal dynamical systems governed by spatiotemporal PDEs, including reaction-diffusion, Burgers’, and Navier-Stokes equations.

Medical Transformer: Universal Encoder for 3-D Brain MRI Analysis.

Transfer learning has attracted considerable attention in medical image analysis because of the limited number of annotated 3-D medical datasets available for training data-driven deep learning models in the real world. We propose Medical Transformer, a novel transfer learning framework that effectively models 3-D volumetric images as a sequence of 2-D image slices. To improve the high-level representation in 3-D-form empowering spatial relations, we use a multiview approach that leverages information from three planes of the 3-D volume, while providing parameter-efficient training. For building a source model generally applicable to various tasks, we pretrain the model using self-supervised learning (SSL) for masked encoding vector prediction as a proxy task, using a large-scale normal, healthy brain magnetic resonance imaging (MRI) dataset. Our pretrained model is evaluated on three downstream tasks: 1) brain disease diagnosis; 2) brain age prediction; and 3) brain tumor segmentation, which are widely studied in brain MRI research. Experimental results demonstrate that our Medical Transformer outperforms the state-of-the-art (SOTA) transfer learning methods, efficiently reducing the number of parameters by up to approximately 92% for classification and regression tasks and 97% for segmentation task, and it also achieves good performance in scenarios where only partial training samples are used.

Robust Stochastic Neural Ensemble Learning with Noisy Labels for Thoracic Disease Classification.

Chest radiography is the most common radiology examination for thoracic disease diagnosis, such as pneumonia. A tremendous number of chest X-rays prompt data-driven deep learning models in constructing computer-aided diagnosis systems for thoracic diseases. However, in realistic radiology practice, a deep learning-based model often suffers from performance degradation when trained on data with noisy labels possibly caused by different types of annotation biases. To this end, we present a novel stochastic neural ensemble learning (SNEL) framework for robust thoracic disease diagnosis using chest X-rays. The core idea of our method is to learn from noisy labels by constructing model ensembles and designing noise-robust loss functions. Specifically, we propose a fast neural ensemble method that collects parameters simultaneously across model instances and along optimization trajectories. Moreover, we propose a loss function that both optimizes a robust measure and characterizes a diversity measure of ensembles. We evaluated our proposed SNEL method on three publicly available hospital-scale chest X-ray datasets. The experimental results indicate that our method outperforms competing methods and demonstrate the effectiveness and robustness of our method in learning from noisy labels. Our code is available at https://github.com/hywang01/SNEL.

LandBench 1.0: A benchmark dataset and evaluation metrics for data-driven land surface variables prediction

The advancements in deep learning methods have presented new opportunities and challenges for predicting land surface variables (LSVs) due to their similarity with computer sciences tasks. However, few researchers focus on the benchmark datasets for LSVs predictions that hampers fair comparisons of different data-driven deep learning models. Hence, we propose a LSVs benchmark dataset and prediction toolbox to boost research in data-driven LSVs modeling and improve the consistency of data-driven deep learning models for LSVs. LSVs benchmark dataset contains a large number of hydrology-related variables, such as global soil moisture, runoff, etc., which can verify the simulation of hydrological processes. Various global data from European Centre for Medium-Range Weather Forecasts reanalysis 5 (ERA5), ERA5-land, global gridded soil information (SoilGrid), soil moisture storage capacity (SMSC), and moderate-resolution imaging spectroradiometer (MODIS) datasets have been pre-processed into daily data at 0.5-, 1-, 2-, and 4-degree resolutions to facilitate their use in data-driven models. Simple statistical metrics, i.e., the root mean squared error and correlation coefficient, are chosen to evaluate the performance of different deep learning (DL) models, including convolutional neural network, long short-term memory and convolution long short-term memory models, with lead times of 1 and 5 days. A processed-based model serves as a physic baseline, soil moisture and surface sensible heat fluxes are taken as the target variables. The developed benchmark dataset and evaluation metrics for predicting LSVs using data-driven approaches, named as the LandBench toolbox, were implemented using Pytorch. This toolbox facilitates the reimplementation of existing methods, the development of novel predictive models, and the utilization of unified evaluation metrics. Additionally, the toolbox incorporates address mapping technology to enable high-resolution global predictions with constrained computing resources. We hope LandBench will not only serves as a standardized framework, fostering equitable model comparisons, but also provides indispensable data and a robust scientific foundation essential for advancing climate change research, disaster management, and sustainable development initiatives.

Real-time stored product insect detection and identification using deep learning: System integration and extensibility to mobile platforms

Existing stored product insect monitoring methods are time-consuming, costly, and often require specialized equipment or training. This study proposed an integrated insect monitoring system that employs a simple RGB camera and data-driven deep-learning models to detect and identify stored product insect species in warehouses, food facilities, and retail environments. Top-down images of six common insect species were acquired with a setup simulating a conceptualized probe-type monitor under varying lighting and background conditions. These images were preprocessed and manually annotated, resulting in an insect dataset of 2630 images with 14,509 labeled insects. A state-of-the-art computer vision model from YOLO family was selected, and six YOLO variants (YOLOv5s/m/l and YOLOv5s/m/l) were trained and evaluated on the insect dataset. All trained YOLO models delivered an impressive performance in terms of high detection accuracy (above 76% for mAP@[0.50:0.95]) and fast inference time (12–36 ms range). Subsequently, the best-performing YOLOv8l model was integrated and deployed on a mobile device, achieving a good detection performance with average detection speeds of 16 and 29 fps on a desktop computer and smartphone, respectively. The study provided an end-to-end framework for automatic and real-time insect detection and identification in a stored product environment. The lightweight variant of YOLO can be deployed on low-cost edge hardware, mobile devices, or cloud computing. The proposed system is relatively fast, accurate, and inexpensive for insect monitoring and may prove an alternate solution to existing methods. The system would serve as a decision-making tool for stored product facilities managers and can be easily scaled and adapted in a variety of stored product environments.

Comparison and analysis of various deep learning models for super-resolution reconstruction of turbulent flows

Single image super-resolution (SR) has become a promising research topic, with many deep learning-based models invented to reconstruct high-fidelity high-resolution (HR) images from low-resolution (LR) images. Motivated by a large amount of turbulent flow field data collected by experimental measurements and numerical simulation, researchers begin investigating the application of these data-driven deep learning models to conduct SR reconstruction of LR flow field data. Due to the limitations of experimental equipment and computing power, sometimes researchers can only obtain LR data. However, deep learning models can quickly reconstruct HR spatial-temporal turbulent data from LR data so that researchers can easily conduct further qualitative and quantitative analyses. This article reviews the development of flow field data SR reconstruction models and the problems encountered from the two aspects of network structure and loss function definition. Finally, we propose the research direction of applying the conditional generative adversarial network (cGAN) to turbulent flow SR reconstruction since seldom study has been conducted in this field.

Monthly Arctic sea ice prediction based on a data-driven deep learning model

There is growing interest in sub-seasonal to seasonal predictions of Arctic sea ice due to its potential effects on midlatitude weather and climate extremes. Current prediction systems are largely dependent on physics-based climate models. While climate models can provide good forecasts for Arctic sea ice at different timescales, they are susceptible to initial states and high computational costs. Here we present a purely data-driven deep learning model, UNet-F/M, to predict monthly sea ice concentration (SIC) one month ahead. We train the model using monthly satellite-observed SIC for the melting and freezing seasons, respectively. Results show that UNet-F/M has a good predictive skill of Arctic SIC at monthly time scales, generally outperforming several recently proposed deep learning models, particularly for September sea-ice minimum. Our study offers a perspective on sub-seasonal prediction of future Arctic sea ice and may have implications for forecasting weather and climate in northern midlatitudes.

Traffic Flow Prediction Based on Spatiotemporal Potential Energy Fields

Traffic flow prediction is a fundamental problem in spatiotemporal data mining. Most of the existing studies focuses on designing statistical models to fit historical traffic data, which are purely data-driven approaches and fail to reveal the underlying mechanisms of urban traffic. To address this issue, we propose the spatiotemporal potential energy field model (ST-PEF+), which applies the field theory for human mobility to interpret the underlying mechanisms of urban traffic, and introduces the theory into data-driven deep learning models. ST-PEF+ consists of a PEF extraction module and a data-driven module. Inspired by the field theory for human mobility, the PEF extraction module adopts an algorithm to decompose the grid-based traffic flow graph into several polytree-based potential energy fields (PEFs), where traffic flows from high potential locations to low potential locations, just as water is driven by the gravity field. We also provide a theoretical analysis to ensure that the polytree decomposition algorithm can decompose any traffic flow graph. In the data-driven module, ST-PEF+ learns a spatiotemporal deep learning model to predict the dynamics of PEFs. The model adopts correlation-adaptive neural network structures, which consists of a temporal component for temporal correlations and a spatial component for spatial correlations. The temporal component employs a GRU and DCN combined structure to capture both short-term autocorrelation and long-term repeating patterns of PEFs. The spatial component extends the GAT using weighted directed attention to model the asymmetric spatial structure in PEFs. The prediction results of traffic flow are finally derived from PEFs that are predicted by the spatiotemporal deep learning model. We conduct extensive evaluations on three real-world traffic datasets. The results show that our model outperforms the state-of-the-art baselines. In addition, case studies confirm that the PEFs learned in our framework can reveal the underlying mechanisms of urban traffic, thus improving the model interpretability.

Rapid prediction of urban flood based on disaster-breeding environment clustering and Bayesian optimized deep learning model in the coastal city

Rapid prediction of urban flood is essential for sustainable city and society development. The data-driven deep learning model is commonly adopted for flood prediction, but it rarely considers the physical causes of flooding. In the study, we proposed an urban flood prediction model based on K-means cluster and Bayesian optimized deep learning algorithm. The K-means cluster algorithm was utilized for classifying the study area into different flooding cause clusters based on disaster-bearing environment factors. The gradient boosting decision tree (GBDT) algorithm was adopted to establish the prediction model for the areas with the same flooding cause. Furthermore, to improve the model accuracy, the study explored Bayesian algorithm to optimize GBDT model's hyperparameters. The proposed model was verified in the area of Haikou, China. Elevation, slope, density of pipe network, distance to the river and curvature were selected as the disaster-bearing environment factors, and the study area was classified into 3 clusters using K-means cluster based on these factors. Different GBDT models were established for each cluster, then Bayesian algorithm was used to optimize the six hyperparameters in the models. The second cluster of area is close to the sea, so the hazard degree of tide level is added when building the GBDT model. The Nash-Sutcliffe efficiency coefficient (NSE) of K-means-GBDT models for three clusters are 0.9628, 0.9580 and 0.9914 respectively, indicating the excellent performance of the proposed model. In comparison to the GBDT model without clustering, the cluster 2 has the highest accuracy improvement, with the NSE improving from 0.8951 to 0.9580. Furthermore, when making predictions using all available data for the whole study area, the NSE of the K-means-GBDT models increased from 0.9361 to 0.9761 compared to the GBDT model without clustering. The proposed model was validated for its applicability and effectiveness in rapid prediction of urban flood, and can provide support for flood control and quick response.

Biology-guided deep learning predicts prognosis and cancer immunotherapy response

Substantial progress has been made in using deep learning for cancer detection and diagnosis in medical images. Yet, there is limited success on prediction of treatment response and outcomes, which has important implications for personalized treatment strategies. A significant hurdle for clinical translation of current data-driven deep learning models is lack of interpretability, often attributable to a disconnect from the underlying pathobiology. Here, we present a biology-guided deep learning approach that enables simultaneous prediction of the tumor immune and stromal microenvironment status as well as treatment outcomes from medical images. We validate the model for predicting prognosis of gastric cancer and the benefit from adjuvant chemotherapy in a multi-center international study. Further, the model predicts response to immune checkpoint inhibitors and complements clinically approved biomarkers. Importantly, our model identifies a subset of mismatch repair-deficient tumors that are non-responsive to immunotherapy and may inform the selection of patients for combination treatments.

An ensemble CNN-LSTM and GRU adaptive weighting model based improved sparrow search algorithm for predicting runoff using historical meteorological and runoff data as input

Accurate prediction of river runoff is of great significance for water resources management, flood prevention and mitigation. The causes of runoff are complex and the mechanisms behind them are difficult to grasp. Building a data-driven deep learning model for runoff prediction is an effective solution. To achieve the fusion of multi-source information, prediction accuracy and wide applicability, a hybrid model based on CNN-LSTM & GRU-ISSA is proposed in this study. In this paper, meteorological data, hydrological data and runoff data are selected and the maximum information coefficient (MIC) is used to calculate the relationship between each variable and runoff in order to reduce the dimensionality of the data. A convolutional neural network (CNN) is used to extract features of the long time series of runoff and a long short-term memory network (LSTM) is used for the prediction of the long time series of runoff. A gated recurrent unit (GRU) is also used for the short time series prediction of runoff. In order to extract the advantages of both prediction models, an adaptive weighting module (AWM) is proposed to dynamically learn the outputs of both models and combine them into the final prediction results. To be able to solve the selection of model hyperparameters, we use the improved sparrow search algorithm (ISSA). This algorithm introduces two improvement points, Lévy Flight and Sine Cosine Algorithm, based on the Sparrow Search Algorithm (SSA), to achieve fast convergence of the algorithm with better global optimal solutions. The proposed model was validated using watersheds with different runoff ranges, e.g., in the Bailong River watershed, the NSE value was 0.90 and the RMSE value was 2.17. The results showed that the proposed model significantly outperformed the other baseline models.

Reconstructing 3D ocean subsurface salinity (OSS) from T–S mapping via a data-driven deep learning model

Despite the increasing volume of oceanographic data, the available ocean salinity data remains patently insufficient which limits studying ocean dynamics, climate change and the calculation of salinity-related ocean elements. Considering that traditional salinity reconstruction methods often suffer from various factors such as additional constraints, priori physical assumptions and large of specific regression coefficients, a generative adversarial networks (GANs) based deep learning (DL) framework is proposed to directly construct a near real-time, high-resolution daily three-dimensional ocean subsurface salinity (3D OSS) dataset from a data-driven perspective in this study. Four models with different structural combinations are designed in the China’s marginal seas. Experimental results demonstrate that the models can successfully reconstruct the high-precision and high-resolution 3D OSS on a daily scale in 12 depth levels (from 2 m to 200 m). The asymmetric inception-3DGAN model with all enhanced structures has the highest accuracy, the average root-mean-squared error (RMSE) is 0.135psu, the average coefficient of determination (R2) is 0.5641 and the percent bias is 0.436%. Comparing with model without enhanced structures, the average RMSE is decreased by 22.41% when adding all the enhancement structures. Besides, temporal and spatial error analysis are also conducted to evaluate the models’ performance from different aspects. Finally, the model results are used to analyze common ocean elements, such as water mass properties, dynamic fields and geostrophic velocity fields, demonstrating that the 3D OSS dataset construction approach proposed in this study not only provides some new insights into ocean observations, but also has high application values.