Unveiling the limits of deep learning models in hydrological extrapolation tasks

Long Short-Term Memory (LSTM) networks have shown strong performance in rainfall–runoff modelling, often surpassing conventional hydrological models in benchmark studies. However, recent studies raise questions about their ability to extrapolate, particularly under extreme conditions that exceed the range of their training data. This study examines the performance of a stand-alone LSTM trained on 196 catchments in Switzerland when subjected to synthetic design precipitation events of increasing intensity and varying duration. The model’s response is compared to that of a hybrid model and evaluated against hydrological process understanding. Our study reiterates that the stand-alone LSTM is characterised by a theoretical prediction limit, and we show that this limit is below the range of the data the model was trained on. We show that saturation of the LSTM cell states alone does not fully account for this characteristic behaviour, as the LSTM does not reach full saturation, particularly for the 1-day events. Instead, its gating mechanisms prevent new information about the current extreme precipitation from being incorporated into the cell states. Adjusting the LSTM architecture, for instance, by increasing the number of hidden states, and/or using a larger, more diverse training dataset can help mitigate the problem. However, these adjustments do not guarantee improved extrapolation performance, and the LSTM continues to predict values below the range of the training data or show hydrologically unfeasible runoff responses during the 1-day design experiments. Despite these shortcomings, our findings highlight the inherent potential of stand-alone LSTMs to capture complex hydro-meteorological relationships. We argue that more robust training strategies and model configurations could address the observed limitations, ensuring the promise of stand-alone LSTMs for rainfall–runoff modelling.

Abstract. Long short-term memory (LSTM) models are recurrent neural networks from the field of deep learning (DL) which have shown promise for time series modelling, especially in conditions when data are abundant. Previous studies have demonstrated the applicability of LSTM-based models for rainfall–runoff modelling; however, LSTMs have not been tested on catchments in Great Britain (GB). Moreover, opportunities exist to use spatial and seasonal patterns in model performances to improve our understanding of hydrological processes and to examine the advantages and disadvantages of LSTM-based models for hydrological simulation. By training two LSTM architectures across a large sample of 669 catchments in GB, we demonstrate that the LSTM and the Entity Aware LSTM (EA LSTM) models simulate discharge with median Nash–Sutcliffe efficiency (NSE) scores of 0.88 and 0.86 respectively. We find that the LSTM-based models outperform a suite of benchmark conceptual models, suggesting an opportunity to use additional data to refine conceptual models. In summary, the LSTM-based models show the largest performance improvements in the north-east of Scotland and in south-east of England. The south-east of England remained difficult to model, however, in part due to the inability of the LSTMs configured in this study to learn groundwater processes, human abstractions and complex percolation properties from the hydro-meteorological variables typically employed for hydrological modelling.

Deep learning (DL)-based hydrological models, particularly those using Long Short-Term Memory (LSTM) networks, typically require large datasets for effective training. In the context of large-scale rainfall-runoff modeling, dataset size can refer to either the number of watersheds or the length of the training period. While it is well established that training a regional model across more watersheds improves performance (Kratzert et al., 2024), the benefits of extending the training period are less clear.Empirical evidence from studies such as Boulmaiz et al. (2020) and Gauch et al. (2021) suggests that longer training periods enhance LSTM performance in rainfall-runoff modeling. This improvement is attributed to the need for extensive datasets to ensure proper model convergence and the ability to capture a wide range of hydrological conditions and events. However, these studies neglected the influence of data recency (or data recentness), which is critical for operational applications that forecast current and future hydrological conditions. In the context of climate change and anthropogenic interventions, the assumption of stationarity (i.e., that historical patterns reliably represent future conditions) may no longer hold for hydrological systems (Shen et al., 2022). Consequently, the selection of training periods should account for potential non-stationarity, as more recent data may better reflect current rainfall-runoff dynamics. Intriguingly, Shen et al. (2022) found that calibrating hydrologic models to the latest data is a superior approach compared to using old data, and completely discarding the oldest data can even improve the performance in streamflow prediction.This study aims to address two research questions: (1) As the number of watersheds increases, is it still necessary to train LSTM models on decades of historical observations? (2) Can LSTM models achieve comparable performance using shorter training periods focused on more recent data? Specifically, we examine whether models trained on recent data outperform those trained on older data and explore how different temporal partitions of historical records affect predictive skill.This study leverages a comprehensive dataset comprising streamflow records from over 1,300 watersheds across North America, representing diverse climatic and hydrological regimes, with streamflow data spanning 1950 to 2023. Training periods are designed to isolate the effects of temporal data recency while keeping period lengths consistent. This approach enables a systematic comparison of model performance using exclusively older (e.g., pre-1980) versus exclusively recent data (e.g., post-1980). This research provides evidence-based recommendations for selecting training data while balancing computational costs, data availability, and prediction accuracy. ReferencesBoulmaiz, T., Guermoui, M., and Boutaghane, H.: Impact of training data size on the LSTM performances for rainfall–runoff modeling, Model Earth Syst Environ, 6, 2153–2164, https://doi.org/10.1007/S40808-020-00830-W/FIGURES/9, 2020.Gauch, M., Mai, J., and Lin, J.: The proper care and feeding of CAMELS: How limited training data affects streamflow prediction, Environmental Modelling and Software, 135, https://doi.org/10.1016/j.envsoft.2020.104926, 2021.Kratzert, F., Gauch, M., Klotz, D., and Nearing, G.: HESS Opinions: Never train an LSTM on a single basin, Hydrology and Earth System Science, https://doi.org/10.5194/hess-2023-275, 2024.Shen, H., Tolson, B. A., and Mai, J.: Time to Update the Split-Sample Approach in Hydrological Model Calibration, Water Resour Res, 58, e2021WR031523, https://doi.org/10.1029/2021WR031523, 2022.

Rainfall-runoff (RR) modeling remains a challenging task in the field of hydrology especially when it comes to regional scale hydrology. Recently, the Long Short-Term Memory (LSTM) - which is known for its ability to learn sequential and temporal relations - has been widely adopted in RR modeling. The Convolutional Neural Networks (CNN) have matured enough in computer vision tasks, and trials were conducted to use them in hydrological applications. Different combinations of CNN and LSTM have proved to work; however, questions remain about suitability of different model architectures, the input variables needed for the model and the interpretability of the learning process of the models for regional scale. In this work we trained a sequential CNN-LSTM deep learning architecture to predict daily streamflow between 1980 and 2014, regionally and simultaneously, over 86 catchments from CAMELS dataset in the US. The model was forced using year-long spatially distributed (gridded) input with precipitation, maximum temperature and minimum temperature for each day, to predict one day streamflow. The model takes advantage of the CNN to encode the spatial patterns in the input tensor, and feed them to the LSTM for learning the temporal relations between them. The trained model was further fine-tuned to predict for 3 local sub-clusters of the 86 stations. This was made in order to test the significance of fine-tuning in the performance and model learning process. Also, to interpret the spatial patterns learning process, a perturbation was introduced in the gridded input data and the sensitivity of the model output to the perturbation was shown in spatial heat maps. Finally, to evaluate the performance of the model, different benchmark models were trained using -as possible- a similar training setup as for the CNN-LSTM model. These models are CNN without the LSTM part (regional model), LSTM without CNN part (regional model), simple single-layer ANN (regional model), and LSTM trained for individual stations (considered as state of the art). All of these benchmark models have been fined-tuned for the 3 clusters as well. CNN-LSTM model, after being fine-tuned, performed well predicting daily streamflow over the test period with a median Nash-Sutcliffe efficiency (NSE) of 0.62 and 65% of the 86 stations with NSE > 0.6 outperforming all benchmark models that were trained regionally using the same training setup. The model also achieved a comparable performance as for the -state of the art- LSTM trained for individual stations. Fine-tuning improved the performance for all of the models during the test period. The CNN-LSTM model, was shown to be more sensitive to input perturbations near the stations in which the prediction is intended. This was even clearer for the fine-tuned model, indicating that the model is learning spatially relevant information from the input gridded data, and fine tuning is helping on guiding the model to focus more on the relevant input.   This work shows the potential of CNN and LSTM for regional Rainfall-runoff modeling by capturing spatiotemporal patterns involved in RR process. The work, also, contributes toward more physically interpretable data-driven modeling paradigm.

<p>Techniques from the field of machine learning have shown considerable promise in rainfall-runoff modelling. This research offers three novel contributions to the advancement of this field: a study of the performance of LSTM based models in a GB hydrological context; a diagnosis of hydrological processes that data-driven models simulate well but conceptual models struggle with; and finally an exploration of methods for interpreting the internal cell states of the LSTMs. </p><p>In this study we train two deep learning models, a Long Short Term Memory (LSTM) Network and an Entity Aware LSTM (EALSTM), to simulate discharge for 518 catchments across Great Britain using a newly published dataset, CAMELS-GB. We demonstrate that the LSTM models are capable of simulating discharge for a large sample of catchments across Great Britain, achieving a mean catchment Nash-Sutcliffe Efficiency (NSE) of 0.88 for the LSTM and 0.86 for the EALSTM, where no stations have an NSE < 0. We compare these models against a series of conceptual models which have been externally calibrated and used as a benchmark (Lane et al., 2019). </p><p>Alongside robust performance for simulating discharge, we note the potential for data-driven methods to identify hydrological processes that are present in the underlying data, but the FUSE conceptual models are unable to capture. Therefore, we calculate the relative improvement of the LSTMs compared to the conceptual models, ∆NSE. We find that the largest improvement of the LSTM models compared to our benchmark is in the summer months and in the South East of Great Britain. </p><p>We also demonstrate that the internal “memory” of the LSTM correlates with soil moisture, despite the LSTM not receiving soil moisture as an input. This process of “concept-formation” offers three interesting findings. It provides a novel method for deriving soil moisture estimates. It suggests the LSTM is learning physically realistic representations of hydrological processes. Finally, this process of concept formation offers the potential to explore how the LSTM is able to produce accurate simulations of discharge, and the transformations that are learned from inputs (temperature, precipitation) to outputs (discharge).</p><p>

Abstract. Rainfall-runoff modeling is of great importance for flood forecast and water management. Hydrological modeling is the traditional and commonly used approach for rainfall-runoff modeling. In recent years, with the development of artificial intelligence technology, deep learning models, such as the long short-term memory (LSTM) model, are increasingly applied to rainfall-runoff modeling. However, current works do not consider the effect of rainfall spatial distribution information on the results. Focusing on 10 catchments from the Catchment Attributes and Meteorology for Large-Sample Studies (CAMELS) dataset, this study compared the performance of LSTM with different look-back windows (7, 15, 30, 180, 365 d) for future 1 d discharges and for future multi-day simulations (7, 15 d). Secondly, the differences between LSTMs as individual models trained independently in each catchment and LSTMs as regional models were also compared across 10 catchments. All models are driven by catchment mean rainfall data and spatially distributed rainfall data, respectively. The results demonstrate that regardless of whether LSTMs are trained independently in each catchment or trained as regional models, rainfall data with spatial information improves the performance of LSTMs compared to models driven by mean rainfall data. The LSTM as a regional model did not obtain better results than LSTM as an individual model in our study. However, we found that using spatially distributed rainfall data can reduce the difference between LSTM as a regional model and LSTM as an individual model. In summary, (a) adding information about the spatial distribution of the data is another way to improve the performance of LSTM where long-term rainfall records are absent, and (b) understanding and utilizing the spatial distribution information can help improve the performance of deep learning models in runoff simulations.

Better understanding the predictive capabilities of hydrological models under contrasting climate conditions will enable more robust decision-making. Here, we tested the ability of the long short-term memory (LSTM) for daily discharge prediction under changing conditions using six snow-influenced catchments in Switzerland. We benchmarked the LSTM using the Hydrologiska Byråns Vattenbalansavdelning (HBV) bucket-type model with two parameterizations. We compared the model performance under changing conditions against constant conditions and tested the impact of the time-series size used in calibration on the model performance. When calibrated, the LSTM resulted in a much better fit than the HBV. However, in validation, the performance of the LSTM dropped considerably, and the fit was as good or poorer than the HBV performance in validation. Using longer time series in calibration improved the robustness of the LSTM, whereas HBV needed fewer data to ensure a robust parameterization. When using the maximum number of years in calibration, the LSTM was considered robust to simulate discharges in a drier period than the one used in calibration. Overall, the HBV was found to be less sensitive for applications under contrasted climates than the data-driven model. However, other LSTM modeling setups might be able to improve the transferability between different conditions.

Use of one-dimensional CNN for input data size reduction in LSTM for improved computational efficiency and accuracy in hourly rainfall-runoff modeling

This study compares the bias correction techniques of empirical quantile mapping (QM) and the Long Short-Term Memory (LSTM) machine learning model for summertime daily rainfall simulation focusing on precipitation-dependent bias and temporal variation. Numerical experiments using Weather Research and Forecasting (WRF) were conducted over South Korea with lateral boundary conditions of ERA5 reanalysis data. For the spatial distribution of mean summertime rainfall, the bias-uncorrected WRF simulation (WRF_RAW) showed dry bias for most of the region of South Korea. The WRF results corrected by QM and LSTM (WRF_QM and WRF_LSTM, respectively) were improved for the mean summer rainfall simulation with the root mean square error values of 0.17 and 0.69, respectively, which were smaller than those of the WRF_RAW (1.10). Although the WRF_QM performed better than the WRF_LSTM in terms of the summertime mean and monthly precipitation, the WRF_LSTM presented a closer interannual rainfall variation to the observation than the WRF_QM. The coefficient of determination for calendar-day mean rainfall was the highest in the following order: the WRF_LSTM (0.451), WRF_QM (0.230), and WRF_RAW (0.201). However, the WRF_LSTM had a limitation in reproducing extreme rainfall exceeding 50 mm/day due to the few cases of extreme precipitation in training data. Nevertheless, the WRF_LSTM better simulated the observed light-to-moderate precipitation (10–50 mm/day) than the others.

Machine learning is increasingly important for rainfall–runoff modelling. In particular, the community started to widely adopt the Long Short-Term Memory (LSTM) network. One of the most important established best practices  in this context is to train the LSTMs on a large number of diverse basins  (Kratzert et al., 2019; 2024). Intuitively, the reason for adopting this practice is that training deep learning models on small and homogeneous data sets (e.g., data from only a single hydrological basin) leads to poor generalization behavior — especially for high-flows.  To examine this behavior, Kratzert et al. (2024) use a theoretical maximum prediction limit for LSTMs. This theoretical limit is computed as the L1 norm (i.e., the sum of the absolute values of each vector component) of the learned weight vector that relates the hidden states to the estimated streamflow. Hence, for random vectors we could simply obtain larger theoretical limits by increasing the size of the network (i.e., the  number of parameters). However, since LSTMs are trained using gradient descent, this relationship is more intricate.  This contribution explores the relationship between the theoretical limit and the network size. In particular, we will look at how increasing the network size in untrained models increases the prediction limit and contrast it to the scaling behavior of trained models.

<p>In recent years, deep learning has been applied to various issues in natural science, including hydrology. These application results show its high applicability. There are some studies that performed rainfall-runoff modeling by means of a deep learning method, LSTM (Long Short-Term Memory). LSTM is a kind of RNN (Recurrent Neural Networks) that is suitable for modeling time series data with long-term dependence. These studies showed the capability of LSTM for rainfall-runoff modeling. However, there are few studies that investigate the effects of input variables on the estimation accuracy. Therefore, this study, investigated the effects of the selection of input variables on the accuracy of a rainfall-runoff model by means of LSTM. As the study watershed, this study selected a snow-dominated watershed, the Ishikari River basin, which is in the Hokkaido region of Japan. The flow discharge was obtained at a gauging station near the outlet of the river as the target data. For the input data to the model, Meteorological variables were obtained from an atmospheric reanalysis dataset, ERA5, in addition to the gridded precipitation dataset. The selected meteorological variables were air temperature, evaporation, longwave radiation, shortwave radiation, and mean sea level pressure. Then, the rainfall-runoff model was trained with several combinations of the input variables. After the training, the model accuracy was compared among the combinations. The use of meteorological variables in addition to precipitation and air temperature as input improved the model accuracy. In some cases, however, the model accuracy was worsened by using more variables as input. The results indicate the importance to select adequate variables as input for rainfall-runoff modeling by LSTM.</p>

Long Short Term Memory (LSTM) is a special kind of Recurrent Neural Networks popularly used in various applications. However, using a single LSTM is often not enough to attain reliable performance on complicated machine learning tasks. This is because LSTM is sensitive to the specifics of the training data. Ensemble learning is a promising approach to improve the performance of LSTMs on complicated tasks. However, it is difficult to design an ensemble of LSTMs. LSTMs that constitute the ensemble should be both accurate and diverse. This paper proposes a new two-phase evolutionary algorithm to design ensembles. The first phase is to evolve best performing LSTMs automatically. A connection weight inheritance approach is used in the first phase to improve the effectiveness and efficiency of the evolutionary process. The second phase is to design ensembles by choosing suitable LSTMs without fixing the ensemble size in advance. We use bagging to train the selected LSTMs to build the ensemble to achieve good diversity among the LSTMs. The proposed approach is evaluated on various classification tasks. The results show the effectiveness of the proposed approach and its significant improvement in performance over many state-of-the-art machine learning models. The results also show the efficiency of the proposed approach in comparison with the baseline algorithm.

Abstract. A deep learning model designed for time series predictions, the long short-term memory (LSTM) architecture, is regularly producing reliable results in local and regional rainfall–runoff applications around the world. Recent large-sample hydrology studies in North America and Europe have shown the LSTM model to successfully match conceptual model performance at a daily time step over hundreds of catchments. Here we investigate how these models perform in producing monthly runoff predictions in the relatively dry and variable conditions of the Australian continent. The monthly time step matches historic data availability and is also important for future water resources planning; however, it provides significantly smaller training datasets than daily time series. In this study, a continental-scale comparison of monthly deep learning (LSTM) predictions to conceptual rainfall–runoff (WAPABA model) predictions is performed on almost 500 catchments across Australia with performance results aggregated over a variety of catchment sizes, flow conditions, and hydrological record lengths. The study period covers a wet phase followed by a prolonged drought, introducing challenges for making predictions outside of known conditions – challenges that will intensify as climate change progresses. The results show that LSTM models matched or exceeded WAPABA prediction performance for more than two-thirds of the study catchments, the largest performance gains of LSTM versus WAPABA occurred in large catchments, the LSTMs struggled less to generalise than the WAPABA models (e.g. making predictions under new conditions), and catchments with few training observations due to the monthly time step did not demonstrate a clear benefit with either WAPABA or LSTM.

<p>The authors propose bagged and boosted convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and compare their performance with the bagged and boosted traditional shallow artificial neural networks (ANNs) for short-term electricity load forecasting. Unlike existing references that mainly compare the performance of ensemble deep learning with single deep learning and machine learning techniques, three further performance comparisons are carried out: (1) bagged CNNs and bagged LSTMs, (2) boosted CNNs and LSTMs, and (3) bagged CNNs and bagged LSTMs, and boosted CNNs and LSTMs. This allows an insight into the individual effects of ensemble learning on CNNs and LSTMs. The proposed models' inputs consist of weather and time-related features in addition to the past load. The use of these features allows CNNs and LSTMs to estimate further complex relationship between them and the load. We implement all these methods and compare their performance on the same New England electricity load forecasting data set via statistical analysis. Effects on the forecasting performance with reduced training data are further shown. The LSTM models have the largest performance variation and are also more sensitive to a reduction in training data. In these models, boosting can improve both prediction accuracy and consistency.</p> <p> </p>

<p>The authors propose bagged and boosted convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and compare their performance with the bagged and boosted traditional shallow artificial neural networks (ANNs) for short-term electricity load forecasting. Unlike existing references that mainly compare the performance of ensemble deep learning with single deep learning and machine learning techniques, three further performance comparisons are carried out: (1) bagged CNNs and bagged LSTMs, (2) boosted CNNs and LSTMs, and (3) bagged CNNs and bagged LSTMs, and boosted CNNs and LSTMs. This allows an insight into the individual effects of ensemble learning on CNNs and LSTMs. The proposed models' inputs consist of weather and time-related features in addition to the past load. The use of these features allows CNNs and LSTMs to estimate further complex relationship between them and the load. We implement all these methods and compare their performance on the same New England electricity load forecasting data set via statistical analysis. Effects on the forecasting performance with reduced training data are further shown. The LSTM models have the largest performance variation and are also more sensitive to a reduction in training data. In these models, boosting can improve both prediction accuracy and consistency.</p> <p> </p>