Transformer Neural Network Interpretable Prediction of Diverse Environmental Time Series Using Weather Forecasts

Abstract

Transformer Neural Networks (TNNs) have caused a paradigm shift in deep learning domains like natural language processing and gathered immense interest due to their versatility in other fields such as time series forecasting (TSF). Most current TSF applications of TNNs use only historic observations to predict future events, ignoring information available in weather forecasts to inform better predictions, and with little attention given to the interpretability of the model’s use of environmental input factors. This work explores the potential for TNNs to perform TSF across multiple environmental variables (streamflow, stage, water temperature, and salinity) in two ecologically important regions: the Peace River watershed (Florida) and the northern Gulf of Mexico (Louisiana). The TNN was tested (and uncertainty quantified) for each response variable from one- to fourteen-day-ahead forecasts using past observations and spatially distributed weather forecasts. Additionally, a sensitivity analysis was performed on the trained TNNs’ attention weights to identify the relative influence of each input variable on each response variable’s prediction. Overall model performance ranged from good to very good (0.78<NSE<0.99 for all variables and forecast horizons). Through the sensitivity analysis, we found that the TNN was able to learn the physical patterns behind the data, adapt the use of the input variables to each forecast, and increasingly use weather forecast information as forecasting windows increased. The TNN excellent performance and flexibility, along with the intuitive interpretability highlighting the logic behind the models’ forecasting decision-making process, provide evidence for the applicability of this architecture to other TSF variables and locations.