Flow-dependent Covariance Research Articles

Implementation of Scale-Dependent Background-Error Covariance Localization in the Canadian Global Deterministic Prediction System

Abstract The approach of applying different amounts of horizontal localization to different ranges of background-error covariance horizontal scales as proposed by Buehner and Shlyaeva was recently implemented in the four-dimensional ensemble–variational (4DEnVar) data assimilation scheme of the global deterministic prediction system (GDPS) at Environment and Climate Change Canada operations. To maximize the benefits from this approach to reduce the sampling noise in the ensemble-derived background-error covariances, it was necessary to adopt a new weighting between the climatological and flow-dependent covariances that increases significantly the role of the latter. Thus, in December 2021 the GDPS became the first operational global deterministic medium-range weather forecasting system to rely completely on flow-dependent covariances in the troposphere and the lower stratosphere. The experiments that led to the adoption of these two related changes and their impacts on the forecasts up to 7 days for various regions of the globe during the boreal summer of 2019 and winter of 2020 are presented here. It is also illustrated that relying more on ensemble-derived covariances amplifies the positive impacts on the GDPS when the background ensemble generation strategy is improved.

The High-Resolution Rapid Refresh (HRRR): An Hourly Updating Convection-Allowing Forecast Model. Part I: Motivation and System Description

Abstract The High-Resolution Rapid Refresh (HRRR) is a convection-allowing implementation of the Advanced Research version of the Weather Research and Forecasting (WRF-ARW) Model with hourly data assimilation that covers the conterminous United States and Alaska and runs in real time at the NOAA/National Centers for Environmental Prediction (NCEP). Implemented operationally at NOAA/NCEP in 2014, the HRRR features 3-km horizontal grid spacing and frequent forecasts (hourly for CONUS and 3-hourly for Alaska). HRRR initialization is designed for optimal short-range forecast skill with a particular focus on the evolution of precipitating systems. Key components of the initialization are radar-reflectivity data assimilation, hybrid ensemble-variational assimilation of conventional weather observations, and a cloud analysis to initialize stratiform cloud layers. From this initial state, HRRR forecasts are produced out to 18 h every hour, and out to 48 h every 6 h, with boundary conditions provided by the Rapid Refresh system. Between 2014 and 2020, HRRR development was focused on reducing model bias errors and improving forecast realism and accuracy. Improved representation of the planetary boundary layer, subgrid-scale clouds, and land surface contributed extensively to overall HRRR improvements. The final version of the HRRR (HRRRv4), implemented in late 2020, also features hybrid data assimilation using flow-dependent covariances from a 3-km, 36-member ensemble (“HRRRDAS”) with explicit convective storms. HRRRv4 also includes prediction of wildfire smoke plumes. The HRRR provides a baseline capability for evaluating NOAA’s next-generation Rapid Refresh Forecast System, now under development. Significance Statement NOAA’s operational hourly updating, convection-allowing model, the High-Resolution Rapid Refresh (HRRR), is a key tool for short-range weather forecasting and situational awareness. Improvements in assimilation of weather observations, as well as in physics parameterizations, have led to improvements in simulated radar reflectivity and quantitative precipitation forecasts since the initial implementation of HRRR in September 2014. Other targeted development has focused on improved representation of the diurnal cycle of the planetary boundary layer, resulting in improved near-surface temperature and humidity forecasts. Additional physics and data assimilation changes have led to improved treatment of the development and erosion of low-level clouds, including subgrid-scale clouds. The final version of HRRR features storm-scale ensemble data assimilation and explicit prediction of wildfire smoke plumes.

Assimilation of MWHS and MWTS radiance data from the FY-3A satellite with the POD-3DEnVar method for forecasting heavy rainfall

An ensemble three-dimensional variational assimilation method based on Proper Orthogonal Decomposition (referred to as POD-3DEnVar) can not only provide flow-dependent covariances through the evolving ensemble of short-range forecasts, but also obtain directly the analysis field without an iterative process. Using the POD-3DEnVar method, a regional hybrid variational ensemble data assimilation system (referred to as POD-HVEDAS) is constructed with the Community Radiative Transfer Model (CRTM) as the observation operator. Observations from Microwave Humidity Sounder (MWHS) and Microwave Temperature Sounder (MWTS) onboard Fengyun-3A are simultaneously assimilated into the Weather Research and Forecasting (WRF) model by the POD-HVEDAS. Bias correction and quality control schemes for MWHS and MWTS microwave radiance are applied in the system. Three experiments (Con, Hybrid-DA, and POD-DA) are designed to investigate the assimilation ability of the POD-HVEDAS for heavy rainfall over the Yangtze River. The assimilation effects of POD-3DEnVar and WRFDA-Hybrid assimilation method are compared. The results show that the POD-HVEDAS can assimilate MWHS and MWTS microwave radiance effectively, and give a better precipitation forecast than that of the WRFDA-Hybrid method. And it is found that the assimilation increments of both POD-3DEnVar and WRFDA-Hybrid methods all show flow dependent characteristic. However, the improvements in the background wind, temperature and humidity fields by the POD-DA experiment are more obvious than those of the Hybrid-DA experiment, which play key roles in improving the accuracy of precipitation forecast. These promising results suggest initiulizing limited-area models with POD-3DEnVar method that assimilate microwave radiances is beneficial. In the future, POD-3DEnVar method will be used to study multiple cases to fully assess the robustness of POD-HVEDAS.

A hybrid ensemble-OI Kalman filter for efficient data assimilation into a 3-D biogeochemical model of the Mediterranean

A hybrid ensemble data assimilation scheme (HYBRID), combining a flow-dependent with a static background covariance, was developed and implemented for assimilating satellite (SeaWiFS) Chl-a data into a marine ecosystem model of the Mediterranean. The performance of HYBRID was assessed against a model free-run, the ensemble-based singular evolutive interpolated Kalman (SEIK) and its variant with static covariance (SFEK), with regard to the assimilated variable (Chl-a) and non-assimilated variables (dissolved inorganic nutrients). HYBRID was found more efficient than both SEIK and SFEK, reducing the Chl-a error by more than 40% in most areas, as compared to the free-run. Data assimilation had a positive overall impact on nutrients, except for a deterioration of nitrates simulation by SEIK in the most productive area (Adriatic). This was related to SEIK pronounced update in this area and the phytoplankton limitation on phosphate that lead to a built up of excess nitrates. SEIK was found more efficient in productive and variable areas, where its ensemble exhibited important spread. SFEK had an effect mostly on Chl-a, performing better than SEIK in less dynamic areas, adequately described by the dominant modes of its static covariance. HYBRID performed well in all areas, due to its “blended” covariance. Its flow-dependent component appears to track changes in the system dynamics, while its static covariance helps maintaining sufficient spread in the forecast. HYBRID sensitivity experiments showed that an increased contribution from the flow-dependent covariance results in a deterioration of nitrates, similar to SEIK, while the improvement of HYBRID with increasing flow-dependent ensemble size quickly levels off.

A weak-constraint 4DEnsembleVar. Part I: formulation and simple model experiments

ABSTRACT4DEnsembleVar is a hybrid data assimilation method which purpose is not only to use ensemble flow-dependent covariance information in a variational setting, but to altogether avoid the computation of tangent linear and adjoint models. This formulation has been explored in the context of perfect models. In this setting, all information from observations has to be brought back to the start of the assimilation window using the space-time covariances of the ensemble. In large models, localisation of these covariances is essential, but the standard time-independent localisation leads to serious problems when advection is strong. This is because observation information is advected out of the localisation area, having no influence on the update.This is part I of a two-part paper in which we develop a weak-constraint formulation in which updates are allowed at observational times. This partially alleviates the time-localisation problem. Furthermore, we provide—for the first time—a detailed description of strong- and weak-constraint 4DEnVar, including implementation details for the incremental form.The merits of our new weak-constraint formulation are illustrated using the Korteweg-de-Vries equation (propagation of a soliton). The second part of this paper deals with experiments in larger and more complicated models, namely the Lorenz (1996) model and a shallow water equations model with simulated convection.

Evaluation of Wind Forecasts and Observation Impacts from Variational and Ensemble Data Assimilation for Wind Energy Applications

Abstract The U.S. Department of Energy Wind Forecast Improvement Project (WFIP) has recently been completed with the aim of 1) understanding the performance of different mesoscale data assimilation systems for lower-atmospheric wind prediction and 2) determining the observation impacts on wind forecasts within the different assimilation systems. Here an ensemble Kalman filter (EnKF) was tested against a three-dimensional variational data assimilation (3DVAR) technique. Forecasts lasting 24 hours were produced for a month-long period to determine the day-to-day performance of each system, as well as over 10 individual wind ramp cases. The observation impacts from surface mesonet and profiler/sodar wind observations aloft were also tested in each system for both the month-long run and the ramp forecasts. It was found that EnKF forecasts verified over a domain including Texas and Oklahoma were better than those of 3DVAR for the month-long experiment throughout the forecast window, presumably from the use of flow-dependent covariances in the EnKF. The assimilation of mesonet data improved both EnKF and 3DVAR early forecasts, but sodar/profiler data showed a degradation (EnKF) or had no effect (3DVAR), with the degradation apparently resulting from a lower-atmospheric wind bias. For the wind ramp forecasts, ensemble averaging appears to overwhelm any improvements flow-dependent assimilation may have on ramp forecasts, leading to better 3DVAR ramp prediction. This suggests that best member techniques within the EnKF may be necessary for improved performance over 3DVAR for forecasts of sharp features such as wind ramps. Observation impacts from mesonet and profiler/sodar observations generally improved EnKF ramp forecasts, but either had little effect on or degraded 3DVAR forecasts.

Some Observing System Simulation Experiments with a Hybrid 3DEnVAR System for Storm-Scale Radar Data Assimilation

AbstractA hybrid three-dimensional ensemble–variational data assimilation (3DEnVAR) algorithm is developed based on the 3D variational data assimilation (3DVAR) and ensemble Kalman filter (EnKF) programs with the Advanced Regional Prediction System (ARPS). The method uses the extended control variable approach to combine the static and ensemble-derived flow-dependent forecast error covariances. The method is applied to the assimilation of simulated data from two radars for a supercell storm. Some sensitivity experiments are performed to answer questions about how flow-dependent covariance estimated from the forecast ensemble can be best used in the hybrid 3DEnVAR scheme. When the ensemble size is relatively small (with 5 or 10 ensemble members), it is found that experiments with a weaker weighting value for the ensemble covariance leads to better analysis results. Even when severe sampling errors exist, introducing ensemble-estimated covariances into the variational method still benefits the analysis. For reasonably large ensemble sizes (50–100 members), a stronger relative weighting (>0.8) for the ensemble covariance leads to better analyses from the hybrid 3DEnVAR. In addition, the sensitivity experiments also indicate that the best results are obtained when the number of the augmented control variables is a function of three spatial dimensions and ensemble members, and is the same for all analysis variables.

Comparison of Surface Wind and Temperature Analyses from an Ensemble Kalman Filter and the NWS Real-Time Mesoscale Analysis System

Abstract Operational, high-resolution analyses are a vital part of the National Weather Service (NWS) forecasting process. Currently, the Real-Time Mesoscale Analysis (RTMA) system fills this need by providing hourly analyses using a two-dimensional variational assimilation scheme. While success has been shown with the RTMA, an ensemble Kalman filter (EnKF) approach should outperform the RTMA, since the EnKF utilizes purely flow-dependent covariances during assimilation. The purpose of this study is to compare surface wind and temperature analyses from an EnKF to those of the RTMA to determine the relative skill of each approach. To reveal the influence of complex terrain, comparisons are performed for both the U.S. Pacific Northwest and Midwest. As expected, EnKF analysis increments reveal structures that align with the instantaneous flow, particularly regarding the wind field, for which the EnKF produces superior analyses. The EnKF is no better than the RTMA in strongly varying terrain, which may be a result of enhanced representativeness error in such regions. In contrast, temperature analysis increments are far less sensitive to flow dependence and are similar for both the EnKF and RTMA. RTMA temperature analyses possess slightly better skill than those produced by the EnKF, likely due to sampling error within the EnKF. Similar results for wind and temperature are found when assimilating significantly more and less observations in both systems. The implications of these results for operational production of finescale analyses are discussed.

Impact of Flow Dependence, Column Covariance, and Forecast Model Type on Surface-Observation Assimilation for Probabilistic PBL Profile Nowcasts

Abstract A probabilistic verification and factor-separation analysis (FSA) elucidate skillful nowcasts of planetary boundary layer (PBL) temperature, moisture, and wind profiles with a single-column model (SCM) and ensemble filter (EF) assimilation of surface observations. Recently, an FSA showed the importance of surface assimilation versus advection and radiation on ensemble-mean skill. That work addressed the necessary complexity of the model and assimilation scheme for improving PBL nowcasts, relative to deterministic-mesoscale predictions. Here, probabilistic ensemble-based SCM forecasts are compared to a simple probabilistic postprocessing scheme termed climatological dressing (CD). CD adjusts a deterministic mesoscale forecast using surface-atmosphere 3D-climatological covariances, a 30-min persistence model, and surface-forecast errors. It also dresses the adjusted profile with an in-sample uncertainty distribution (obtained from archives) scaled by the 30-min forecast error. Superior deterministic skill from SCM/EF results during night when flow-dependent covariances are more accurate than climatological covariances. CD is deterministically more skillful for temperature and moisture profiles during daytime because SCM/PBL parameterization yields biased covariances. SCM/EF is most probabilistically skillful because (a) the EF covariances accommodate large seasonal variability, (b) the 30-min error persistence assumption fails during nighttime, and (c) vertical error covariance estimates from archived forecasts are generally poor estimates of actual error covariances. A probabilistic FSA of the SCM/EF shows the relative importance of surface assimilation, radiation parameterization, and advection during night. Results confirm surface assimilation as the most important factor. A factor can be deterministically beneficial and probabilistically detrimental, or vice versa, depending on its role in reducing mean error or improving sharpness. Assimilation results in notable probabilistic improvement for nowcasts of low-level jet structures.

Application of the WRF Hybrid ETKF–3DVAR Data Assimilation System for Hurricane Track Forecasts

Abstract A hybrid ensemble transform Kalman filter (ETKF)–three-dimensional variational data assimilation (3DVAR) system developed for the Weather Research and Forecasting Model (WRF) was studied for the forecasts of the tracks of two major hurricanes, Ike and Gustav, in 2008 over the Gulf of Mexico. The impacts of the flow-dependent ensemble covariance generated by the ETKF were revealed by comparing the forecasts, analyses, and analysis increments generated by the hybrid data assimilation method with those generated by the 3DVAR that used the static background covariance. The root-mean-square errors of the track forecasts by the hybrid data assimilation (DA) method were smaller than those by the 3DVAR for both Ike and Gustav. Experiments showed that such improvements were due to the use of the flow-dependent covariance provided by the ETKF ensemble in the hybrid DA system. Detailed diagnostics further revealed that the increments produced by the hybrid and the 3DVAR were different for both the analyses of the hurricane itself and its environment. In particular, it was found that the hybrid, using the flow-dependent covariance that gave the hurricane-specific error covariance estimates, was able to systematically adjust the position of the hurricane during the assimilation whereas the 3DVAR was not. The study served as a pilot study to explore and understand the potential of the hybrid method for hurricane data assimilation and forecasts. Caution needs to be taken to extrapolate the results to operational forecast settings.

Intercomparison of Variational Data Assimilation and the Ensemble Kalman Filter for Global Deterministic NWP. Part I: Description and Single-Observation Experiments

Abstract An intercomparison of the Environment Canada variational and ensemble Kalman filter (EnKF) data assimilation systems is presented in the context of global deterministic NWP. In an EnKF experiment having the same spatial resolution as the inner loop in the four-dimensional variational data assimilation system (4D-Var), the mean of each analysis ensemble is used to initialize the higher-resolution deterministic forecasts. Five different variational data assimilation experiments are also conducted. These include both 4D-Var and 3D-Var (with first guess at appropriate time) experiments using either (i) prescribed background-error covariances similar to those used operationally, which are static in time and include horizontally homogeneous and isotropic correlations; or (ii) flow-dependent covariances computed from the EnKF background ensembles with spatial covariance localization applied. The fifth variational data assimilation experiment is a new approach called the Ensemble-4D-Var (En-4D-Var). This approach uses 4D flow-dependent background-error covariances estimated from EnKF ensembles to produce a 4D analysis without the need for tangent-linear or adjoint versions of the forecast model. In this first part of a two-part paper, results from a series of idealized assimilation experiments are presented. In these experiments, only a single observation or vertical profile of observations is assimilated to explore the impact of various fundamental differences among the EnKF and the various variational data assimilation approaches considered. In particular, differences in the application of covariance localization in the EnKF and variational approaches are shown to have a significant impact on the assimilation of satellite radiance observations. The results also demonstrate that 4D-Var and the EnKF can both produce similar 4D background-error covariances within a 6-h assimilation window. In the second part, results from medium-range deterministic forecasts for the study period of February 2007 are presented for the EnKF and the five variational data assimilation approaches considered.

The Choice of Variable for Atmospheric Moisture Analysis

The implications of using different control variables for the analysis of moisture observations in a global atmospheric data assimilation system are investigated. A moisture analysis based on either mixing ratio or specific humidity is prone to large extrapolation errors, due to the high variability in space and time of these parameters and to the difficulties in modeling their error covariances. Using the logarithm of specific humidity does not alleviate these problems, and has the further disadvantage that very dry background estimates cannot be effectively corrected by observations. Relative humidity is a better choice from a statistical point of view, because this field is spatially and temporally more coherent and error statistics are therefore easier to obtain. If, however, the analysis is designed to preserve relative humidity in the absence of moisture observations, then the analyzed specific humidity field depends entirely on analyzed temperature changes. If the model has a cool bias in the stratosphere this will lead to an unstable accumulation of excess moisture there. A pseudo-relative humidity can be defined by scaling the mixing ratio by the background saturation mixing ratio. A univariate pseudo-relative humidity analysis will preserve the specific humidity field in the absence of moisture observations. A pseudorelative humidity analysis is shown to be equivalent to a mixing ratio analysis with flow-dependent covariances. In the presence of multivariate (temperature-moisture) observations it produces analyzed relative humidity values that are nearly identical to those produced by a relative humidity analysis. Based on a time series analysis of radiosonde observed-minus-background differences it appears to be more justifiable to neglect specific humidity-temperature correlations (in a univariate pseudo-relative humidity analysis) than to neglect relative humidity-temperature correlations (in a univariate relative humidity analysis). A pseudo-relative humidity analysis is easily implemented in an existing moisture analysis system, by simply scaling observed-minus background moisture residuals prior to solving the analysis equation, and rescaling the analyzed increments afterward.

A Hybrid Ensemble Kalman Filter–3D Variational Analysis Scheme

A hybrid ensemble Kalman filter–three-dimensional variational (3DVAR) analysis scheme is demonstrated using a quasigeostrophic model under perfect-model assumptions. Four networks with differing observational densities are tested, including one network with a data void. The hybrid scheme operates by computing a set of parallel data assimilation cycles, with each member of the set receiving unique perturbed observations. The perturbed observations are generated by adding random noise consistent with observation error statistics to the control set of observations. Background error statistics for the data assimilation are estimated from a linear combination of time-invariant 3DVAR covariances and flow-dependent covariances developed from the ensemble of short-range forecasts. The hybrid scheme allows the user to weight the relative contributions of the 3DVAR and ensemble-based background covariances. The analysis scheme was cycled for 90 days, with new observations assimilated every 12 h. Generally, it was found that the analysis performs best when background error covariances are estimated almost fully from the ensemble, especially when the ensemble size was large. When small-sized ensembles are used, some lessened weighting of ensemble-based covariances is desirable. The relative improvement over 3DVAR analyses was dependent upon the observational data density and norm; generally, there is less improvement for data-rich networks than for data-poor networks, with the largest improvement for the network with the data void. As expected, errors depend on the size of the ensemble, with errors decreasing as more ensemble members are added. The sets of initial conditions generated from the hybrid are generally well calibrated and provide an improved set of initial conditions for ensemble forecasts.