4D Variational Data Assimilation Research Articles

Data assimilation with machine learning for dynamical systems: Modelling indoor ventilation

Data assimilation is a method of combining physical observations with prior knowledge (for instance, a computational simulation) in order to produce an improved estimate of the state of a system; that is, improved over what the physical observations or the computational simulation, alone, could offer. Recently, machine learning techniques have been deployed in order to address the significant computational burden that is associated with the procedures involved in data assimilation.In this paper we propose an approach that uses a non-intrusive reduced-order model (NIROM) as a surrogate for a high-resolution model thereby saving computational effort. The mismatch between observations and the surrogate model is propagated forwards and backwards in time in a manner similar to 4D-variational data assimilation methods. The observations and prior are reconciled in a new way which takes full advantage of the neural network used in the NIROM and also means that there is no need to form the sensitivities explicitly when propagating the mismatch. Instead, the observations are part of the input and output of the network. The key advantages of this data assimilation approach are its simplicity (as we can avoid differentiating the forward model and bypass the use of an optimisation method), and its ability to integrate with control and uncertainty quantification methodologies.Modelling the air quality in a school classroom is the test case for our demonstration. After comparing the proposed data assimilation approach with 4D Variational Data Assimilation, we investigate two scenarios. The first of these is a dual-twin type experiment, for which the proposed approach is shown to perform very well, and the second is a test case which assimilates predictions from the NIROM with observations collected from a classroom in Houndsfield Primary School.

Foundations for Universal Non‐Gaussian Data Assimilation

AbstractIn many applications of data assimilation, especially when the size of the problem is large, a substantial assumption is made: all variables are well‐described by Gaussian error statistics. This assumption has the advantage of making calculations considerably simpler, but it is often not valid, leading to biases in forecasts or, even worse, unphysical predictions. We propose a simple, but effective, way of replacing this assumption, by making use of transforming functions, while remaining consistent with Bayes' theorem. This method allows the errors to have any value of the skewness and kurtosis, and permits physical bounds for the variables. As such, the error distribution can conform better to the underlying statistics, reducing biases introduced by the Gaussian assumption. We apply this framework to a 3D variational data assimilation method, and find improved performance in a simple atmospheric toy model (Lorenz‐63), compared to an all‐Gaussian technique.

Tropospheric ozone data assimilation in the NASA GEOS Composition Forecast modeling system (GEOS-CF v2.0) using satellite data for ozone vertical profiles (MLS), total ozone columns (OMI), and thermal infrared radiances (AIRS, IASI)

The NASA Goddard Earth Observing System Composition Forecast system (GEOS-CF) provides global near-real-time analyses and forecasts of atmospheric composition. The current version of GEOS-CF builds on the GEOS general circulation model with Forward Processing assimilation of meteorological data (GEOS-FP) and includes detailed GEOS-Chem tropospheric and stratospheric chemistry. Here we add 3D variational data assimilation in GEOS-CF to assimilate satellite observations of ozone including MLS vertical profiles, OMI total columns, and AIRS and IASI hyperspectral 9.6 μm radiances. We focus our evaluations on the troposphere. We find that the detailed tropospheric chemistry in GEOS-CF significantly improves the simulated background ozone fields relative to previous versions of the GEOS model, allowing for specification of smaller background errors in assimilation and resulting in smaller assimilation increments to correct the simulated ozone. Assimilation increments are largest in the upper troposphere and are consistent between satellite data sets. The OMI and MLS ozone data generally provide more information than the AIRS and IASI radiances except at high latitudes where the radiances provide more information. Comparisons to independent ozonesonde and aircraft (ATom-4) observations for 2018 show significant GEOS-CF improvement from the assimilation, particularly in the extratropical upper troposphere.

A data-assimilative modeling investigation of Gulf Stream variability

An advanced data-assimilative ocean circulation model is used to investigate Gulf Stream (GS) variability during 2017–2018. The modeling system applies a strong-constraint, 4D variational data assimilation algorithm. It assimilates satellite-based sea surface height and sea surface temperature measurements and in situ temperature and salinity profiles. Model skill assessment metrics along with comparisons of GS position and GS's three-dimensional mean kinetic energy with historical observations are applied to validate the data-assimilative model. The resulting time- and space-continuous ocean state estimates are used to diagnose eddy kinetic energy conversion and cross-stream eddy heat and salt fluxes over the two-year study period. The processes leading to kinetic energy conversion are primarily due to GS meanders. Significant inverse energy cascading (EKE→MKE and EKE→EPE) can occur during GS-eddy interactions, particularly during onshore intrusions or offshore meanderings of the GS. Throughout the two-year study period, the cross-stream eddy heat and salt fluxes off Cape Hatteras were predominantly positive (onshore). Both GS offshore meandering (occurring 44% of the time and associated with shelf/slope water export) and GS intrusion (occurring 56% of the time) contribute to onshore heat and salt transport. Improved understanding of these processes and dynamics requires strong integration of an advanced observational infrastructure that combines remote sensing; fixed, mobile, and shore-based observing components; and high-resolution data assimilative models.

Sensitivity Analysis of the Data Assimilation-Driven Decomposition in Space and Time to Solve PDE-Constrained Optimization Problems

This paper is presented in the context of sensitivity analysis (SA) of large-scale data assimilation (DA) models. We studied consistency, convergence, stability and roundoff error propagation of the reduced-space optimization technique arising in parallel 4D Variational DA problems. The results are helpful to understand the reliability of DA, to assess what confidence one can have that the simulation results are correct and to determine its configuration in any application. The main contributions of the present work are as follows. By using forward error analysis, we derived the number of conditions of the parallel approach. We found that the parallel approach reduces the number of conditions, revealing that it is more appropriate than the standard approach usually implemented in most operative software. As the background values are used as initial conditions of local PDE models, we analyzed stability with respect to time direction. Finally, we proved consistency of the proposed approach by analyzing local truncation errors of each computational kernel.

Inverse Modeling of Formaldehyde Emissions and Assessment of Associated Cumulative Ambient Air Exposures at Fine Scale

Among air toxics, formaldehyde (HCHO) is an important contributor to urban cancer risk. Emissions of HCHO in the United States are systematically under-reported and may enhance atmospheric ozone and particulate matter, intensifying their impacts on human health. During the 2021 Michigan-Ontario Ozone Source Experiment (MOOSE), mobile real-time (~1 s frequency) measurements of ozone, nitrogen oxides, and organic compounds were conducted in an industrialized area in metropolitan Detroit. The measured concentrations were used to infer ground-level and elevated emissions of HCHO, CO, and NO from multiple sources at a fine scale (400 m horizontal resolution) based on the 4D variational data assimilation technique and the MicroFACT air quality model. Cumulative exposure to HCHO from multiple sources of both primary (directly emitted) and secondary (atmospherically formed) HCHO was then simulated assuming emissions inferred from inverse modeling. Model-inferred HCHO emissions from larger industrial facilities were greater than 1 US ton per year while corresponding emission ratios of HCHO to CO in combustion sources were roughly 2 to 5%. Moreover, simulated ambient HCHO concentrations depended significantly on wind direction relative to the largest sources. The model helped to explain the observed HCHO concentration gradient between monitoring stations at Dearborn and River Rouge in 2021.

The set‐up and evaluation of fine‐scale data assimilation for the urban climate of Amsterdam

AbstractOngoing urbanization highlights the need for a better understanding and high resolution modelling of the urban climate. In this study, we combine rural observations by WMO surface stations, weather radar data and urban crowd‐sourced observations with very fine‐scale modelling efforts for Amsterdam, The Netherlands. As a model, we use the Weather Research and Forecasting (WRF) mesoscale model with 3D variational data assimilation at a 100‐m resolution in the innermost model domain. In order to enable the assimilation of observations within the urban canopy, we develop a scheme to reduce urban temperature biases by adjusting urban fabric temperatures. The scheme is tested against independent urban observations for the summer month of July 2014 and specifically for a hot period and an extreme precipitation event. We find data assimilation reduces biases in temperature and wind speed. Within the city, the most significant improvement is the reduction of negative temperature biases during clear nights, which implies a better prediction of the Urban Heat Island (UHI). Concerning precipitation, the fractional skill score improves incrementally when additional observations are assimilated, and the largest impact is seen from the assimilation of weather radar observations.

Variational parameter estimation in a two-equation turbulence model: A case study with a 3D primitive-equation ocean model

A three-dimensional and complete adjoint model of the Princeton Ocean Model with a generalized coordinate system (POMgcs) is developed to construct the 4D-Variational data assimilation (4D-Var) algorithm in this study. Uncertain parameters in the Mellor-Yamada 2.5 turbulence submodel (MY-2.5) which is enclosed in POMgcs, are tentatively estimated via the 4D-VAR algorithm within a biased model framework. Here, the control variables in the biased model are set to two uncertain wave-affected parameters (wave energy factor α and Charnock coefficient β ) in the MY-2.5 turbulence model, which play a crucial role in modulating the heat content distribution in the upper coastal sea. First of all, the ocean temperature and salinity in a typical coastal sea, Bohai Sea, are simulated by the model to validate the rationally of the MY-2.5 parameterization scheme for both constructing the “truth model” and generating the “pseudo-observations” in the data assimilation studies. Then, after thoroughly testing the ability of the 4D-Var to optimize the initial state fields of the POMgcs model, a series of parameter estimation experiments are carried out to investigate whether and to what degree the parameters embedded in high-order turbulence models can be significantly optimized. Results of parameter estimation with perfect initial fields show the two estimated parameters in the MY-2.5 submodel can successfully converge to the “truth” value. The local minimum of the cost function can be effectively and efficiently jumped out once two kinds of optimization algorithms, LBFGS and LMBM, are jointly used. In addition, the estimated parameter will converge to the optimal value rather than the truth one to compensate for the initial field error when the state-parameter are estimated simultaneously. Further, the performance of the parameter estimation is also deeply discussed when the observation samples are noised. Finally, prescribing the initial field and parameter as error source, a forecasting experiment for sea temperature is performed. The experiment results indicate that assimilating “pseudo-observations” to the model based on 4D-Var can significantly improve the sea temperature simulation. Moreover, adjusting the initial field and parameter leads to a better result than the only initial field, and this conclusion is more evident at the surface than in the deeper ocean.

The Effect of Assimilating AMSU-A Radiance Data from Satellites and Large-Scale Flows from GFS on Improving Tropical Cyclone Track Forecast

This study aimed to investigate the effect of assimilating either AMSU-A radiance data from satellites, large-scale flows from the Global Forecast System (GFS), or both together, on improving the track forecast of tropical cyclone (TC). The scale-selective data assimilation (SSDA) approach was employed for the assimilation of large-scale GFS flows, while the conventional 3D variational data assimilation (3DVAR) method was used for that of AMSU-A radiance data. The results show that assimilating either AMSU-A radiance data or large-scale GFS flows has a significant improvement on TC track forecast, but the improvement occurs within the first 72 h and after 72 h, respectively. When assimilating both AMSU-A radiance data and large-scale GFS flows, the forecast can take advantage of both data and thus lead to the smallest 5-day mean errors of the track forecast. These results are instructive to future operational TC track forecasting.

Medicane Ianos: 4D-Var Data Assimilation of Surface and Satellite Observations into the Numerical Weather Prediction Model WRF

This work investigates extreme weather events such as the onset of medicanes, which can cause severe socioeconomic impacts, along with their predictability. In order to accurately forecast such events, the Weather Research and Forecasting (WRF) model and its state-of-the-art data assimilation modeling framework (WRFDA) were set up to produce high-resolution forecasts for the case study of Medicane Ianos, which affected Greece between 17 and 19 September 2020. Information from weather stations and the satellite precipitation product IMERG was blended with the background model information from the Global Forecast System (GFS) using the 4D variational data assimilation (4D-Var) technique. New fields in an 18 km spatial resolution domain covering Europe were generated and utilized as improved initial conditions for the forecast model. Forecasts were issued based on these improved initial conditions at two nested domains of 6 km and 2 km spatial resolution, with the 2 km domain enclosing Greece. Denial experiments, where no observational data were assimilated in the initial boundary conditions, showed that the temperature fields benefited throughout the forecasting horizon from the assimilation (ranging from a 5 to 10% reduction in the average MAE values), while neutral to slightly positive (ranging from a 0.4 to 2% reduction in the average MAE values) improvement was found for wind, although not throughout the forecast horizon. The increase in spatial resolution did not significantly reduce the forecast error, but was kept at the same small order of magnitude. A tendency of the model to overpredict precipitation regardless of assimilation was observed. The assimilation of the IMERG data improved the precipitation forecasting ability up to the 18th hour of forecast. When compared to assimilation experiments that excluded IMERG data, the assimilation of IMERG data produced a better representation of the spatial distribution of the precipitation fields.

Rapid reconstruction of temperature and salinity fields based on machine learning and the assimilation application

Satellite observations play important roles in ocean operational forecasting systems, however, the direct assimilation of satellite observations cannot provide sufficient constraints on the model underwater structure. This study adopted the indirect assimilation method. First, we created a 3D temperature and salinity reconstruction model that took into account the advantage of the nonlinear regression of the generalized regression neural network with the fruit fly optimization (abbreviated as FOAGRNN). Compared with the reanalysis product and the WOA13 climatology data, the synthetic T/S (temperature and salinity) profiles had sufficient accuracy and could better describe the characteristics of mesoscale eddies. Then, the synthetic T/S profiles were assimilated into the Regional Ocean Model System (ROMS) using the Incremental Strong constraint 4D Variational (I4D-Var) data assimilation algorithm. The quantitative and qualitative analysis results indicated that compared with the direct assimilation of satellite observations, the root mean square errors (RMSEs) of temperature and salinity were reduced by 26.0% and 23.1% respectively by assimilating the synthetic T/S profiles. Furthermore, this method significantly improved the simulation effect of the model underwater structure, especially in the 300 m to 500 m water layer. Compared with the National Marine Data Center’s real-time analysis data, the machine learning-based assimilation system demonstrated a significant advantage in the simulation of underwater salinity structure, while showing a similar performance in the simulation of underwater temperature structure.

Variational data assimilation for a sea dynamics model

Abstract The 4D variational data assimilation technique is presented for modelling the sea dynamics problems, developed at the Marchuk Institute of Numerical Mathematics of the Russian Academy of Sciences (INM RAS). The approach is based on the splitting method for the mathematical model of sea dynamics and the minimization of cost functionals related to the observation data by solving an optimality system that involves the adjoint equations and observation and background error covariances. Efficient algorithms for solving the variational data assimilation problems are presented based on iterative processes with a special choice of iterative parameters. The technique is illustrated for the Black Sea dynamics model with variational data assimilation to restore the sea surface heat fluxes.

Predictive skill comparative assessment of WRF 4DVar and 3DVar data assimilation: An Indian Ocean tropical cyclone case study

Tropical cyclones (TCs) predictions are crucial and of significant scientific and socio-economic interest. To improve the TC's simulation, this study aimed to use the Weather Research and Forecasting (WRF) model to assimilate satellite radiances and surface and upper-air conventional observations cyclically at the intervals of 6 h. The predictive skill of WRF with 4D variational (4DVar) and 3D variational (3DVar) data assimilation (DA) systems are investigated to examine the added value of 4DVar considering four cyclones namely Vardah, Titli, Bulbul, and Yaas over the North Indian Ocean (NIO). Each analysis was subjected to a short-range free forecast, and the estimates obtained from the 4DVar DA showed more realistic TC characteristics in terms of structure, intensity, rainfall, and track. The simulation of mean sea-level pressure and maximum sustained wind has demonstrated significant improvements in the 4DVar run for all the cases. Furthermore, the vector displacement error in cyclone Vardah was reduced by 54%–57% in the 4DVar simulation, whereas this improvement was found to be 9%–18% in the case of Titli. More or less similar results were found for Bulbul and Yaas also. Simulation of rainfall also showed marked improvement from the 4DVar scheme. Thus, the performance of the WRF model integrated with the 4DVar DA method enhanced the simulations of TC intensity, tracks, and structures over the NIO.

A Scalable Space-Time Domain Decomposition Approach for Solving Large Scale Nonlinear Regularized Inverse Ill Posed Problems in 4D Variational Data Assimilation

We address the development of innovative algorithms designed to solve the strong-constraint Four Dimensional Variational Data Assimilation (4DVar DA) problems in large scale applications. We present a space-time decomposition approach which employs the whole domain decomposition, i.e. both along the spacial and temporal direction in the overlapping case, and the partitioning of both the solution and the operator. Starting from the global functional defined on the entire domain, we get to a sort of regularized local functionals on the set of sub domains providing the order reduction of both the predictive and the Data Assimilation models. The algorithm convergence is developed. Performance in terms of reduction of time complexity and algorithmic scalability is discussed on the Shallow Water Equations on the sphere. The number of state variables in the model, the number of observations in an assimilation cycle, as well as numerical parameters as the discretization step in time and in space domain are defined on the basis of discretization grid used by data available at repository Ocean Synthesis/Reanalysis Directory of Hamburg University.

4D Technology of Variational Data Assimilation for Sea Dynamics Problems

The technology aimed at high-performance computing is presented for modeling the sea dynamics problems based on 4D variational data assimilation technique developed at the Marchuk Institute of Numerical Mathematics, Russian Academy of Sciences (INM RAS). The technology is based on the multicomponent splitting method for the mathematical model of sea dynamics and the minimization of cost functionals related to the observation data by solving an optimality system that involves the adjoint equations with observation data and observation error covariances. Efficient algorithms for solving the variational data assimilation problems are presented based on modern iterative processes with a special choice of iterative parameters. The technology is illustrated for the Baltic Sea dynamics model with variational data assimilation to restore the initial states and the heat fluxes on the sea surface.

Comparing 3DVAR and hybrid radar data assimilation methods for heavy rain forecast

Radar data are assimilated into numerical prediction models to improve precipitation forecasts using three-dimensional (3D) variational (VAR) and hybrid data assimilation (DA) methods. The effect of radar DA (RDA) and prediction accuracy according to DA methods are evaluated by conducting experiments of three heavy rainfall events and analyzing the spatial pattern of accumulated precipitation, profiles of hydrometeors, and cloud microphysical processes. Increments of the initial analysis revealed that the hybrid method simulated the convective band, wind convergence in front of the cold front and a higher amount of water vapor mixing ratio compared to 3DVAR. Based on numerical experiments, the root mean square error of the total cumulative precipitation in 3DVAR was 9.68 mm lower than that of the no-RDA experiment (CTRL), whereas that of the Hybrid was 11.12 mm lower than that of CTRL. RDA improved precipitation forecasts, and Hybrid showed better precipitation forecast accuracy than 3DVAR. Although RDA indirectly changed the water vapor, rain, snow, and graupel mixing ratios, the change in the water vapor amount was the major factor affecting microphysical processes. The increase (decrease) in the water vapor mixing ratio had the greatest impact on precipitation formation (dissipation), whereas the effect of snow, rain, and graupel mixing ratios was relatively small. The results indicate that it is critical to create an environment in which water vapor can transform into precipitating hydrometeors and the hydrometeors can grow through cloud microphysical processes. The accuracy of heavy rain forecast can be improved by hybrid DA method, which can consider the model error in real time, making it more adaptable to the day-to-day changing weather conditions.

A multi-physics ensemble approach for short-term precipitation forecasts at convective permitting scales based on sensitivity experiments over southern parts of peninsular India

The southern peninsular India is characterized by unique climatology with rainfall processes throughout the year from land–ocean contrasts. In addition, the complex terrain induces localized effects causing huge spatial and temporal variability in the observed precipitation. This study aims at evaluating the sensitivity of the high-resolution Weather Research and Forecasting (WRF) model (4 km) to multi-physics parameterizations, 3D variational data assimilation, and domain configuration, in the study domain covering southern peninsular India. Furthermore, the study focusses on the formulation of an ensemble method to improve the simulation of precipitation across seasons. A total of 120 experiments were set up across four crucial rainfall events, of varying spatial extent and duration, dominated by different rainfall generation mechanisms. The assessment of the experiments shows that the model’s cumulus and microphysics schemes have the highest impact on the location, intensity, and spread of the simulated 4-day long Quantitative Precipitation Forecasts (QPFs). Applying cumulus schemes at all domains represented the variability in the QPFs, across space and time, for the precipitation events dominated by convective activity. The cases without cumulus schemes at the convective scale domain (4 km), captured the higher intensity rains during organized cyclonic circulations in the north-east monsoon period. Hence, a 10-member multi-physics ensemble approach including members with and without cumulus parameterization at the fine resolution domain was adopted. The preliminary results demonstrate that the mean from the suggested ensemble approach (n-MPP) performed well in capturing the dynamics of QPFs across the rainfall events, as opposed to a single-member deterministic simulation and mean from larger member conventional multi-physics ensemble approach (c-MPP) without cumulus parameterization at the convective scale. The rank histogram, delta semi-variance plots, and outlier statistics at various lead times clearly showed that the suggested n-MPP was able to capture the high-intensity rainfall, increasing the spread of precipitation forecasts and consequently reducing the occurrence of outliers.

Assessment of extremely severe cyclonic storms over Bay of Bengal and performance evaluation of ARW model in the prediction of track and intensity

A detailed assessment was carried out for extremely severe cyclonic storms (ESCSs) that developed over the Bay of Bengal during 1990–2020 using the India Meteorological Department (IMD) best-fit track data. During the past 30 years, the maximum intensity of land-falling ESCS has increased 26%, which is an average of about 8% increase per decade. Analysis signifies an increasing trend in the life cycle, duration of ESCS stage, and maximum wind speed of land-falling ESCSs. It is observed that the stage of land-falling ESCSs was very severe, and therefore, reliable forecast of land-falling ESCSs is very important having wide socioeconomic implications. Furthermore, a case study investigated the performance of Advanced Research of Weather Research and Forecasting (ARW) model for a 6-day forecast of ESCS Hudhud that developed over the Bay of Bengal during 2014. Performance evaluation was based on the impact of model domain size, vertical resolution, data assimilation, sea surface temperature (SST), gravity wave option (GWO), and air-sea flux parameterization schemes. Initial condition was improved through WRF 3D variational data assimilation (WRF-3DVAR) system. Thereafter, the simulated track, intensity, and intensification of the storm were validated against available IMD best-fit track datasets. Results from the ARW model indicate that the track and intensity of ESCS Hudhud were influenced by domain size, vertical resolution, data assimilation, SST, GWO, and air-sea flux schemes. Study deciphers that bigger domain, higher vertical resolution, data assimilation, and air-sea flux scheme with enthalpy coefficients provided a better forecast for ESCS Hudhud. The track errors on day 1 to day 4 was 61 km, 73 km, 85 km, and 96 km, respectively, and absolute error in terms of MSW was about 8 ms−1, 7 ms−1, 2 ms−1, and 8 ms−1 respectively at 9-km horizontal resolution.

Impacts on Initial Condition Modification from Hyperspectral Infrared Sounding Data Assimilation: Comparisons between Full-Spectrum and Channel-Selection Scheme Based on Two-Month Experiments Using CrIS and IASI Observation

This paper discusses the performance difference between full-spectrum and channel-selection assimilation scheme of hyperspectral infrared observation, e.g. CrIS and IASI, on improving the accuracy of initial condition in numerical weather prediction. To accomplish this, we develop a 3D-Variational data assimilation system whose observation operator is a principal-component based fast radiative transfer model, which equips the direct assimilation of full-channel radiance from hyperspectral infrared sounders with high computational efficiency. This project’s primary goal is to demonstrate that assimilation of infrared observation in a full-channel mode could improve the accuracy of initial condition compared to selected-channel assimilation. Results show that full-channel assimilation performs better than selected-channel assimilation in modifying low and middle troposphere (1000 - 700 hPa, 700 - 400 hPa) temperature and water vapor field, while marginal improvements from temperature and water vapor field could be found over upper troposphere (400 - 100 hPa). This research also proves the feasibility of an alternative path to data assimilation for the full usage of hyperspectral infrared sounding observation in numerical weather prediction.

Model reduction in space and time for ab initio decomposition of 4D variational data assimilation problems

We present an innovative approach for solving time dependent Four Dimensional Variational Data Assimilation (4D VAR DA) problems. The proposed approach performs a decomposition of the whole physical domain, i.e. both along spatial and temporal directions; a reduction in space and time of both the Partial Differential Equations-based model and the Data Assimilation functional; finally it uses a modified regularization functional describing restricted 4D VAR DA problems on the domain decomposition. Innovation mainly lies in the introduction ab initio, i.e. on the numerical model – of a domain decomposition approach in space and time joining the idea of Schwarz's method and Parallel in Time (PinT)–based approaches. We provide the numerical framework of this method including convergence analysis and error propagation. A validation analysis is performed discussing computational results on a case study relying on Shallow Water Equations.