Use In Data Assimilation Research Articles

Latent diffusion models for parameterization of facies-based geomodels and their use in data assimilation

Geological parameterization entails the representation of a geomodel using a small set of latent variables and a mapping from these variables to grid-block properties such as porosity and permeability. Parameterization is useful for data assimilation (history matching), as it maintains geological realism while reducing the number of variables to be determined. Diffusion models are a new class of generative deep-learning procedures that have been shown to outperform previous methods, such as generative adversarial networks, for image generation tasks. Diffusion models are trained to “denoise,” which enables them to generate new geological realizations from input fields characterized by random noise. Latent diffusion models, which are the specific variant considered in this study, provide dimension reduction through use of a low-dimensional latent variable. The model developed in this work includes a variational autoencoder for dimension reduction and a U-net for the denoising process. Our application involves conditional 2D three-facies (channel-levee-mud) systems. The latent diffusion model is shown to provide realizations that are visually consistent with samples from geomodeling software. Quantitative metrics involving spatial and flow-response statistics are evaluated, and general agreement between the diffusion-generated models and reference realizations is observed. Stability tests are performed to assess the smoothness of the parameterization method. The latent diffusion model is then used for ensemble-based data assimilation. Two synthetic “true” models are considered. Significant uncertainty reduction, posterior P10–P90 forecasts that generally bracket observed data, and consistent posterior geomodels, are achieved in both cases.

Information geometry of physics-informed statistical manifolds and its use in data assimilation

The data-aware method of distributions (DAMD) is a low-dimensional data assimilation procedure to forecast the behavior of dynamical systems described by differential equations. The core of DAMD is the minimization of a distance between an observation and a prediction in distributional terms, with prior and posterior distributions constrained to a statistical manifold defined by the method of distributions (MD). We leverage the information-geometric properties of the statistical manifold to reduce predictive uncertainty via data assimilation. Specifically, we exploit the information-geometric structures induced by two discrepancy metrics, the Kullback-Leibler divergence and the Wasserstein distance, which explicitly yield natural gradient descent. The use of a deep neural network as a surrogate model for MD enables automatic differentiation, further accelerating optimization. The manifold's geometry is quantified without sampling, yielding an accurate approximation of the gradient descent direction. Our numerical experiments demonstrate that accounting for the manifold's geometry significantly reduces the computational cost of data assimilation by both facilitating the calculation of gradients and reducing the number of required iterations.

The Effects of Heavy Precipitation on Polarimetric Radio Occultation (PRO) Bending Angle Observations

Abstract Following the successful launch of the Spanish PAZ mission the proof of concept experiment “Radio Occultation and Heavy Precipitation with PAZ” (ROHP-PAZ) started operating in May 2018. The ROHP-PAZ observations demonstrated that precise measurements of the phase shift between horizontal and vertical polarizations from Global Navigation Satellite System (GNSS) L-band signals are sensitive to oriented hydrometeors along the ray paths. While this differential phase shift measurement as a function of time has proven very useful, the regular radio occultation (RO) intermediate products from different polarized channels, such as bending angle and phase retrievals on the domain of impact parameter, have never been exploited. In this research, we studied the characteristics of polarimetric phase and bending angle difference retrieved by the radio-holographic (RH) method to mitigate atmospheric multipath effect and to explore their use in data assimilation. To validate RH approach in polarimetric retrievals, we performed end-to-end simulations where the hydrometeors are modeled by the effective refractivity with different horizontal extents. The simulation results demonstrate that the strong precipitation (>15 mm h−1) with 40-km horizontal extent can be detected with the retrieved bending angle shift. The calibration process on the impact parameter domain has also been developed to extract the differential phase and bending angle shift from the actual polarimetric RO data. Statistics from the PAZ data shows that the mean retrieved RH polarimetric phase shift with various horizontal extent is approximately proportional to the tangent point location rain rate at a ratio of 0.02 rad (mm h−1)−1.

Machine learning techniques to construct patched analog ensembles for data assimilation

Using generative models from the machine learning literature to create artificial ensemble members for use within data assimilation schemes has been introduced in Grooms (2021) [1] as constructed analog ensemble optimal interpolation (cAnEnOI). Specifically, we study general and variational autoencoders for the machine learning component of this method, and combine the ideas of constructed analogs and ensemble optimal interpolation in the data assimilation piece. To extend the scalability of cAnEnOI for use in data assimilation on complex dynamical models, we propose using patching schemes to divide the global spatial domain into digestible chunks. Using patches makes training the generative models possible and has the added benefit of being able to exploit parallelism during the generative step. Testing this new algorithm on a 1D toy model, we find that larger patch sizes make it harder to train an accurate generative model (i.e. a model whose reconstruction error is small), while conversely the data assimilation performance improves at larger patch sizes. There is thus a sweet spot where the patch size is large enough to enable good data assimilation performance, but not so large that it becomes difficult to train an accurate generative model. In our tests the new patched cAnEnOI method outperforms the original (unpatched) cAnEnOI, as well as the ensemble square root filter results from [1].

Assimilation of GNSS reflectometry delay‐Doppler maps with a two‐dimensional variational analysis of global ocean surface winds

AbstractDirect remote‐sensing observations (e.g., radar backscatter, radiometer brightness temperature, or radio occultation bending angle) are often more effective for use in data assimilation (DA) than the corresponding geophysical retrievals (e.g., ocean surface winds, soil moisture, or atmospheric water vapor). In the particular case of Global Navigation Satellite System Reflectometry (GNSS‐R), the lower‐level delay‐Doppler map (DDM) observable shows a complicated relationship with the ocean surface wind field. Prior studies have demonstrated DA using GNSS‐R wind retrievals inferred from DDMs. The complexity of the DDM dependence on winds, however, suggests that the alternative approach of ingesting DDM observables directly into DA systems, without performing a wind retrieval, may be beneficial. We demonstrate assimilation of DDM observables from the NASA Cyclone Global Navigation Satellite System (CYGNSS) mission into global ocean surface wind analyses using a two‐dimensional variational analysis method. Bias correction and quality‐control methods are described. Several models for the required observation‐error covariance matrix are developed and evaluated, with the conclusion that a diagonal matrix performs as well as a fully populated matrix empirically tuned to a large ensemble of CYGNSS observation data. The 10‐m surface winds from the European Centre for Medium‐Range Weather Forecasts (ECMWF) operational forecast are used as the background (i.e., prior in the variational analysis). Results are compared with independent scatterometer (the advanced scatterometer (ASCAT), the oceansat‐2 Scatterometer (OSCAT)) winds. For one month (June 2017) of data, the root‐mean‐square difference (RMSD) was reduced from 1.17 to 1.07 m·s−1 and bias from −0.14 to −0.08 m·s−1 for the wind speed at the specular point. Within a 150‐km wide swath along the specular point track, the RMSD was reduced from 1.20 to 1.13 m·s−1. These RMSD and bias statistics are smaller than other CYGNSS wind products available at this time.

Toward a variational assimilation of polarimetric radar observations in a convective-scale numerical weather prediction (NWP) model

Abstract. This paper presents the potential of nonlinear and linear versions of an observation operator for simulating polarimetric variables observed by weather radars. These variables, deduced from the horizontally and vertically polarized backscattered radiations, give information about the shape, the phase and the distributions of hydrometeors. Different studies in observation space are presented as a first step toward their inclusion in a variational data assimilation context, which is not treated here. Input variables are prognostic variables forecasted by the AROME-France numerical weather prediction (NWP) model at convective scale, including liquid and solid hydrometeor contents. A nonlinear observation operator, based on the T-matrix method, allows us to simulate the horizontal and the vertical reflectivities (ZHH and ZVV), the differential reflectivity ZDR, the specific differential phase KDP and the co-polar correlation coefficient ρHV. To assess the uncertainty of such simulations, perturbations have been applied to input parameters of the operator, such as dielectric constant, shape and orientation of the scatterers. Statistics of innovations, defined by the difference between simulated and observed values, are then performed. After some specific filtering procedures, shapes close to a Gaussian distribution have been found for both reflectivities and for ZDR, contrary to KDP and ρHV. A linearized version of this observation operator has been obtained by its Jacobian matrix estimated with the finite difference method. This step allows us to study the sensitivity of polarimetric variables to hydrometeor content perturbations, in the model geometry as well as in the radar one. The polarimetric variables ZHH and ZDR appear to be good candidates for hydrometeor initialization, while KDP seems to be useful only for rain contents. Due to the weak sensitivity of ρHV, its use in data assimilation is expected to be very challenging.

Additions to the Last Millennium Reanalysis Multi-Proxy Database

Progress in paleoclimatology increasingly occurs via data syntheses. We describe additions to a collection prepared for use in paleoclimate state estimation, specifically the Last Millennium Reanalysis (LMR). The 2290 additional series include 2152 tree ring chronologies and 138 other series. They supplement the collection used previously and together form a database titled LMRdb 1.0.0. The additional data draws from lake core, ice core, coral, speleothem, and tree ring archives, using published data primarily from the NOAA Paleoclimatology archive and a set of tree ring width chronologies standardized from raw International Tree Ring Data Bank ring width series. In contrast to many previous paleo compilations, the data were not selected (screened) on the basis of their environmental correlation, multi-century length, or other attributes. The inclusion of proxies sensitive to moisture and other environmental variables expands their use in data assimilation. A preliminary calibration using linear regression with mean annual temperature reveals characteristics of the proxy series and their relationship to temperature, as well as the noise and error characteristics of the records. The additional records are structured as individual files in the NOAA Paleoclimatology format and archived at NOAA Paleoclimatology (Anderson et al. 2018) and will continue to be improved and expanded as part of the LMR Project. The additions represent a four-fold increase in the number of records available for assimilation, provide expanded geographic coverage, and add additional proxy variables. Applications include data assimilation, proxy system model development, and paleoclimate reconstruction using climate field reconstruction and other methods.

Collecting Empirically Derived SAR Characteristic Values over One Year of Sea Ice Environments for Use in Data Assimilation

Abstract A new tool has been developed to calculate dynamic, state-specific tie points, to aid in the assimilation of various types of satellite data into Environment and Climate Change Canada’s Regional Ice Ocean Prediction System. These tie points are referred to as characteristic values (CVs). In this study, CVs are calculated for RadarSat-2 ScanSAR-Wide-A HH-HV backscatter data from October 2010 to September 2011. In this collection, the mean, standard deviation, and percentile distribution of backscatter at locations and times identified as being either ice or open water are represented over different relevant categories affecting the signal. The resulting water CVs are compared with modeled backscatter values, and are in close agreement at midrange wind speeds (5–14 m s−1), where wind slicks are not present. When compared against previously reported values, the ice CVs correspond best for ice conditions with fairly uniform backscatter distributions, such as the Arctic during the spring. When the ice and water CVs are compared to each other, the best cases for the assimilation of RadarSat-2 data are evident. In these cases, the CV distributions at a given incidence angle and wind speed are well separated from each other, such as in the far range (40°–50°) at midrange wind speeds. This set of CVs will be used for an initial assimilation of binary ice and open water retrievals. Future work will include a more complex treatment of ice CVs to address mixed ice types, and the application of CVs to other types of satellite data, including those from passive microwave sensors.

Lidar-Based Height Correction for the Assimilation of Atmospheric Motion Vectors

AbstractThis study uses lidar observations from the polar-orbiting Cloud–Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) satellite to correct operational atmospheric motion vector (AMV) pressure heights. This intends to reduce the height assignment error of AMVs for their use in data assimilation. Additionally, AMVs are treated as winds in a vertical layer as proposed by several recent studies. Corrected and uncorrected AMV winds are evaluated using short-term forecasts of the global forecasting system of the German Weather Service. First, a direct lidar-based height reassignment of AMVs with collocated CALIPSO observations is evaluated. Assigning AMV winds from Meteosat-10 to ~120-hPa-deep layers below the lidar cloud top reduces the vector root-mean-square (VRMS) differences of AMVs from Meteosat-10 by 8%–15%. However, such a direct reassignment can only be applied to collocated AMV–CALIPSO observations that compose a comparably small subset of all AMVs. Second, CALIPSO observations are used to derive statistical height bias correction functions for a general height correction of all operational AMVs from Meteosat-10. Such a height bias correction achieves on average about 50% of the reduction of VRMS differences of the direct height reassignment. Results for other satellites are more ambiguous but still encouraging. Given that such a height bias correction can be applied to all AMVs from a geostationary satellite, the method exhibits a promising approach for the assimilation of AMVs in numerical weather prediction models in the future.

Error model for the assimilation of cloud‐affected infrared satellite observations in an ensemble data assimilation system

Infrared satellite observations are strongly affected by clouds, which complicates their effective use in data assimilation. While observation minus first‐guess (FG departure) statistics for cloud‐free data are close to a normal (Gaussian) distribution, the occurrence of clouds leads to strongly increased uncertainty, systematic differences between observations and model forecasts and subsequently a clear deviation of the FG departures from the Gaussianity that is usually assumed in data assimilation. This study aims to classify the cloud impact on Meteosat Second Generation (MSG) Spinning Enhanced Visible and Infrared Imager (SEVIRI) infrared brightness temperature observations and model equivalents to mitigate the issues of non‐Gaussian FG departure statistics for data assimilation. A threshold brightness temperature is introduced that allows us to quantify the cloud impact and to derive an error estimate for FG departures as a function of the cloud impact. The use of the dynamic error estimate leads to substantially more Gaussian FG departure statistics. Based on the dynamic error estimate, an observation error model is derived for the assimilation of infrared brightness temperature observations in an all‐sky approach. The proposed method allows us to treat cloud‐free and cloud‐affected observations in a uniform way, without the need for cloud screening.

Improving the description of the suspended particulate matter concentrations in the southern North Sea through assimilating remotely sensed data

The integration of remote sensing data of suspended particulate matter (SPM) into numerical models is useful to improve the understanding of the temporal and spatial behaviour of SPM in dynamic shelf seas. In this paper a generic method based on the Ensemble Kalman Filtering (EnKF) for assimilating remote sensing SPM data into a transport model is presented. The EnKF technique is used to assimilate SPM data of the North Sea retrieved from the MERIS sensor, into the computational water quality and sediment transport model, Delft3D-WAQ. The satellite data were processed with the HYDROPT algorithm that provides SPM concentrations and error information per pixel, which enables their use in data assimilation. The uncertainty of the transport model, expressed in the system noise covariance matrix, was quantified by means of a Monte Carlo approach. From a case study covering the first half of 2003, it is demonstrated that the MERIS observations and transport model application are sufficiently robust for a successful generic assimilation. The assimilation results provide a consistent description of the spatial-temporal variability of SPM in the southern North Sea and show a clear decrease of the model bias with respect to independent in-situ observations. This study also identifies some shortcomings in the assimilated results, such as over prediction of surface SPM concentrations in regions experiencing periods of rapid stratification/de-stratification. Overall this feasibility study leads to a range of suggestions for improving and enhancing the model, the observations and the assimilation scheme.

Observability of Flow-Dependent Structure Functions for Use in Data Assimilation

Abstract One of the objectives of data assimilation is to produce initial conditions that will improve the quality of forecasts. Studies on singular vectors and sensitivity studies have shown that small changes to the initial conditions can sometimes lead to exponential error growth. This has motivated research to include flow-dependent structures within the assimilation that would have the characteristics to correctly predict the growth or decay of meteorological systems. This relates to the characterization of precursors to atmospheric instability. In this paper, the observability of such structures by observations is discussed. Several studies have shown that deploying observations over regions where changes in the initial conditions may impact the forecast the most do not lead to the expected benefit. In this paper, it is shown that given the small magnitude of the signal to be detected, it is important to take into account the accuracy of the observations. If the signal-to-noise ratio is too low, observations cannot detect and characterize precursors to forecast error growth. From that perspective, the assimilation only has the possibility to extract information about evolved structures of error growth. Experiments with a simple one-dimensional variational data assimilation (1D-Var) system are presented and, then, an adapted three-dimensional variational data assimilation (3D-Var) system with different sensitivity structure functions is used. The results have been obtained by adapting the variational assimilation system of Environment Canada.

Synchronization of radar observations with multi-scale storm tracking

The 3-D radar reflectivity data has become increasingly important for use in data assimilation towards convective scale numerical weather prediction as well as next generation precipitation estimation. Typically, reflectivity data from multiple radars are objectively analyzed and mosaiced onto a regional 3-D Cartesian grid prior to being assimilated into the models. One of the scientific issues associated with the mosaic of multi-radar observations is the synchronization of all the observations. Since radar data is usually rapidly updated (∼every 5–10 min), it is common in current multi-radar mosaic techniques to combine multiple radar’ observations within a time window by assuming that the storms are steady within the window. The assumption holds well for slow evolving precipitation systems, but for fast evolving convective storms, this assumption may be violated and the mosaic of radar observations at different times may result in inaccurate storm structure depictions. This study investigates the impact of synchronization on storm structures in multiple radar data analyses using a multi-scale storm tracking algorithm.

Accuracy of ACARS Wind and Temperature Observations Determined by Collocation

A collocation study of meteorological reports from commercial aircraft relayed through the Aircraft Communications, Addressing, and Reporting System (ACARS) has been performed to estimate standard deviations of observation errors for wind and temperature. ACARS observations were collected over an area in the western and central United States for a 13-month period, and this dataset was examined for pairs of reports within small spatial (#10 km) and temporal (#10 min) windows. The results showed an observation error of a single horizontal component of wind of 1.1 m s21 and 0.5 K for temperature above the boundary layer. Within the boundary layer, the rms difference of wind and temperature between aircraft was larger, presumably due to larger smallscale variations in the atmosphere and, in the case of wind, from aircraft maneuvers. These observation error estimates are valuable for use in data assimilation and for determination of forecast error from ACARS observation-minus-forecast differences. By comparing standard deviations at different levels, estimates of mesoscale variability at a 10-km scale in the lower troposphere were also calculated. These values (rms vector error of 1.8 ms 21 for wind, rms error of 0.5 K for temperature) can be interpreted as estimates of the 10-km lowertropospheric error of representativeness, also useful for data assimilation.

Alternative Forms of Humidity Information in Global Data Assimilation

Global data analysis procedures were developed to perform data assimilation for observed geopotential heights wind components, and relative humidity. These procedures were implemented in conjunction with a global spectral forecast model (GSM) and normal mode initialization procedure to produce global analyses at six-hour intervals. A set of five-day experiments were conducted to assess the impact of several alternative sources of humidity analysis procedure. Satellite moisture retrievals, surface weather observations, and Air Force Global Weather Central 3-D nephanalysis cloud amounts were used as sources of upper-air relative humidity as to permit their use in data assimilation. The 3-D nephanalysis-inferred humidities, and to a lesser extent the surface weather observation-inferred humidities, were found to have a beneficial impact on the analyses. When used in a manner to maximize their impact, their inclusion in the analysis resulted in an analysis that agreed better with withheld radiosonde relative humidity than did the uncorrected first guess.