Standard 4D-Var Research Articles

A hybrid approach to generating search subspaces in dynamically constrained 4-dimensional data assimilation

Development and maintenance of the linearized and adjoint code for advanced circulation models is a challenging issue, requiring a significant proportion of total effort in operational data assimilation (DA). The ensemble-based DA techniques provide a derivative-free alternative, which appears to be competitive with variational methods in many practical applications. This article proposes a hybrid scheme for generating the search subspaces in the adjoint-free 4-dimensional DA method (a4dVar) that does not use a predefined ensemble. The method resembles 4dVar in that the optimal solution is strongly constrained by model dynamics and search directions are supplied iteratively using information from the current and previous model trajectories generated in the process of optimization. In contrast to 4dVar, which produces a single search direction from exact gradient information, a4dVar employs an ensemble of directions to form a subspace in order to proceed. In the earlier versions of a4dVar, search subspaces were built using the leading EOFs of either the model trajectory or the projections of the model-data misfits onto the range of the background error covariance (BEC) matrix at the current iteration. In the present study, we blend both approaches and explore a hybrid scheme of ensemble generation in order to improve the performance and flexibility of the algorithm. In addition, we introduce balance constraints into the BEC structure and periodically augment the search ensemble with BEC eigenvectors to avoid repeating minimization over already explored subspaces. Performance of the proposed hybrid a4dVar (ha4dVar) method is compared with that of standard 4dVar in a realistic regional configuration assimilating real data into the Navy Coastal Ocean Model (NCOM). It is shown that the ha4dVar converges faster than a4dVar and can be potentially competitive with 4dvar both in terms of the required computational time and the forecast skill.

Reducing the prediction uncertainties of high-impact weather and climate events: An overview of studies at LASG

This paper summarizes recent progress at the State Key Laboratory of Numerical Modeling for Atmospheric Sciences and Geophysical Fluid Dynamics (LASG), Institute of Atmospheric Physics, Chinese Academy of Sciences in studies on targeted observations, data assimilation, and ensemble prediction, which are three effective strategies to reduce the prediction uncertainties and improve the forecast skill of weather and climate events. Considering the limitations of traditional targeted observation approaches, LASG researchers have developed a conditional nonlinear optimal perturbation-based targeted observation strategy to optimize the design of the observing network. This strategy has been employed to identify sensitive areas for targeted observations of the El Nino–Southern Oscillation, Indian Ocean dipole, and tropical cyclones, and has been demonstrated to be effective in improving the forecast skill of these events. To assimilate the targeted observations into the initial state of a numerical model, a dimension-reducedprojection- based four-dimensional variational data assimilation (DRP-4DVar) approach has been proposed and is used operationally to supply accurate initial conditions in numerical forecasts. The performance of DRP-4DVar is good, and its computational cost is much lower than the standard 4DVar approach. Besides, ensemble prediction, which is a practical approach to generate probabilistic forecasts of the future state of a particular system, can be used to reduce the prediction uncertainties of single forecasts by taking the ensemble mean of forecast members. In this field, LASG researchers have proposed an ensemble forecast method that uses nonlinear local Lyapunov vectors (NLLVs) to yield ensemble initial perturbations. Its application in simple models has shown that NLLVs are more useful than bred vectors and singular vectors in improving the skill of the ensemble forecast. Therefore, NLLVs represent a candidate for possible development as an ensemble method in operational forecasts. Despite the considerable efforts made towards developing these methods to reduce prediction uncertainties, much challenging but highly important work remains in terms of improving the methods to further increase the skill in forecasting such weather and climate events.

Relationships among Four-Dimensional Hybrid Ensemble–Variational Data Assimilation Algorithms with Full and Approximate Ensemble Covariance Localization

Abstract Ensemble–variational data assimilation algorithms that can incorporate the time dimension (four-dimensional or 4D) and combine static and ensemble-derived background error covariances (hybrid) are formulated in general forms based on the extended control variable and the observation-space-perturbation approaches. The properties and relationships of these algorithms and their approximated formulations are discussed. The main algorithms discussed include the following: 1) the standard ensemble 4DVar (En4DVar) algorithm incorporating ensemble-derived background error covariance through the extended control variable approach, 2) the 4DEnVar neglecting the time propagation of the extended control variable (4DEnVar-NPC), 3) the 4D ensemble–variational algorithm based on observation space perturbation (4DEnVar), and 4) the 4DEnVar with no propagation of covariance localization (4DEnVar-NPL). Without the static background error covariance term, none of the algorithms requires the adjoint model except for En4DVar. Costly applications of the tangent linear model to localized ensemble perturbations can be avoided by making the NPC and NPL approximations. It is proven that En4DVar and 4DEnVar are mathematically equivalent, while 4DEnVar-NPC and 4DEnVar-NPL are mathematically equivalent. Such equivalences are also demonstrated by single-observation assimilation experiments with a 1D linear advection model. The effects of the non-flow-following or stationary localization approximations are also examined through the experiments. All of the above algorithms can include the static background error covariance term to establish a hybrid formulation. When the static term is included, all algorithms will require a tangent linear model and an adjoint model. The first guess at appropriate time (FGAT) approximation is proposed to avoid the tangent linear and adjoint models. Computational costs of the algorithms are also discussed.

Four-dimensional ensemble variational (4D-En-Var) data assimilation for the HIgh Resolution Limited Area Model (HIRLAM)

Abstract. A four-dimensional ensemble variational (4D-En-Var) data assimilation has been developed for a limited area model. The integration of tangent linear and adjoint models, as applied in standard 4D-Var, is replaced with the use of an ensemble of non-linear model states to estimate four-dimensional background error covariances over the assimilation time window. The computational costs for 4D-En-Var are therefore significantly reduced in comparison with standard 4D-Var and the scalability of the algorithm is improved. The flow dependency of 4D-En-Var assimilation increments is demonstrated in single simulated observation experiments and compared with corresponding increments from standard 4D-Var and Hybrid 4D-Var ensemble assimilation experiments. Real observation data assimilation experiments carried out over a 6-week period show that 4D-En-Var outperforms standard 4D-Var as well as Hybrid 4D-Var ensemble data assimilation with regard to forecast quality measured by forecast verification scores.

A Consistent Hybrid Variational-Smoothing Data Assimilation Method: Application to a Simple Shallow-Water Model of the Turbulent Midlatitude Ocean

Abstract In the standard four-dimensional variational data assimilation (4D-Var) algorithm the background error covariance matrix remains static over time. It may therefore be unable to correctly take into account the information accumulated by a system into which data are gradually being assimilated. A possible method for remedying this flaw is presented and tested in this paper. A hybrid variational-smoothing algorithm is based on a reduced-rank incremental 4D-Var. Its consistent coupling to a singular evolutive extended Kalman (SEEK) smoother ensures the evolution of the matrix. In the analysis step, a low-dimensional error covariance matrix is updated so as to take into account the increased confidence level in the state vector it describes, once the observations have been introduced into the system. In the forecast step, the basis spanning the corresponding control subspace is propagated via the tangent linear model. The hybrid method is implemented and tested in twin experiments employing a shallow-water model. The background error covariance matrix is initialized using an EOF decomposition of a sample of model states. The quality of the analyses and the information content in the bases spanning control subspaces are also assessed. Several numerical experiments are conducted that differ with regard to the initialization of the matrix. The feasibility of the method is illustrated. Since improvement due to the hybrid method is not universal, configurations that benefit from employing it instead of the standard 4D-Var are described and an explanation of the possible reasons for this is proposed.

Four‐dimensional variational assimilation in the unstable subspace and the optimal subspace dimension

AbstractKey apriori information used in 4D‐Var is the knowledge of the system's evolution equations. In this article we propose a method for taking full advantage of the knowledge of the system's dynamical instabilities in order to improve the quality of the analysis. We present an algorithm for four‐dimensional variational assimilation in the unstable subspace (4D‐Var – AUS), which consists of confining in this subspace the increment of the control variable. The existence of an optimal subspace dimension for this confinement is hypothesized. Theoretical arguments in favour of the present approach are supported by numerical experiments in a simple perfect nonlinear model scenario. It is found that the RMS analysis error is a function of the dimension N of the subspace where the analysis is confined and is a minimum for N approximately equal to the dimension of the unstable and neutral manifold. For all assimilation windows, from 1 to 5 d, 4D‐Var – AUS performs better than standard 4D‐Var. In the presence of observational noise, the 4D‐Var solution, while being closer to the observations, is farther away from the truth. The implementation of 4D‐Var – AUS does not require the adjoint integration. Copyright © 2010 Royal Meteorological Society

A Method of Successive Corrections of the Control Subspace in the Reduced-Order Variational Data Assimilation*

Abstract A version of the reduced control space four-dimensional variational method (R4DVAR) of data assimilation into numerical models is proposed. In contrast to the conventional 4DVAR schemes, the method does not require development of the tangent linear and adjoint codes for implementation. The proposed R4DVAR technique is based on minimization of the cost function in a sequence of low-dimensional subspaces of the control space. Performance of the method is demonstrated in a series of twin-data assimilation experiments into a nonlinear quasigeostrophic model utilized as a strong constraint. When the adjoint code is stable, R4DVAR’s convergence rate is comparable to that of the standard 4DVAR algorithm. In the presence of strong instabilities in the direct model, R4DVAR works better than 4DVAR whose performance is deteriorated because of the breakdown of the tangent linear approximation. Comparison of the 4DVAR and R4DVAR also shows that R4DVAR becomes advantageous when observations are sparse and noisy.

Coupling ensemble Kalman filter with four-dimensional variational data assimilation

This study examines the performance of coupling the deterministic four-dimensional variational assimilation system (4DVAR) with an ensemble Kalman filter (EnKF) to produce a superior hybrid approach for data assimilation. The coupled assimilation scheme (E4DVAR) benefits from using the state-dependent uncertainty provided by EnKF while taking advantage of 4DVAR in preventing filter divergence: the 4DVAR analysis produces posterior maximum likelihood solutions through minimization of a cost function about which the ensemble perturbations are transformed, and the resulting ensemble analysis can be propagated forward both for the next assimilation cycle and as a basis for ensemble forecasting. The feasibility and effectiveness of this coupled approach are demonstrated in an idealized model with simulated observations. It is found that the E4DVAR is capable of outperforming both 4DVAR and the EnKF under both perfect- and imperfect-model scenarios. The performance of the coupled scheme is also less sensitive to either the ensemble size or the assimilation window length than those for standard EnKF or 4DVAR implementations.

An Ensemble-Based Four-Dimensional Variational Data Assimilation Scheme. Part I: Technical Formulation and Preliminary Test

Abstract Applying a flow-dependent background error covariance (𝗕 matrix) in variational data assimilation has been a topic of interest among researchers in recent years. In this paper, an ensemble-based four-dimensional variational (En4DVAR) algorithm, designed by the authors, is presented that uses a flow-dependent background error covariance matrix constructed by ensemble forecasts and performs 4DVAR optimization to produce a balanced analysis. A great advantage of this En4DVAR design over standard 4DVAR methods is that the tangent linear and adjoint models can be avoided in its formulation and implementation. In addition, it can be easily incorporated into variational data assimilation systems that are already in use at operational centers and among the research community. A one-dimensional shallow water model was used for preliminary tests of the En4DVAR scheme. Compared with standard 4DVAR, the En4DVAR converges well and can produce results that are as good as those with 4DVAR but with far less computation cost in its minimization. In addition, a comparison of the results from En4DVAR with those from other data assimilation schemes [e.g., 3DVAR and ensemble Kalman filter (EnKF)] is made. The results show that the En4DVAR yields an analysis that is comparable to the widely used variational or ensemble data assimilation schemes and can be a promising approach for real-time applications. In addition, experiments were carried out to test the sensitivities of EnKF and En4DVAR, whose background error covariance is estimated from the same ensemble forecasts. The experiments indicated that En4DVAR obtained reasonably sound analysis even with larger observation error, higher observation frequency, and more unbalanced background field.

Generalization of the dual variational data assimilation algorithm to a nonlinear layered quasi-geostrophic ocean model

In this paper, we present a generalization to nonlinear models of the four-dimensional variational dual method, the 4D-PSAS algorithm. The idea of 4D-PSAS (physical space analysis system) is to perform the minimization in the space of the observations, rather than in the model space as in the primal 4D-VAR scheme. Despite the formal equivalence between 4D-VAR and 4D-PSAS in a linear situation (both for model equations and observation operators), the dual method has several important advantages: in oceanographic cases, the observation space is smaller than the model space, which should improve the minimization process; for no additional cost, it provides an estimation of the model error; and finally, it does not have any singularities when the covariance error matrices tend to zero. The idea of this paper is to extend this algorithm to a fully nonlinear situation, as has been done in previous years with other classical data assimilation schemes: the 4D-VAR and the Kalman filter. For this purpose, we consider a nonlinear multi-layer quasi-geostrophic ocean model, which mimics quite well the mid-latitude circulation. We recall the standard primal 4D-VAR scheme applied to this model, and then introduce an extended 4D-PSAS algorithm in the particular case of this nonlinear QG model. We report then the results of extensive numerical experiments that have been carried out to compare this extended algorithm to the classical variational formulation, and to study its sensitivity to many parameters such as the nonlinearities, the number of available observations, the presence of an unknown term in the assimilation model and to study the detection of the model error. As a matter of fact, it is found that this extended algorithm has kept the same advantages as in the linear case (model error detection, smaller sensitivity to various perturbations, more efficient minimization process). All these experiments suggest that it is an efficient assimilation scheme for oceanographic problems.

Four-dimensional variational data assimilation for mesoscale and storm-scale applications

¶The status and progress of the four-dimensional variational data assimilation (4DVAR) are briefly reviewed focusing on application to prediction of mesoscale/storm-scale atmospheric phenomena. Theoretical background is provided for each important component of the 4DVAR system – forecast and adjoint models, observations, background, cost function, preconditioning, and minimization. An overview of practical issues specific for mesoscale/storm-scale 4DVAR is then presented in terms of high-resolution observations, nonlinearity and discontinuity problem, model error, errors from lateral boundary condition, and precipitation assimilation. Practical strategies for efficient and simplified 4DVAR are also introduced, e.g., incremental 4DVAR, poor man’s 4DVAR, and inverse 3DVAR. A new concept on hybrid approach is proposed to combine an efficient 4DVAR scheme and the standard 4DVAR scheme aiming at reducing computational demand required by the standard 4DVAR while improving the accuracy of the simplified 4DVAR. Applications to both hydrostatic and nonhydrostatic models are illustrated and our vision on opportunities and directions for future research is provided.

Performance of 4D-Var with Different Strategies for the Use of Adjoint Physics with the FSU Global Spectral Model

A set of four-dimensional variational data assimilation (4D-Var) experiments were conducted using both a standard method and an incremental method in an identical twin framework. The full physics adjoint model of the Florida State University global spectral model (FSUGSM) was used in the standard 4D-Var, while the adjoint of only a few selected physical parameterizations was used in the incremental method. The impact of physical processes on 4D-Var was examined in detail by comparing the results of these experiments. The inclusion of full physics turned out to be significantly beneficial in terms of assimilation error to the lower troposphere during the entire minimization process. The beneficial impact was found to be primarily related to boundary layer physics. The precipitation physics in the adjoint model also tended to have a beneficial impact after an intermediate number (50) of minimization iterations. Experiment results confirmed that the forecast from assimilation analyses with the full physics adjoint model displays a shorter precipitation spinup period. The beneficial impact on precipitation spinup did not result solely from the inclusion of the precipitation physics in the adjoint model, but rather from the combined impact of several physical processes. The inclusion of full physics in the adjoint model exhibited a detrimental impact on the rate of convergence at an early stage of the minimization process, but did not affect the final convergence. A truncated Newton-like incremental approach was introduced for examining the possibility of circumventing the detrimental aspects using the full physics in the adjoint model in 4D-Var but taking into account its positive aspects. This algorithm was based on the idea of the truncated Newton minimization method and the sequential cost function incremental method introduced by Courtier et al., consisting of an inner loop and an outer loop. The inner loop comprised the incremental method, while the outer loop consisted of the standard 4D-Var method using the full physics adjoint. The limited-memory quasi-Newton minimization method (L-BFGS) was used for both inner and outer loops, while information on the Hessian of the cost function was jointly updated at every iteration in both loops. In an experiment with a two-cycle truncated Newton-like incremental approach, the assimilation analyses turned out to be better than those obtained from either the standard 4D-Var or the incremental 4D-Var in all aspects examined. The CPU time required by this two-cycle approach was larger by 35% compared with that required by the incremental 4D-Var without almost any physics in the adjoint model, while the CPU time required by the standard 4D-Var with the full physics adjoint model was more than twice that required by the incremental 4D-Var. Finally, several hypotheses concerning the impact of using standard 4D-Var full physics on minimization convergence were advanced and discussed.

Application of the Quasi-Inverse Method to Data Assimilation

Four-dimensional variational data assimilation (4D-Var) seeks to find an optimal initial field that minimizes a cost function defined as the squared distance between model solutions and observations within an assimilation window. For a perfect linear model, Lorenc showed that the 4D-Var forecast at the end of the window coincides with a Kalman filter analysis if two conditions are fulfilled: (a) addition to the cost function of a term that measures the distance to the background at the beginning of the assimilation window, and (b) use of the Kalman filter background error covariance in this term. The standard 4D-Var requires minimization algorithms along with adjoint models to compute gradient information needed for the minimization. In this study, an alternative method is suggested based on the use of the quasi-inverse model that, for certain applications, may help accelerate the solution of problems close to 4D-Var. The quasi-inverse approach for the forecast sensitivity problem is introduced, and then a closely related variational assimilation problem using the quasi-inverse model is formulated (i.e., the model is integrated backward but changing the sign of the dissipation terms). It is shown that if the cost function has no background term, and has a complete set of observations (as assumed in many classical 4D-Var papers), the new method solves the 4D-Var-minimization problem efficiently, and is in fact equivalent to the Newton algorithm but without having to compute a Hessian. If the background term is included but computed at the end of the interval, allowing the use of observations that are not complete, the minimization can still be carried out very efficiently. In this case, however, the method is much closer to a 3D-Var formulation in which the analysis is attained through a model integration. For this reason, the method is called ‘‘inverse 3D-Var’’ (I3D-Var). The I3D-Var method was applied to simple models (viscous Burgers’ equation and Lorenz model), and it was found that when the background term is ignored and complete fields of noisy observations are available at multiple times, the inverse 3D-Var method minimizes the same cost function as 4D-Var but converges much faster. Tests with the Advanced Regional Prediction System (ARPS) indicate that I3D-Var is about twice as fast as the adjoint Newton method and many times faster than the quasi-Newton LBFGS algorithm, which uses the adjoint model. Potential problems (including the growth of random errors during the integration back in time) and possible applications to preconditioning, and to problems such as storm-scale data assimilation and reanalysis are also discussed.