Adaptive Bias Correction Research Articles

Introducing large‐scale analysis constraints in regional hybrid EnVar data assimilation for the prediction of triple typhoons

AbstractLarge‐scale environment fields play an important role in the accurate prediction of typhoons. However, regional predictions for typhoons often suffer from inadequate representation of large‐scale flow pattern such as those from global models due to limited domain size and observations employed in regional models, especially when multiple typhoons that interact concurrently occur in a regional domain. This study merges the large‐scale information from global model forecasts with the mesoscale information from regional model forecasts in the hybrid ensemble‐variational (EnVar) data assimilation by adding an analysis constraint in the EnVar cost function, which is defined by the departure of the regional model EnVar analysis from the global model fields and takes advantage of flow‐dependent ensemble background error covariance for the introduction of large scales using data assimilation. The EnVar assimilation impacts of the large‐scale fields on predictions of triple typhoons are assessed by conducting cycling assimilation and forecast experiments for a 13‐day‐long period in July 2015 when three typhoons concurrently occurred. Results show that the large‐scale constraint for EnVar can clearly improve the triple‐typhoons' track and intensity forecasts of the regional model. The large‐scale information introduced by the proposed method is also shown to reduce forecast errors of wind, temperature and humidity, respectively. Predictions of the rainfall caused by typhoons are also ameliorated. Besides, the analysis‐constrained regional predictions provide better model dynamic fields in terms of sea surface pressure, geopotential height, and water vapor transport, as well as developed typhoon structures. In addition, the adaptive bias correction for radiance assimilation presents a stable performance under the influence of introducing extra background large‐scale fields. The results indicate that the large‐scale analysis constraint introduced in the hybrid EnVar takes advantages of the multiscale information from the global model and the regional model respectively, thus improving the final results of the predictions of multiple typhoons.

Intermittent Demand Forecasting with Adaptive Bias Correction Based on the Modified Corston Method

Intermittent Demand Forecasting with Adaptive Bias Correction Based on the Modified Corston Method

Bayesian safety surveillance with adaptive bias correction.

Postmarket safety surveillance is an integral part of mass vaccination programs. Typically relying on sequential analysis of real-world health data as they accrue, safety surveillance is challenged by sequential multiple testing and by biases induced by residual confounding in observational data. The current standard approach based on the maximized sequential probability ratio test (MaxSPRT) fails to satisfactorily address these practical challenges and it remains a rigid framework that requires prespecification of the surveillance schedule. We develop an alternative Bayesian surveillance procedure that addresses both aforementioned challenges using a more flexible framework. To mitigate bias, we jointly analyze a large set of negative control outcomes that are adverse events with no known association with the vaccines in order to inform an empirical bias distribution, which we then incorporate into estimating the effect of vaccine exposure on the adverse event of interest through a Bayesian hierarchical model. To address multiple testing and improve on flexibility, at each analysis timepoint, we update a posterior probability in favor of the alternative hypothesis that vaccination induces higher risks of adverse events, and then use it for sequential detection of safety signals. Through an empirical evaluation using six US observational healthcare databases covering more than 360 million patients, we benchmark the proposed procedure against MaxSPRT on testing errors and estimation accuracy, under two epidemiological designs, the historical comparator and the self-controlled case series. We demonstrate that our procedure substantially reduces Type 1 error rates, maintains high statistical power and fast signal detection, and provides considerably more accurate estimation than MaxSPRT. Given the extensiveness of the empirical study which yields more than 7 million sets of results, we present all results in a public R ShinyApp. As an effort to promote open science, we provide full implementation of our method in the open-source R package EvidenceSynthesis.

Adaptive bias correction for improved subseasonal forecasting

Subseasonal forecasting—predicting temperature and precipitation 2 to 6 weeks ahead—is critical for effective water allocation, wildfire management, and drought and flood mitigation. Recent international research efforts have advanced the subseasonal capabilities of operational dynamical models, yet temperature and precipitation prediction skills remain poor, partly due to stubborn errors in representing atmospheric dynamics and physics inside dynamical models. Here, to counter these errors, we introduce an adaptive bias correction (ABC) method that combines state-of-the-art dynamical forecasts with observations using machine learning. We show that, when applied to the leading subseasonal model from the European Centre for Medium-Range Weather Forecasts (ECMWF), ABC improves temperature forecasting skill by 60–90% (over baseline skills of 0.18–0.25) and precipitation forecasting skill by 40–69% (over baseline skills of 0.11–0.15) in the contiguous U.S. We couple these performance improvements with a practical workflow to explain ABC skill gains and identify higher-skill windows of opportunity based on specific climate conditions.

Field scale spatial wheat yield forecasting system under limited field data availability by integrating crop simulation model with weather forecast and satellite remote sensing

CONTEXTAn accurate crop yield forecast with sufficient lead time is critical for various applications, such as crop management, resources mobilization, agri-commodity trading, crop insurance, etc. Accurate yield forecasting well ahead of harvest at field scale with minimal field input data remains a challenge. OBJECTIVEThis study aimed to develop a novel prototype wheat yield forecasting system by assimilating remote sensing (RS) derived crop parameters and weather forecast into InfoCrop-Wheat crop simulation model (CSM), using minimum field measurements. METHODSThe CSM was calibrated and validated at both research farm and farmers' fields. The crop LAI was retrieved through inversion of the PROSAIL radiative transfer model from Sentinel-2A and Landsat-8 images and validated using in-situ LAI measurements. The CSM was modified to test assimilation of RS derived LAI through “Ensemble Kalman Filtering” (EnKF) and “Forcing” strategies at multiple time-steps. The RS derived LAI was not only used to correct/replace model simulated LAI but other model state variables were also adjusted accordingly. A major challenge of adjusting crop phenology based on RS derived LAI was also attempted. The WRF weather forecast was bias-corrected and incorporated into the modified model-LAI assimilation framework. Generic crop management inputs were specified to the model. Finally, the study demonstrated a workable prototype of a field scale wheat growth and yield forecasting system under limited field data availability. RESULTS AND CONCLUSIONSThe inversion of PROSAIL showed an RMSE of 0.56 m2/m2 in LAI retrievals. Model validation with measured inputs showed normalized error (NE) of 6‐–8% in grain yield. The proposed framework showed only 2%, 5%, 3% and 1% higher NE in simulating days to anthesis, days to physiological maturity, dry matter and grain yield, respectively, than with measured inputs. The “EnkF” outperformed “Forcing” for predicting crop yield as well as phenology and growth of wheat using generic management inputs. The system showed acceptable accuracy in forecasting phenology, dry matter and yield of wheat at field scale when weighted adaptive bias-correction of weather forecast was incorporated with a 15 days lead time. SIGNIFICANCEThe prototype can be scaled-up for wheat and other crops for predicting real-time crop condition and yield losses at farmers' field for a range of applications, notably, crop-insurance, resources allocation, targeted agro-advisories and triggering contingency plans. It offers considerable potential for objective assessment of crops in the marginal and smallholder systems supporting the smart farming paradigm.

Variational bias correction for Mode-S aircraft derived winds

A variational bias correction (VarBC) technique is proposed within the three-dimensional variational (3D-Var) assimilation system of the Météo-France convective scale model AROME to remove biases present in horizontal wind components derived from aircraft Mode-S data. A set of two predictors derived from the geometry between ground and air speeds has been chosen in order to correct the aircraft speed and heading. Data collected from a set of 8 ADS-B antennas located over France for two periods in 2018 (April and October) have been used to assess the VarBC. Thanks to the availability of anchoring data (radiosoundings, AMDAR aircraft reports), the adaptive bias correction scheme is very efficient for improving the quality of Mode-S derived winds to a level that is similar to those from AMDAR and Mode-S bias corrected by the KNMI. The heading bias correction is the most important one in order to reach background departure values around 2.5 m/s. The behaviour of the VarBC is assessed more precisely for specific aircraft where its capacity to adapt to rapid changes due for example to aircraft maintenance is demonstrated. Moreover, when examining separately predictors for both wind components they appear to converge most of the time towards similar speed and heading corrections. Additional quality controls have allowed to reach background departure values of 2.2 m/s while increasing by a factor of ten the number of aircraft observations to be assimilated in the AROME-France domain.

The use of GNSS zenith total delays in operational AROME/Hungary 3D-Var over a central European domain

Abstract. The delay of satellite signals broadcasted by Global Navigation Satellite System (GNSS) provides unique atmospheric observations which endorse numerical weather prediction from global to limited-area models. Due to the possibility of its frequent and near-real-time estimation, the zenith total delays (ZTDs) are valuable information for any state-of-the-art data assimilation system. This article introduces the data assimilation of ZTDs in a Hungarian numerical weather prediction system, which was carried out by taking into account observations from central European GNSS analysis and processing centres. The importance of ZTD observations is described and shown by a diagnostic tool in the 3-hourly updated 3D-Var assimilation scheme. Furthermore, observing system experiments are done to evaluate the impact of GNSS ZTDs on mesoscale limited-area forecasts. The results of the use of GNSS ZTDs showed a clear added value to improve screen-level temperature and humidity forecasts when the bias is accurately estimated and corrected in the data assimilation scheme. The importance of variational, i.e. adaptive bias correction, is highlighted by verification scores compared to static bias correction. Moreover, this paper reviews the quality control of GNSS ground-based stations inside the central European domain, the calculation of optimal thinning distance and the preparation of the two above-mentioned bias correction methods. Finally, conclusions are drawn on different settings of the forecast and analysis experiments with a brief future outlook.

Comparison of assimilating all‐sky and clear‐sky infrared radiances from Himawari‐8 in a mesoscale system

This study examines the advantages of infrared all‐sky radiance (ASR) assimilation over traditional clear‐sky radiance (CSR) assimilation using a mesoscale LETKF data assimilation system. To effectively assimilate ASR data from the Himawari‐8 geostationary satellite, a cloud‐dependent quality‐control procedure and an observation error model were developed. A single humidity band to be assimilated and thinning distance were determined based on observation error statistics. The operational pre‐processing and parameter settings, such as observation errors and an adaptive bias correction for CSR, were incorporated into the LETKF data assimilation system.A comparison of the impacts of assimilating ASRs and CSRs was accomplished using single‐cycle and 10‐day cycle assimilation experiments. Study results revealed that ASR assimilation provided a higher degree of improvement in the first‐guess fit for conventional observations and satellite retrievals with respect to temperature, moisture and wind. Furthermore, ASR assimilation displayed a more stable improvement in the prediction of a severe rainfall event because it has more universal data coverage than CSR. Adaptive bias correction schemes with two different sets of predictors for ASRs were tested and revealed the difficulty in extracting additional information for the assimilation of no‐bias corrected ASR due to complicated bias factors. This was in contrast to the CSR assimilation, where bias correction had a positive impact.

Assessment and Adaptive Correction of Observations in Atmospheric Sounding Channels of the Satellite Microwave Radiometer MTVZA-GY

The microwave radiometer MTVZA-GY on board the Russian polar orbiting meteorological satellite Meteor-M N2 is briefly described. Errors and biases in antenna brightness temperatures are documented. Observation errors are found to be dependent on the solar angles. An adaptive bias correction technique for MTVZA-GY antenna temperatures is motivated and developed. The technique accounts for the solar angles and sequentially assimilates observed minus simulated radiances in a perpetual 24 h cycle in order to estimate up-to-date correction coefficients defined to be functions of the zenith and azimuth solar angles. The simulated radiances are computed by the RTTOV radiative transfer model from three-dimensional numerical weather prediction fields. The correction technique is implemented for atmospheric temperature and humidity sounding channels of MTVZA-GY. The corrected observations are shown to be significantly more accurate as compared with raw antenna temperatures and with observations that undergo simpler and more traditional corrections. The accuracy of corrected MTVZA-GY observations is compared with the accuracy of AMSU-A and MHS data.

Development of an Observation Processing Package for Data Assimilation in KIAPS

A new observation processing system, the Korea Institute of Atmospheric Prediction Systems (KIAPS) Package for Observation Processing (KPOP), has been developed to provide optimal observation datasets to the data assimilation (DA) system for the Korean Integrated Model, KIM. This paper presents the KPOP’s conceptual design, how the principal modules have been developed, and some of their preliminary results. Currently, the KPOP is capable of processing almost all observation types used by the Korea Meteorological Administration (KMA) and some new observation types that have a positive impact in other operational centers. We have developed an adaptive bias correction (BC) method that only uses the background of the analysis time and selects the best observations through the consecutive iteration of BC and quality control (QC); it has been verified that this method will be the best suited for the KIAPS DA system until the development of variational BC (VarBC) has been completed. The requirement of considering the radiosonde balloon drift in the DA according to the increase of spatial resolution of the NWP model was accounted for using a balloon drift estimation method that considers the pressure difference and wind speed; thus the distance error was less than 1% in the sample test. Some kind of widely used methods were tested for height adjustment of the SURFACE observation, and a new method for temperature adjustment was outlined that used the correlation between temperature and relative humidity. In addition, three types of map projection were compared: the cubed-sphere (CS), equidistance (ED), and equirectangular (ER) projection for thinning. Data denial experiments were conducted to investigate how the KPOP affected the quality of the analysis fields in the three-dimensional variational data assimilation system (3D-Var). Qualified observations produced by the KPOP had a positive impact by reducing the analysis error.

Adaptive bias correction of advanced infrared sounding radiance assimilation in a regional model and its impact on typhoon forecast

Hyperspectral infrared remote sensing can provide the information about temperature and humidity of the atmosphere at high vertical resolution and high accuracy. To assimilate its radiances directly, we must correct biases between the observed radiances and the simulated ones from the model first guess, caused by systematic error of radiances and by the radiative transfer model and assimilating system. The method used for bias correction was developed for global models, and its adaptation to regional models raises further questions. This study is based on coupling the mesoscale numerical model weather research and forecast and gridpoint statistical interpolation assimilation system, using adaptive variational bias correction (VarBC) to the scan angle and air-mass factor, and investigates the characteristics of bias correction coefficients for the regional model. It was found that advanced infrared sounder (AIRS) channels located in the 15-μm CO2 absorption band had large scan bias, and that its nadir bias had a time dependence, which is probably due to the bias from the radiative transfer model. By contrast, other channels had small scan bias and weak time dependence. In air-mass bias correction, predictors of zenith and temperature lapse rate had huge oscillations due to variations in data coverage from the regional models. The effect of this scheme on correction in a regional model was verified via the histogram analysis of innovation. The verification showed that correction on most of the channels got satisfactory results except for several land surface channels. The corrected histogram satisfied the requirement of an unbiased normal distribution. In a typhoon forecast experiment, the influence of radiance bias correction on forecast result was tested. It showed that, compared with parameters from the global model, regional radiance correction parameters study improved the prediction of the typhoon 72-h forecast.

Prospects for flash flood forecasting in mountainous regions – An investigation of Tropical Storm Fay in the Southern Appalachians

Summary The sensitivity of Quantitative flash-Flood Estimates (QFEs) and Quantitative flash-Flood Forecasts (QFFs) to Quantitative Precipitation Estimates (QPEs) and Quantitative Precipitation Forecasts (QPFs) was investigated in three headwater catchments with different topographic and hydro-geomorphic characteristics during the passage of Tropical Storm Fay, 2008 over the Southern Appalachian Mountains in North Carolina, USA. QFEs and QFFs were generated by a high-resolution hydrologic model (250 × 250 m 2 ) with coupled surface–subsurface physics and rainfall forcing from the Next Generation Multi-sensor QPE (Q2) spatial rainfall (1 × 1 km 2 ) product, and from National Digital Forecast Database (NDFD) operational QPF product (5 × 5 km 2 ). Optimal QPE products (Q2+) were derived by assimilating rainfall observations from a high density raingauge network through adaptive bias correction. Deterministic QFEs simulated by the hydrologic model agree well with streamgauge observations (15-min intervals) regarding total water volume and peak flow with Nash–Sutcliffe (NS) coefficients 0.8–0.9, thus suggesting that the model without calibration captures well the dominant flash-flood physics. The propagation of uncertainty in storm rainfall to rainfall–runoff response was subsequently evaluated through model simulations forced by Monte Carlo replications of the QPEs to generate QFE distributions. Analysis of the joint QPE–QFE distributions shows that flood response at the catchment scale is highly non-linear, and exhibits strong dependence on basin physiography, initial soil moisture conditions (transient basin storage capacity), the space–time organization of runoff generation and conveyance mechanisms, and in particular interflow dynamics, with respect to the space–time structure of rainfall. QFFs for 6- to 1-h lead times using precipitation composites of Q2 QPE and NDFD QPF to drive the hydrology model in operational mode exhibited ubiquitous lack of skill yielding consistently negative NS scores. An experiment consisting of merging satellite-like observations into operational QPE/QPF showed significant improvement in QFF performance (e.g. 5–50% relative NS increases), especially when the timing of satellite overpass is such that it captures transient episodes of heavy rainfall during the event. Future advances in QFF remain principally constrained by progress in QPE and QPF at the spatial resolution necessary to resolve rainfall-interflow dynamics in mountainous regions.

Ensemble Kalman Filter and 4D-Var Intercomparison with the Japanese Operational Global Analysis and Prediction System

Abstract The local ensemble transform Kalman filter (LETKF) is implemented and assessed with the experimental operational system at the Japanese Meteorological Agency (JMA). This paper describes the details of the LETKF system and verification of deterministic forecast skill. JMA has been operating a four-dimensional variational data assimilation (4D-Var) system for global numerical weather prediction since 2005. The main purpose of this study is to make a reasonable comparison between the LETKF and the operational 4D-Var. Several forecast–analysis cycle experiments are performed to find sensitivity to the parameters of the LETKF. The difference between additive and multiplicative error covariance inflation schemes is investigated. Moreover, an adaptive bias correction method for satellite radiance observations is proposed and implemented, so that the LETKF is equipped with functionality similar to the variational bias correction used in the operational 4D-Var. Finally, the LETKF is compared with the operational 4D-Var. Although forecast verification scores of the two systems relative to each system’s own analyses and to radiosonde observations show some disagreement, the overall conclusion indicates that the LETKF and 4D-Var have essentially comparable performance. The LETKF shows larger temperature bias in the lower troposphere mainly over the ocean, which is related to a well-known JMA model bias that plays an important role in the significant degradation of the forecast scores in the SH. The LETKF suffers less of a performance degradation than 4D-Var in the absence of satellite radiance assimilation. This suggests that better treatment of satellite radiances would be important in future developments toward operational use of the LETKF. Developing both LETKF and 4D-Var at JMA has shown significant benefits by the synergistic effect and is the recommended strategy for the moment.

Correcting for Surface Pressure Background Bias in Ensemble-Based Analyses

Model error is the component of the forecast error that is due to the difference between the dynamics of the atmosphere and the dynamics of the numerical prediction model. The systematic, slowly varying part of the model error is called model bias. This paper evaluates three different ensemble-based strategies to account for the surface pressure model bias in the analysis scheme. These strategies are based on modifying the observation operator for the surface pressure observations by the addition of a bias-correction term. One estimates the correction term adaptively, while another uses the hydrostatic balance equation to obtain the correction term. The third strategy combines an adaptively estimated correction term and the hydrostatic-balance-based correction term. Numerical experiments are carried out in an idealized setting, where the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS) model is integrated at resolution T62L28 to simulate the evolution of the atmosphere and the T30L7 resolution Simplified Parameterization Primitive Equation Dynamics (SPEEDY) model is used for data assimilation. The results suggest that the adaptive bias-correction term is effective in correcting the bias in the data-rich regions, while the hydrostatic-balance-based approach is effective in data-sparse regions. The adaptive bias-correction approach also has the benefit that it leads to a significant improvement of the temperature and wind analysis at the higher model levels. The best results are obtained when the two bias-correction approaches are combined.

Adaptive bias correction for satellite data in a numerical weather prediction system

AbstractAdaptive bias corrections for satellite radiances need to separate the observation bias from the systematic errors in the background in order to prevent the analysis from drifting towards its own climate. The variational bias correction scheme (VarBC) is a particular adaptive scheme that is embedded inside the assimilation system.VarBC is compared with an offline adaptive and a static bias correction scheme. In simulation, the three schemes are exposed to artificial shifts in the observations and the background. The VarBC scheme can be considered as a good compromise between the static and the offline adaptive schemes. It shows some skill in distinguishing between the background‐error and the observation biases when other unbiased observations are available to anchor the system. Tests of VarBC in a real numerical weather prediction (NWP) environment show a significant reduction in the misfit with radiosonde observations (especially in the stratosphere) due to NWP model error. The scheme adapts to an instrument error with only minimal disruption to the analysis.In VarBC, the bias is constrained by the fit to observations—such as radiosondes—that are not bias‐corrected to the NWP model. In parts of the atmosphere where no radiosonde observations are available, the radiosonde network still imposes an indirect constraint on the system, which can be enhanced by applying a mask to VarBC. Copyright © 2007 Royal Meteorological Society

Application of the NCEP Ensemble Prediction System to Medium-Range Forecasting in South Africa: New Products, Benefits, and Challenges

Abstract The National Centers for Environmental Prediction (NCEP) Ensemble Forecasting System (EFS) is used operationally in South Africa for medium-range forecasts up to 14 days ahead. The use of model-generated probability forecasts has a clear benefit in the skill of the 1–7-day forecasts. This is seen in the forecast probability distribution being more successful in spanning the observed space than a single deterministic forecast and, thus, substantially reducing the instances of missed events in the forecast. In addition, the probability forecasts generated using the EFS are particularly useful in estimating confidence in forecasts. During the second week of the forecast the EFS is used as a heads-up for possible synoptic-scale events and also for predicting average weather conditions and probability density distributions of some elements such as maximum temperature and wind. This paper assesses the medium-range forecast process and the application of the NCEP EFS at the South African Weather Service. It includes a description of the various medium-range products, adaptive bias-correction methods applied to the forecasts, verification of the forecast products, and a discussion on the various challenges that face researchers and forecasters alike.

Validation of an EnKF System for OGCM Initialization Assimilating Temperature, Salinity, and Surface Height Measurements

Abstract Results are presented from a decade-long assimilation run with a 64-member OGCM ensemble in a global configuration. The assimilation system can be used to produce ocean initial conditions for seasonal forecasts. The ensemble is constructed with the Max Planck Institute Ocean Model, where each member is forced by differently perturbed 40-yr European Centre for Medium-Range Weather Forecasts Re-Analysis atmospheric fields over sequential 10-day intervals. Along-track altimetric data from the European Remote Sensing and the Ocean Topography Experiment (TOPEX)/Poseidon satellites, as well as quality-controlled subsurface temperature and salinity profiles, are subsequently assimilated using the standard formulation of the ensemble Kalman filter. The applied forcing perturbation method and data selection and processing procedures are described, as well as a framework for the construction of appropriate data constraint error models for all three data types. The results indicate that the system is stable, does not experience a tendency toward ensemble collapse, and provides smooth analyses that are closer to withheld data than an unconstrained control run. Subsurface bias and time-dependent errors are reduced by the assimilation but not entirely removed. Time series of assimilation and ensemble statistics also indicate that the model is not very strongly constrained by the data because of an overspecification of the data errors. A comparison of equatorial zonal velocity profiles with in situ current meter data shows mixed results. A shift in the time-mean profile in the central Pacific is primarily associated with an assimilation-induced bias. The use of an adaptive bias correction scheme is suggested as a solution to this problem.