Improved diagnostics for NWP verification in the tropics

Abstract

The root‐mean‐square error (RMSE) is often used to verify forecasts. However, its strong dependence on the observation variability makes it unsuitable for comparing model performance between regions where observation variability is much different, e.g., across vertical levels or between the midlatitudes and the tropics. The alpha index based on the tensor variance of forecast‐observation discrepancy was formulated to improve on RMSE (and the closely related bias‐corrected RMSE). An “error ellipse” was used to represent the random error in vector wind, yielding two other diagnostics: eccentricity and orientation. These diagnostics were applied to verify Naval Research Laboratory's limited‐area model, Coupled Ocean/Atmospheric Mesoscale Prediction System (COAMPS), for the first time in Southeast Asia. COAMPS forecasts were verified against radiosonde data from South China Sea Monsoon Experiment (SCSMEX), May–June 1998. Results revealed falling model performance as forecast time increases but little difference between forecasts at 18‐km and 54‐km resolution. Systematic errors in the model dynamics were suggested from the biases. The alpha indices show that (after bias correction) COAMPS performs best for wind, followed by temperature and then by dew point depression. In this tropical region, 1‐day persistence forecasts were only outperformed by the model for wind predictions between 400 mb and 850 mb at forecast times less than 24 hr. The RMSE diagnostic was shown to sometimes yield misleading evaluation of the model's performance. The wind error ellipses revealed that the random error tended to align more with the background flow than with the model bias, possibly indicating a dynamical reason for its existence.