Using XGBoost & SHAP feature importance to understand the drivers of the Southern Ocean cloud-radiation bias

Abstract

The Southern Ocean radiation bias has been a multigenerational problem in climate and weather models. This radiation bias, where too much sunlight is allowed to reach the surface, has been attributed to the incorrect simulation of cloud properties, particularly their phase. Allowing too much sunlight to reach the surface of the Southern Ocean has significant effects both locally (e.g. sea surface temperatures) and globally. Traditional model evaluation techniques of the Southern Ocean often focus on one particular cloud type or synoptic situation or have been limited temporally or spatially. Additionally, it is difficult to quantify non-linear relationships with more traditional evaluation techniques. In this work, we present an evaluation of the Southern Ocean cloud-radiation bias using an XGBoost model and SHapley Additive exPlanations feature importance analysis. We propose that this method can provide as much information as traditional evaluation techniques and more by handling large data sets and non-linear relationships. We find that our XGBoost model can explain 55% of the summertime daily cloud radiative bias of the entire Southern Ocean (1x1˚ grid) over 5 years, where biases in cloud properties are used as predictors. Using SHAP feature importance we find that the bias in cloud liquid water path is the most important predictor, though this varies by cloud type and latitude, and that the SHAP feature analysis can identify and quantify important complex relationships. We also show that this method can be useful in evaluating model perturbations, in particular with respect to how changes in a model may impact different clouds or regions in a different way. Overall, we suggest that this method can be used to evaluate large and complex systems in a holistic way, streamlining model evaluation and development testing.