Mode testing, also called error tracking, is a key requirement for collateral behavior forecasting models in securitized product, including models of prepayment, default, response, utilization, etc. The high dimensionality and statistical noise associated with agency mortgage-backed securities prepayment behavior make error tracking a complex task. The traditional method focuses on single dimension, for example, along vintage coupon, and does not provide a clear measure of model accuracy and effectiveness. The new method, a rank-based error-tracking methodology, provides an efficient and comprehensive approach to measure model performance. It provides a clear definition of accuracy-which allows clear measure to compare among models. This is superior to the existing one-dimensional method, and other supplement methods (e.g., lift curve method for loans). We discuss these issues as well as applicable statistical theory, and potential applications. <b>TOPICS:</b>MBS and residential mortgage loans, factor-based models <b>Key Findings</b> • The proposed ranked-based error tracking provides a comprehensive and consistent measure for model performance across macroeconomic environment and across pool/loan attributes, superior to the prevailing one-dimensional error-tracking methodology. • Two key measures are the steepness and accuracy of the ranking curve. The steepness measures quantify the model’s ability differentiating across macro and pool variables. The accuracy measures quantify the model’s ability forecasting the prepayment intensity. • The rank-based methodology can be applied to any selected sample pool universe and performance periods. It can be systematically used for model review and model comparison.
Read full abstract