Comparing and contrasting choice model and machine learning techniques in the context of vehicle ownership decisions

Abstract

In recent years, planners have started considering Machine Learning (ML) techniques as an alternative to discrete choice models (CM). ML techniques are primarily data-driven and typically achieve better prediction accuracy compared to CM. However, it is hypothesized that since the ML techniques do not have the strong grounding to economic theory as the CMs, they may not perform well in contexts that are radically different from the ‘training’ scenario. It is also hypothesized that the relative prediction performance may be affected by the metrics used for comparing the models. This research aims to test these two hypotheses empirically by modelling vehicle ownership choices using household survey data from Dhaka, Bangladesh collected in 2004, 2010 and 2019. The performances of CM (multinomial logit) and ML techniques (neural networks and gradient boosting trees) have been compared using log-likelihood and mean absolute percentage error of market shares. The results indicate that the multinomial logit model (MNL) with a piecewise linear transformation of the household income, has the best performance in terms of log-likelihood and mean absolute percentage error of market shares. This is followed by Neural Networks (NN) and Gradient Boosting Trees (GBT). The results thus provide empirical evidence that the ML techniques do not consistently outperform CM. Moreover, the difference in the performance of the models further increases if the prediction scenario is substantially different. This reinforces the hypothesis that CMs, with their behavioural underpinning, are better suited for long-term forecasting than data-driven ML approaches, especially if the population and network attributes are expected to change substantially. These findings will be useful for planners and policy makers in the selection of the appropriate tool for forecasting travel demand.