Model Predictive Control has gained much attention due to its potential to improve building operations by reducing costs, integrating renewable energy sources, and increasing thermal comfort. This paper aims to compare the accuracy of grey-box models based on resistance–capacitance (RC) networks and Long-Short-Term Memory (LSTM) neural networks in the prediction of the buildings’ thermal response, which is a key feature for the successful implementation of predictive controllers. Indoor air temperature prediction tests have been performed on simulated and measured data from buildings with different thermal insulation and thermal mass during both heating and cooling seasons. Results show that neural networks have, on average, a better prediction performance than grey-box models. Both modelling approaches are affected by the building characteristics and by the season considered. The grey-box models require less training data, although the latter seems to play a role only in the worse-performing tests. When user setpoint changes in the testing phase, the LSTM neural network shows a significant drop in the root mean square error. In conclusion, although LSTM outperforms grey-box models on average, the reduced training data and higher reliability under normal operating conditions, as well as their linearity, make RC models a strong alternative for predictive controllers.