The new era of the Internet of Things is promoting the evolution of self-driving vehicles into connected and autonomous vehicles (CAVs). The deployment of CAVs in smart cities is highly dependent on the performance of their underlying networks known as vehicular networks. Designing an effective clustering approach is of great importance in such dynamic networks as it can significantly improve the reliability and scalability of the routing protocols. In this paper, we consider a heterogeneous vehicular network architecture that supports vehicle-to-vehicle and vehicle-to-infrastructure communications based on IEEE 802.11p and cellular networks (LTE/5G) with direct communications, namely cellular vehicle-to-everything (C-V2X) technologies. We introduce a novel clustering scheme for real-time routing based on the proposed network architecture. We formulate the problem of optimal clustering for connected and autonomous vehicles (OCCAV) and show that the problem is NP-hard. We then develop a clusters' stability maximization algorithm (CSM), which utilizes the stability degree of vehicles over a prediction horizon to efficiently solve the optimization problem in real-time. The algorithm is used within a rolling horizon framework for continuously solving the problem, making the resulting clusters adaptive to future traffic dynamics. We propose a hybrid routing protocol based on our clustering scheme, aiming to improve the packet delivery ratio and reduce the average delivery delay. For evaluation purposes, we use extensive realistic simulations based on mobility scenarios validated using real vehicular trajectories. The results demonstrate that our clustering scheme improves the alternative algorithms in terms of the average cluster head duration, cluster head change rate, cluster member duration, and overall stability by 61%, 62%, 44%, and 52%, respectively. CSM also outperforms clustering overhead by 54%. Compared to the other cluster-based routing, our scheme also achieves a higher packet delivery ratio by up to 30%, and a lower average delay by up to 52%. To provide an in-depth analysis of the optimality of our scheme and its alternatives, we also use the Gurobi optimizer to find an optimal solution to the OCCAV problem. The results suggest that our scheme can achieve near-optimal cluster stability.