Providing accurate and real-time bus travel time information is crucial for both passengers and public transportation managers. However, in the traditional bus travel time prediction model, due to the lack of consideration of the influence of different bus drivers’ driving styles on the bus travel time, the prediction result is not ideal. In the traditional bus travel time prediction model, the historical travel data of all drivers in the entire bus line are usually used for training and prediction. Due to great differences in individual driving styles, the eigenvalues of drivers’ driving parameters are widely distributed. Therefore, the prediction accuracy of the model trained by this dataset is low. At the same time, the training time of the model is too long due to the large sample size, making it difficult to provide a timely prediction in practical applications. However, if only the historical dataset of a single driver is used for training and prediction, the amount of training data is too small, and it is also difficult to accurately predict travel time. To solve these problems, this paper proposes a method to predict bus travel times based on the similarity of drivers’ driving styles. Firstly, the historical travel time data of different drivers are clustered, and then the corresponding types of drivers’ historical data are used to predict the travel time, so as to improve the accuracy and speed of the travel time prediction. We evaluated our approach using a real-world bus trajectory dataset collected in Shenyang, China. The experimental results show that the accuracy of the proposed method is 13.4% higher than that of the traditional method.