Accurate vehicle acceleration prediction is useful for developing reliable Advanced Driving Assistance Systems (ADAS) and improving road safety. The existence of driver heterogeneity magnifies the variations in acceleration data, leading to consequential impacts on the precision of vehicle acceleration prediction. However, few studies have fully considered the driver heterogeneity when predicting vehicle acceleration. To model the characteristics of individual drivers, this study first identifies the driving behavior semantics which is defined as the underlying patterns of driving behaviors. The analysis results from the coupled hidden Markov model (CHMM) are used to evaluate the driving behavior differences between different drivers by Wasserstein distance. Then the convolutional neural network (CNN) and long short-term memory (LSTM) network are applied to predict vehicle acceleration. To validate the accuracy of the proposed prediction framework, vehicle acceleration data in car-following conditions is extracted from the safety pilot model deployment (SPMD) dataset. The segmentation results indicate that the CHMM possesses a robust capacity for modeling driving behavior. The prediction results demonstrate that the proposed framework, which incorporates driver clustering before prediction, significantly improves the accuracy of predictions. And the CNN-LSTM outperforms the LSTM in predicting vehicle acceleration during car-following scenarios. The findings from this study can enhance the development of personalized functionalities within ADAS to promote its deployment, thereby improving its acceptance and safety.