Different individuals may move to different regions over time, but every individual has several fixed travel positions or unique travel patterns. Predicting destinations of each individual facilitates traffic demand management, which has great research value. Based on the data of multi-day GPS and passengers’ travel survey, a hidden Markov model (HMM) is employed in this paper to predict trip destination for weekdays and weekends. Firstly, the habit of destination choice among consecutive days and weeks can be discovered by identifying frequently visited destinations. Then, on the basis of Viterbi algorithm, this paper takes frequently visited destinations as one of the factors of the predicting process and constructs a travel destination prediction model based on HMM. Then, the HMM is calibrated with Baum-Welch algorithm and passengers’ travel destination characteristics are effectively analyzed. Finally, the HMM was compared with several classical algorithms. The results show that the place of residence and work are the most probable activities to occur and workplace dominates the activities when duration is longer than 8 h. Moreover, the results of frequently visited destinations identification indicate that the patterns of destination choice on weekdays and weekends are different from each other. In addition, the results show that the prediction accuracy on weekdays is higher than that on weekends and HMM outperforms other prevailing algorithms. The method proposed in this paper can be applied to real-time travel navigation applications, as well as supporting health and safety fields, such as epidemic prevention and control.