Accurate destination prediction over sub-trajectories is essential for a wide range of location-based services. Traditional trip matching methods fail to capture temporal dependence hidden in trajectories and may suffer from data sparsity problems. With the help of massive trajectory data, state-of-the-art approaches based on deep learning (DL) have achieved great success. However, existing DL approaches rarely consider the influence of individual travel preferences in destination prediction. When the trip is long but the known partial trajectory is short, DL models are unable to produce satisfactory results. Thus, we design a feature extraction mechanism to extract useful temporal features, spatial features, and static covariates for destination prediction, among which the spatial features characterize individual travel preferences by considering two main movement patterns in daily travel. Then, a hierarchical model including multiple modules is proposed to finely process heterogeneous features. Extensive experiments conducted on two public datasets demonstrate the superior performance of the proposed model compared to the state-of-the-art methods. Moreover, further experimental results show that the proposed model still performs well when trajectory prefix is short or travel duration is long, which confirms the effectiveness of integrating individual travel preferences.