A precise vehicle dynamics model is critical for simulation and algorithm testing. Neural networks have been widely used to build high-fidelity vehicle dynamics models due to the excellent learning ability. However, data starvation is a common phenomenon in neural networks. With limited data, it is difficult for neural networks to achieve precise predictions. To address these problems, a hybrid model combining physics and dual attention neural networks is developed to model vehicle lateral dynamics. First, due to the interpretability of the physical model, linear lateral dynamics model is regarded as a prior model. However, due to the imperfect prior knowledge, there are residuals between the prior and the actual vehicle dynamics. Therefore, neural networks are used to characterize the residuals to achieve recalibration of vehicle dynamics model. Modeling vehicle residual dynamics with neural networks is a time series forecasting problem. The GRU with a dual attention mechanism and adaptive initial hidden states (DA-AGRU) is designed to capture spatial and temporal correlations in the vehicle dynamics data. In particular, considering the unique auto regressive structure of the hybrid model, a spatial attention mechanism with a feature fusion module is designed, so as to globally compute the weights of different channel features. The dataset used to train and validate the model is recorded from a vehicle platform, and the experimental results show that the proposed hybrid model can accurately predict vehicle dynamics states in a data-scarce environment.