In an ultra-dense network (UDN), how to associate the mobile user equipments (UEs) with the appropriate low power nodes (LPNs) is a very essential and challenging problem, especially when the UEs are moving and may suffer from frequent handovers. In this paper, the UE association problem is investigated from a long-term perspective, where the UE's mobility pattern, i.e., the moving velocity and direction, is taken into consideration. To predict the UE's positions based on their previous activity data, a practical scheme, i.e., the classification and batch online learning method (CBOM), is proposed for achieving the UE's velocity and direction transition matrices. Then the UE association problem with the objective of maximizing the UEs’ expected achievable rate while minimizing the number of handovers is formulated. Since the proposed problem is a NP-hard mixed integer nonlinear programming problem, we reformulate it into a matching game, and propose a mobility-based long-term matching game algorithm to find the near-optimal solutions. We prove that our proposed algorithm is convergent and has low computational complexity. Simulation results show our two proposed schemes outperform the existing schemes in terms of the achieved data rate, the number of handovers, and the overall system utility.