Low Earth Orbit (LEO) mega-constellation networks, exemplified by Starlink, are poised to play a pivotal role in future mobile communication networks, due to their low latency and high capacity. With the massively deployed satellites, ground users now can be covered by multiple visible satellites, but also face complex handover issues with such massive high-mobility satellites in multi-layer. The end-to-end routing is also affected by the handover behavior. In this paper, we propose an intelligent handover strategy dedicated to multi-layer LEO mega-constellation networks. Firstly, an analytic model is utilized to rapidly estimate the end-to-end propagation latency as a key handover factor to construct a multi-objective optimization model. Subsequently, an intelligent handover strategy is proposed by employing the Dueling Double Deep Q Network (D3QN)-based deep reinforcement learning algorithm for single-layer constellations. Moreover, an optimal cross-layer handover scheme is proposed by predicting the latency-jitter and minimizing the cross-layer overhead. Simulation results demonstrate the superior performance of the proposed method in the multi-layer LEO mega-constellation, showcasing reductions of up to 8.2% and 59.5% in end-to-end latency and jitter respectively, when compared to the existing handover strategies.