The lunar probe, Chang’E-2, is the first one to successfully achieve both the transfer to Sun–Earth libration point orbit and the flyby of near-Earth asteroid Toutatis. This paper, taking the Chang’E-2’s asteroid flyby mission as an example, provides a method to design low-energy transfers from the libration point orbit to an asteroid. The method includes the analysis of transfer families and the design of optimal impulse transfers. Firstly, the one-impulse transfers are constructed by correcting the initial guesses, which are obtained by perturbing in the direction of unstable eigenvector. Secondly, the optimality of one-impulse transfers is analyzed and the optimal impulse transfers are built by using the primer vector theory. After optimization, the transfer families, including the slow and the fast transfers, are refined to be continuous and lower-cost transfers. The method proposed in this paper can be also used for designing transfers from an arbitrary Sun–Earth libration point orbit to a near-Earth asteroid in the Sun–Earth–Moon system.