The parameters optimization is the key issue for directional drilling trajectory design in oil and gas fields development, and there are three main challenges in multi-objective and multi-constraint optimization: (1) how to establish a multi-objective optimization model based on geological constraints; (2) how to design an appropriate optimization algorithm and solve the optimization model effectively; (3) how to select the desired result from the obtained Pareto solution and meet the engineering requirements. To build a safe and cost-efficient directional drilling trajectory, a new multi-objective optimization model is established in this paper. The effective objective functions to evaluate the drilling trajectory are summarized as the minimum trajectory length, torque, and strain energy. Moreover, the new model takes the wellbore stability based on Mohr–Coulomb criterion as constraint to prevent the borehole from collapsing. A novel adaptive grid-based multi-objective particle swarm optimization(AGMOPSO) is presented to achieve a set of Pareto optimal solutions of the established optimization model. In this algorithm, a new particle flight mode based on arcsine function of inertia weight and Gaussian mutation strategy are introduced to further improve the global searching ability and obtain more non-inferior solutions. To ensure the uniformity of non-inferior solutions, the adaptive grid based on density control factor is designed to map the space of objective functions to the grid space and adaptively adjust the density of non-inferior solutions in the external archive. Besides, a linear weighted summation function is developed to realize leader selection and archive maintenance of non-inferior solutions. The optimization results on the Pareto front indicate that AGMOPSO has better convergence and uniformity than the unmodified algorithm and reported results. To be concluded, AGMOPSO achieves a better optimization performance and obtain a better trajectory for drilling trajectory optimization model with geological constraints, which has good practical and theoretical significance for directional drilling trajectory optimization.