ABSTRACT The orbital dynamics in the vicinity of a binary asteroid system has been studied extensively, motivated by the special dynamical environment and possible exploration missions. Equilibrium points, periodic orbits, and invariant manifolds have been investigated in many studies based on the model of the Restricted Full Three Body Problem (RF3BP). In this paper, a new semi-analytical orbital dynamical model around the primary of a binary system is developed as a perturbed two-body problem. The solution includes the effect of the primary's oblateness and the secondary's third-body gravity. The semi-analytical dynamical model, also denoted as the averaged model, is obtained by using the averaging process and Lagrange planetary equations in terms of the Milankovitch orbital elements. This semi-analytical model enables much faster orbital propagations than the non-averaged counterpart, and is particularly useful in orbital stability analysis and the design of long-term passively stable orbits and orbits with specific performance, e.g. frozen orbits. The applicability of the semi-analytical model is then discussed. Two parameters describing relative magnitudes of both perturbations w.r.t. the primary's point mass gravity and the third parameter related to the orbital period ratio w.r.t. the secondary are defined to provide indicators for the validity of the averaged model. The validity boundaries in terms of the three parameters are given based on numerical simulations, by comparing with the full orbital model. The application to a real binary system, 2003 YT1, has shown that the averaged solution has a high precision in the long-term orbital propagation.