This study considers a relative orbit estimation problem wherein an observing spacecraft navigates with respect to a target space object at a large separation distance (several kilometers) using only the bearing angles obtained by a single onboard camera. Generally, the extended Kalman filter (EKF), which is based on linear relative motion equations such as the Clohessy–Wiltshire equation, is used for the relative navigation of satellites. The EKF linearizes the estimation error around the current estimate and applies the Kalman filter equations to this linearized system. However, it has been shown that nonlinearities of the orbit determination problem can make the linearization assumption insufficient to represent the actual uncertainty. Therefore, an analytical second-order extended Kalman filter (ASEKF) for relative orbit estimation is proposed in this study. The ASEKF, to sequentially estimate the relative states of satellites and their associated uncertainties, is formulated based on a second-order analytic relative-motion equation under J2-perturbtation, which can overcome the deficiencies of existing approaches that mainly focus on applications in two-body, near-circular, and linearized orbit dynamics. Numerical results show that the proposed method provides superior robustness and mean-square error performance compared to linear estimators under the conditions considered.