Angles-only orbit determination during space situational awareness missions for non-cooperative targets such as space debris is subject to the well-known observability problem, that is, the orbital state is weakly observable or even unobservable because of missing range measurements. The key point to solve this problem is how to quickly capture the nonlinearity of orbit dynamics if only one space-based passive sensor is available to observe the target and translational maneuvers are not executed. To solve the short-arc angles-only problem, this study develops a simple and fast orbit determination method by exploiting the advantages of radial basis function neural network (RBFNN). Emphasis was placed on the design of the RBFNN, which was trained to construct a mapping network model from line-of-sight (LOS) measurements to the relative orbit. First, the perturbed relative orbit dynamics considering the Earth's non-spherical J2 term and air drag was established with the LOS measurement model. Second, a RBFNN-based angles-only orbit determination frame was developed, in which the generator of the training dataset, data preprocessing algorithm, and RBF network structure were designed. Finally, the proposed method was verified and tested using a set of numerical simulations of low-Earth orbit cases. The anti-noise and generalization performance of the designed network and the sensitivity of the solution accuracy to the measurement frequency and span were presented and discussed. Additionally, the advantages of the RBFNN-based method were confirmed by comparison with the results obtained using the BPNN (Back Propagation Neural Network)-based method. The simulation results show that the average orbit prediction error for LEO scenarios is approximately 0.08% using the index of initial relative distance, where only three sets of LOS angles are measured within a span of 24 s.