In order to solve the problem that the star point positioning accuracy of the star sensor in near space is decreased due to atmospheric background stray light and rapid maneuvering of platform, this paper proposes a star point positioning algorithm based on the capsule network whose input and output are both vectors. First, a PCTL (Probability-Coordinate Transformation Layer) is designed to represent the mapping relationship between the probability output of the capsule network and the star point sub-pixel coordinates. Then, Coordconv Layer is introduced to implement explicit encoding of space information and the probability is used as the centroid weight to achieve the conversion between probability and star point sub-pixel coordinates, which improves the network’s ability to perceive star point positions. Finally, based on the dynamic imaging principle of star sensors and the characteristics of near-space environment, a star map dataset for algorithm training and testing is constructed. The simulation results show that the proposed algorithm reduces the MAE (Mean Absolute Error) and RMSE (Root Mean Square Error) of the star point positioning by 36.1% and 41.7% respectively compared with the traditional algorithm. The research results can provide important theory and technical support for the scheme design, index demonstration, test and evaluation of large dynamic star sensors in near space.