Remote-sensing object detection is a challenging task due to the difficulties of separating the objects with arbitrary direction from complex backgrounds. Though substantial progress has been made, there still exist challenges for object detection under the scenario of small scale, large aspect ratio, and dense distribution. Besides, the current mainstream approach falls under anchor-based multi-stage method, which has a serious shortcoming of slower inference speed. To conquer the aforementioned issues, this paper used RoPoints (points in rotation objects), a new better representation of objects as a set of sample points to perform object localization and classification. Then, we propose an anchor-free refined rotation detector:ROPDet based on RoPoints for more accurate and faster object detection. In our method, there is no need to predefine a large number anchors with different shapes. We only need to learn RoPoints for each object followed by converting to the corresponding bounding box, which greatly accelerates the inference process. Extensive experiments on two public remote-sensing datasets DOTA and HRSC-2016 demonstrate the competitive ability in terms of accuracy and inference speed.