Unmanned surface vehicles (USVs) offer significant value through their capability to undertake hazardous and time-consuming missions across water surfaces. Recently, as the application of USVs has been extended to nearshore waterways, object tracking is vital to the safe navigation of USVs in offshore scenes. However, existing tracking systems for USVs are mainly based on cameras or LiDAR sensors, which suffer from drawbacks such as lack of depth perception or high deployment costs. In contrast, millimeter-wave (MMW) radar offers advantages in terms of low cost and robustness in all weather and lighting conditions. In this work, to construct a robust and low-cost tracking system for USVs in complex offshore scenes, we propose a novel MMW radar-based object tracking method (ROTracker). The proposed ROTracker combines the physical properties of MMW radar with traditional tracking systems. Specifically, we introduce the radar Doppler velocity and a designed motion discriminator to improve the robustness of the tracking system toward low-speed targets. Moreover, we conducted real-world experiments to validate the efficacy of the proposed ROTracker. Compared to other baseline methods, ROTracker achieves excellent multiple object tracking accuracy in terms of 91.9% in our collected dataset. The experimental results demonstrated that the proposed ROTracker has significant application potential in both accuracy and efficiency for USVs, addressing the challenges posed by complex nearshore environments.