Target tracking is a key task for maintenance of maritime rights and interests in the coastal countries, which mainly depends on the cruises of large official ships, and their deficiencies are high cost and less stealth. USVs (Unmanned Surface Vehicles) have been gradually employed in maritime ISR(Intelligence, Surveillance and Reconnaissance) due to autonomy, safety and flexibility. In this paper, a target tracking algorithm for USV's vision is proposed, which considers motion features of the USV, such as velocity, pitch and heading. Firstly, a CM(Centroid Matching) method for the target detection is proposed based on FT(Frequency-Tuned) saliency method, which can remove environment disturbances. Secondly, a fDSST(fast Discriminative Scale Space Tracker) is improved for target tracking by considering motion information of the USV from aboard GPS and IMU(Inertial Measurement Unit), where the learning rates of the correlation filters are adjusted adaptively according to variety of the velocity, pitch and heading of the USV. Lastly, the modified target detection and tracking are integrated to acquire a robust tracking result. Based on our USV called as `Jiuhang 490', sea tracking trials for rubber boat in four encountering scenes are executed at Nanjiang port in Qingdao city, China in July 2018. The favorable performance of the proposed algorithm is quantitatively validated comparing to the original fDSST algorithm in the accuracy of position, scale of the target and the tracking speed.