This paper addresses the method for estimating maneuvering mathematical models in waves, and uses captive model tests to verify it. To obtain the wave force and moment, two kinds of captive model tests are performed. One is straight towing tests with different towing directions to consider various incident wave angles, while the other is circular motion tests at different regular wave frequency. These captive model tests provide added resistance and wave-induced steady force considering both different wave direction and ship speed, and different wave drift angles. The wave drift forces and moment obtained are validated by performing the maneuvering simulation in waves, and comparison with the results of free running model tests. This paper introduces the application of the Estimation-Before-Modeling (EBM) approach, one of the System Identification methods, to estimate mathematical models for ship maneuverability targeting KRISO Very Large Crude oil Carrier (KVLCC2). The most important force component for the prediction of the maneuverability of a ship is the accuracy of the wave-induced steady force acting on the ship. Here, the EBM approach is divided into two steps: First, the motion variables and forces are estimated by Extended Kalman Filter and Modified Bryson-Frazier smoother. Second, the wave-induced steady forces are determined by removing the hydrodynamic force acting on the ship during motion. Also, during the second step of the EBM approach, the database consisting of wave length and incident wave angles is constructed. All estimated variables are based on data obtained by free running model tests in regular waves. Finally, the estimated wave-induced steady force is interpolated, added to the hydrodynamic force component of the 3-DoF equations of motion, and the turning simulation of the ship in waves is then performed. Comparison of the results with the results of free running model tests confirms the validity of the established model.