Lateral vibration of bottom hole assembly (BHA) severely affects the drilling efficiencies of downhole tools and even leads to downhole accidents. To further study the lateral motion states of BHA during rotary drilling, a new experimental setup was built based on the similarity criterion that was derived using the Buckingham π-theorem. Meanwhile, four characteristic parameters were introduced to distinguish the types of motion states and highlight their dynamic characteristics. Moreover, a systematical experiment was conducted to analyze the effects of well inclination, rotation speed, weight on bit (WOB), measurement position, and friction coefficient on the motion states. In addition, the occurrence mechanism of the motion states was explained through the principle of least energy dissipation, and the numerical simulation was conducted to calculate the energy dissipation. The results show that the motion states in vertical wells contain forward whirl, random motion, and backward whirl, and the main motion state in deviated wells is circular arc swing. The rotation speed and friction coefficient have major effects and WOB has moderate effects on the motion states in vertical wells. The larger rotation speed, friction coefficient, and WOB, the easier is the occurrence of backward whirl. Meanwhile, well inclination has suppressing effects on lateral vibration, while rotation speed and friction coefficient have promoting effects. WOB has promoting effects on lateral vibration in vertical wells and relatively small effects in deviated wells. The stabilizer can effectively limit lateral vibration. Moreover, the change of the motion states depends on which motion state consumes the least external work. In particular, the conditions of forming backward whirl are more rigorous than those of maintaining backward whirl. Thus backward whirl is a stable motion state and is difficult to stop once it occurs. In this paper, the study can help to understand the dynamic characteristics and occurrence mechanism of the motion states during rotary drilling.