In this paper, the Lie symmetry analysis and the dynamical system method are performed on an integrable evolution equation for surface waves in deep water $$\begin{aligned} 2\sqrt{\frac{k}{g}}u_{xxt}=k^2u_x-\frac{3}{2}k(uu_x)_{xx}. \end{aligned}$$ All of the geometric vector fields of the equation are presented, as well as some exact similarity solutions with an arbitrary function of t are obtained by using a special symmetry reduction and the dynamical system method. Different kinds of traveling wave solutions also be found by selecting the function appropriately.