A six degrees of freedom (6-DOF) controller is a commonly used input device in three-dimensional user interface (3DUI) applications. However, for the fundamental task of target selection in 3D space, the selection accuracy decreases owing to the Heisenberg effect during the manipulation of the 6-DOF controller. Based on the pointing action of a 6-DOF device, we establish the mathematical model for raycasting and analyze the possibility of using a rotation gesture to select a target. This study proposes a target selection method using the axial rotation of the user's wrist, which reduces the negative impact of a discrete input for triggering the selection on accuracy. The detection model can identify the start time of the user's rotation action. The custom designed control display gain (CD gain) function can maintain the ray's stability during the rotation gesture. This method was verified using a 6-DOF pen and Vive controller in three experiments. The results show that the accuracy of the proposed rotation gesture-based raycasting target selection method is superior to that of the traditional button-pressing method, and it can be integrated into existing tracking systems for further application.