This paper improves accuracy of fisheye camera's intrinsic parameter estimation method using camera rotation by adding constraint to the camera rotation. The method takes advantage of trajectories of feature points in the scene. The trajectories of feature points are obtained by a rotation movement of the camera in a specific plane. The method therefore can utilize rich feature points for calibration, and furthermore, specific calibration targets are not required. The effectiveness of the proposed method is evaluated by simulations and experiments using a real fisheye camera. The improvement of the proposed method is verified by converting fisheye images to perspective images using the estimated parameters. Comparison with an existing method is also performed.