Accurate lateral localization is the basis of intelligent vehicle decision making and control and one of the core problems of autonomous driving. In the process of vehicle driving, the vehicle body vibration, road bumps, and slope change will affect the motion parameters of the camera, which influences the localization accuracy. Therefore, we investigate a vehicle visual localization method based on the motion state estimation of vehicle camera by Gaussian Bayes sphere. This method uses Gaussian spherical crown sampling and maximum likelihood search to estimate the motion state of the vehicle camera in real time. Then, the real-time homography matrix between the sampling ground plane and the pixel plane is calculated according to the results to convert the coordinate points of the lane lines to the sampling ground plane for lateral localization of vehicles. Last, the localization results are restored to the real scale by the lane line width. The road experimental results show that the average error of the proposed method for lateral localization of vehicles is 5.7 cm under different road conditions on campus roads, and that under different road conditions on urban roads is 5.9 cm. At the same time, the localization accuracy of the proposed method is tested on the slope and under different weather conditions, and good results are obtained.