The control of autonomous mobile robots is an important topic in the literature of intelligent robotics. With great progress in intelligent systems, mobile robots are often autonomously driven. The computer vision replaces the human eye to be the feedback component and, through image processing, the necessary information can be deciphered. However, owing to the complexity of the whole system, a mathematical model is usually not available. Hence, many model—based robust control rules cannot be applied to vision—guided autonomous mobile robots. At the same time, the calculation of image processing takes up a long time, from fetching the image to creating the image information. In order to solve these problems, this paper proposes a novel pseudo—model following integral variable structure control (PMF—IVSC). By the grey model (GM(1, 2)) method, i.e. the grey model with one output and two inputs, the plant and the reference model are converted into an accumulated model and a pseudo—reference model respectively. Applying the IVSC, the accumulated model is governed to follow the pseudo—reference model. The PMF—IVSC will successfully solve the unknown system's model following control. Finally, an experimental mobile robot is developed to examine the potential of the proposed control scheme. This paper carries out the vision—based experiment of automatically driving while lane following. The experimental results show that the proposed controller will eliminate the swinging phenomenon and increase the accuracy while tracking. The results also show the practical capability of the proposed rule in intelligent robotic systems.