To utilize autonomous driving tractors for precision agriculture, the cultivable farmland area and its boundary information must be determined. Currently, autonomous driving is performed using the global positioning system (GPS) data of four boundary points; however, the operator must drive and obtain the boundary information. In this study, a vision-based autonomous driving algorithm was developed. A stereo camera and a MobileNetV3-based segmentation algorithm were used to segment the cultivable area and boundary, perform autonomous driving, and store the boundary information. Inverse perspective mapping was performed to store the farmland information in a two-dimensional space, and occupancy grid mapping was performed according to the heading angle of the tractor obtained from Inertial Measurement Unit (IMU) and GPS. By using the preview distance concept, a target heading angle of the tractor was generated at a point away from the boundary, and Proportional-Differential controller was used for the path-tracking algorithm.