In agricultural operation scenarios, the diversity of farmland terrain, crops and other forms, as well as uncertain factors such as weather changes and crop growth during agricultural operation, can have an impact on the construction of high-precision maps. To address these challenges and analyze operational scenarios based on the characteristics of agricultural scenarios, this paper proposes a point cloud map construction algorithm for plant point removal and locatability estimation. Based on the existing Simultaneous Localization and Mapping (SLAM) framework, plant point removal and locatability estimation are improved. Firstly, Red, Green, Blue (RGB) images and Near Infrared (NIR) images are fused to identify and remove plant point clouds, preserving effective inter frame matching information, reducing the impact of dynamic points on inter frame matching, and achieving high front-end motion estimation accuracy. Then, the localization estimation method based on learning is used to determine the motion estimation status and determine whether to execute the backend optimization algorithm. Finally, the back-end optimization algorithm based on Factor graph is designed, and the Factor graph, constraint relationship and optimization function are constructed to optimize the pose of all frames. The optimized map construction algorithm reduces the re projection errors between field roads, paths, and crop rows by 10.27%, 20.76%, and 14.36% compared to before optimization. To verify the actual operational effectiveness of the point cloud map construction algorithm, the hardware part of the multi-sensor information collection system was designed, and sensor internal and external parameter calibration were also carried out. A map information collection vehicle was built and field experiments were conducted. The results showed that the positioning error of the point cloud map construction method proposed in this paper is less than 0.5°, and the cumulative error of 30 m translation is less than 12 cm, which meets the actual operational requirements.