A Novel Two-Stage Approach for Automatic Extraction and Multi-View Generation of Litchis

Abstract

Obtaining consistent multi-view images of litchis is crucial for various litchi-related studies, such as data augmentation and 3D reconstruction. This paper proposes a two-stage model that integrates the Mask2Former semantic segmentation network with the Wonder3D multi-view generation network. This integration aims to accurately segment and extract litchis from complex backgrounds and generate consistent multi-view images of previously unseen litchis. In the first stage, the Mask2Former model is utilized to predict litchi masks, enabling the extraction of litchis from complex backgrounds. To further enhance the accuracy of litchi branch extraction, we propose a novel method that combines the predicted masks with morphological operations and the HSV color space. This approach ensures accurate extraction of litchi branches even when the semantic segmentation model’s prediction accuracy is not high. In the second stage, the segmented and extracted litchi images are passed as input into the Wonder3D network to generate multi-view of the litchis. After comparing different semantic segmentation and multi-view synthesis networks, the Mask2Former and Wonder3D networks demonstrated the best performance. The Mask2Former network achieved a mean Intersection over Union (mIoU) of 79.79% and a mean pixel accuracy (mPA) of 85.82%. The Wonder3D network achieved a peak signal-to-noise ratio (PSNR) of 18.89 dB, a structural similarity index (SSIM) of 0.8199, and a learned perceptual image patch similarity (LPIPS) of 0.114. Combining the Mask2Former model with the Wonder3D network resulted in an increase in PSNR and SSIM scores by 0.21 dB and 0.0121, respectively, and a decrease in LPIPS by 0.064 compared to using the Wonder3D model alone. Therefore, the proposed two-stage model effectively achieves automatic extraction and multi-view generation of litchis with high accuracy.