PlaneAC: Line-guided planar 3D reconstruction based on self-attention and convolution hybrid model

Abstract

Planar 3D reconstruction aims to simultaneously extract plane instances and reconstruct the local 3D model through the estimated plane parameters. Existing methods achieve promising results either through self-attention or convolution neural network (CNNs), but usually ignore the complementary properties of them. In this paper, we propose a line-guided planar 3D reconstruction method PlaneAC, which leverages the advantage of self-attention and CNNs to capture long-range dependencies and alleviate the computational burden. In addition, explicit connection between two adjacent attention layers is built for better leveraging the transferable knowledge and facilitating the information flow between tokenized feature from different layers. Therefore, the subsequent attention layer can directly interact with previous results. Finally, a line segment filtering method is presented to remove irrelevant guiding information from indistinctive line segments extracted from the image. Extensive experiments on ScanNet and NYUv2 public datasets demonstrate the preferable performance of our proposed method, and the results show that PlaneAC achieves a better trade-off between accuracy and computation cost compared with other state-of-the-art methods.