In laparoscopic liver surgery, accurately predicting the displacement of key intrahepatic anatomical structures is crucial for informing the doctor's intraoperative decision-making. However, due to the constrained surgical perspective, only a partial surface of the liver is typically visible. Consequently, the utilization of non-rigid volume to surface registration methods becomes essential. But traditional registration methods lack the necessary accuracy and cannot meet real-timerequirements. To achieve high-precision liver registration with only partial surface information and estimate the displacement of internal liver tissues inreal-time. We propose a novel neural network architecture tailored for real-time non-rigid liver volume to surface registration. The network utilizes a voxel-based method, integrating sparse convolution with the newly proposed points of interest (POI) linear attention module. POI linear attention module specifically calculates attention on the previously extracted POI. Additionally, we identified the most suitable normalization methodRMSINorm. We evaluated our proposed network and other networks on a dataset generated from real liver models and two real datasets. Our method achieves an average error of 4.23mm and a mean frame rate of 65.4fps in the generation dataset. It also achieves an average error of 8.29mm in the human breathing motiondataset. Our network outperforms CNN-based networks and other attention networks in terms of accuracy and inferencespeed.