Breast cancer is the most common cancer among women. Accurate and intelligent ultrasound image tumor segmentation can help physicians efficiently determine the location, shape and features of tumors and then develop appropriate treatment plans. To this end, we propose a deep learning method combining interlayer information and dual attention improvement for accurate and intelligent segmentation of breast tumors in ABVS image sequences. We extracted the slices containing tumors in ABVS images and formed a three-channel image input network with the current frame and the corresponding before and after frames. The network combines interlayer information during computation, and the before and after frames can guide the current frame image for more accurate tumor segmentation. In addition, we improved the channel attention module by replacing the full connection with a one-dimensional convolution and applying global max pooling on the basis of the original channel attention module, which improved the segmentation performance of the module while reducing the number of parameters. The improved channel attention module was added to the U-Net skip-connection section to assign different weight coefficients to each feature channel according to the characteristics of the input three-channel image. Our method outperformed other mainstream tumor segmentation methods on the dataset in both centers, achieving a Dice similarity coefficient (DSC) of 84.97 %, a Jaccard coefficient (JC) of 74.63 %, a 95 % Hausdorff distance (95HD) of 6.22 mm on the inner center, and DSC of 80.79 %, JC of 69.74 %, 95HD of 9.82 mm on the outer dataset.