Precise segmentation of breast ultrasound images is essential for early breast cancer screening. However, the segmentation process is challenging due to the diverse morphology of tumors, blurred boundaries, and the similar grey intensity distribution between lesions and normal tissues. To overcome these obstacles, we propose the Asym-UNet, an asymmetric U-shaped network tailored for segmenting breast lesions with high accuracy and reliability. This network features a Multi-branch Residual Encoder (MRE), an External Attention Module (EAM), and a Boundary Detection Module (BDM). The MRE is designed to capture distinctive representations of breast lesions, facilitating accurate subsequent segmentation. The EAM directs the network to focus on the lesion areas while filtering out irrelevant information, thereby enhancing diagnostic precision. Furthermore, the BDM, incorporated to refine the boundaries of predicted masks, provides additional supervision and guidance through a multi-level prediction layer. We conducted evaluations of the Asym-UNet against thirteen other segmentation methods using two publicly available datasets. The results indicate that our network achieves state-of-the-art performance in breast tumor segmentation.