Pavement cracks are the primary type of distress that cause road damage, and deep-learning-based pavement crack segmentation is a critical technology for current pavement maintenance and management. To address the issues of segmentation discontinuity and poor performance in the segmentation of irregular cracks faced by current semantic segmentation models, this paper proposes an irregular pavement crack segmentation method based on multi-scale convolutional attention aggregation. In this approach, GhostNet is first introduced as the model backbone network for reducing parameter count, with dynamic convolution enhancing GhostNet’s feature extraction capability. Next, a multi-scale convolutional attention aggregation module is proposed to cause the model to focus more on crack features and thus improve the segmentation effect on irregular cracks. Finally, a progressive up-sampling structure is used to enrich the feature information by gradually fusing feature maps of different depths to enhance the continuity of segmentation results. The experimental results on the HGCrack dataset show that GMDNet has a lighter model structure and higher segmentation accuracy than the mainstream semantic segmentation algorithms, achieving 75.16% of MIoU and 84.43% of F1 score, with only 7.67 M parameters. Therefore, the GMDNet proposed in this paper can accurately and efficiently segment irregular cracks on pavements that are more suitable for pavement crack segmentation scenarios in practical applications.