Intelligent defect detection systems based on deep learning have great potential for industrial applications to ensure product quality. However, the success of such data-driven approaches relies heavily on well-established datasets that require considerable time and labor for fine-grained annotation. Actual industrial datasets are often roughly labeled with many inaccurate annotations, thus seriously hindering the performance of defect detection. In particular, the boundaries of surface defects are not only difficult to label reliably, but also to detect accurately. In this paper, a novel approach termed BTDNet (Boundary-guided Task Decomposition Network) is proposed to tackle the challenge posed by inaccurate annotations. BTDNet decomposes the inaccurate original annotation into two accurate annotations, introducing two reliable auxiliary tasks. These auxiliary tasks are supervised separately using the new annotations, enabling BTDNet to mitigate the negative impact of inaccurate annotations during training and learn robust features. An efficient statistics recalibration module is constructed in BTDNet to customize input features for different tasks, while an adaptive feature summation module is employed to merge the outputs of all tasks. As an essential part of BTDNet, both modules are designed with a strong focus on lightweight. Extensive experiments on three publicly available industrial datasets confirm the effectiveness of our method, showing that BTDNet can consistently achieve performance gains across different baselines and input scales. For example, BTDNet clearly improves the F1-score of DeepLabv3 and GCNet methods on the Magnetic-tile dataset by 5.77% and 5.40%, respectively.