To reduce the risk of head trauma to workers working in high-risk workplaces such as construction sites, we designed a new automated lightweight end-to-end convolutional neural network to identify whether all people on a construction site are wearing helmets. Firstly, we used GhostNet as the backbone feature extraction network to take advantage of its low running cost and make the model lighter overall while ensuring efficient automatic feature extraction. Secondly, we designed a multi-scale segmentation and feature fusion network (MSFFN) in the feature-processing stage to improve the algorithm’s robustness in detecting objects at different scales. In contrast, the design of the feature fusion network can enrich the diversity of helmet features and improve the accuracy of helmet detection when distance changes, viewpoint changes, and occlusion phenomena occur. Thirdly, we proposed an improved version of the attention mechanism, the lightweight residual convolutional attention network version 2 (LRCA-Netv2). The main idea of the improvement is implemented around the spatial dimension by fusing the combined features along with the horizontal and vertical directions and then weighting them separately. Such an operation allows the establishment of dependencies between the more distant features with improved accuracy compared to the original LRCA-Net. Finally, when tested on the dataset, the proposed lightweight helmet-wearing detection network has a mAP and FPS of 93.5% and 42.