Almost one out of five people, including children, suffers from musculoskeletal disorders. It is the second leading cause of disability worldwide. It affects the musculoskeletal system’s major areas, represented by the shoulder, forearm, and wrist. It causes severe pain, joint noises, and disability. To detect the abnormality, the radiologist analyzes the patient’s anatomy through X-rays of different views and projections. To automatically diagnose the abnormality in the musculoskeletal system is a challenging task. Previously, various researchers detected the abnormality in the musculoskeletal system from radiographic images by using several deep learning techniques. They used a capsule network, 169-layer convolutional neural network, and group normalized convolutional neural network in musculoskeletal abnormality detection. However, to propose methods for improving abnormality detection, further work needs to be done because the accuracy of the conventional methods is far away from 90%. This paper presents an ensemble learning-based classification system for detecting abnormality in wrist radiographs. Tags in radiographs may result in learning noisy features hence reducing the performance. Therefore, tags are segmented and removed using UNet trained on the annotated ground truths. Segmented images are then used for voting-based diagnosis. The simulation results show that the proposed methodology improves testing accuracy by 1.5%-4.5% compared to the available wrist abnormality detection methods. The proposed methodology can be used for any kind of musculoskeletal abnormality detection.