X-ray security image contraband classification is widely used to assist in maintaining aviation and transportation security. This paper suggests an end-to-end X-ray security inspection image classification method that takes sample imbalance into account in order to address the issues of different scales of contraband in X-ray images, challenging samples, and unbalanced positive and negative samples inherent in passenger baggage security inspection. The feature fusion module is used to enhance the model's ability to express picture edge and texture features while the multi-scale feature extraction network is used to capture the features of numerous sorts of illegal goods with various scales. Based on the cost-sensitive idea, the loss function is designed to solve the problem of dataset imbalance, and improve the classification accuracy of difficult samples. experimental results of the subset constructed on the public dataset SIXray show that the proposed method improves the mean AP index by compared with the current optimal end-to-end classification model, especially for hard-to-classify samples such as scissors, the AP index has a significant improvement effect.