Grouping of tobacco is a key technology in intelligent management of tobacco planting and fine blending in cigarette industry. However, existing grouping methods based on deep learning are limited by lacking key feature expression due to excessive abstraction at higher scales. In addition, the inter-class discrimination ability is limited and the model prefers to majority classes during training. To tackle these problems, we propose a real-time tobacco grouping network (TGNet) based on channel weighting of higher scale features and dynamic loss regulation. Firstly, the industrial array camera equipped on a machine vision detection platform is used to capture the original tobacco images, which including the orange (F), lemon (L), variegated (K), greenish (V), and green-yellow (GY) categories manually labeled by grading experts. Then, to tackle the deficiency of key feature expression in the higher scale abstraction layers, a feature channel weighting module is proposed, which generates weighting factors proportional to different channels to re-encode higher scale features and emphatically highlight the key area and color information. The lightweight structure of feature channel weighting module compresses the model volume and improves forward inference speed. Finally, aiming at the problems of inter-group ambiguity and class imbalance dilemma, the focal and dynamic margin (FADM) loss with focusing on minority classes training and adaptative margin is proposed. The dynamic margin is adaptatively encoded by a Cosine function of similarity, which uses tobacco category recognition probability to measure the similarity of tobacco. Extensive performance evaluations show that TGNet outperforms other classification models in terms of the average classification accuracy 77.16%, inference time 112FPS, model size 2.5MB, model computation 0.12GFlops and model parameter number 0.613 M. The classification accuracy of minority classes V and GY is improved.