Driver distraction detection not only effectively prevents traffic accidents but also promotes the development of intelligent transportation systems. In recent years, thanks to the powerful feature learning capabilities of deep learning algorithms, driver distraction detection methods based on deep learning have increased significantly. However, for resource-constrained onboard devices, real-time lightweight models are crucial. Most existing methods tend to focus solely on lightweight model design, neglecting the loss in detection performance for small targets. To achieve a balance between detection accuracy and network lightweighting, this paper proposes a driver distraction detection method that combines enhancement and global saliency optimization. The method mainly consists of three modules: context fusion enhancement module (CFEM), channel optimization feedback module (COFM), and channel saliency distillation module (CSDM). In the CFEM module, one-dimensional convolution is used to capture information between distant pixels, and an injection mechanism is adopted to further integrate high-level semantic information with low-level detail information, enhancing feature fusion capabilities. The COFM module incorporates a feedback mechanism to consider the impact of inter-layer and intra-layer channel relationships on model compression performance, achieving joint pruning of global channels. The CSDM module guides the student network to learn the salient feature information from the teacher network, effectively balancing the model’s real-time performance and accuracy. Experimental results show that this method outperforms the state-of-the-art methods in driver distraction detection tasks, demonstrating good performance and potential application prospects.