Due to the poor lighting conditions and the presence of a large amount of suspended dust in coal mines, obtained video has problems with uneven lighting and low differentiation of facial features. In order to address these problems, an improved image enhancement method is proposed. Firstly, the characteristics of underground coal mine images are analyzed, and median filtering is selected for noise removal. Then, the gamma function and fractional order operator are introduced, and an image enhancement algorithm based on particle swarm optimization is proposed. Finally, several experiments are conducted, and the results show that the proposed improved algorithm outperforms classical image enhancement algorithms, such as MSR, CLAHE and HF. Compared with the original image, the evaluation metrics of the enhanced Yale face images, including average local standard deviation, average gradient, information entropy and contrast, are improved by 113.1%, 63.8%, 22.8% and 24.1%, respectively. Moreover, the proposed algorithm achieves a superior enhancement effect in the simulated coal mine environment.