To improve the efficiency of coal mine gas extraction, it is necessary to optimize the performance of sealing materials. In this paper, fly ash, nano-silica, and triethanolamine was selected to optimize the pore structure of the sealing materials. The qualitative and quantitative analysis of the seal material was performed by low-temperature liquid nitrogen adsorption experiments, scanning electron microscopy, and combined with fractal theory. The results show that the pores of common sealing materials are mainly mesoporous, accounting for 52.19%, and the volume of micropores is the smallest, accounting for 9.88%. The single doping of fly ash made the pore volume of the material decrease by 9.5%, and the percentage of mesopores increased by 1.3%, but the percentage of micropores decreased by 0.9%. Co-doping with nano-silica and fly ash increased the microporous percentage of the material by 0.77% and the peak stress by 17.43%. The synergistic effect of fly ash, nano-silica, and triethanolamine was most effective in optimizing the pore structure of the sealing material. The pore volume was reduced to 0.08198 cm3/g, and the fractal dimensions D1 and D2 were increased by 2.88% and 2.48%, respectively. The microscopic morphology of the material was also flatter and denser, and the hydration of fly ash was also more complete.