Lignin, as a renewable energy source, has attracted much attention due to its high valorization potential. Due to the complex composition of lignin, vanillin, an important intermediate of lignin, was chosen as a model compound in this study. The study examined the influence of the HZSM-5 molecular sieve on the quality of pyrolysis oils during the catalytic pyrolysis of vanillin at 600 °C using a horizontal tube furnace and Py-GC/MS analysis. The results revealed that the HZSM-5 catalyst significantly increased the yields of toluene, guaiacol, and 4-hydroxyisophthalaldehyde by 2.49 %, 19.26 %, and 1.8 %, respectively, whereas reducing the contents of phenol and 3,4-dimethoxyphenol by 6.01 % and 3.79 %, respectively. These findings indicate that the HZSM-5 molecular sieve effectively improves the quality of pyrolysis oils during the catalytic pyrolysis of vanillin. The evolution of the main functional groups in the pyrolysis of vanillin was analyzed using an in situ diffuse reflectance infrared Fourier transform spectroscopy. The addition of HZSM-5 promoted the cleavage of oxygen-containing functional groups and the generation of aromatic hydrocarbons during the pyrolysis of vanillin. Furthermore, the pyrolysis reaction pathways of vanillin were simulated under the B3LYP/6-311 g (d, p) basis set using the density functional theory. The acidic sites of HZSM-5 interacted with vanillin via the hydrogen bonds that affected the energy barriers of the pyrolysis reaction pathways of vanillin. HZSM-5 inhibited the generation of phenol and 3,4-dimethoxyphenol by increasing the energy barriers by 32.95 and 31.03 kJ/mol, respectively; however, HZSM-5 promoted toluene production by decreasing the energy barrier by 37.38 kJ/mol. The effects of HZSM-5 on the products in the simulation results were consistent with the experimental results.