The protein-rich defatted microalgae (Nannochloropsis gaditana) after lipid extraction was used as feedstock in catalytic pyrolysis to optimize economic returns of microalgae waste utilization. The effects of Ni- and Co-modified HZSM-5 on the formation of aromatic hydrocarbons (AHs), deoxygenation and denitrogenation in catalytic fast pyrolysis were studied via pyrolysis-gas chromatography and mass spectrometry (Py-GC/MS) at 500°C. Monometallic modified catalysts were synthesized by loading 0.5 wt%, 1 wt%, and 2 wt% of metal species onto HZSM-5 using ion exchange methods. The composition and pyrolysis pathways of primary volatiles detectable by gas chromatography (GC) were investigated, which included phenols, aliphatic hydrocarbons, AHs, acids, furans, N-heterocyclic compounds (N-HCs), amides and nitriles. The relative yields in AHs increased about 10.9 (0.5 wt% Ni/HZSM-5 or Ni0.5), 10.2 (Ni1), 6.8 (Ni2), 7.3 (Co0.5), 9.7 (Co1) and 7.8 (Co2) times compared to the relative yield from defatted microalgae pyrolysis. The doping of metals onto the HZSM-5 altered its porosity and acidity, leading to an impaired diffusion of reactants but also the creation of new acidic sites, which enhanced the selectivity towards monoaromatic hydrocarbons (MAHs) by aromatization and cleavage of N-containing heterocycles via the H radicals formation. Ni-loaded HZSM-5 outperformed Co-modified zeolite with respect to AHs production, deoxygenation and denitrogenation. The specific surface area (SBET) and pore volume were positively correlated with the formation of AHs, while strong acidity showed a strong positive correlation with the generation of nitriles and N-HCs. Relatively low loadings of Ni and Co (0.5–1 wt%) on HZSM-5 (Si/Al = 23) were recommended to decrease the N-containing compounds in pyrolysis vapors from defatted microalgae. The findings may shed light on the development of a high-performance HZSM-5 catalyst for the production of AHs and denitrogenation from protein-rich defatted microalgae.