This study investigates the effects of serration angle on the three-dimensional flow field and noise reduction mechanism of a NACA6512-63 airfoil using large eddy simulation (LES) and spectral proper orthogonal decomposition (SPOD) techniques. The serration angle is defined as the ratio of serration wavelength (λ) to serration amplitude (h). The mean flow field analysis on suction side of serration reveals the outward flow towards the serration edges and the downwash flow between the valleys that generate the counter-rotating vortex pair. This flow alteration changes the serration effective angle of the serration, consequently affecting the noise reduction performance. Additionally, by comparing the general solution of Lyu’s semi-analytical equation with the SPOD mode results, the noise reduction mechanism is examined, and the frequency range in which noise reduction occurs is analyzed. The linearly distributed multipole-like pressure patterns, similar to those in the semi-analytical solution, are observed in the low-frequency range for all three cases, with the λ/h = 0.3 case exhibiting the most effective noise reduction. The findings suggest that the local deviation of the turbulence axis, caused by the strong outward flow near the serration edges, can hinder the formation of the optimal pressure field distribution required for effective noise reduction.