The influence of serration angle on the three-dimensional flow characteristics and noise attenuation mechanisms of a NACA6512-63 airfoil is explored in this research using large eddy simulation (LES) and spectral proper orthogonal decomposition (SPOD) methods. Analysis of the mean flow field reveals the formation of counter-rotating vortex pairs, resulting from the outward flow towards the serration edges and the downwash flow in the valleys. This flow modification alters the effective angle of the serration, which in turn impacts its noise reduction performance. Furthermore, the noise reduction mechanism is investigated by comparing the general solution of Lyu's semi-analytical equation with the SPOD mode results, and the frequency range associated with noise attenuation is analyzed. The findings indicate that the case with λ/h=0.15 demonstrates the most significant noise reduction effect. This research offers valuable insights into the interplay between serration geometry and the resulting flow characteristics and noise reduction mechanisms.