We show theoretically and experimentally that scattered light by thermal phonons inside a second-order nonlinear crystal is the source of additional phase noise observed in optical parametric oscillators. This additional phase noise reduces the quantum correlations and has hitherto hindered the direct production of multipartite entanglement in a single nonlinear optical system. We cooled the nonlinear crystal and observed a reduction in the extra noise. Our treatment of this noise can be successfully applied to different systems in the literature.