A mechanism for the phonon Hall effect (PHE) in nonmagnetic insulators under an external magnetic field is theoretically studied. PHE is known in (para)magnetic compounds, where the magnetic moments and spin-orbit interaction play an essential role. In sharp contrast, we here discuss that the PHE also occurs in nonmagnetic band insulators subject to the magnetic field. We find that a correction to the Born-Oppenheimer approximation gives rise to a Raman-type interaction between the magnetic field and the phonons; this interaction gives rise to the Berry curvature of a phonon band. This Berry curvature results in the finite thermal Hall conductivity κ_{H} in nonmagnetic band insulators. The value of κ_{H} is calculated for square and honeycomb lattices. The order of the magnitude estimation for κ_{H} is given for Si at room temperature.