Green functions scattering method is generalized to consider mixing of electromagnetic polarizations after reflection from the plane boundary between different media and applied to derivation of the Casimir-Polder potential in systems with Chern-Simons plane boundary layers. The method is first applied to derive the Casimir-Polder potential of an anisotropic atom in the presence of a Chern-Simons plane boundary layer on a dielectric half-space. Then a general result for the Casimir-Polder potential of an anisotropic atom between two dielectric half-spaces with Chern-Simons plane parallel boundary layers is derived. The Casimir-Polder potential of an anisotropic atom between two Chern-Simons plane parallel layers in vacuum is expressed through special functions. Novel P-odd three-body vacuum effects are discovered and analyzed in the system of two Chern-Simons plane parallel layers and a neutral atom in its ground state between the layers. Remarkably, P-odd three-body vacuum effects arising due to 180 degree rotation of one of the Chern-Simons layers can be verified in experiments with neutral atoms having QED dipole interaction with an electromagnetic field.