In most experiments to test the weak equivalence principle, the test objects coupled with gravity are treated as point particles. To study whether the weak equivalence principle is obeyed by nonlocal objects, especially wave packets with interesting physical configurations, one must first properly define the trajectory of such objects. A phenomenon called the geometric spin Hall effect suggests that one can directly use the energy-momentum tensor to describe the motion of wave packets. In this paper, we construct spin-polarized free-falling electromagnetic wave packets in gravity, and calculate the evolution of the center of the energy-momentum tensor. We find that the electromagnetic wave packets with opposite helicity are separated in the direction perpendicular to spin and gravity. This behavior is thus a kind of gravitational birefringence, and it means that motions of these spin-polarized wave packets in gravity do violate the weak equivalence principle. Furthermore, we find that the trajectories defined by different components and expressions of the energy-momentum tensor are not the same, and they are all different from the trajectory given by the Mathisson-Papapetrou-Dixon equations with the constraint ${p}_{\ensuremath{\mu}}{S}^{\ensuremath{\mu}\ensuremath{\rho}}=0$. This suggests that the trajectory of spin-polarized wave packets in gravity, and its confrontation with the weak equivalence principle, depend not only on the gravitational interaction but also strongly on how the wave packets are measured and analyzed.