Non-dipole effects in strong-field photoelectron momentum spectra have been revealed experimentally (Smeenk et al 2011 Phys. Rev. Lett. 106 193002; Ludwig et al 2014 Phys. Rev. Lett. 113 243001). For certain laser parameters and photoelectron momenta the spectra were found to be shifted against the laser propagation direction whereas one would naively assume that the radiation pressure due to the -force pushes electrons always in the propagation direction. Only the interplay between Lorentz and Coulomb forces may give rise to such counterintuitive dynamics. In this work, we calculate the momentum-dependent shift in and against the propagation direction by extending the quantum trajectory-based Coulomb-corrected strong-field approximation beyond the dipole approximation. A semi-analytical treatment where both magnetic and Coulomb forces are treated perturbatively but simultaneously reproduces the results from the full numerical solution of the equations of motion.