We calculate the Casimir-Polder frequency shift and decay rate for an atom in front of a nonreciprocal medium by using macroscopic quantum electrodynamics. The results are a generalization of the respective quantities for matter with broken time-reversal symmetry which does not fulfill the Lorentz reciprocity principle. As examples, we contrast the decay rates, the resonant and nonresonant frequency shifts of a perfectly conducting (reciprocal) mirror to those of a perfectly reflecting nonreciprocal mirror. We find different power laws for the distance dependence of all quantities in the retarded and nonretarded limits. As an example of a more realistic nonreciprocal medium, we investigate a topological insulator subject to a time-symmetry breaking perturbation.