Recent experimental realization of strongly imbalanced mixtures of ultracold atoms opens new possibilities for studying impurity dynamics in a controlled setting. We discuss how the techniques of atomic physics can be used to explore new regimes and manifestations of Anderson's orthogonality catastrophe (OC), which could not be accessed in solid state systems. We consider a system of impurity atoms localized by a strong optical lattice potential and immersed in a sea of itinerant Fermi atoms. Ramsey interference experiments with impurity atoms probe OC in the time domain, while radio-frequency (RF) spectroscopy probes OC in the frequency domain. The OC in such systems is universal for all times and is determined by the impurity scattering length and Fermi wave vector of itinerant fermions. We calculate the universal Ramsey response and RF absorption spectra. In addition to the standard power-law contribution, which corresponds to the excitation of multiple particle-hole pairs near the Fermi surface, we identify a novel contribution to OC that comes from exciting one extra particle from the bottom of the itinerant band. This gives rise to a non-analytic feature in the RF absorption spectra, which evolves into a true power-law singularity with universal exponent 1/4 at the unitarity. Furthermore, we discuss the manifestations of OC in spin-echo experiments, as well as in the energy counting statistic of the Fermi gas following a sudden quench of the impurity state. Finally, systems in which the itinerant fermions have two or more hyperfine states provide an even richer playground for studying non-equilibrium impurity physics, allowing one to explore non-equilibrium OC and to simulate quantum transport through nano-structures. This provides a useful connection between cold atomic systems and mesoscopic quantum transport.