In materials with strong local Coulomb interactions, simple defects such as atomic substitutions strongly affect both macroscopic and local properties of the system. A nonmagnetic impurity, for instance, is seen to induce magnetism nearby. Even without disorder, models of such correlated systems are generally not soluble in 2 or 3 dimensions, and so few exact results are known for the properties of such impurities. Nevertheless, some simple physical ideas have emerged from experiments and approximate theories. Here, we first review what we can learn about this problem from 1D antiferromagnetically correlated systems. We then discuss experiments on the high Tc cuprate normal state which probe the effect of impurities on local charge and spin degrees of freedom, and compare with theories of single impurities in correlated hosts, as well as phenomenological effective Kondo descriptions. Subsequently, we review theories of impurities in d-wave superconductors including residual quasiparticle interactions, and compare with experiments in the superconducting state. We argue that existing data exhibit a remarkable similarity to impurity-induced magnetism in the 1D case, implying the importance of electronic correlations for the understanding of these phenomena, and suggesting that impurities may provide excellent probes of the still poorly understood ground state of the cuprates.