Let (X,g) be a compact Riemannian manifold with boundary Mn and σ a defining function of M. To these data, we associate natural conformally covariant polynomial one-parameter families of differential operators C∞(X)→C∞(M). They arise through a residue construction which generalizes an earlier construction in the framework of Poincaré-Einstein metrics and are referred to as residue families. Residue families may be viewed as curved analogs of conformal symmetry breaking differential operators. The main ingredient of the definition of residue families are eigenfunctions of the Laplacian of the singular metric σ−2g. We prove that if σ is an approximate solution of a singular Yamabe problem, i.e., if σ−2g has constant scalar curvature −n(n+1), up to a sufficiently small remainder, these families can be written as compositions of certain degenerate Laplacians (Laplace-Robin operators). This result implies that the notions of extrinsic conformal Laplacians and extrinsic Q-curvature introduced in recent works by Gover and Waldron can naturally be rephrased in terms of residue families. This spectral theoretical perspective allows easy new proofs of several results of Gover and Waldron. Moreover, it allows us to relate the extrinsic conformal Laplacians and the critical extrinsic Q-curvature to the scattering operator of the asymptotically hyperbolic metric σ−2g, extending the work of Graham and Zworski. The relation to the scattering operators implies that the extrinsic conformal Laplacians are self-adjoint. We describe the asymptotic expansion of the volume of a singular Yamabe metric in terms of Laplace-Robin operators (reproving results of Gover and Waldron). We also derive new local holographic formulas for all extrinsic Q-curvatures (critical and sub-critical ones) in terms of renormalized volume coefficients, the scalar curvature of the background metric, and the asymptotic expansions of eigenfunctions of the Laplacian of the singular metric σ−2g. These results naturally extend earlier results in the Poincaré-Einstein case. Furthermore, we prove a new formula for the singular Yamabe obstruction Bn. The simple structure of these formulas shows the benefit of the systematic use of so-called adapted coordinates. We use the latter formula for Bn to derive explicit expressions for the obstructions in low-order cases (confirming earlier results). Finally, we relate the obstruction Bn to the supercritical Q-curvature Qn+1.