For a boundary conformal field theory to give a good approximation to the bulk flat-space $S$-matrix, a number of conditions need to be satisfied: some of those are investigated here. In particular, one would like to identify an appropriate set of approximate asymptotic scattering states, constructed purely via boundary data. We overview, elaborate, and simplify obstacles encountered with existing proposals for these. Those corresponding to normalizable wave functions undergo multiple interactions; we contrast this situation with that needed for a flat-space Lehmann-Symanzik-Zimmermann treatment. Non-normalizable wave functions can have spurious interactions, due either to power-law tails of wave packets or to their non-normalizable behavior, which obscure $S$-matrix amplitudes we wish to extract, although in the latter case we show that such gravitational interactions can be finite, as a result of gravitational redshift. We outline an illustrative construction of arbitrary normalizable wave packets from boundary data that also yields such spurious interactions. Another set of nontrivial questions regard the form of unitarity relations for the bulk $S$-matrix, and in particular its normalization and multiparticle cuts. These combined constraints, together with those found earlier on a boundary singularity structure needed for bulk momentum conservation and other physical/analytic properties, are a nontrivial collection of obstacles to surmount if a fine-grained $S$-matrix, as opposed to a coarse-grained construction, is to be defined purely from boundary data.