The Kerr/CFT correspondence is a possible route to gain insight into the quantum theory of gravity in the near-horizon region of a Kerr black hole via a dual holographic conformal field theory (CFT). Predictions of the black hole entropy, scattering cross-section and the quasi normal modes from the dual holographic CFT corroborate this proposed correspondence. More recently, it has been suggested that quantum gravitational effects in the near-horizon region of a black hole may drastically modify the classical general relativistic description, leading to potential observable consequences. In this paper, we study the absorption cross-section and quasi normal modes of a horizonless Kerr-like exotic compact object (ECO) in the dual CFT picture. Our analysis suggests that the near-horizon quantum modifications of the black hole can be understood as finite size and/or finite $N$ effects in the dual CFT. Signature of the near-horizon modification to a black hole geometry manifests itself as delayed echoes in the ringdown (i.e. the postmerger phase) of a binary black hole coalescence. From our dual CFT analysis we show how the length of the circle, on which the dual CFT lives, must be related to the echo time-delay that depends on the position of the near-horizon quantum structure. We further derive the reflectivity of the ECO membrane in terms of the CFT modular parameters, showing that it takes the Boltzmann form.