Distances in the conformal manifold, the space of CFTs related by marginal deformations, can be measured in terms of the Zamolodchikov metric. Part of the CFT Distance Conjecture posits that points in this manifold where part of the spectrum becomes free, called higher-spin points, can only be at infinite distance from the interior. There, an infinite tower of operators become conserved currents, and the conformal symmetry is enhanced to a higher-spin algebra. This proposal was initially motivated by the Swampland Distance Conjecture, one of pillars of the Swampland Program. In this work, we show that the conjecture can be tackled using only methods from the conformal toolkit, and without relying on the existence of a weakly-coupled gravity dual. Via conformal perturbation theory combined with properties of correlators and of the higher-spin algebra, we establish that higher-spin points are indeed at infinite distance in the conformal manifold. We make no assumptions besides the usual properties of local CFTs, such as unitarity and the existence of an energy-momentum tensor. In particular, we do not rely on a specific dimension of spacetime (although we assume d > 2), nor do we require the presence of supersymmetry.