The low-frequency L1 assumption has been extensively applied to the large-time asymptotics of solutions to the compressible Navier-Stokes equations and incompressible Navier-Stokes equations since the classical efforts due to Kawashima, Matsumura, Nishida, Ponce, Schonbek and Wiegner. In this paper, we establish a sharp decay characterization for the compressible Navier-Stokes equations in the critical Lp framework. Precisely, it is proved that the Besov space B˙2,∞σ1-boundedness condition (with d2−2dp≤σ1<d2−1) of the low-frequency part of initial perturbation is not only sufficient, but also necessary to achieve those upper bounds of time-decay estimates. Furthermore, we show that the upper and lower bounds of time-decay estimates hold if and only if the low-frequency part of initial perturbation belongs to a nontrivial subset of B˙2,∞σ1. To the best of our knowledge, our work is the first one addressing the inverse problem for the large-time asymptotics of compressible viscous fluids.