We report an ultrasonic study of the magneto-elastic coupling of the hydrogenated and deuterated (TMTTF)$_2$PF$_6$ organic salts. For both salts the temperature dependence of the longitudinal velocity along the c* axis displays a monotonic stiffening of the $C_{33}$ compressibility modulus upon cooling. Below the characteristic temperature scale 40 K the modulus stiffening becomes markedly enhanced, in concomitance with the reduction of spin degrees of freedom previously seen in magnetic measurements as low dimensional precursors of the spin-Peierls transition. The magneto-elastic coupling appears to be much weaker in the hydrogenated salt due to the highly inhomogeneous elastic behavior induced by the proximity of the charge ordering transition to the spin-Peierls phase. For the deuterated salt, an important anomaly in the ultrasound velocity is observed below the spin-Peierls transition temperature $T_{\rm SP}$ in agreement with scaling of the elastic deformation with the spin-Peierls order parameter. In spite of the weakly inhomogeneous character of the spin-Peierls phase transition, the magnetic field dependence of $T_{\rm SP}$ is well captured with the mean-field prediction for the lattice distorted Heisenberg spin chain.