The envelope of the time-lagged cross-correlation of an underwater noise field between two hydrophones can under certain conditions be used as a proxy for active acoustic receptions between the two locations enabling the study of ocean variability. Previous work looked at the sensitivity of cross-correlation peak amplitudes with respect to the distribution of the noise sources. The present study examines the sensitivity of the cross-correlation envelope peak times with respect to changes in the sound-speed distribution. A wave-theoretic scheme allowing for finite-frequency calculations in two and three dimensions, combined with the Born approximation for perturbations of the Green's function and the peak arrival approach, is used to obtain sensitivity kernels with respect to environmental (sound-speed) changes. These kernels provide a way to infer ocean structure from the cross-correlation peaks, considered as observables on their own and valid even in cases where the cross-correlation function does not approximate the time-domain Green's function between the two receivers. The sensitivity behavior is studied for different propagation conditions and noise-source distributions, ranging from spatially distributed uncorrelated noise sources to point sources, such as individual ships. Deviations from linearity are addressed and discussed.