The cable serves as the primary structural component in cable-stayed bridges, suspension bridges, and other cable-supported bridge designs. Accurate measurement of cable tension holds paramount importance for ensuring the overall safety and stability of bridge structures. Cable tension can be inferred from the vibration frequency, making precise measurement of cable vibration frequency of utmost practical significance. Presently, video-based methodologies, such as phase-based vibration extraction, present notable advantages in terms of speed, efficiency, and non-contact capabilities over conventional acceleration sensor approaches. However, these methods face challenges in accurately discerning the micro-vibration of cables in complex background interference. To address this problem, a phase magnification-based feature signal extraction (PMFSE) algorithm is proposed, which is suitable for detecting small vibration in complex background. The overall idea is to combine the small vibration enhancement method in spatial domain with the signal separation method in time domain. Specifically, in the spatial domain, edge phase information is amplified through phase amplitude weighting and phase enhancement techniques to mitigate non-edge noise. In time domain, cable signals are extracted based on distinguishing features between the cable signal and noise signal. The efficacy of PMFSE is evaluated in various outdoor experiments. Compared with the state-of-the-art methods, PMFSE exhibits substantial improvements on the cable micro-vibration frequency measurement in complex background.