In passive sonar systems, deviations from an ideal linear configuration can significantly impair the beamforming performance of towed hydrophone arrays. This paper presents a method aimed at improving the underwater acoustic signals in the presence of array distortion. The method is centered on a high-order time-delay difference estimation technique utilizing time-frequency autofocus. Initially, a detailed signal model is established that captures the distinctive characteristics of distorted arrays. Subsequently, an algorithm is introduced for high-order time-delay difference estimation to enhance signal fidelity by leveraging phase information within narrowband components originating from incidental acoustic sources. Additionally, a quality metric to evaluate these components is introduced, facilitating the practical implementation of the method. The effectiveness of the proposed approach is validated through both simulation and experimental results, demonstrating its superiority over existing techniques. Importantly, this method does not require prior knowledge of the distortion pattern, making it adaptable to various non-linear array configurations.