In this study, the effect of gap distance on the aerodynamic interference effect of two neighboring buildings has been systematically investigated. Firstly, the variation of aerodynamic features of principal building with different gap distances (from 1.5B to 6B) and wind directions (from 0°to 180°) are analyzed. Then, Proper Orthogonal Decomposition (POD) and Independent Component Analysis (ICA) methods are employed to further unveil the interference mechanism under tandem and side-by-side arrangement. The spatial-spectral feature of surface pressure field, including pressure pattern, modal coefficients and correlation are comprehensively compared to highlight the differences of POD and ICA. The results show that 3B is recognized as the critical gap whose pressure pattern is more similar to the small gap type than the large gap type. Moreover, two new phenomena, “mode competition” and “mode transition”, are respectively captured by POD and ICA in tandem arrangement, and quantitively interpreted by modal load component ratio. For side-by-side arrangement, the variation of channeling effect with gap distance is well reflected by both POD and ICA pressure patterns. Compared with POD, ICA provides a more variety of information on pressure patterns by capturing their variation with gap distance on all four side-surfaces. These conclusions can enhance understanding of fluid–structure interaction mechanism and provide reference for wind-induced responses control measures for interference buildings.