We determine a detailed 3-D crustal S-wave velocity structure in and around the Xianshuihe (XSH) fault zone using the direct ambient-noise surface-wave tomography. The continuous waveform data recorded at 22 portable seismic stations of our newly deployed around the XSH fault zone and 13 Chinese provincial stations are used in the data collection during September to December 2020. Our results show obvious low-velocity (low-V) anomalies under the Sichuan basin and high-velocity (high-V) anomalies under the Songpan-Ganzi block at 5–10 km depths but high-V anomalies under the Sichuan basin and low-V anomalies under the Chuandian block at 20–40 km depths. The XSH faut zone shows a clear boundary of low-V and high-V anomalies. Furthermore, there is a clear relationship between velocity structure and Luding mainshock. The Luding mainshock and its aftershocks occurred at the boundary of high-V anomalies in the upper crust but underlain by low-V anomalies in the mid-lower crust. These low-V anomalies indicate that the fluids have reached in the source area. These results suggest that fluids might reduce the friction force due to the decrease of both frictional coefficient and effective normal stress on the fault planes and trigger the strike-slip movement of the fault zone and the occurrence of the Luding earthquake. These fluids could be contained in the hot and wet mantle upwelling flow in the big mantle wedge caused by the deep subduction of the India slab down to the mantle transition zone, in addition to the crustal channel flow due to the Indo-Eurasian collision.