Earthquake swarms with the possible largest magnitude greater than 5.0 frequently strike the Brawley seismic zone in southern California. The latest swarm of earthquakes occurred on 5 June 2021, including a magnitude 5.2 quake. The Salton Sea Geothermal Field lies at the northern end of the Brawley seismic zone. A good understanding of the detailed subsurface structure is necessary for seismic hazard assessment and the monitoring of geothermal resources exploration in this region. We apply the novel adjoint-state traveltime tomography method to image the crustal P- and S-wave velocity structures of the Brawley seismic zone and the adjacent area to the west. This new method requires no ray tracing but maintains a high level of accuracy in complex media. High-resolution P- and S-wave velocity models are constructed by inverting 85,321 first P-wave arrival times and 43,912 first S-wave arrival times, respectively. The velocity models reveal strong crustal heterogeneities. At 5–12 km depth the Brawley seismic zone is occupied by earthquake-prone high-velocity rocks; while the near surface is dominated by low-velocity sedimentary rocks. The source zone of the 1987 Superstition Hills earthquake sequence is filled with low-velocity rocks, possibly suggesting the presence of fluids. Fluids may have also played a critical role in the generation of earthquake swarms in the Brawley seismic zone.