SUMMARY We perform a joint inversion of arrival time data generated by direct P and fault zone (FZ) head waves in the San Andreas Fault south of Hollister, CA, to obtain a high-resolution local velocity structure. The incorporation of head waves allows us to obtain a sharp image of the overall velocity contrast across the fault as a function of depth, while the use of near-fault data allows us to resolve internal variations in the FZ structure. The data consist of over 9800 direct P and over 2700 head wave arrival times from 450 events at up to 54 stations of a dense temporary seismic array and the permanent northern California seismic network in the area. One set of inversions is performed upon the whole data set, and five inversion sets are performed on various data subsets in an effort to resolve details of the FZ structure. The results imply a strong contrast of P-wave velocities across the fault of ∼50 per cent in the shallow section, and lower contrasts of 10‐20 per cent below 3 km, with the southwest being the side with faster velocities. The presence of a shallow low velocity zone around the fault, which could corresponds to the damage structures imaged in trapped wave studies, is detected by inversions using subsets of the data made up of only stations close to the fault. The faster southwest side of the fault shows the development of a shallow low velocity FZ layer in inversions using instruments closer and closer to the fault (<5 and <2 km). Such a feature is not present in results of inversions using only stations at greater distances from the fault. On the slower northeast side of the fault, the presence of a low velocity shallow layer is only detected in the inversions using the stations within 2 km of the fault. We interpret this asymmetry across the fault as a possible indication of a preferred propagation direction of earthquake ruptures in the region. Using events from different portions of the fault, the head wave inversions also resolve small-scale features of the fault visible in the surface geology and relocated seismicity.