Full waveform inversion (FWI) has become a popular and efficient tool for near-surface site characterization. SH- and Love-wave inversion has recently gained attention due to the good sensitivity to mass density and S-wave velocity and the computing advantage over Rayleigh-wave inversion. In this study, we present a new time-domain Gauss-Newton FWI (GN-FWI) method of SH- and Love-waves to extract both mass density and S-wave velocity of soil and rock. The advantage of Gauss-Newton optimization is that the inverse of the regularized Hessian matrix filters, balances, and regularizes the gradient vector to reduce shallow artifacts and resolve deeper structures. To overcome the main challenge of Gauss-Newton approach as high computing demand, we derive analytical formula to compute Jacobian matrix efficiently based on the virtual source and reciprocity theory. The presented method is demonstrated on realistic synthetic and field experiments. The synthetic experiment indicates that the method can characterize a challenging reverse profile with four variable layers of high and low mass density and S-wave velocity. The density and S-wave velocity values and layer interfaces are recovered. The field experiment reveals an 18-m depth subsurface profile, which includes a softer soil layer underlain by stiffer weathered limestone layer. Comparing to invasive Standard Penetration Tests (SPT), the trends of both density and S-wave velocity generally agree with the SPT N-values, including a weak soil zone and weathered limestone. To our best understanding, this is the first study of Gauss-Newton FWI on SH- and Love-waves at any scales.