The detection of bolt loosening using vibro-acoustic modulation (VAM) has been increasingly investigated in the past decade. However, conventional nonlinear coefficients, derived from theoretical analysis, are usually based on the assumption of ideal wave–surface interactions at the joint interfaces. Such coefficients show a poor correlation with the tightening torque when the joint is under the combined influences of structural and material nonlinearities. A reliable inspection method of residual bolt torque is proposed in this study using support vector regression (SVR) with acoustic features from VAM. By considering the material intrinsic nonlinearity (MIN) and dissipative nonlinearity (DN), the responses of aluminum–aluminum and composite–composite bolted joints during the VAM test were accurately simulated. The SVRs were subsequently established based on the database built by combining simulated and experimental nonlinear spectral features when the joints were inspected at different scenarios. The results show that the evaluation of residual torque using the SVR models driven by the acoustic nonlinear responses had higher accuracy compared to the conventional nonlinear coefficients. Requiring limited experimental data, the proposed method can achieve a reliable inspection of bolt torque by including the simulated data in the machine training.