Drive-by bridge inspection using acceleration responses of a passing vehicle has great potential for bridge structural health monitoring. It is, however, known that the road surface roughness is a big challenge for the practical application of this indirect approach. This paper presents a new two-step method for the bridge damage identification from only the dynamic responses of a passing vehicle without the road surface roughness information. A state-space equation of the vehicle model is derived based on the Newmark-[Formula: see text] method. In the first step, the road surface roughness is estimated from the dynamic responses of a passing vehicle using the dual Kalman filter (DKF). In the second step, the bridge damage is identified based on the interaction force sensitivity analysis with Tikhonov regularization. A vehicle–bridge interaction model with a wireless monitoring system has been built in the laboratory. Experimental investigation has been carried out for the interaction force and bridge surface roughness identification. Results show that the proposed method is effective and reliable to identify the interaction force and bridge surface roughness. Numerical simulations have also been conducted to study the effectiveness of the proposed method for bridge damage detection. The vehicle is modeled as a 4-degrees-of-freedom half-car and the bridge is modeled as a simply-supported beam. The local bridge damage is simulated as an elemental flexural stiffness reduction. Effects of measurement noise, surface roughness and vehicle speed on the identification are discussed.The results show that the proposed drive-by inspection strategy is efficient and accurate for a quick review on the bridge conditions.