The extraction of damage-sensitive features based on deep learning for the vibration response caused by vehicle-bridge interaction (VBI) has become a mainstream method in damage identification. However, most studies remain at the stage of qualitative damage assessment, not achieving precise numerical quantification of damage. They also typically involve multiple frequent runs using only a single dedicated vehicle. In this study, utilizing a deep neural network designed for multi-objective regression tasks, a novel method for bridge damage localization and quantification is presented with a multi-objective regressor designed to create a direct mapping between the curve of bridge deflection in time domain and the damage information matrix. The final quantitative damage values are obtained through statistical analysis. The method is evaluated on a simulated data set generated from VBI simulation using various 3D heavy vehicle models. The results demonstrate that this method can accurately locate and quantify damage even with unseen vehicle load conditions, damage scenarios, and measurement noise. The structure and depth affect the effectiveness of extracting damage-sensitive features from time-domain deflection during training. The study shows potential for real-time damage identification under normal operating conditions of real bridges in the rapid development of intelligent transportation systems.