The determination of internal material damage is always an arduous challenge. Non-destructive monitoring methods show great potential in quantitatively determining the internal material properties, whereas most of the studies relying on external observations remain in a qualitative stage. They either violate the basic thermodynamic assumptions or are difficult to guide engineering practice. In this paper, following the theory of continuum mechanics, an elastoplastic damage model based on non-destructive monitoring methods (i.e., acoustic emission and ultrasonic wave velocity measurement) has been developed. To capture the continuous and precise damage evolution inside rock mass, P wave velocity obtained by ultrasonic wave measurement was utilised and then considered as an input for the proposed elastoplastic damage model. Triaxial loading test results on six Gosford sandstone samples were analysed first to characterise critical stresses along the stress–strain loading curves, such as crack closure stress, stable crack propagation stress and unstable crack propagation stress. The drop of ultrasonic wave velocity can be seen as an indicator to represent the damage evolution inside rock material. Damage initiation is also closely related to the confining stress and dilation induced volumetric expansion. The test results also suggested that the Drucker–Prager criterion is sufficient to describe the plastic yielding surface and the following material hardening. A non-associated plastic flow assumption was adopted, considering the essence of microcrack shearing in rock failure and the effect of hydrostatic pressure on plastic deformation. A modified Drucker–Prager plastic potential was also introduced to track the orientation of plastic increment with material hardening. A scalar damage variable was derived from ultrasonic wave measurement results to indirectly represent the deterioration of rock properties (modulus). The proposed model was used to match lab test results with high consistency, and the main features of rock behaviour in triaxial loading tests were successfully captured by the model. Finally, the damage evolution of rock samples was analysed, which indicates that damage is dependent on its conjugate force, namely damage energy release rate Y. This study proves that P wave velocity can be an effective approach to measure and forecast the internal damage evolution inside rock mass, which has broad prospects for engineering applications.