We introduce an improved non-destructive and non-contact measurement technique for the hardness penetration depth of thermally hardened steel parts based on all-optical laser ultrasound. The hardening of the near-surface region results in a stratified system with layers of different microstructural characteristics caused by changes in grain size. We employ spatially resolved ultrasonic backscattering signals to provide subsurface imaging of the axial and lateral hardness profile. Spatial scans over a range of 30 mm with a lateral step size of 100 μm were obtained within 30 s. We investigated three common steel grades with different microstructural characteristics and hardened layer thicknesses ranging from about 3 to 10 mm. The hardened surface layer and the unhardened core material can be identified. The data evaluation is assisted by a supervised machine learning approach that takes advantage of the spatio-temporal ultrasonic backscattering information. This method also provides an alternative route to define the hardness penetration depth using spatio-temporal scattering characteristics.