Semi-circular Bending (SCB) test has been widely used to evaluate the fracture performance of asphalt mixtures, but its potential in evaluating the low-temperature performance of asphalt mixtures at mortar scale has been seldom investigated. This study aims to explore the feasibility and evaluate the performance of SCB tests in evaluating the low-temperature performance of asphalt mortar. To quantify the influence of the crushing and local creep at the top of the SCB test samples on testing results, the Digital Image Correlation (DIC) method was used to characterize the displacement field of mortar samples during the test under different loading rates at −10 °C. The effects of binder film thickness, filler content, and aging condition on low-temperature performance of asphalt mortar were investigated by SCB test. The experimental design set a fixed nominal maximum aggregate size of 4.75 mm. The test matrix considered three binder film thicknesses (9, 10 and 11 µm) and four filler contents (8.5%, 9.4%, 10.5% and 11%) at two aging conditions (short-term and long-term aging). Two cracking indexes including before-peak slope (Sbp) and fracture energy (Gf) were used to evaluate the low-temperature performance. According to the results of DIC analysis, the loading rate of 3 mm/min was proposed to be the optimum for the SCB mortar test at −10 °C. It was concluded that the SCB mortar test can effectively differentiate the low-temperature cracking performance of different asphalt mortars under different aging conditions. Thicker binder film thickness can significantly decrease the aging rate of asphalt mortar from short-term aging to long-term aging, whereas the effect of filler content on ageing rate is not statistically significant.