Previous research revealed that wave interferometry can be more effective than existing ultrasonic techniques in monitoring the strength development of early-age concrete. To improve its accuracy, this study aims to examine the impact of aggregate size on the relative velocity change of high-frequency ultrasound during the concrete curing process. Concrete cubic specimens with 150 mm sides were prepared using two different maximum aggregate sizes, 4.75 mm and 20 mm. These specimens were subjected to water curing for the first 28 days, followed by air curing until 150 days. Ultrasonic tests employing a 1 MHz frequency were performed throughout the curing process, and the measured signals were analyzed by applying wave interferometry techniques. The test results suggest that the influence of aggregate size was evident in both the initial water curing and the subsequent air curing. During the initial curing, the specimen with larger aggregates exhibited a greater increase in relative velocity change. Thereafter, the velocity change decreased during the subsequent air curing in both specimens. Based on the results, the correlation between relative compressive strength and relative velocity change was proposed considering the maximum aggregate size. The findings offer insights into the use of ultrasonic methods to evaluate the characteristics of early-age concrete.