Ultrasonic testing (UT) is widely used in many engineering fields because of its unique merits (e.g. high accuracy, non-destructive, versatility, portability etc.). Moreover, UT can be used to detect a variety of internal flaws/defects, including cracks, voids, inclusions, and material degradation. Therefore, UT can be successfully used in evaluation of concrete status. However, the high degree of non-homogeneity of concrete impairs the reliability and accuracy of UT detection. And also, UT is a frequency-dependent method. Its resolution and accuracy can be highly influenced by the frequency. Thus, the existence of aggregate in concrete will introduce the uncertainty to the detection, which is often ignored. In this paper, to improve the accuracy and reliability of UT in concrete, the influence of aggregate and frequency were investigated. Through the parametric study, the inherent correlation between the stochastic effect introduced by coarse aggregates and UT signal was revealed based on the combination of experimental and numerical model. Therefore, the UT features for concrete detection are properly selected, which has less influence by aggregate and more sensitive to the internal flaws. The proposed strategy of UT data analysis can provide a guidance for on-site detection in concrete.