The workability of asphalt mixtures significantly influences the construction of the asphalt pavement. In this study, a mixing rheometer was developed to characterize the workability of asphalt mixtures during mixing and further to investigate the effect of coarse aggregates. In addition, the corresponding rheological indices for evaluating the mixing rheological properties were proposed, including the b value related to the initial mixing resistance and the k value related to the steady-state mixing resistance. Moreover, the correlation between the geometric properties of the coarse aggregate and the mixing rheological indices was established. Results show that the sieving diameter/shape index of single-size aggregates (SA) and single-size aggregate mixtures (SAM) exhibits a strong linear correlation with the rheological indice k value. The fluctuation of SAM’s mixing rheological indices was reduced by about 1 time compared with SA’s. Therefore, asphalt plays a lubricating role in the mixture to reduce mixing resistance and reduce the sensitivity of aggregate geometric characteristics to mixing rheological indices. As the aggregate size increases, the mixing rheological properties of the mixture deteriorate. During the initial mixing, the b values of SA, SAM and GAM are most sensitive to the aggregate shape index, angularity and sieving diameter, respectively. During the steady-state mixing, the aggregate sieving diameter is most sensitive to the mixing rheological indice k value, followed by the shape index, while the angularity is less sensitive to the k value.