Due to low density, large pores, and numerous cracks, recycled aggregate mortar has defects of lower fluidity and lower strength compared to natural aggregate mortar. In order to modify inherent defects of recycled aggregates, nanosilica (NS) with different dosages (0%, 0.6%, 1.2%, 1.8%, 2.4% and 3%) was added to recycled aggregate mortar (RNC), and 40% natural sand in RNC was replaced by recycled aggregates. Then consistency test, compressive strength test and microscopic tests (SEM, EDS and XRD) were conducted to investigate working performance and microscopic mechanism. It was found that the addition of NS can improve the flowability of RNC, and the consistency value reaches the maximum (53 mm) while NS content is 2.4%, which increases by 14% comparing to the specimen without NS. The compressive strength of RNC increases and decreases with increasing NS content in the ranges of 0–1.8% and 1.8–3% respectively, and compressive strength is 3.3 and 5.9 MPa at curing ages of 7 d and 28 d respectively while NS content is 1.8%, which increase 50% and 14% comparing to the specimen without NS. Meanwhile, the consistency value of the specimen with NS content of 1.8% is 51 mm, which is close the maximum value corresponding to NS content of 2.4%. For that, 1.8% is suggested as the optimal content of NS. Moreover, the addition of NS can reduce the brittleness index and mechanical damage degree, and enhance the energy absorption effect. Microscopic analysis reveals that NS can not only fill the inherent cracks and pores of recycled aggregates, but also promote cement hydration reaction, and more hydration products (C-S-H) can efficiently reduce the cracks and pores of recycled aggregates. It is believed that the finds of this paper can provide theoretical support for the application of recycled aggregates in practical engineering.