Reclaimed Asphalt Pavement (RAP) materials are used as substitutes for new materials in asphalt pavement construction, leveraging the engineering and commercial benefits of the aged binders and aggregate matrixes in RAP. These asphalt mixtures impart significant variations in volumetric properties and asphalt mixture characteristics. The current study investigates the Marshall properties, moisture susceptibility, and rutting behavior of 24 recycled asphalt mixtures developed with nanosilica and nanoclay. RAP material percent, nanomaterial content, binder grade, and extra binder were considered the factors influencing asphalt mixture performance. The above factors were analyzed using the Response Surface Methodology (RSM) to predict the Marshall and volumetric properties. Also, this investigation covers the moisture susceptibility and rut characteristics of recycled nanomaterial-modified Hot Mix Asphalt (HMA) and Warm Mix Asphalt (WMA) mixes developed with Viscosity Grade 30 (VG-30) and Polymer-Modified Bitumen-40 (PMB-40). The chemical additive Zycotherm was used to develop WMA mixes. The test results indicate that adding RAP material at higher percentages and modifying the binder with nanomaterials affected moisture susceptibility with reduced moisture damage. Recycled nanosilica-modified HMA mixes developed with PMB-40 at higher RAP percentages reported higher tensile strength ratio (TSR) values in contrast with VG-30 mixes, indicating their greater susceptibility toward moisture-induced damage. The rutting potential of all of the recycled asphalt mixture combinations was enhanced by densely packed aggregate structures optimized with nanomaterials, total binder content, and RAP materials developed using the Marshall method. Overall, the nanosilica-modified recycled asphalt mixes developed with PMB40 at higher RAP percentages showed better performance in terms of strength and durability.