There remains a scarcity of studies on the utilization and optimization of core–shell plasmonic nanofluid filter although it shows potential in solar energy exploitation. To fill this research gap, this paper introduces an optimization methodology founded on merit function (MF) to optimize the Au@SiO2-based nanofluid filters for enhancing the photovoltaic/photothermal (PV/T) system performance. Initially, the MF model of the nanofluid filtered-PV/T system was constructed based on a reliable optical model and validated by referenced experimental results. Using these models, the effect of nanofluid parameters on the system performance and the mechanism of optimization procedure were investigated. Results indicated that modifying SiO2 shell thickness is a cost-effective means of tailoring the transmittance of the nanofluids compared to adjusting the Au core diameter. The optimization mechanism can be summarized as enhancing electrical efficiency or improving thermal efficiency to offset electricity losses. Furthermore, the MF of the Au@SiO2/EG nanofluid increased from 1.3737 to 1.3844 with optical thickness decreasing from 44 mm to 40 mm by mixing with Ag@SiO2 nanoparticles, indicating the cost-effectivity of blended nanofluid. The MF of the optimized blended nanofluids are 0.007–0.084 higher than those of the referenced nanofluids. This work offers a feasible pathway toward achieving high-performance nanofluid optical filters.