Ag@SiO2 nanofluid is widely used in spectral splitting PV/T system. Its core–shell structure has great influence on the optical properties. In this work, we focus on the comprehensive analysis and structure optimization of Ag@SiO2 nanofluid to achieve its optimal spectral performance. DDA method is used to predict the optical properties of Ag@SiO2 nanofluid and an optimization model based on filter efficiency is proposed. The effect of the SiO2 shell thickness and Ag core mass concentration is analyzed. It indicates that the spectral performance of Ag@SiO2 nanofluid can be improved with SiO2 shell thickness of 15–40 nm and Ag core mass concentration of 81–135 mg/L. To achieve the same theoretical merit function of 1.46, the usage of Ag mass can be reduced by 25/33/44/62 % with SiO2 coating of 10/20/40/70 nm. The optimal structure to achieve the highest filter efficiency η of 37.8 % is with a shell thickness of 20 nm and a mass concentration of 113.9 mg/L. An indoor PV/T operation testing is conducted to verify the optimization results. The merit function of Ag-based nanofluids increases from 1.58 to 1.598 and a reduction in Ag usage of 17 % is achieved with a SiO2 coating shell of 17.8 nm. Operation stability is also enhanced with no aggregation observed during the working cycle and 7-day static experiment.