The worldwide energy demand is continuously increasing, prompting experts to recommend using alternative energy sources to conserve natural gas, fossil fuels, and electricity. Photovoltaic thermal (PVT) systems emerge as a viable solution, generating electrical and heat energy simultaneously while freeing carbon dioxide (CO2) emissions. These systems offer sustainable green technology for supplying renewable electricity and heat to commercial and domestic applications. This study delves into the performance of a photovoltaic thermal (PVT) system featuring an isosceles triangular-shaped absorber design. It considers size variations of 0.02 and 0.03 m while maintaining a constant aspect ratio. Water-based nanofluids such as CuO/w, MgO/w, and ZnO/w, with a nanoparticle volume portion of 4%, alongside pure water as a coolant, are utilized with a variation of mass flow rate ranges from 0.028 kg/s to 0.11 kg/s, allowing for an exploration of its impact on performance parameters. A numerical model is established to comprehensively analyze the system's performance, applying an energy balance equation to its components. An economic analysis is also conducted to assess the system's cost-effectiveness and determine the energy payback time. Results indicate that the highest overall daily performance is achieved with ZnO/w nanofluid at a mass flow rate of 0.112 kg/s and a fluid flow channel size of 0.02 m. Comparatively, compared to other nanofluids and pure water, the average electrical, thermal, and overall performances achieved are 14.57%, 22.36%, and 36.40%, respectively. The energy payback periods are 5.5, 5.2, 5.4, and 4.8 years for CuO/w, MgO/w, ZnO/w, and Pure water, respectively. Furthermore, it is observed that a higher mass flow rate correlates with higher system performance parameters.