This research explores the fluorescence properties and photostability of boron nitrogen co-doped graphene quantum dots (BN-GQDs), evaluating their effectiveness as sensors for rutin (RU). BN-GQDs are biocompatible and exhibit notable absorbance and fluorescence characteristics, making them suitable for sensing applications. The study utilized various analytical techniques to investigate the chemical composition, structure, morphology, optical attributes, elemental composition, and particle size of BN-GQDs. Techniques included X-ray diffraction (XRD), energy-dispersive X-ray spectroscopy (EDS), transmission electron microscopy (TEM), scanning electron microscopy (SEM), and atomic force microscopy (AFM). The average particle size of the BN-GQDs was determined to be approximately 3.5 ± 0.3nm. A clear correlation between the emission intensity ratio and RU concentration was identified across the range of 0.42 to 4.1μM, featuring an impressively low detection limit (LOD) of 1.23nM. The application of BN-GQDs as fluorescent probes has facilitated the development of a highly sensitive and selective RU detection method based on Förster resonance energy transfer (FRET) principles. This technique leverages emission at 465nm. Density Functional Theory (DFT) analyses confirm that FRET is the primary mechanism behind fluorescence quenching, as indicated by the energy levels of the lowest unoccupied molecular orbitals (LUMOs) of BN-GQDs and RU. The method's effectiveness has been validated by measuring RU concentrations in human serum samples, showing a recovery range between 97.8% and 103.31%. Additionally, a smartphone-based detection method utilizing BN-GQDs has been successfully implemented, achieving a detection limit (LOD) of 49nM.