The Hydroxyl-functionalized hexagonal boron nitride quantum dots (h-BN QDs) were synthesized using a hydrothermal technique in a water medium, exhibiting stability in water for over 90 days. The present study demonstrates two simple fluorescence and colorimetric techniques for sensing organophosphorus (OP) and organochloride (OC) pesticides (atrazine, simazine and chlorpyrifos) in an aqueous medium using hydroxyl-functionalized h-BN QDs. The sensing capability of h-BN QDs for these pesticides was determined to be in the range of 5–10 µM using both techniques. The hydroxyl groups on the surface of the QDs enhanced dispersibility and enabled peroxidase-like catalytic activity in acidic conditions. The radical generation using hydroxyl-functionalized h-BN QDs in the presence of H2O2 upon 2,2′-azino-bis (3-ethylbenzothiazoline-6-sulfonic acid) diammonium salt (ABTS) oxidation plays a significant role in the peroxidase reaction. The hydrogen bonding and π-π interactions between the pesticide molecules and ABTS can enhance the pesticide sensing performance. The limit of detection (LOD) using ABTS for atrazine, simazine, and chlorpyrifos was 6.11, 6.56, and 2.47 µM, respectively. Fluorescence sensing, based on a turn ON/OFF mechanism, demonstrated high specificity and sensitivity with LODs of 5.59, 6.45, and 8.08 µM for the same pesticides. The DFT studies revealed non-covalent bonding, such as hydrogen bonding, π-π interactions between the pesticides and the h-BN QDs, indicating binding affinities and stabilization energies of the complexes which can detect pesticides effectively. Atrazine exhibited higher binding affinity towards h-BN QDs, followed by simazine and chlorpyrifos. The h-BN QDs provide a new insight into bonding with pesticides, enhancing the peroxidase mimetic and fluorescence activity.