Colloidal quantum dots/graphene (QD/Gr) nanohybrids have been studied intensively for photodetection in a broadband spectrum including ultraviolet, visible, near-infrared, and shortwave infrared (UV-vis-NIR-SWIR). Since the optoelectronic process in the QD/Gr nanohybrid relies on the photogenerated charge carrier transfer from QDs to graphene, understanding the role of the QD-QD and QD-Gr interfaces is imperative to the QD/Gr nanohybrid photodetection. Herein, a systematic study is carried out to probe the effect of these interfaces on the noise, photoresponse, and specific detectivity in the UV-vis-NIR-SWIR spectrum. Interestingly, the photoresponse has been found to be negligible without a 3-mercaptopropionic acid (MPA) ligand exchange, moderate with a single ligand exchange after all QD layers are deposited on graphene, and maximum if it is performed after each QD layer deposition up to five layers of total QD thickness of 260-280 nm. Furthermore, exposure of graphene to C-band UV (UVC) for a short period of 4-5 min before QD deposition leads to improved photoresponse via removal of polar molecules at the QD/Gr interface. With the combination of the MPA ligand exchange and UVC exposure, optimal optoelectronic properties can be obtained on the PbS QD/Gr nanohybrids with high specific detectivity up to 2.6 × 1011, 1.5 × 1011, 5 × 1010, and 1.9 × 109 Jones at 400, 550, 1000, and 1700 nm, respectively, making the nanohybrids promising for broadband photodetection.