Fluorescence sensors detect veterinary drug residues in milk, which are usually interfered with by the complex milk matrix. The specific DNA template can produce DNA stabilized silver nanoclusters (DNA-AgNCs) emitting in the visible red wavelength range. This will reduce the fluorescence interference of milk by avoiding the fluorescence signal generated by the inherent component of milk in the relatively short wavelength range. Furthermore, the red fluorescence has relatively high penetration, which can also reduce the impact of interfering substances. Here, on the basis of fluorescence resonance energy transfer (FRET), we designed a fluorescent aptasensor using DNA-AgNCs and gold nanoparticles (AuNPs) to detect kanamycin (KAN) residue in milk. Due to electrostatic interaction, DNA adsorbs on the AuNPs. The decrease in the distance between DNA-AgNCs and AuNPs generates FRET, which causes the fluorescence of DNA-AgNCs to be quenched. When KAN is added, the aptamer and KAN can specifically bind with high affinity, resulting in a weakening of the electrostatic interaction between DNA-AgNCs and AuNPs. This results in a decrease in FRET between DNA-AgNCs and AuNPs, and recovery of the fluorescent signal intensity of DNA-AgNCs. The designed sensor has satisfactory specificity and sensitivity for the detection of KAN residues. The applied strategy showed the limit of KAN detection to be as low as 22.6 nM. Moreover, this work provides a way for reducing the interference of the milk background fluorescence signal in the fluorescence sensor and has broad application prospects.