Upconversion nanoparticles (UCNPs) have shown great promise in bioanalytical applications owing to their excellent optical properties. Generally, most analytical applications are based on the fluorescence resonance energy transfer (FRET) principle to quench the fluorescence of UCNPs. However, each UCNP contains thousands of emission center ions, and most of them exceed the FRET critical distance, which hinders FRET efficiency and leads to a low signal-to-background ratio (SBR). Herein, a novel nanoprobe for the detection of Xanthine (XA) based on inner filter effects (IFE) and cascade signal amplification strategy was constructed by decorating UCNP with trypsin-chymotrypsin-stabilized gold nanoparticles-gold nanoclusters (Try-chy-AuNPs-AuNCs) monometallic nanohybrids. The Try-chy-AuNPs-AuNCs prepared by ultrafast (3 min) and green synthesis method have efficient upconversion fluorescence quenching ability (the quenching efficiency up to 90.9%), which can effectively improve the SBR of the probe, so as to improve the sensitivity. In addition, the Try-chy-AuNPs-AuNCs have a unique spatial structure, which can effectively prevent the interaction between large-size biothiol (glutathione) and the probe, thus improving its selectivity. Besides, combined with the excellent optical performance of UCNPs and cascaded signal amplification strategy, the sensitivity of the probe can be further improved. Under the optimized conditions, the linear response range of the probe was obtained from 0.05 to 50 μM, 0.06–80 μM and with the low detection limit of 22.6 nM and 26.3 nM for H2O2 and XA, respectively. Meanwhile, the developed method has been further applied to the detection of XA in human serum with satisfactory results.