Coagulation factor XIa (FXIa) is associated with a low risk of bleeding and has been identified as an effective and safe target for the development of novel anticoagulant drugs. In this study, we established an ultrasensitive competitive dual-enzyme cascade signal amplification method for the quantitative analysis and screening of FXIa inhibitors. Due to the specific recognition of FXIa's active site by the aptamer AptE40, the AptE40-QDs-EK recognition probe modified with enterokinase (EK) and the aptamer AptE40, was attached to the MNPs-FXIa capture probe. When FXIa inhibitor was present, it competed with AptE40 for binding to FXIa, resulting in the detachment of AptE40-QDs-EK from MNPs-FXIa. After magnetic separation, the enterokinase of AptE40-QDs-EK in the supernatant hydrolyzed N-terminal hexapeptide of trypsinogen, leading to the production of a large amount of trypsin as part of the first-stage signal cascade amplification. Next, trypsin could hydrolyze the hexameric arginine peptide (RRRRRR, R6), leading to the dissociation of RQDs from the R6-RQDs signal probe; this resulted in a dramatic increase in the fluorescence intensity of the supernatant as the second-stage signal cascade was amplified. The feasibility of the method was investigated using the FXIa inhibitor aptamer FELIAP as a positive model drug. Furthermore, the method was applied to screen the FXIa inhibitors in Eupolyphaga sinensis Walker. Two fractions with more active anticoagulated ingredients were successfully identified and validated via the conventional method, and the results were consistent. The established method provides a key technique for the sensitive detection, high-throughput analysis, and screening of the FXIa inhibitors.